Ukunquma usayizi ofanele weqoqo le-Kafka e-Kubernetes

Qaphela. transl.: Kulesi sihloko, i-Banzai Cloud yabelana ngesibonelo sokuthi amathuluzi ayo angokwezifiso angasetshenziswa kanjani ukwenza i-Kafka ibe lula ukuyisebenzisa ngaphakathi kwe-Kubernetes. Imiyalo elandelayo ikhombisa ukuthi unganquma kanjani usayizi ophelele wengqalasizinda yakho futhi ulungiselele i-Kafka ngokwayo ukuze uzuze umphumela odingekayo.

I-Apache Kafka iyinkundla yokusakaza esabalalisiwe yokudala amasistimu okusakaza esikhathi sangempela athembekile, angakala futhi asebenza kahle kakhulu. Amandla ayo ahlaba umxhwele anganwetshwa kusetshenziswa i-Kubernetes. Ngalokhu sithuthukile Umthombo ovulekile we-Kafka opharetha kanye nethuluzi elibizwa Amashubhu. Zikuvumela ukuthi usebenzise i-Kafka ku-Kubernetes futhi usebenzise izici zayo ezihlukahlukene, njengokulungisa kahle ukumiswa komthengisi, ukukala okusekelwe kumethrikhi ngokulinganisa kabusha, ukuqwashisa ngama-rack, "soft" (umusa) ukukhipha izibuyekezo, njll.

Zama ama-Supertubes kuqoqo lakho:

curl https://getsupertubes.sh | sh и supertubes install -a --no-democluster --kubeconfig <path-to-eks-cluster-kubeconfig-file>

Noma othintana naye imibhalo. Ungafunda futhi ngamanye amakhono e-Kafka, umsebenzi ozenzakalelayo usebenzisa i-Supertubes ne-Kafka opharetha. Sesivele sibhale ngabo kubhulogi:

• Hhayi bo! Enye futhi i-opharetha ye-Kafka ye-Kubernetes;
• Qapha futhi usebenzise i-Kafka ngokusekelwe kumamethrikhi e-Prometheus;
• Ukuqwashisa kwe-Kafka rack ku-Kubernetes;
• Ukugijima i-Apache Kafka phezu kwe-Istio - ibhentshimakhi;
• Umsebenzisi ugunyazwe futhi ufinyelele amaqoqo alawulwayo ngo-opharetha we-Kafka;
• Ukuthuthukiswa kwe-Kafka rolling kanye nokucushwa okuguquguqukayo ku-Kubernetes;
• Isihlungi sephrothokholi ye-Evoy ye-Kafka, ene-meshed.

Uma uthatha isinqumo sokuphakela iqoqo le-Kafka ku-Kubernetes, cishe uzobhekana nenselelo yokunquma usayizi ophelele wengqalasizinda engaphansi kanye nesidingo sokulungisa kahle ukulungiselelwa kwe-Kafka yakho ukuze uhlangabezane nezimfuneko zokusebenza. Ukusebenza okuphezulu komdayisi ngamunye kunqunywa ukusebenza kwezingxenye zengqalasizinda eziyisisekelo, njengememori, iphrosesa, isivinini sediski, umkhawulokudonsa wenethiwekhi, njll.

Ngokufanelekile, ukucushwa komdayisi kufanele kube ngendlela yokuthi zonke izici zengqalasizinda zisetshenziswe ngamakhono azo aphezulu. Nokho, empilweni yangempela lokhu kusetha kuyinkimbinkimbi. Kungenzeka kakhulu ukuthi abasebenzisi bazomisa abathengisi ukuze bakhulise ukusetshenziswa kwengxenye eyodwa noma ezimbili (idiski, inkumbulo, noma iphrosesa). Ngokuvamile, umthengisi ubonisa ukusebenza okuphezulu lapho ukucushwa kwayo kuvumela ingxenye enensa kakhulu ukuthi isetshenziswe ngezinga layo eligcwele. Ngale ndlela singathola umbono ongemuhle womthwalo umthengisi oyedwa angawuphatha.

Ngokwetiyori, singaphinda silinganisele inani labadayisi abadingekayo ukuze kuphathwe umthwalo othile. Kodwa-ke, ekusebenzeni kunezinketho eziningi zokumisa emazingeni ahlukene kangangokuthi kunzima kakhulu (uma kungenakwenzeka) ukuhlola ukusebenza okungenzeka kokucushwa okuthile. Ngamanye amazwi, kunzima kakhulu ukuhlela ukucushwa okusekelwe ekusebenzeni okuthile okunikeziwe.

Kubasebenzisi beSupertubes, sivame ukuthatha le ndlela elandelayo: siqala ngokucushwa okuthile (ingqalasizinda + izilungiselelo), bese sikala ukusebenza kwayo, silungise izilungiselelo zomthengisi bese siphinda inqubo futhi. Lokhu kwenzeka kuze kube yilapho ingxenye ehamba kancane yengqalasizinda isisetshenziswa ngokugcwele.

Ngale ndlela, sithola umbono ocacile wokuthi bangaki ama-broker iqoqo elidinga ukuphatha umthwalo othile (inani labadayisi libuye lincike kwezinye izici, njengenani elincane lokuphindaphinda kwemiyalezo ukuze kuqinisekiswe ukuqina, inani lokuhlukaniswa. abaholi, njll.). Ngaphezu kwalokho, sithola ukuqonda ukuthi yiziphi izingxenye zengqalasizinda ezidinga ukulinganiswa okuqondile.

Lesi sihloko sizokhuluma ngezinyathelo esizithathayo ukuze sizuze okuningi ezingxenyeni ezihamba kancane ekucushweni kokuqala futhi silinganise ukuphuma kweqoqo le-Kafka. Ukucushwa okuqinile kakhulu kudinga okungenani ama-broker amathathu asebenzayo (min.insync.replicas=3), isatshalaliswa kuzo zonke izindawo zokufinyeleleka ezintathu ezihlukene. Ukumisa, ukukala nokuqapha ingqalasizinda ye-Kubernetes, sisebenzisa inkundla yethu yokuphatha iziqukathi zamafu ayingxubevange - Iphayiphi. Isekela endaweni (insimbi engenalutho, i-VMware) nezinhlobo ezinhlanu zamafu (Alibaba, AWS, Azure, Google, Oracle), kanye nanoma iyiphi inhlanganisela yazo.

Imicabango mayelana nengqalasizinda yeqoqo le-Kafka kanye nokucushwa

Ezibonelweni ezingezansi, sikhethe i-AWS njengomhlinzeki wamafu kanye ne-EKS njengokusabalalisa kwe-Kubernetes. Ukucushwa okufanayo kungenziwa kusetshenziswa P.K.E. - Ukusatshalaliswa kwe-Kubernetes kusuka ku-Banzai Cloud, kugunyazwe yi-CNCF.

idiski

I-Amazon inikeza ezihlukahlukene Izinhlobo zevolumu ye-EBS. Emqondweni gp2 и io1 kukhona amadrayivu e-SSD, noma kunjalo, ukuze kuqinisekiswe ukuphuma okuphezulu gp2 idla amakhredithi anqwabelene (Amakhredithi e-I/O), ngakho sincamele uhlobo io1, okunikeza ukusebenza okuphezulu okungaguquki.

Izinhlobo zezibonelo

Ukusebenza kwe-Kafka kuncike kakhulu kunqolobane yekhasi lesistimu yokusebenza, ngakho-ke sidinga izimo ezinememori eyanele yabathengi (JVM) kanye nenqolobane yekhasi. Isibonelo c5.2okukhulu - isiqalo esihle, njengoba ine-16 GB yememori futhi ilungiselelwe ukusebenza ne-EBS. Ububi bayo ukuthi ikwazi kuphela ukuhlinzeka ngokusebenza okuphezulu okungaphezu kwemizuzu engama-30 njalo ngamahora angama-24. Uma umsebenzi wakho udinga ukusebenza okuphezulu kakhulu isikhathi eside, ungase uthande ukucabangela ezinye izinhlobo zezibonelo. Yilokho impela esakwenza, ukuma kukho c5.4okukhulu. Ihlinzeka nge-throughput enkulu 593,75 Mb/s. Umkhawulo ophezulu wokukhipha umthamo wevolumu ye-EBS io1 phezulu kunesibonelo c5.4okukhulu, ngakho-ke into ehamba kancane yengqalasizinda kungenzeka kube i-I/O yalolu hlobo lwesibonelo (okumele kuqinisekiswe ukuhlolwa komthwalo wethu).

Inethiwekhi

Ukuphuma kwenethiwekhi kufanele kube kukhulu ngokwanele uma kuqhathaniswa nokusebenza kwesibonelo se-VM nediski, ngaphandle kwalokho inethiwekhi iba ibhodlela. Esimweni sethu, i-interface yenethiwekhi c5.4okukhulu isekela isivinini esingafika ku-10 Gb/s, ephakeme kakhulu kunokuphuma kwe-I/O kwesibonelo se-VM.

Ukuthunyelwa Kwebroker

Ama-broker kufanele athunyelwe (ahlelwe ku-Kubernetes) ezindaweni ezizinikele ukuze kugwenywe ukuncintisana nezinye izinqubo ze-CPU, inkumbulo, inethiwekhi, nezinsiza zediski.

Inguqulo ye-Java

Inketho enengqondo i-Java 11 ngoba ihambisana ne-Docker ngomqondo wokuthi i-JVM inquma kahle amaphrosesa nenkumbulo etholakala esitsheni lapho umthengisi asebenza khona. Ngokwazi ukuthi imikhawulo ye-CPU ibalulekile, i-JVM isetha ngaphakathi futhi ngokusobala inani lemicu ye-GC nemicu ye-JIT. Sisebenzise isithombe se-Kafka banzaicloud/kafka:2.13-2.4.0, okufaka inguqulo ye-Kafka 2.4.0 (Scala 2.13) ku-Java 11.

Uma ungathanda ukufunda okwengeziwe nge-Java/JVM ku-Kubernetes, bheka okuthunyelwe kwethu okulandelayo:

• Kungani uhlelo lwami lwe-Java luyi-OOMKilled;
• Ungawenza kanjani usayizi olungile weziqukathi zezinhlelo zokusebenza ze-Java 10.

Izilungiselelo zememori yomthengisi

Kunezici ezimbili ezibalulekile zokumisa inkumbulo yomthengisi: izilungiselelo ze-JVM neze-Kubernetes pod. Umkhawulo wememori osethelwe i-pod kufanele ube mkhulu kunosayizi omkhulu wenqwaba ukuze i-JVM ibe nendawo ye-Java metaspace ehlala kumemori yayo kanye nenqolobane yekhasi lesistimu yokusebenza esetshenziswa i-Kafka. Ezivivinyweni zethu sethule abathengisi be-Kafka ngamapharamitha -Xmx4G -Xms2G, futhi umkhawulo wememori we-pod wawungu 10 Gi. Sicela uqaphele ukuthi izilungiselelo zememori ze-JVM zingatholwa ngokuzenzakalelayo kusetshenziswa -XX:MaxRAMPercentage и -X:MinRAMPercentage, ngokusekelwe kumkhawulo wememori we-pod.

Izilungiselelo zephrosesa yomthengisi

Ngokuvamile, ungathuthukisa ukusebenza ngokwandisa ukufana ngokwandisa inani lezintambo ezisetshenziswa i-Kafka. Amaphrosesa engeziwe atholakalayo e-Kafka, angcono. Ekuhlolweni kwethu, siqale ngomkhawulo wamaphrosesa angu-6 futhi kancane kancane (ngokuphindaphinda) sakhuphula inombolo yawo yaba ngu-15. Ngaphezu kwalokho, sibeka num.network.threads=12 kuzilungiselelo zomthengisi ukwandisa inani lezintambo ezithola idatha kunethiwekhi futhi ziyithumele. Ngokushesha lapho bethola ukuthi abalandeli bama-broker abakwazanga ukuthola izifanekiso ngokushesha ngokwanele, baphakamisa num.replica.fetchers kuya ku-4 ukuze kukhuliswe isivinini lapho abadayisi babalandeli bephindaphinda khona imilayezo evela kubaholi.

Layisha Ithuluzi Lokukhiqiza

Kufanele uqinisekise ukuthi ijeneretha yokulayisha ekhethiwe ayiphelelwa amandla ngaphambi kokuthi iqoqo le-Kafka (elilinganiswayo) lifinyelele umthwalo walo omkhulu. Ngamanye amazwi, kuyadingeka ukwenza ukuhlolwa kokuqala kwamakhono ethuluzi lokukhiqiza umthwalo, futhi ukhethe izinhlobo zezibonelo zalo ngenani elanele lamaphrosesa nenkumbulo. Kulokhu, ithuluzi lethu lizokhiqiza umthwalo omningi ukwedlula iqoqo le-Kafka elingawuphatha. Ngemva kokuhlola okuningi, sahlala emakhophi amathathu c5.4okukhulu, ngayinye eyayinejeneretha esebenzayo.

Ukulinganisa izilinganiso

Isilinganiso sokusebenza siyinqubo ephindaphindayo ehlanganisa izigaba ezilandelayo:

• ukumisa ingqalasizinda (iqoqo le-EKS, iqoqo le-Kafka, ithuluzi lokukhiqiza umthwalo, kanye ne-Prometheus ne-Grafana);
• ukukhiqiza umthwalo isikhathi esithile ukuze kuhlungwe ukuchezuka okungahleliwe ezinkomba zokusebenza eziqoqiwe;
• ukulungisa ingqalasizinda yomthengisi kanye nokucushwa okusekelwe kuzinkomba zokusebenza eziqashiwe;
• ukuphinda inqubo kuze kube yilapho izinga elidingekayo le-Kafka cluster throughput lifinyelelwa. Ngesikhathi esifanayo, kufanele iphindaphindeke njalo futhi ibonise ukuhluka okuncane kokuphumayo.

Isigaba esilandelayo sichaza izinyathelo ezenziwe phakathi nenqubo yokulinganisa yeqoqo lokuhlola.

Amathuluzi

Amathuluzi alandelayo asetshenziswa ukuze kusetshenziswe ngokushesha ukucushwa kwesisekelo, ukukhiqiza imithwalo, nokulinganisa ukusebenza:

• I-Banzai Cloud Pipeline ngokuhlela iqoqo le-EKS elivela e-Amazon c Prometheus (ukuqoqa i-Kafka kanye namamethrikhi engqalasizinda) kanye UGrafana (ukubona ngeso lengqondo lawa mamethrikhi). Sasizakala kuhlanganisiwe в Iphayiphi izinsiza ezihlinzeka ngokuqapha okuhlangene, ukuqoqwa kwelogi endaweni eyodwa, ukuskena ubungozi, ukutakula kwezinhlekelele, ukuphepha kwebhizinisi nokunye okuningi.
• I-Sangrenel - ithuluzi lokuhlola umthwalo weqoqo le-Kafka.
• Amadeshibhodi e-Grafana okubuka ngeso lengqondo amamethrikhi we-Kafka nengqalasizinda: Kubernetes Kafka, I-Node Exporter.
• I-Supertubes CLI yendlela elula yokusetha iqoqo le-Kafka ku-Kubernetes. I-Zookeeper, i-opharetha ye-Kafka, i-Envoy nezinye izingxenye eziningi zifakiwe futhi zalungiselelwa kahle ukuze ziqhube iqoqo le-Kafka elilungele ukukhiqiza ku-Kubernetes.
  • Ukufakwa Ama-supertube CLI sebenzisa imiyalelo enikeziwe lapha.

Iqoqo le-EKS

Lungiselela iqoqo le-EKS elinamanodi abasebenzi abazinikele c5.4okukhulu ezindaweni ezihlukene zokutholakala kwama-pods nabathengi be-Kafka, kanye namanodi azinikele ejeneretha yokulayisha kanye nengqalasizinda yokuqapha.

banzai cluster create -f https://raw.githubusercontent.com/banzaicloud/kafka-operator/master/docs/benchmarks/infrastructure/cluster_eks_202001.json

Uma iqoqo le-EKS selivuliwe futhi lisebenza, vumela ukuhlanganiswa kwalo isevisi yokuqapha - uzosebenzisa u-Prometheus noGrafana babe yiqoqo.

Izingxenye zesistimu ye-Kafka

Faka izingxenye zesistimu ye-Kafka (Zookeeper, i-kafka-opharetha) ku-EKS usebenzisa ama-supertubes CLI:

supertubes install -a --no-democluster --kubeconfig <path-to-eks-cluster-kubeconfig-file>

Iqoqo le-Kafka

Ngokuzenzakalelayo, i-EKS isebenzisa imiqulu ye-EBS yohlobo gp2, ngakho-ke udinga ukudala ikilasi lesitoreji elihlukile ngokusekelwe kumavolumu io1 okweqoqo le-Kafka:

kubectl create -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1
  iopsPerGB: "50"
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer
EOF

Setha ipharamitha yabadayisi min.insync.replicas=3 futhi usebenzise ama-pods omthengisi kumanodi ezindaweni ezintathu ezihlukene ezitholakalayo:

supertubes cluster create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f https://raw.githubusercontent.com/banzaicloud/kafka-operator/master/docs/benchmarks/infrastructure/kafka_202001_3brokers.yaml --wait --timeout 600

Izihloko

Sisebenzise izikhathi ezintathu zokuphehla ukulayisha ngokuhambisanayo. Ngamunye wabo ubhala esihlokweni sakhe, okungukuthi, sidinga izihloko ezintathu sezizonke:

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
  name: perftest1
spec:
  name: perftest1
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
    name: perftest2
spec:
  name: perftest2
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
  name: perftest3
spec:
  name: perftest3
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

Esihlokweni ngasinye, isici sokuphindaphinda singu-3—inani elincane elinconyiwe lezinhlelo zokukhiqiza ezitholakala kakhulu.

Layisha Ithuluzi Lokukhiqiza

Sethule amakhophi amathathu ejeneretha yokulayisha (ngalinye libhale esihlokweni esihlukile). Ukuze uthole ama-pods generator, udinga ukusetha ukuhambisana kwe-node ukuze ahlelwe kuphela kuma-node abelwe wona:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: loadtest
  name: perf-load1
  namespace: kafka
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: loadtest
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: loadtest
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodepool.banzaicloud.io/name
                operator: In
                values:
                - loadgen
      containers:
      - args:
        - -brokers=kafka-0:29092,kafka-1:29092,kafka-2:29092,kafka-3:29092
        - -topic=perftest1
        - -required-acks=all
        - -message-size=512
        - -workers=20
        image: banzaicloud/perfload:0.1.0-blog
        imagePullPolicy: Always
        name: sangrenel
        resources:
          limits:
            cpu: 2
            memory: 1Gi
          requests:
            cpu: 2
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

Amaphuzu ambalwa okumele uwaqaphele:

• Ijeneretha yokulayisha ikhiqiza imilayezo engamabhayithi angu-512 ubude futhi iyishicilele e-Kafka ngamaqoqo emilayezo engu-500.
• Ukusebenzisa ingxabano -required-acks=all Ukushicilelwa kuthathwa njengempumelelo lapho zonke izifaniso ezivumelanisiwe zomlayezo zamukelwe futhi ziqinisekiswa ngabathengi be-Kafka. Lokhu kusho ukuthi kubhentshimakhi asikalanga nje isivinini sabaholi abathola imilayezo, kodwa nabalandeli babo abaphindaphinda imilayezo. Inhloso yalokhu kuhlola akukhona ukuhlola isivinini sokufunda komthengi (abathengi) isanda kuthola imilayezo esasele kunqolobane yekhasi le-OS, kanye nokuqhathaniswa kwayo nesivinini sokufunda semilayezo egcinwe kudiski.
• I-load generator isebenzisa abasebenzi abangu-20 ngokuhambisana (-workers=20). Isisebenzi ngasinye siqukethe abakhiqizi abangu-5 ababelana ngoxhumano lwesisebenzi kuqoqo le-Kafka. Ngenxa yalokho, i-generator ngayinye inabakhiqizi abayi-100, futhi bonke bathumela imilayezo kuqoqo le-Kafka.

Ukuqapha impilo yeqoqo

Ngesikhathi sokuhlolwa komthwalo weqoqo le-Kafka, siphinde saqapha impilo yalo ukuze siqinisekise ukuthi akukho ukuqalisa kabusha kwe-pod, azikho izifaniso ezingaphandle kokuvumelanisa, kanye nokuphuma okuphezulu okunokuguquguquka okuncane:

• Ijeneretha yokulayisha ibhala izibalo ezijwayelekile mayelana nenani lemilayezo eshicilelwe kanye nezinga lamaphutha. Izinga lephutha kufanele lihlale linjalo 0,00%.
• I-Cruise Control, ekhishwe yi-kafka-opharetha, inikeza ideshibhodi lapho singaphinda sigade isimo seqoqo. Ukuze ubuke leli phaneli yenza:

  supertubes cluster cruisecontrol show -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file>

• Izinga le-ISR (inombolo yezifaniso “ezivunyelanisiwe”) ukuncipha nokunwebeka kulingana no-0.

Imiphumela yokulinganisa

3 wabathengi, usayizi womlayezo - 512 bytes

Ngokuhlukaniswa okusatshalaliswe ngokulinganayo kubathengi abathathu, sikwazile ukuzuza ukusebenza ~500 Mb/s (imiyalezo elinganiselwa ku-990 XNUMX ngomzuzwana):

Ukusetshenziswa kwenkumbulo yomshini we-JVM we-virtual akuzange kudlule u-2 GB:

Ukuphuma kwediski kufinyelele umkhawulo ophezulu wenodi ye-I/O kuzo zonke izimo ezintathu lapho abathengisi bebesebenza khona:

Kusuka kudatha ekusetshenzisweni kwememori ngamanodi, kulandela ukuthi ukugcina kumthamo wesistimu kanye nokugcinwa kwesikhashana kuthathe ~10-15 GB:

3 wabathengi, usayizi womlayezo - 100 bytes

Njengoba usayizi womlayezo uncipha, ukudlulisa kwehla cishe ngo-15-20%: isikhathi esichithwe ukucubungula umlayezo ngamunye siyawuthinta. Ngaphezu kwalokho, umthwalo weprosesa cishe uphindwe kabili.

Njengoba ama-broker node asenama-cores angasetshenzisiwe, ukusebenza kungathuthukiswa ngokushintsha ukucushwa kwe-Kafka. Lona akuwona umsebenzi olula, ngakho-ke ukwandisa ukudlula kungcono ukusebenza ngemilayezo emikhulu.

4 wabathengi, usayizi womlayezo - 512 bytes

Ungakwazi ukwandisa kalula ukusebenza kweqoqo le-Kafka ngokumane ungeze ama-broker amasha nokugcina ibhalansi yama-partitions (lokhu kuqinisekisa ukuthi umthwalo usatshalaliswa ngokulinganayo phakathi kwabathengi). Esimweni sethu, ngemva kokwengeza umthengisi, i-throughput ye-cluster ikhuphuke yaba ~580 Mb/s (~1,1 million imilayezo ngomzuzwana). Ukukhula kube kuncane kunalokho obekulindelwe: lokhu kuchazwa ikakhulukazi ukungalingani kwama-partitions (akubona bonke abadayisi abasebenza phezulu kwamakhono abo).

Ukusetshenziswa kwememori komshini we-JVM kwahlala kungaphansi kwe-2 GB:

Umsebenzi wabadayisi abanamadrayivu uthintwa ukungalingani kokuhlukaniswa:

okutholakele

Indlela yokuphindaphinda eyethulwe ngenhla inganwetshwa ukuze ihlanganise izimo eziyinkimbinkimbi ezibandakanya amakhulukhulu abathengi, ukuhlukaniswa kabusha, ukuvuselelwa kabusha, ukuqaliswa kabusha kwe-pod, njll. Konke lokhu kusivumela ukuthi sihlole imikhawulo yamakhono eqoqo le-Kafka ezimeni ezihlukahlukene, sihlonze izingqinamba ekusebenzeni kwayo futhi sithole izindlela zokulwa nazo.

Siklame ama-Supertubes ukuthi akhiphe iqoqo ngokushesha futhi kalula, alilungise, engeze/asuse ama-broker nezihloko, aphendule izexwayiso, futhi aqinisekise ukuthi i-Kafka ngokuvamile isebenza kahle ku-Kubernetes. Umgomo wethu ukukusiza ukuthi ugxile emsebenzini oyinhloko ("khiqiza" futhi "udle" imilayezo ye-Kafka), futhi ushiye wonke umsebenzi onzima ku-Supertubes kanye no-opharetha we-Kafka.

Uma unentshisekelo kubuchwepheshe be-Banzai Cloud namaphrojekthi womthombo ovulekile, bhalisa enkampanini ethi GitHub, I-LinkedIn noma Twitter.

I-PS evela kumhumushi

Funda futhi kubhulogi yethu:

• «Indaba eyodwa no-opharetha we-Redis kuma-K8 kanye nokubuyekezwa okuncane kwezinsiza zokuhlaziya idatha evela kulesi sizindalwazi.";
• «Ukufuduka okungenamthungo kwe-RabbitMQ kuya e-Kubernetes";
• «zetcd evela ku-CoreOS: Ibuyisela i-ZooKeeper nge... njll".

Source: www.habr.com