Ukumisela ubungakanani obufanelekileyo beqela leKafka eKubernetes

Phawula. transl.: Kule nqaku, i-Banzai Cloud yabelana ngomzekelo wendlela izixhobo zayo eziqhelekileyo ezinokusetyenziswa ngayo ukwenza i-Kafka ibe lula ukuyisebenzisa ngaphakathi kwe-Kubernetes. Le miyalelo ilandelayo ibonisa indlela onokumisela ngayo ubukhulu besiseko sakho kwaye uqwalasele i-Kafka ngokwayo ukufezekisa i-throughput efunekayo.

I-Apache Kafka liqonga lokusasaza elisasazwayo lokudala iinkqubo ezinokuthenjwa, ezinokwehla kunye nokusebenza okuphezulu ngexesha langempela lokusasaza. Ubuchule bayo obunomtsalane bunokwandiswa kusetyenziswa iKubernetes. Ngenxa yoku siphuhlisele Open Source Kafka umsebenzisi kunye nesixhobo esibizwa ngokuba Iityhubhu ezinkulu. Bakuvumela ukuba usebenzise i-Kafka kwi-Kubernetes kwaye usebenzise iimpawu zayo ezahlukeneyo, ezinjengokulungisa kakuhle ulungelelwaniso lomthengisi, ukulinganisa okusekwe kwimetric ngokulinganisa kwakhona, ukwazi irack, "ithambile" (nobabalo) ukukhupha uhlaziyo, njl.

Zama ii-Supertubes kwiqela lakho:

curl https://getsupertubes.sh | sh и supertubes install -a --no-democluster --kubeconfig <path-to-eks-cluster-kubeconfig-file>

Okanye uqhagamshelane amaxwebhu. Unokufunda malunga nobunye ubuchule beKafka, umsebenzi owenziwe ngokuzenzekelayo usebenzisa iSupertubes kunye neKafka opharetha. Sele sibhale malunga nabo kwibhlog:

• Oh hayi! Ukanti omnye umqhubi weKafka weKubernetes;
• Beka iliso kwaye usebenzise i-Kafka esekwe kwi-Prometheus metrics;
• Kafka rack ulwazi kwi Kubernetes;
• Ukubaleka i-Apache Kafka ngaphezulu kwe-Istio-ibhentshi;
• Umsebenzisi oqinisekisiweyo kunye nokufikelela kumaqela alawulwayo kunye nomsebenzisi weKafka;
• Uphuculo oluqengqelekayo lweKafka kunye nolungelelwaniso oluguqukayo kwiKubernetes;
• Isihluzi seprotocol yomthunywa weKafka, enemeshed.

Xa uthatha isigqibo sokuthumela iqela le-Kafka kwi-Kubernetes, uya kujongana nomceli mngeni wokumisela ubungakanani obufanelekileyo beziseko ezingundoqo kunye nesidingo sokulungisa kakuhle ubumbeko bakho be-Kafka ukuhlangabezana neemfuno zokugqiba. Ukusebenza okuphezulu komthengisi ngamnye kuchongwa ngokusebenza kwamacandelo eziseko ezisisiseko, njengememori, iprosesa, isivinini sediski, i-bandwidth yenethiwekhi, njl.

Ngokufanelekileyo, ulungelelwaniso lomthengisi kufuneka lube lolo hlobo lokuba zonke izinto zeziseko zophuhliso zisetyenziselwe ubuninzi bamandla abo. Nangona kunjalo, kubomi bokwenyani olu cwangciso lunzima kakhulu. Kunokwenzeka ukuba abasebenzisi baqwalasele ii-brokers ukwandisa usetyenziso lwecandelo elinye okanye ezimbini (idiski, imemori, okanye iprosesa). Ngokubanzi, umthengisi ubonisa ukusebenza okuphezulu xa ubumbeko lwayo luvumela elona candelo licothayo ukuba lisetyenziswe ngokupheleleyo. Ngale ndlela sinokufumana umbono orhabaxa womthwalo onokuphathwa ngumthengisi omnye.

Ngokwethiyori, sinokuthelekelela inani labarhwebi abafunekayo ukuphatha umthwalo onikiweyo. Nangona kunjalo, ekusebenzeni kukho iinketho ezininzi zokucwangcisa kumanqanaba ahlukeneyo ukuba kunzima kakhulu (ukuba akunakwenzeka) ukuvavanya ukusebenza okunokwenzeka koqwalaselo oluthile. Ngamanye amazwi, kunzima kakhulu ukucwangcisa uqwalaselo olusekwe kumsebenzi othile onikiweyo.

Kubasebenzisi beSupertubes, sihlala sithatha le ndlela ilandelayo: siqala ngolungelelwaniso (iinfrastructure + setting), emva koko ulinganise ukusebenza kwayo, uhlengahlengise useto lwe-broker kwaye uphinde inkqubo kwakhona. Oku kwenzeka de elona candelo licothayo lesiseko lisetyenziswe ngokupheleleyo.

Ngale ndlela, sifumana umbono ocacileyo wokuba bangaphi na abarhwebi abadinga ukujongana nomthwalo othile (inani labarhwebi likwaxhomekeke kwezinye izinto, ezinje ngenani elincinane leempendulo zomyalezo ukuqinisekisa ukomelela, inani lokwahlulahlula. iinkokeli, njalo njalo). Ukongeza, sifumana ukuqonda ukuba ngawaphi amacandelo eziseko zophuhliso afuna ukulinganiswa ngokuthe nkqo.

Eli nqaku liza kuthetha malunga namanyathelo esiwathathayo ukufumana uninzi lwawona macandelo acothayo kuqwalaselo lokuqala kunye nokulinganisa i-throughput yeqela le-Kafka. Uqwalaselo oluluqilima kakhulu lufuna ubuncinci ii-broker ezintathu ezisebenzayo (min.insync.replicas=3), isasazwe kwiindawo ezintathu ezahlukeneyo zofikelelo. Ukuqwalasela, ukukala kunye nokubeka iliso kwisiseko se-Kubernetes, sisebenzisa eyethu iqonga lolawulo lwesikhongozeli samafu axubileyo - Iipayipi. Ixhasa kwisiseko (intsimbi engenanto, VMware) kunye neentlobo ezintlanu zamafu (Alibaba, AWS, Azure, Google, Oracle), kunye nayo nayiphi na indibaniselwano yazo.

Iingcamango kwiziseko zophuhliso lweqela laseKafka kunye noqwalaselo

Kwimizekelo engezantsi, sikhethe i-AWS njengomboneleli welifu kunye ne-EKS njengosasazo lwe-Kubernetes. Ubumbeko obufanayo bunokuphunyezwa ngokusetyenziswa P.K.E. - Ukuhanjiswa kwe-Kubernetes kwi-Banzai Cloud, eqinisekisiwe yi-CNCF.

Drive

IAmazon ibonelela ngeendlela ezahlukeneyo Iintlobo zevolumu ye-EBS. Embindini gp2 и io1 kukho iidrives ze-SSD, nangona kunjalo, ukuqinisekisa ukuhamba okuphezulu gp2 isebenzisa iikhredithi eziqokelelweyo (I/O credits), ngoko sikhethe uhlobo io1, enikezela nge-throughput ephezulu engaguqukiyo.

Iintlobo zemizekelo

Ukusebenza kwe-Kafka kuxhomekeke kakhulu kwi-cache yekhasi lenkqubo yokusebenza, ngoko sifuna iimeko ezinememori eyaneleyo yabathengi (JVM) kunye ne-cache yephepha. Umzekelo c5.2 enkulu - isiqalo esihle, ekubeni ine-16 GB yememori kunye ilungiselelwe ukusebenza ne-EBS. Ukungalungi kwayo kukuba iyakwazi ukubonelela ukusebenza okuphezulu kuphela kwimizuzu engama-30 rhoqo kwiiyure ezingama-24. Ukuba umthwalo wakho womsebenzi ufuna ukusebenza okuphezulu ngexesha elide, unokufuna ukuqwalasela ezinye iindidi zemizekelo. Yiloo nto kanye esayenzayo, simisa kuyo c5.4 enkulu. Ibonelela ngemveliso ephezulu 593,75 Mb/s. Esona siphumo siphezulu somthamo we-EBS io1 ngaphezulu kunomzekelo c5.4 enkulu, ngoko ke eyona elementi icothayo yesiseko inokuba yi-I/O yokuphuma kolu hlobo lomzekelo (ekufuneka uvavanyo lwethu lomthwalo nalo luqinisekise).

Inethiwekhi

I-network throughput kufuneka ibe nkulu ngokwaneleyo xa kuthelekiswa nokusebenza komzekelo we-VM kunye nediski, kungenjalo inethiwekhi iba yingxaki. Kwimeko yethu, ujongano lwenethiwekhi c5.4 enkulu ixhasa izantya zokuya kuthi ga kwi-10 Gb/s, ephezulu kakhulu kune-I/O yokuphuma komzekelo we-VM.

Ukusasazwa kwee-Broker

Ii-brokers kufuneka zibekwe (zicwangciswe kwi-Kubernetes) kwiindawo ezizinikeleyo ukuphepha ukukhuphisana nezinye iinkqubo ze-CPU, imemori, inethiwekhi, kunye nezixhobo zediski.

Uguqulelo lweJava

Ukhetho olunengqiqo yiJava 11 kuba iyahambelana neDocker ngengqiqo yokuba i-JVM imisela ngokuchanekileyo abaqhubekisi kunye nememori ekhoyo kwisikhongozeli apho umthengisi asebenza khona. Ukwazi ukuba imida ye-CPU ibalulekile, i-JVM yangaphakathi kwaye ibeka ngokucacileyo inani lemicu ye-GC kunye neentambo ze-JIT. Sisebenzise umfanekiso weKafka banzaicloud/kafka:2.13-2.4.0, equka i-Kafka version 2.4.0 (Scala 2.13) kwiJava 11.

Ukuba ungathanda ukufunda ngakumbi malunga neJava/JVM kwiKubernetes, jonga izithuba zethu ezilandelayo:

• Kutheni isicelo sam seJava si-OOMKilled;
• Uzenza njani ngokuchanekileyo izikhongozeli zeJava 10 izicelo.

Useto lwememori ye-Broker

Kukho imiba emibini ephambili yokuqwalasela imemori ye-broker: izicwangciso ze-JVM kunye ne-Kubernetes pod. Umda wememori omiselwe i-pod kufuneka ube mkhulu kunobungakanani benqwaba yemfumba ukuze i-JVM ibe nendawo ye-Java metaspace, ehlala kwimemori yayo, kunye ne-cache yekhasi lenkqubo yokusebenza, esetyenziswa nguKafka ngokusebenzayo. Kwiimvavanyo zethu saqalisa i-Kafka brokers kunye neeparameters -Xmx4G -Xms2G, kwaye umda wememori wepod wawu 10 Gi. Nceda uqaphele ukuba useto lwememori ye-JVM inokufumaneka ngokuzenzekelayo kusetyenziswa -XX:MaxRAMPercentage и -X:MinRAMPercentage, ngokusekelwe kumda wememori yepod.

Iisetingi zeprosesa ye-Broker

Ngokubanzi, unokuphucula ukusebenza ngokunyusa ukuhambelana ngokunyusa inani lemisonto esetyenziswa yiKafka. Iiprosesa ezininzi ezikhoyo kwi-Kafka, ngcono. Kuvavanyo lwethu, saqala ngomda weeprosesa ze-6 kwaye ngokuthe ngcembe (ngokuphindaphinda) baphakamisa inani labo kwi-15. Ukongeza, sibeka num.network.threads=12 kwizicwangciso zomthengisi ukwandisa inani lemisonto efumana idatha kwinethiwekhi kwaye uyithumele. Ngoko nangoko bafumanisa ukuba abalandeli abarhwebi abanakufumana iikopi ngokukhawuleza ngokwaneleyo, baphakamisa num.replica.fetchers ukuya ku-4 ukwandisa isantya apho abalandeli bee-brokers baphindaphinda imiyalezo evela kwiinkokeli.

IsiXhobo sokuLayisha

Kufuneka uqinisekise ukuba ijenereyitha yomthwalo ekhethiweyo ayiphelelwa ngumthamo phambi kokuba iqela le-Kafka (elibekwe uphawu) lifike kumthwalo walo omkhulu. Ngamanye amazwi, kuyimfuneko ukwenza uvavanyo lokuqala lwezakhono zesixhobo sokuvelisa umthwalo, kwaye ukhethe iindidi zemizekelo yazo kunye nenani elaneleyo leeprosesa kunye nememori. Kule meko, isixhobo sethu siya kuvelisa umthwalo ongaphezulu kunokuba iqela leKafka linokusingatha. Emva kovavanyo oluninzi, sazinza kwiikopi ezintathu c5.4 enkulu, nganye kuzo yayinejenereyitha esebenzayo.

Ukuthelekisa

Umlinganiselo wokwenziwa komsebenzi yinkqubo ephindaphindwayo equka ezi zigaba zilandelayo:

• ukuseka iziseko zophuhliso (iqela le-EKS, iqela laseKafka, isixhobo sokuvelisa umthwalo, kwakunye nePrometheus neGrafana);
• ukuvelisa umthwalo kwixesha elithile ukuze kuhluzwe ukutenxa ngokungaqhelekanga kwiimpawu zokusebenza eziqokelelweyo;
• ukulungelelanisa iziseko ezingundoqo zomrhwebi kunye nolungelelwaniso olusekwe kwizalathi zokusebenza eziqatshelweyo;
• ukuphinda inkqubo de inqanaba elifunekayo le-Kafka cluster throughput liphunyeziwe. Kwangaxeshanye, kufuneka iphinde ikwazi ukuphinda iphinde iphinde iphinde ibonise umahluko omncinci kwi-output.

Icandelo elilandelayo lichaza amanyathelo awenziweyo ngexesha lenkqubo yokulinganisa yeqela lovavanyo.

Zixhobo

Ezi zixhobo zilandelayo zisetyenziselwe ukuhambisa ngokukhawuleza ubumbeko lwesiseko, ukuvelisa imithwalo, kunye nokulinganisa ukusebenza:

• I-Banzai Cloud Pipeline ukulungiselela iqela le-EKS elivela eAmazon c Prometheus (ukuqokelela iKafka kunye neemetrics zeziseko zophuhliso) kunye IGrafana (ukubona ezi metrics). Sathatha ithuba idityanisiwe в Iipayipi iinkonzo ezibonelela ngokujongwa okudibeneyo, ukuqokelelwa kwelog kwindawo esembindini, ukuskena ukuba sesichengeni, ukubuyiswa kweentlekele, ukhuseleko lomgangatho woshishino nokunye okuninzi.
• Sangrenel - isixhobo sokuvavanya umthwalo kwiqela leKafka.
• Iideshibhodi zaseGrafana zokujonga iimetrics zeKafka kunye neziseko zophuhliso: Kubernetes Kafka, I-Node yangaphandle.
• Supertubes CLI yeyona ndlela ilula yokuseta iqela leKafka kwiKubernetes. I-Zookeeper, umqhubi we-Kafka, uMthunywa kunye namanye amacandelo amaninzi afakwe kwaye aqwalaselwe ngokufanelekileyo ukuqhuba imveliso-elungele i-Kafka cluster kwi-Kubernetes.
  • Yofakelo ii-supertubes CLI sebenzisa imiyalelo enikiweyo apha.

Iqela le-EKS

Lungiselela iqela le-EKS elinabasebenzi abazinikeleyo c5.4 enkulu kwiindawo ezifumanekayo ezahlukeneyo zeepods ezinee-brokers ze-Kafka, kunye neendawo ezinikezelweyo zejenereyitha yomthwalo kunye neziseko zokuhlola.

banzai cluster create -f https://raw.githubusercontent.com/banzaicloud/kafka-operator/master/docs/benchmarks/infrastructure/cluster_eks_202001.json

Nje ukuba iqela le-EKS liphakame kwaye lisebenze, vumela ukuba lidityaniswe inkonzo yokubeka iliso - uya kuthumela iPrometheus kunye neGrafana kwiqela.

Amacandelo enkqubo yeKafka

Faka amacandelo enkqubo yeKafka (Zookeeper, kafka-operator) kwi-EKS usebenzisa ii-supertubes CLI:

supertubes install -a --no-democluster --kubeconfig <path-to-eks-cluster-kubeconfig-file>

Iqela laseKafka

Ngokungagqibekanga, i-EKS isebenzisa imiqulu ye-EBS yohlobo gp2, ngoko kufuneka udale iklasi yokugcina eyahlukileyo ngokusekelwe kwimiqulu io1 yeqela laseKafka:

kubectl create -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1
  iopsPerGB: "50"
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer
EOF

Seta iparameter yabarhwebi min.insync.replicas=3 kwaye ubeke iipod zomthengisi kwiindawo ezifumanekayo kwiindawo ezintathu ezahlukeneyo zokufumaneka:

supertubes cluster create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f https://raw.githubusercontent.com/banzaicloud/kafka-operator/master/docs/benchmarks/infrastructure/kafka_202001_3brokers.yaml --wait --timeout 600

Imixholo

Siqhube imizekelo emithathu yejenereyitha ngokuhambelanayo. Ngamnye kubo ubhala kwisihloko sakhe, oko kukuthi, sifuna izihloko ezithathu zizonke:

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
  name: perftest1
spec:
  name: perftest1
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
    name: perftest2
spec:
  name: perftest2
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
  name: perftest3
spec:
  name: perftest3
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

Kwisihloko ngasinye, i-replication factor yi-3-ixabiso elincinci elicetyiswayo kwiinkqubo zokuvelisa ezifumaneka kakhulu.

IsiXhobo sokuLayisha

Saqalisa iikopi ezintathu zejenereyitha yomthwalo (ngamnye wabhala kwisihloko esahlukileyo). Kwimibhobho yejenereyitha yomthwalo, kufuneka usete i-node affinity ukuze icwangciswe kuphela kwiindawo ezabelwe zona:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: loadtest
  name: perf-load1
  namespace: kafka
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: loadtest
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: loadtest
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodepool.banzaicloud.io/name
                operator: In
                values:
                - loadgen
      containers:
      - args:
        - -brokers=kafka-0:29092,kafka-1:29092,kafka-2:29092,kafka-3:29092
        - -topic=perftest1
        - -required-acks=all
        - -message-size=512
        - -workers=20
        image: banzaicloud/perfload:0.1.0-blog
        imagePullPolicy: Always
        name: sangrenel
        resources:
          limits:
            cpu: 2
            memory: 1Gi
          requests:
            cpu: 2
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

Amanqaku ambalwa ekufuneka uwaqaphele:

• Ijeneretha yomthwalo ivelisa imiyalezo ye-512 bytes ubude kwaye ipapashe kwi-Kafka kwiibhetshi zemiyalezo ye-500.
• Ukusebenzisa ingxabano -required-acks=all Upapasho luthathwa njengoluyimpumelelo xa zonke iikopi ezidityanisiweyo zomyalezo zamkelwa kwaye ziqinisekiswa ngabathengisi beKafka. Oku kuthetha ukuba kwi-benchmark asilinganisanga kuphela isantya seenkokeli ezifumana imiyalezo, kodwa nabalandeli babo baphindaphinda imiyalezo. Injongo yolu vavanyo ayikokuvavanya isantya sokufunda kwabathengi (abathengi) kutshanje ifunyenwe imiyalezo eshiyekileyo kwi-cache yephepha le-OS, kunye nothelekiso lwayo nesantya sokufunda semiyalezo egcinwe kwidiski.
• Ijenereyitha yomthwalo iqhuba abasebenzi abangama-20 ngaxeshanye (-workers=20). Umsebenzi ngamnye uqulethe abavelisi aba-5 ababelana ngoqhagamshelwano lomsebenzi kwiqela le-Kafka. Ngenxa yoko, i-generator nganye inabakhiqizi be-100, kwaye bonke bathumela imiyalezo kwi-cluster yaseKafka.

Ukubeka iliso kwimpilo yeqela

Ngexesha lokuvavanywa komthwalo weqela le-Kafka, siye sabeka iliso kwimpilo yalo ukuqinisekisa ukuba akukho pod iphinda iqalwe, akukho zikopi ziphuma ngaphandle kwe-sync, kunye nokuphuma okuphezulu kunye nokuguquguquka okuncinci:

• Umenzi womthwalo ubhala amanani asemgangathweni malunga nenani lemiyalezo epapashwe kunye nezinga lempazamo. Izinga lempazamo kufuneka lihlale lifana 0,00%.
• Ulawulo lokuhamba emanzini, isetyenziswe yi-kafka-operator, ibonelela ngedeshibhodi apho sinokubeka iliso kwimeko yeqela. Ukujonga eli qela lolawulo yenza:

  supertubes cluster cruisecontrol show -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file>

• Inqanaba le-ISR (inani le "in-sync" replicas) Ukuncipha kunye nokwandiswa kuyalingana no-0.

Iziphumo zomlinganiselo

Iibroker ezi-3, ubungakanani bomyalezo - 512 bytes

Ngokwahlulahlula ngokulinganayo kubo bonke abarhwebi abathathu, sakwazi ukuphumeza ukusebenza ~500 Mb/s (malunga nama-990 amawaka emiyalezo ngomzuzwana):

Ukusetyenziswa kwememori yomatshini obonakalayo we-JVM awudlulanga i-2 GB:

Ugqithiso lwediski lufikelele kowona mthamo uphezulu we-I/O wenode kuzo zontathu iimeko apho abathengisi bebeqhuba:

Ukusuka kwidatha ekusetyenzisweni kwememori ngamaqhuqhuva, kulandela ukuba inkqubo yebuffering kunye necaching ithathe ~10-15 GB:

Iibroker ezi-3, ubungakanani bomyalezo - 100 bytes

Njengoko ubungakanani bomyalezo buncipha, umthamo we-output wehla malunga ne-15-20%: ixesha elichithwe ukusetyenzwa komyalezo ngamnye liyawuchaphazela. Ukongeza, umthwalo weprosesa uphantse waphinda kabini.

Ekubeni ii-broker nodes zisenazo ii-cores ezingasetyenziswanga, ukusebenza kunokuphuculwa ngokutshintsha uqwalaselo lwe-Kafka. Lo ayingomsebenzi ulula, ngoko ke ukonyusa i-throughput kungcono ukusebenza ngemiyalezo emikhulu.

Iibroker ezi-4, ubungakanani bomyalezo - 512 bytes

Unokwandisa ngokulula ukusebenza kweqela le-Kafka ngokungongeza nje ii-brokers ezintsha kunye nokugcina ibhalansi yezahlulo (oku kuqinisekisa ukuba umthwalo usasazwa ngokulinganayo phakathi kwee-brokers). Kwimeko yethu, emva kokongeza i-broker, i-throughput ye-cluster yanda ukuya ~580 Mb/s (~1,1 million imiyalezo ngomzuzwana). Ukukhula kuye kwaba ngaphantsi kunokuba bekulindelwe: oku kuchazwa ikakhulu ngokungalingani kwezahlulo (ayingabo bonke abarhwebi abasebenza kwincopho yezakhono zabo).

Ukusetyenziswa kwememori yomatshini we-JVM kwahlala kungaphantsi kwe-2 GB:

Umsebenzi wabarhwebi abaneedrive wachatshazelwa kukungalingani kwezahlulo:

ezifunyanisiweyo

Indlela yokuphindaphinda evezwe ngasentla inokwandiswa ukugubungela iimeko ezinzima ngakumbi ezibandakanya amakhulukhulu abathengi, ukwahlulahlula, uhlaziyo oluqengqelekayo, ukuqaliswa ngokutsha kweepod, njl.njl. Konke oku kusivumela ukuba sihlole imida yezakhono zeqela le-Kafka kwiimeko ezahlukeneyo, ukuchonga iibhotile ekusebenzeni kwayo kwaye ufumane iindlela zokulwa nazo.

Siye sayila iiSupertubes ukuba zikhawuleze kwaye zisebenzise iqela ngokukhawuleza, liyiqwalasele, yongeze/isuse iibrokhwe kunye nezihloko, iphendule kwizilumkiso, kwaye iqinisekise ukuba iKafka ngokubanzi isebenza ngokufanelekileyo kwiKubernetes. Injongo yethu kukukunceda ukuba ugxininise kumsebenzi ophambili ("ukuvelisa" kwaye "udle" imiyalezo ye-Kafka), kwaye ushiye wonke umsebenzi onzima kwi-Supertubes kunye nomqhubi we-Kafka.

Ukuba unomdla kwitekhnoloji yeBanzai Cloud kunye neeprojekthi zoMthombo oVulekileyo, bhalisa kwinkampani GitHub, LinkedIn okanye Twitter.

PS evela kumguquleli

Funda nakwibhlog yethu:

• «Ibali elinye kunye nomsebenzisi weRedis kwi-K8s kunye nophononongo oluncinci lwezinto eziluncedo zokuhlalutya idatha kule database.";
• «I-RabbitMQ engenamthungo ukuya kwi-Kubernetes imfuduko";
• «zetcd ukusuka kwi-CoreOS: Ukutshintsha i-ZooKeeper nge... njl njl».

umthombo: www.habr.com