Ingabe i-Kafka ku-Kubernetes inhle?

Sawubona, Habr!

Ngesinye isikhathi, saba ngabokuqala ukwethula isihloko emakethe yaseRussia KAFKA futhi uqhubeke ithrekhi ukuze ithuthuke. Ikakhulukazi, sithole isihloko sokusebenzisana phakathi kweKafka ne Kubernetes. Kuyaqapheleka (futhi ngokucophelela) indatshana lesi sihloko sashicilelwa ku-Confluent blog emuva ngo-Okthoba nyakenye ngaphansi kokubhalwa kukaGwen Shapira. Namuhla sithanda ukudonsela ukunaka kwakho esihlokweni sakamuva esivela ku-April ngu-Johann Gyger, okuthi nakuba engenalo uphawu lombuzo esihlokweni, uhlola isihloko ngendlela eqinile, ehambisana nombhalo ngezixhumanisi ezithakazelisayo. Sicela usithethelele ukuhunyushwa kwamahhala kwe-"chaos monkey" uma ukwazi!

Isingeniso

I-Kubernetes yakhelwe ukuphatha imithwalo yemisebenzi engapheli. Imvamisa, imithwalo yemisebenzi enjalo yethulwa ngendlela ye-microservice architecture, ayisindi, ilinganisa kahle ngokuvundlile, ilandela imigomo yezinhlelo zokusebenza eziyi-12, futhi ingasebenza nama-circuit breaker kanye nezinkawu ze-chaos.

I-Kafka, ngakolunye uhlangothi, isebenza njengesizindalwazi esabalalisiwe. Ngakho-ke, lapho usebenza, kufanele ubhekane nombuso, futhi unzima kakhulu kune-microservice. I-Kubernetes isekela imithwalo yemfanelo, kodwa njengoba u-Kelsey Hightower ebonisa kuma-tweets amabili, kufanele aphathwe ngokucophelela:

Abanye abantu banomuzwa wokuthi uma ufaka i-Kubernetes emthwalweni womsebenzi omuhle, iba isizindalwazi esiphethwe ngokugcwele esiqhudelana ne-RDS. Lokhu akulungile. Mhlawumbe, uma usebenza kanzima ngokwanele, engeza izingxenye ezengeziwe futhi uhehe ithimba lonjiniyela be-SRE, uzokwazi ukwakha i-RDS phezu kwe-Kubernetes.

Ngihlala ngincoma ukuthi wonke umuntu aqaphe ngokwedlulele lapho esebenzisa imithwalo enzima yemisebenzi ku-Kubernetes. Abantu abaningi ababuza ukuthi “ngingakwazi yini ukwenza umsebenzi omningi ku-Kubernetes” abanalo ulwazi olwanele nge-Kubernetes, futhi ngokuvamile ngomthwalo womsebenzi ababuza ngawo.

Ngakho-ke, kufanele usebenzise i-Kafka ku-Kubernetes? Umbuzo ophikisayo: ingabe iKafka izosebenza kangcono ngaphandle kukaKubernetes? Kungakho ngifuna ukugqamisa kulesi sihloko ukuthi uKafka noKubernetes baphelelisana kanjani, nokuthi yiziphi izingibe ezingafika ngokuzihlanganisa.

Isikhathi sokuqeda

Ake sikhulume ngento eyisisekelo - imvelo yesikhathi sokusebenza ngokwayo

Inqubo

Abathengi be-Kafka banobungani be-CPU. I-TLS ingase yethule okunye okungaphezulu. Nokho, amaklayenti e-Kafka angase abe ne-CPU ejulile uma esebenzisa ukubethela, kodwa lokhu akuthinti ama-broker.

inkumbulo

Abathengi be-Kafka badla inkumbulo. Usayizi wenqwaba ye-JVM ngokuvamile ukhawulelwe ku-4-5 GB, kodwa uzodinga futhi inkumbulo yesistimu eningi njengoba i-Kafka isebenzisa inqolobane yekhasi kakhulu. Ku-Kubernetes, setha insiza yesiqukathi bese ucela imikhawulo ngokufanele.

Isitolo sedatha

Ukugcinwa kwedatha ezitsheni kungokwesikhashana - idatha iyalahleka lapho iqalwa kabusha. Ngedatha ye-Kafka ungasebenzisa ivolumu emptyDir, futhi umphumela uzofana: idatha yakho yomthengisi izolahleka ngemva kokuqedwa. Imilayezo yakho isengagcinwa kwabanye abadayisi njengezifaniso. Ngakho-ke, ngemva kokuqalisa kabusha, umthengisi ohlulekile kufanele aqale aphindaphinde yonke idatha, futhi le nqubo ingathatha isikhathi esiningi.

Yingakho kufanele usebenzise ukugcinwa kwedatha yesikhathi eside. Makube okungesona okwendawo isitoreji sesikhathi eside ngohlelo lwefayela le-XFS noma, ngokuqondile, i-ext4. Ungasebenzisi i-NFS. Ngikuxwayisile. Izinguqulo ze-NFS v3 noma i-v4 ngeke zisebenze. Ngamafuphi, umthengisi we-Kafka uzophahlazeka uma engakwazi ukususa uhla lwedatha ngenxa yenkinga "yokuqamba kabusha oyisiphukuphuku" ku-NFS. Uma ngingakakukholisanga, ngokucophelela funda lesi sihloko. Isitolo sedatha kufanele singabi esasendaweni ukuze u-Kubernetes akwazi ukukhetha kalula indawo entsha ngemva kokuqalisa kabusha noma ukuthuthwa.

Inethiwekhi

Njengamasistimu amaningi asabalalisiwe, ukusebenza kwe-Kafka kuncike kakhulu ekugcineni ukubambezeleka kwenethiwekhi kubuncane kanye nomkhawulokudonsa ufike phezulu. Ungazami ukusingatha bonke abadayisi endaweni eyodwa, njengoba lokhu kuzonciphisa ukutholakala. Uma i-node ye-Kubernetes ihluleka, lonke iqoqo le-Kafka lizohluleka. Futhi, ungasabalalisi iqoqo le-Kafka kuzo zonke izikhungo zedatha. Okufanayo kuya kwiqoqo le-Kubernetes. Ukuvumelana okuhle kuleli cala ukukhetha izindawo zokutholakala ezihlukene.

Ukucushwa

I-manifestos evamile

Iwebhusayithi ye-Kubernetes ine umhlahlandlela omuhle kakhulu mayelana nendlela yokumisa i-ZooKeeper usebenzisa i-manifest. Njengoba i-ZooKeeper iyingxenye ye-Kafka, lena indawo enhle yokuqala ukujwayelana nemiqondo ye-Kubernetes esebenza lapha. Uma usuqonda lokhu, ungasebenzisa imiqondo efanayo neqoqo le-Kafka.

• Ngaphansi: I-pod iyunithi encane esebenzisekayo e-Kubernetes. I-pod iqukethe umsebenzi wakho, futhi i-pod ngokwayo ihambisana nenqubo yeqoqo lakho. I-pod iqukethe isitsha esisodwa noma ngaphezulu. Iseva ngayinye ye-ZooKeeper ekuhlanganisweni kanye nomthengisi ngamunye kuqoqo le-Kafka uzosebenza ku-pod ehlukile.
• StatefulSet: I-StatefulSet iyinto ye-Kubernetes ephatha imithwalo yemisebenzi eminingi, futhi imithwalo enjalo idinga ukuxhumana. I-StatefulSets inikeza iziqinisekiso mayelana noku-oda ama-pods kanye nokuhluka kwawo.
• Amasevisi angenakhanda: Amasevisi akuvumela ukuthi ukhiphe ama-pod kumakhasimende usebenzisa igama elinengqondo. U-Kubernetes kuleli cala unesibopho sokulinganisa ukulayisha. Kodwa-ke, lapho usebenzisa imithwalo yemisebenzi emihle, efana ne-ZooKeeper ne-Kafka, amaklayenti adinga ukuxhumana nesenzakalo esithile. Yilapho izinsizakalo ezingenakhanda zisiza khona: kulokhu, iklayenti lisazoba negama elinengqondo, kodwa ngeke kudingeke ukuthi uxhumane ne-pod ngokuqondile.
• Ivolumu yesitoreji eside: Lawa mavolumu ayadingeka ukuze kumiswe isitoreji esiqhubekayo sokuvimba okungesona esasendaweni okukhulunywe ngenhla.

In Yolean Ihlinzeka ngesethi ephelele yama-manifest ukukusiza ukuthi uqalise nge-Kafka ku-Kubernetes.

Amashadi e-Helm

UHelm ungumphathi wephakheji ye-Kubernetes engafaniswa nabaphathi bephakheji ye-OS njenge-yum, i-apt, i-Homebrew noma i-Chocolatey. Kwenza kube lula ukufaka amaphakheji esoftware achazwe ngaphambilini kumashadi e-Helm. Ishadi le-Helm elikhethwe kahle lenza umsebenzi onzima wokuthi ungamisa kanjani kahle yonke imingcele ukuze usebenzise i-Kafka ku-Kubernetes kalula. Kunemidwebo eminingana ye-Kafka: esemthethweni itholakala esimweni se-incubator, kukhona okuvela Ukudideka, omunye ngaphezulu - kusuka Bitnami.

Abasebenzisi

Ngenxa yokuthi iHelm inokushiyeka okuthile, elinye ithuluzi lithola ukuthandwa okukhulu: Ama-opharetha e-Kubernetes. Umsebenzisi akagcini nje ngokupakisha isoftware ye-Kubernetes, kodwa futhi ikuvumela ukuthi usebenzise isoftware enjalo futhi uyilawule.

Ohlwini opharetha emangalisayo Kukhulunywa ngama-opharetha amabili e-Kafka. Omunye wabo - Strimzi. Nge-Strimzi, kulula ukwenza iqoqo lakho le-Kafka lisebenze futhi ngamaminithi. Cishe akukho ukucushwa okudingekayo, ngaphezu kwalokho, u-opharetha ngokwawo uhlinzeka ngezici ezinhle, isibonelo, ukubethela kwe-point-to-point ye-TLS ngaphakathi kweqoqo. I-Confluent iphinde inikeze opharetha uqobo.

Ukukhiqiza

Kubalulekile ukuhlola ukusebenza ngokulinganisa isibonelo sakho se-Kafka. Ukuhlolwa okunjalo kuzokusiza ukuthi uthole izingqinamba ezingase zibe khona ngaphambi kokuba kuvele izinkinga. Ngenhlanhla, i-Kafka isivele ihlinzeka ngamathuluzi amabili okuhlola ukusebenza: kafka-producer-perf-test.sh и kafka-consumer-perf-test.sh. Zisebenzise ngokugcwele. Ukuze uthole ireferensi, ungabheka imiphumela echazwe ku lokhu okuthunyelwe Jay Kreps, noma gxila lokhu kubuyekezwa Amazon MSK by Stéphane Maarek.

Ukusebenza

Ukuqapha

Ukucaca ohlelweni kubaluleke kakhulu - ngaphandle kwalokho ngeke uqonde ukuthi kwenzekani kulo. Namuhla kunekhithi yamathuluzi eqinile ehlinzeka ngokuqapha okusekelwe kumamethrikhi ngesitayela somdabu samafu. Amathuluzi amabili adumile ale njongo i-Prometheus ne-Grafana. I-Prometheus ingaqoqa amamethrikhi kuzo zonke izinqubo ze-Java (i-Kafka, i-Zookeeper, i-Kafka Connect) isebenzisa isithekelisi se-JMX - ngendlela elula. Uma ungeza amamethrikhi e-cAdvisor, ungaqonda ngokugcwele ukuthi izinsiza zisetshenziswa kanjani ku-Kubernetes.

U-Strimzi unesibonelo esikahle kakhulu sedeshibhodi ye-Grafana ye-Kafka. Ibona ngeso lengqondo amamethrikhi angukhiye, isibonelo, mayelana nemikhakha engaphindaphindeki kangako noma leyo engaxhunyiwe ku-inthanethi. Konke kucace kakhulu lapho. Lawa mamethrikhi ahambisana nokusetshenziswa kwensiza kanye nolwazi lokusebenza, kanye nezinkomba zokuzinza. Ngakho uthola ukuqapha okuyisisekelo kweqoqo le-Kafka mahhala!

Source: streamzi.io/docs/master/#kafka_dashboard

Kungaba kuhle ukwenezela konke lokhu ngokuqapha kwamakhasimende (amamethrikhi kubathengi nabakhiqizi), kanye nokuqapha ukubambezeleka (kwalokhu kukhona Burrow) kanye nokuqapha kokugcina - kulokhu kusetshenziswa I-Kafka Monitor.

Ukugawula

Ukugawula omunye umsebenzi obalulekile. Qiniseka ukuthi zonke iziqukathi ekufakweni kwakho kwe-Kafka zifakwe kuzo stdout и stderr, futhi uqinisekise ukuthi iqoqo lakho le-Kubernetes lihlanganisa wonke amalogi abe nengqalasizinda yokugawulwa kwemithi emaphakathi, isb. Islastiki.

Ukusebenza kokuhlolwa

I-Kubernetes isebenzisa ukuphila nokulungela ukuhlola ukuthi ama-pods akho asebenza ngokujwayelekile yini. Uma ukuhlola ukuphila kwehluleka, i-Kubernetes izomisa leso siqukathi bese isiqala kabusha ngokuzenzakalelayo uma inqubomgomo yokuqalisa kabusha isethwe ngokufanele. Uma ukuhlolwa kokulungela kwehluleka, i-Kubernetes ihlukanisa i-pod ezicelweni zokunikeza isevisi. Ngakho-ke, ezimweni ezinjalo, ukungenelela okwenziwa ngesandla akusadingeki nhlobo, okuyi-plus enkulu.

Ikhipha izibuyekezo

I-StatefulSets isekela izibuyekezo ezizenzakalelayo: uma ukhetha isu le-RollingUpdate, ngalinye elingaphansi kwe-Kafka lizobuyekezwa ngokulandelana. Ngale ndlela, isikhathi sokuphumula singancishiswa sibe ziro.

Ukukala

Ukukala iqoqo le-Kafka akuwona umsebenzi olula. Kodwa-ke, i-Kubernetes yenza kube lula kakhulu ukukala ama-pods enanini elithile le-replicas, okusho ukuthi ungachaza ngokusobala abathengisi abaningi be-Kafka ngendlela othanda ngayo. Into enzima kakhulu kuleli cala ukwaba kabusha imikhakha ngemuva kokwenyusa noma ngaphambi kokwehlisa. Futhi, uKubernetes uzokusiza ngalo msebenzi.

Ukuphatha

Imisebenzi ehlobene nokuphatha iqoqo lakho le-Kafka, njengokudala izihloko nokwabela kabusha imikhakha, ingenziwa kusetshenziswa imibhalo yegobolondo ekhona ngokuvula isixhumi esibonakalayo somugqa womyalo kuma-pods akho. Nokho, lesi sixazululo asisihle kakhulu. I-Strimzi isekela ukuphatha izihloko isebenzisa u-opharetha ohlukile. Kukhona indawo yokuthuthukisa lapha.

Yenza isipele futhi ubuyisele

Manje ukutholakala kwe-Kafka nakho kuzoncika ekutholakaleni kwe-Kubernetes. Uma iqoqo lakho le-Kubernetes lehluleka, khona-ke esimweni esibi kakhulu, iqoqo lakho le-Kafka nalo lizohluleka. Ngokomthetho ka-Murphy, lokhu kuzokwenzeka nakanjani, futhi uzolahlekelwa idatha. Ukunciphisa lolu hlobo lobungozi, yiba nomqondo omuhle wokusekelayo. Ungasebenzisa i-MirrorMaker, enye inketho ukusebenzisa i-S3 yalokhu, njengoba kuchazwe kulokhu okuthunyelwe from Zalando.

isiphetho

Uma usebenza namaqoqo e-Kafka amancane ukuya kosayizi omaphakathi, kufanelekile impela ukusebenzisa i-Kubernetes njengoba ihlinzeka ngokuguquguquka okwengeziwe futhi yenza kube lula ukuzizwisa komsebenzisi. Uma unezidingo ezibaluleke kakhulu zokubambezeleka ezingasebenzi kanye/noma zokuphuma, kungase kube ngcono ukucabangela enye inketho yokusebenzisa.

Source: www.habr.com