Molo, Habr!
Ngesinye isikhathi, saba ngabokuqala ukwazisa isihloko kwimarike yaseRashiya kwaye uqhubeke kuphuhliso lwayo. Ngokukodwa, sifumene isihloko sokusebenzisana phakathi kweKafka kunye . Kuyaqapheleka (kwaye ulumke kakhulu) esi sihloko sapapashwa kwiblogi ye-Confluent emva kwe-Okthobha kulo nyaka uphelileyo phantsi kolawulo lukaGwen Shapira. Namhlanje singathanda ukutsalela ingqalelo yakho kwinqaku lakutshanje ukususela ngo-Apreli ngu-Johann Gyger, owathi, nangona engenalo uphawu lombuzo kwisihloko, uhlola isihloko ngendlela engaphezulu, ehamba kunye nesicatshulwa kunye nezikhonkco ezinomdla. Nceda usixolele inguqulelo yasimahla ye "chaos monkey" ukuba unako!
Intshayelelo
I-Kubernetes yenzelwe ukuphatha imisebenzi emininzi. Ngokuqhelekileyo, imisebenzi enjalo ibonakaliswe ngendlela ye-microservice architecture, ilula, ilinganisa kakuhle ngokuthe tye, ilandele imigaqo ye-12-factor applications, kwaye inokusebenza kunye nabaphuli beesekethe kunye neenkawu ze-chaos.
I-Kafka, kwelinye icala, isebenza njengesiseko sedatha esabiweyo. Ke, xa usebenza, kufuneka ujongane norhulumente, kwaye inzima kakhulu kune-microservice. I-Kubernetes ixhasa imithwalo enzima, kodwa njengoko uKelsey Hightower echaza kwiitweets ezimbini, kufuneka ziphathwe ngononophelo:
Abanye abantu bavakalelwa kukuba ukuba uqengqa i-Kubernetes ibe ngumthwalo onzima womsebenzi, iba yindawo egciniweyo elawulwa ngokupheleleyo ekhuphisana ne-RDS. Oku akulunganga. Mhlawumbi, ukuba usebenza nzima ngokwaneleyo, yongeza izinto ezongezelelweyo kwaye utsale iqela leenjineli ze-SRE, uya kuba nakho ukwakha i-RDS phezu kwe-Kubernetes.
Ndihlala ndicebisa ukuba wonke umntu alumke kakhulu xa eqhuba umsebenzi onzima kwi-Kubernetes. Uninzi lwabantu ababuza ukuba "ndingaqhuba umsebenzi omninzi kwi-Kubernetes" abanawo amava aneleyo nge-Kubernetes, kwaye bahlala benomthwalo womsebenzi abawubuzayo.
Ke, ngaba kufuneka uqhube iKafka kwiKubernetes? Umbuzo ophikisayo: ngaba iKafka iya kusebenza ngcono ngaphandle kweKubernetes? Yiyo loo nto ndifuna ukuqaqambisa kweli nqaku ukuba uKafka kunye noKubernetes bancedisana njani, kwaye yeyiphi imigibe enokuza ngokuyidibanisa.
Ixesha lokugqitywa
Makhe sithethe ngento esisiseko - indawo yokuqhuba ixesha ngokwayo
Inkqubo
Abarhwebi baseKafka banobuhlobo kwi-CPU. I-TLS inokwazisa i-overhead ethile. Nangona kunjalo, abathengi be-Kafka banokuba nzulu ngakumbi kwi-CPU ukuba basebenzisa i-encryption, kodwa oku akuchaphazeli abathengisi.
Imemori
Abathengisi beKafka batya inkumbulo. Ubungakanani bemfumba ye-JVM buhlala bukhawulelwe kwi-4-5 GB, kodwa uya kufuna imemori eninzi yenkqubo kuba iKafka isebenzisa i-cache yephepha kakhulu. Kwi-Kubernetes, seta isibonelelo sesikhongozeli kwaye ucele imida ngokufanelekileyo.
Ivenkile yedatha
Ukugcinwa kwedatha kwizikhongozeli yi-ephemeral - idatha ilahlekile xa iqalwa kwakhona. Kwidatha yeKafka ungasebenzisa umthamo emptyDir, kwaye umphumo uya kufana: idatha yakho yomthengisi iya kulahleka emva kokugqitywa. Imiyalezo yakho isenokugcinwa kwezinye iibroker njenge replicas. Ngoko ke, emva kokuqalisa kwakhona, umthengisi ongaphumelelanga kufuneka aqale aphindaphinde yonke idatha, kwaye le nkqubo inokuthatha ixesha elininzi.
Kungenxa yoko le nto kufuneka usebenzise ukugcinwa kwedatha ixesha elide. Mayibe ingeyiyo indawo yokugcina ixesha elide kunye nenkqubo yefayile yeXFS okanye, ngokuchanekileyo, ext4. Musa ukusebenzisa i-NFS. Ndikulumkisile. Iinguqulelo ze-NFS v3 okanye i-v4 aziyi kusebenza. Ngamafutshane, umthengisi we-Kafka uya kuphazamiseka ukuba akakwazi ukucima isikhombisi sedatha ngenxa yengxaki "yegama elibi" kwi-NFS. Ukuba andikakuqinisekisi, ngononophelo olukhulu . Isitoreji sedatha kufuneka singabi sendawo ukwenzela ukuba i-Kubernetes ikwazi ukukhetha ngokuguquguqukayo i-node entsha emva kokuqalisa ngokutsha okanye ukufuduswa.
Inethiwekhi
Njengoko uninzi lweenkqubo ezisasazwayo, ukusebenza kweKafka kuxhomekeke kakhulu ekugcineni i-latency yenethiwekhi ukuya kubuncinci kunye ne-bandwidth ukuya phezulu. Ungazami ukusingatha zonke ii-broker kwindawo enye, njengoko oku kuya kunciphisa ukufumaneka. Ukuba i-node ye-Kubernetes iyasilela, yonke i-cluster ye-Kafka iya kusilela. Kwakhona, musa ukusasaza iqela le-Kafka kuwo wonke amaziko edatha. Okufanayo kuya kwiqela leKubernetes. Ukulungelelaniswa okulungileyo kule meko kukukhetha imimandla eyahlukeneyo yokufumaneka.
Isimo
Iimanifestos rhoqo
Iwebhusayithi yeKubernetes ine malunga nendlela yokuqwalasela iZooKeeper usebenzisa imiboniso. Kuba iZooKeeper iyinxalenye yeKafka, le yindawo entle yokuqalisa ukuqhelana negama leKubernetes elisebenza apha. Nje ukuba uqonde oku, ungasebenzisa iikhonsepthi ezifanayo kunye neqela leKafka.
- Ngaphantsi: I-pod yeyona yunithi incinci enokuhanjiswa eKubernetes. I-pod iqulethe umthwalo wakho womsebenzi, kwaye i-pod ngokwayo ihambelana nenkqubo kwiqela lakho. I-pod iqulethe isikhongozeli esinye okanye ngaphezulu. Iseva nganye ye-ZooKeeper kwi-ensemble kunye nomthengisi ngamnye kwiqela le-Kafka liya kuqhuba kwi-pod eyahlukileyo.
- StatefulSet: I-StatefulSet yinto ye-Kubernetes ephethe imithwalo yemisebenzi emininzi, kwaye loo misebenzi ifuna ulungelelwaniso. I-StatefulSets ibonelela ngeziqinisekiso malunga nokuodolwa kweepods kunye nokwahluka kwazo.
- Iinkonzo ezingenantloko: Iinkonzo zikuvumela ukuba ukhuphe iipod kubathengi usebenzisa igama elinengqiqo. I-Kubernetes kule meko inoxanduva lokulinganisa umthwalo. Nangona kunjalo, xa usebenza ngomthwalo onzima, onje ngeZooKeeper kunye neKafka, abathengi kufuneka banxibelelane nomzekelo othile. Kulapho iinkonzo ezingenantloko zingena khona: kulo mzekelo, umxhasi uya kuba negama elinengqiqo, kodwa akuyi kufuneka uqhagamshelane ngqo ne-pod.
- Umthamo wokugcina ixesha elide: Le mithamo iyafuneka ukuze kuqwalaselwe ibhloko engekho yendawo yokugcina eqhubekayo ekhankanywe ngasentla.
phezu Ibonelela ngeseti ebanzi yokubonisa ukukunceda ukuba uqalise ngeKafka kwiKubernetes.
Iitshathi zeHelm
I-Helm ngumphathi wephakheji ye-Kubernetes enokuthelekiswa nabaphathi bephakheji ye-OS njenge-yum, i-apt, i-Homebrew okanye i-Chocolatey. Kwenza kube lula ukufaka iipakethe zesoftware echazwe kwangaphambili kwiitshathi zeHelm. Itshathi yeHelm ekhethwe kakuhle yenza umsebenzi onzima wendlela yokuqwalasela ngokufanelekileyo zonke iiparameters ukusebenzisa iKafka kwiKubernetes lula. Kukho imizobo emininzi yeKafka: esemthethweni ikhona , kukho enye evela , enye ngaphezulu - ukusuka .
Ba sebenzi
Ngenxa yokuba iHelm ineziphene ezithile, esinye isixhobo sifumana ukuthandwa okukhulu: Abaqhubi beKubernetes. Umsebenzisi akapakishi kuphela isoftware ye-Kubernetes, kodwa ikuvumela ukuba usebenzise isoftware enjalo kwaye uyilawule.
Kuluhlu Abaqhubi ababini beKafka bakhankanyiwe. Omnye wazo - . Nge-Strimzi, kulula ukufumana iqela lakho le-Kafka kwaye lisebenze ngemizuzu. Phantse akukho lungelelwaniso lufunekayo, ukongeza, umqhubi ngokwakhe ubonelela ngeempawu ezintle, umzekelo, i-point-to-point TLS encryption ngaphakathi kweqela. I-Confluent ikwabonelela .
Imveliso
Kubalulekile ukuvavanya ukusebenza ngokuthelekisa umzekelo wakho weKafka. Iimvavanyo ezinjalo ziya kukunceda ufumane imiqobo enokubakho ngaphambi kokuba kuvele iingxaki. Ngethamsanqa, iKafka sele ibonelela ngezixhobo ezimbini zokuvavanya ukusebenza: kafka-producer-perf-test.sh и kafka-consumer-perf-test.sh. Zisebenzise ngenkuthalo. Ngereferensi, ungabhekisa kwiziphumo ezichazwe kwi Jay Kreps, okanye ugxininise IAmazon MSK nguStéphane Maarek.
UkuSebenza
Ukubeka iliso
Ukucaca kwinkqubo kubaluleke kakhulu - kungenjalo awuyi kuqonda into eyenzekayo kuyo. Namhlanje kukho isixhobo esiluqilima esibonelela ngokujongwa okusekwe kwimethrikhi kwisitayile somthonyama samafu. Izixhobo ezibini ezidumileyo kule njongo yiPrometheus kunye neGrafana. I-Prometheus inokuqokelela i-metrics kuzo zonke iinkqubo zeJava (i-Kafka, i-Zookeeper, i-Kafka Connect) isebenzisa umthengisi we-JMX - ngendlela elula. Ukuba wongeza i-cAdvisor metrics, unokuqonda ngokupheleleyo ukuba izixhobo zisetyenziswa njani kwi-Kubernetes.
U-Strimzi unomzekelo oluncedo kakhulu wedeshibhodi yaseGrafana yeKafka. Ijonga iimethrikhi eziphambili, umzekelo, malunga namacandelo angaphindaphindekiyo okanye lawo angekho kwi-intanethi. Yonke into icace gca apho. Ezi metrics zincediswa kusetyenziso lwezibonelelo kunye neenkcukacha zokusebenza, kunye nezalathi zokuzinza. Ke ufumana iliso elisisiseko leqela leKafka ngaphandle kwento!
umthombo:
Kuya kuba kuhle ukongeza konke oku ngokubeka iliso kubathengi (iimetrics kubathengi kunye nabavelisi), kunye nokubeka iliso kwi-latency (koku kukho ) kunye nokubekwa esweni kokuphela - kolu setyenziso .
Ukugawulwa kwemithi
Ukugawulwa kwemithi ngomnye umsebenzi obalulekileyo. Qinisekisa ukuba zonke izikhongozeli kufakelo lwakho lwe-Kafka zifakwe kuzo stdout и stderr, kwaye uqinisekise ukuba iqela lakho le-Kubernetes lidibanisa zonke iilogi kwiziseko ezingundoqo zokugawulwa kwemithi, umz. .
Umsebenzi osebenzayo
I-Kubernetes isebenzisa ubomi kunye nokulungela ukuphonononga ukujonga ukuba iipods zakho zisebenza ngesiqhelo. Ukuba ujongo lokuphila aluphumeleli, i-Kubernetes iyakumisa eso sikhongozeli kwaye emva koko siyiqale ngokutsha ngokuzenzekelayo ukuba umgaqo-nkqubo wokuqalisa ngokutsha usetwe ngokufanelekileyo. Ukuba ukukhangela ukulungela kusilela, i-Kubernetes ihlukanisa i-pod kwizicelo zenkonzo. Ngaloo ndlela, kwiimeko ezinjalo, ukungenelela kwezandla akusafuneki konke konke, eyona nto idibanisa kakhulu.
Ukukhupha uhlaziyo
I-StatefulSets ixhasa uhlaziyo oluzenzekelayo: ukuba ukhetha isicwangciso seRollingUpdate, nganye phantsi kweKafka iya kuhlaziywa ngokulandelelana. Ngale ndlela, ixesha lokuphumla lingancitshiswa libe ngu-zero.
Ukukala
Ukukala iqela leKafka akuwona umsebenzi olula. Nangona kunjalo, i-Kubernetes yenza kube lula kakhulu ukukala iipods ukuya kwinani elithile leekopi, okuthetha ukuba ungachaza ngokubhengeza uninzi lwabathengi be-Kafka njengoko uthanda. Eyona nto inzima kulo mzekelo kukwabela amacandelo kwakhona emva kokunyusa okanye ngaphambi kokuthoba. Kwakhona, iKubernetes iya kukunceda ngalo msebenzi.
Ulawulo
Imisebenzi enxulumene nokulawula iqela lakho le-Kafka, elifana nokudala izihloko kunye nokwabela kwakhona amacandelo, inokwenziwa ngokusebenzisa izikripthi ezikhoyo zeqokobhe ngokuvula ujongano lomgca womyalelo kwiipods zakho. Nangona kunjalo, esi sisombululo asihle kakhulu. I-Strimzi ixhasa ukulawula izihloko usebenzisa umsebenzisi owahlukileyo. Kukho indawo yokuphucula apha.
Gcina kwaye ubuyisele
Ngoku ukufumaneka kweKafka kuya kuxhomekeka ekufumanekeni kweKubernetes. Ukuba iqela lakho leKubernetes liyasilela, ke kwimeko embi kakhulu, iqela lakho leKafka nalo liya kusilela. Ngokomthetho kaMurphy, oku kuya kwenzeka ngokuqinisekileyo, kwaye uya kulahlekelwa idatha. Ukunciphisa olu hlobo lomngcipheko, yiba nengcamango elungileyo yokugcina. Ungasebenzisa i-MirrorMaker, enye inketho kukusebenzisa i-S3 kule nto, njengoko kuchaziwe kule evela eZalando.
isiphelo
Xa usebenza kunye neqela elincinci ukuya kwephakathi leKafka, ngokuqinisekileyo kufanelekile ukusebenzisa i-Kubernetes njengoko ibonelela ngokuguquguquka okongezelelweyo kwaye yenza lula amava omqhubi. Ukuba uneemfuno ezibalulekileyo ezingasebenziyo zokulinda kunye/okanye ukugqiba, kungangcono ukuqwalasela enye indlela yokusasazwa.
umthombo: www.habr.com