Puas yog Kafka ntawm Kubernetes zoo?

Nyob zoo, Habr!

Nyob rau hauv ib lub sij hawm, peb yog thawj zaug los qhia lub ntsiab lus rau Lavxias teb sab kev ua lag luam Kafka thiab txuas ntxiv mus taug qab rau nws txoj kev loj hlob. Tshwj xeeb, peb pom lub ntsiab lus ntawm kev sib cuam tshuam ntawm Kafka thiab Kubernetes. Observable (thiab ceev faj heev) ib tsab xov xwm cov ncauj lus no tau luam tawm ntawm Confluent blog rov qab rau lub Kaum Hli xyoo tas los raws li kev sau ntawv ntawm Gwen Shapira. Niaj hnub no peb xav kos koj cov xim rau ib tsab xov xwm tsis ntev los no los ntawm lub Plaub Hlis los ntawm Johann Gyger, uas, txawm hais tias tsis muaj cov lus nug hauv lub npe, tshuaj xyuas cov ncauj lus hauv ntau qhov tseem ceeb, nrog rau cov ntawv nyeem nrog kev sib txuas zoo. Thov zam txim rau peb qhov kev txhais dawb ntawm "chaos liab" yog tias koj ua tau!

Taw qhia

Kubernetes yog tsim los tswj cov haujlwm tsis muaj zog. Feem ntau, cov khoom ua haujlwm zoo li no tau nthuav tawm nyob rau hauv daim ntawv ntawm microservice architecture, lawv yog lub teeb yuag, scale zoo horizontally, ua raws li cov hauv paus ntsiab lus ntawm 12-factor applications, thiab tuaj yeem ua haujlwm nrog Circuit Court breakers thiab chaos liab.

Kafka, ntawm qhov tod tes, qhov tseem ceeb ua raws li cov ntaub ntawv faib tawm. Yog li, thaum ua haujlwm, koj yuav tsum ua haujlwm nrog lub xeev, thiab nws hnyav dua li microservice. Kubernetes txhawb nqa lub xeev loads, tab sis raws li Kelsey Hightower taw qhia hauv ob tweets, lawv yuav tsum tau saib xyuas:

Qee tus neeg xav tias yog tias koj dov Kubernetes rau hauv lub xeev ua haujlwm, nws dhau los ua cov ntaub ntawv tswj hwm tag nrho uas sib tw RDS. Qhov no tsis yog lawm. Tej zaum, yog tias koj ua haujlwm hnyav txaus, ntxiv cov khoom ntxiv thiab nyiam pab pawg SRE engineers, koj tuaj yeem tsim RDS rau saum Kubernetes.

Kuv ib txwm xav kom txhua tus siv ceev faj thaum ua haujlwm ua haujlwm ntawm Kubernetes. Cov neeg feem coob uas nug "Kuv puas tuaj yeem khiav cov haujlwm tseem ceeb ntawm Kubernetes" tsis muaj kev paub txaus nrog Kubernetes, thiab feem ntau nrog cov haujlwm uas lawv nug txog.

Yog li, koj yuav tsum khiav Kafka ntawm Kubernetes? Counter question: Puas yuav Kafka ua haujlwm zoo dua yam tsis muaj Kubernetes? Tias yog vim li cas kuv thiaj xav hais qhia hauv tsab xov xwm no yuav ua li cas Kafka thiab Kubernetes ntxiv rau ib leeg, thiab dab tsi pitfalls tuaj yeem tuaj nrog kev sib koom ua ke.

Lub sijhawm ua tiav

Cia peb tham txog qhov yooj yim tshaj plaws - lub sijhawm ua haujlwm ib puag ncig nws tus kheej

txheej txheem

Kafka brokers yog CPU tus phooj ywg. TLS tuaj yeem qhia qee qhov nyiaj siv ua haujlwm. Txawm li cas los xij, Kafka cov neeg siv khoom yuav yog CPU siv ntau dua yog tias lawv siv encryption, tab sis qhov no tsis cuam tshuam rau tus broker.

nco

Kafka brokers noj nco. JVM heap loj feem ntau txwv rau 4-5 GB, tab sis koj tseem yuav xav tau ntau lub cim xeeb vim Kafka siv nplooj ntawv cache hnyav heev. Hauv Kubernetes, teeb tsa cov peev txheej ntim thiab thov txwv kom haum.

Khaws cov ntaub ntawv

Cov ntaub ntawv khaws cia hauv cov ntim khoom yog ephemeral - cov ntaub ntawv ploj thaum rov pib dua. Rau cov ntaub ntawv Kafka koj tuaj yeem siv lub ntim emptyDir, thiab cov nyhuv yuav zoo sib xws: koj cov ntaub ntawv broker yuav ploj tom qab ua tiav. Koj cov lus tseem tuaj yeem muab khaws cia rau ntawm lwm tus broker ua cov ntawv luam. Yog li ntawd, tom qab rov pib dua, tus broker ua tsis tiav yuav tsum xub rov ua dua tag nrho cov ntaub ntawv, thiab cov txheej txheem no tuaj yeem siv sijhawm ntau.

Qhov no yog vim li cas koj yuav tsum siv cov ntaub ntawv ntev ntev. Cia nws tsis nyob hauv lub sijhawm ntev cia nrog XFS cov ntaub ntawv system lossis, ntau dua, ext4. Tsis txhob siv NFS. kuv ceeb toom koj. NFS versions v3 lossis v4 yuav tsis ua haujlwm. Hauv luv luv, Kafka broker yuav tsoo yog tias nws tsis tuaj yeem tshem tawm cov ntaub ntawv teev npe vim qhov teeb meem "rename ruam" hauv NFS. Yog tias kuv tseem tsis tau ntseeg koj, ua tib zoo saib nyeem tsab xov xwm no. Cov ntaub ntawv khaws cia yuav tsum tsis yog nyob hauv zos kom Kubernetes tuaj yeem hloov kho tau yooj yim xaiv qhov chaw tshiab tom qab rov pib dua lossis hloov chaw.

Network

Raws li nrog rau feem ntau cov tshuab, Kafka qhov kev ua tau zoo yog nyob ntawm kev ua kom lub network latency kom tsawg thiab bandwidth mus txog qhov siab tshaj plaws. Tsis txhob sim tuav txhua tus broker ntawm tib lub node, vim qhov no yuav txo qhov muaj. Yog tias Kubernetes node ua tsis tiav, tag nrho Kafka pawg yuav ua tsis tiav. Tsis tas li ntawd, tsis txhob cuam tshuam cov Kafka pawg thoob plaws tag nrho cov chaw zov me nyuam. Ib yam mus rau Kubernetes pawg. Ib qho kev sib haum xeeb zoo nyob rau hauv rooj plaub no yog xaiv cov chaw muaj nyob sib txawv.

Configuration

Tsis tu ncua manifestos

Kubernetes lub vev xaib muaj zoo heev qhia hais txog yuav ua li cas teeb tsa ZooKeeper siv manifests. Txij li thaum ZooKeeper yog ib feem ntawm Kafka, qhov no yog qhov chaw zoo pib kom paub txog cov ntsiab lus Kubernetes siv ntawm no. Thaum koj nkag siab qhov no, koj tuaj yeem siv tib lub ntsiab lus nrog Kafka pawg.

• Hauv qab: Lub taub hau yog qhov tsawg tshaj plaws deployable unit nyob rau hauv Kubernetes. Lub plhaub taum muaj koj cov haujlwm ua haujlwm, thiab lub plhaub taum nws tus kheej sib raug rau cov txheej txheem hauv koj pawg. Ib lub kaus mom muaj ib lossis ntau lub thawv. Txhua tus neeg rau zaub mov ZooKeeper hauv pawg thiab txhua tus broker hauv Kafka pawg yuav khiav hauv ib qho chaw cais.
• StatefulSet: Lub StatefulSet yog Kubernetes cov khoom uas ua haujlwm ntau lub xeev ua haujlwm, thiab cov haujlwm no yuav tsum muaj kev sib koom tes. StatefulSets muab kev lees paub txog kev txiav txim ntawm cov pods thiab lawv qhov tshwj xeeb.
• Cov kev pabcuam tsis muaj taub hau: Cov kev pabcuam tso cai rau koj tshem cov pods ntawm cov neeg siv khoom siv lub npe logic. Kubernetes nyob rau hauv cov ntaub ntawv no yog lub luag hauj lwm rau load ntsuas. Txawm li cas los xij, thaum ua haujlwm cov haujlwm tseem ceeb, xws li ZooKeeper thiab Kafka, cov neeg siv khoom yuav tsum sib txuas lus nrog ib qho piv txwv. Qhov no yog qhov kev pabcuam tsis muaj taub hau tuaj yeem ua ke: qhov no, tus neeg siv khoom tseem yuav muaj lub npe zoo, tab sis koj tsis tas yuav hu rau lub pod ncaj qha.
• Lub sij hawm ntev cia ntim: Cov ntim no yog xav tau los teeb tsa cov khoom tsis nyob hauv zos tsis tu ncua uas tau hais los saum toj no.

rau Yolean Muab cov txheej txheem nthuav dav los pab koj pib nrog Kafka ntawm Kubernetes.

Helm kab kos

Helm yog tus thawj tswj pob rau Kubernetes uas tuaj yeem muab piv rau OS cov tswj hwm pob xws li yum, apt, Homebrew lossis Chocolatey. Nws ua rau nws yooj yim rau nruab predefined pob software piav nyob rau hauv Helm kab kos. Ib daim ntawv qhia Helm uas tau xaiv zoo ua rau txoj haujlwm nyuaj ntawm yuav ua li cas thiaj li teeb tsa txhua qhov tsis zoo siv Kafka ntawm Kubernetes yooj yim. Muaj ob peb daim duab Kafka: qhov chaw nyob nyob rau hauv incubator mob, muaj ib los ntawm Caum, ib tug ntxiv - los ntawm Bitnami.

Cov

Vim Helm muaj qee qhov tsis txaus, lwm lub cuab yeej tau txais txiaj ntsig ntau: Kubernetes cov neeg ua haujlwm. Tus neeg teb xov tooj tsis tsuas yog pob software rau Kubernetes, tab sis kuj tso cai rau koj siv cov software no thiab tswj nws.

Hauv daim ntawv amazing neeg ua haujlwm Ob tus neeg ua haujlwm rau Kafka tau hais. Ib tug ntawm lawv - Strimzi. Nrog Strimzi, nws yooj yim kom tau koj Kafka pawg thiab khiav hauv feeb. Tsis tas li yuav tsum muaj kev teeb tsa zoo, ntxiv rau, tus neeg teb xov tooj nws tus kheej muab qee yam zoo, piv txwv li, point-to-point TLS encryption hauv pawg. Confluent kuj muab tus kheej ua haujlwm.

Tsim tau

Nws yog ib qho tseem ceeb rau kev ntsuas kev ua tau zoo los ntawm kev ntsuas koj li Kafka piv txwv. Cov kev ntsuam xyuas zoo li no yuav pab koj nrhiav pom muaj peev xwm fwj tsis tau ua ntej teeb meem pib. Luckily, Kafka twb muab ob qho kev ntsuas kev ua tau zoo: kafka-producer-perf-test.sh и kafka-consumer-perf-test.sh. Ua kom nquag siv ntawm lawv. Rau kev siv, koj tuaj yeem xa mus rau cov txiaj ntsig tau piav qhia hauv tsab ntawv no Jay Kreps, los yog tsom rau qhov kev tshuaj xyuas no Amazon MSK los ntawm Stéphane Maarek.

Haujlwm

Saib xyuas

Transparency nyob rau hauv lub system yog ib qho tseem ceeb heev - txwv tsis pub koj yuav tsis to taub dab tsi tshwm sim nyob rau hauv nws. Niaj hnub no muaj cov cuab yeej cuab tam uas muab cov ntsuas ntsuas raws li kev soj ntsuam hauv huab cua ib txwm muaj. Ob lub cuab yeej nrov rau lub hom phiaj no yog Prometheus thiab Grafana. Prometheus tuaj yeem sau cov metrics los ntawm tag nrho cov txheej txheem Java (Kafka, Zookeeper, Kafka Txuas) siv JMX exporter - hauv txoj kev yooj yim tshaj plaws. Yog tias koj ntxiv cAdvisor metrics, koj tuaj yeem nkag siab ntau dua li cas cov peev txheej siv hauv Kubernetes.

Strimzi muaj qhov piv txwv yooj yim ntawm Grafana dashboard rau Kafka. Nws pom cov metrics tseem ceeb, piv txwv li, hais txog cov haujlwm uas tsis tau hloov dua tshiab lossis cov uas tsis muaj. Txhua yam yog meej heev nyob ntawd. Cov kev ntsuas no tau ua tiav los ntawm kev siv cov peev txheej thiab cov ntaub ntawv ua haujlwm, nrog rau cov ntsuas kev ruaj ntseg. Yog li koj tau txais kev saib xyuas Kafka pawg tsis muaj dab tsi!

Tau qhov twg los: streamzi.io/docs/master/#kafka_dashboard ua

Nws yuav zoo rau ntxiv tag nrho cov no nrog kev saib xyuas cov neeg siv khoom (kev ntsuas ntawm cov neeg siv khoom thiab cov neeg tsim khoom), nrog rau kev saib xyuas latency (rau qhov no muaj Burrow) thiab qhov kawg-rau-kawg saib xyuas - rau qhov kev siv no Kafka Saib.

Kev sau npe

Kev sau npe yog lwm txoj haujlwm tseem ceeb. Xyuas kom tseeb tias tag nrho cov ntim hauv koj lub Kafka installation tau nkag mus rau stdout и stderr, thiab tseem xyuas kom meej tias koj cov Kubernetes pawg sib sau ua ke tag nrho cov ntawv teev npe rau hauv lub hauv paus kev nkag mus rau hauv qhov chaw, xws li. Elasticsearch.

Kev Ntsuas Kev Ntsuas

Kubernetes siv qhov kev ua neej nyob thiab kev npaj txhij los soj ntsuam seb koj cov pods puas tau khiav ib txwm. Yog tias qhov kev kuaj xyuas tsis muaj sia nyob, Kubernetes yuav nres lub thawv ntawd thiab tom qab ntawd rov pib dua nws yog tias txoj cai rov pib dua tau teeb tsa raws li. Yog tias qhov kev kuaj npaj ua tsis tiav, Kubernetes cais lub plhaub los ntawm kev thov kev pabcuam. Yog li, nyob rau hauv cov ntaub ntawv no, phau ntawv kev cuam tshuam tsis tas yuav tsum tau lawm, uas yog ib tug loj ntxiv.

Rolling tawm tshiab

StatefulSets txhawb kev hloov kho tsis siv neeg: yog tias koj xaiv RollingUpdate lub tswv yim, txhua tus hauv Kafka yuav raug hloov kho tshiab. Ua li no, downtime yuav raug txo kom xoom.

Scaling

Scaling ib pawg Kafka tsis yog ib txoj haujlwm yooj yim. Txawm li cas los xij, Kubernetes ua rau nws yooj yim heev los ntsuas cov pods rau qee tus lej ntawm cov khoom siv, uas txhais tau tias koj tuaj yeem tshaj tawm txhais tau ntau tus neeg ua haujlwm Kafka raws li koj nyiam. Qhov nyuaj tshaj plaws nyob rau hauv cov ntaub ntawv no yog reassigning sectors tom qab scaling nce los yog ua ntej scaling down. Ntxiv dua thiab, Kubernetes yuav pab koj ua txoj haujlwm no.

Kev tswj hwm

Cov haujlwm uas cuam tshuam txog kev tswj hwm koj pawg Kafka, xws li tsim cov ncauj lus thiab rov muab cov haujlwm, tuaj yeem ua tiav siv cov ntawv plhaub uas twb muaj lawm los ntawm kev qhib cov kab hais kom ua hauv koj cov pods. Txawm li cas los xij, qhov kev daws teeb meem no tsis zoo nkauj heev. Strimzi txhawb kev tswj cov ntsiab lus siv tus neeg teb xov tooj sib txawv. Muaj qee qhov kev txhim kho ntawm no.

Thaub qab thiab restore

Tam sim no qhov muaj ntawm Kafka kuj tseem nyob ntawm qhov muaj Kubernetes. Yog tias koj pawg Kubernetes ua tsis tiav, tom qab ntawd hauv qhov xwm txheej phem tshaj plaws, koj pawg Kafka kuj yuav ua tsis tiav. Raws li Murphy txoj cai, qhov no yuav tshwm sim tiag tiag, thiab koj yuav poob cov ntaub ntawv. Txhawm rau txo qhov kev pheej hmoo no, muaj lub tswv yim zoo thaub qab. Koj tuaj yeem siv MirrorMaker, lwm qhov kev xaiv yog siv S3 rau qhov no, raws li tau piav qhia hauv qhov no ncej los ntawm Zalando.

xaus

Thaum ua hauj lwm nrog me me mus rau nruab nrab-qhov loj me Kafka pawg, nws yog qhov tsim nyog siv Kubernetes vim nws muab kev yooj yim ntxiv thiab ua kom yooj yim rau cov neeg siv kev paub. Yog tias koj muaj qhov tseem ceeb tsis ua haujlwm latency thiab / lossis cov kev xav tau dhau los, ces nws yuav zoo dua los xav txog qee qhov kev xaiv xa mus.

Tau qhov twg los: www.hab.com