Kodi Kafka pa Kubernetes ndi yabwino?

Moni, Habr!

Panthawi ina, tinali oyamba kuyambitsa mutuwu kumsika waku Russia Kafka ndi kupitiriza njira za chitukuko chake. Makamaka, tapeza mutu wa kuyanjana pakati pa Kafka ndi Kubernetes. Zowoneka (ndi mosamala kwambiri) nkhani mutuwu unasindikizidwa pa Confluent blog mmbuyo mu October chaka chatha molembedwa ndi Gwen Shapira. Lero tikufuna kukukokerani chidwi chanu ku nkhani yaposachedwa kwambiri ya Epulo yolembedwa ndi Johann Gyger, yemwe, ngakhale kuti alibe funso pamutuwu, amawunika mutuwo mozama, kutsagana ndi mawuwo ndi maulalo osangalatsa. Chonde tikhululukireni kumasulira kwaulere kwa "nyani chisokonezo" ngati mungathe!

Mau oyamba

Kubernetes idapangidwa kuti izigwira ntchito zopanda malire. Nthawi zambiri, zolemetsa zotere zimaperekedwa ngati kamangidwe ka microservice, ndizopepuka, zimakula bwino mopingasa, zimatsata mfundo za 12-factor application, ndipo zimatha kugwira ntchito ndi owononga dera ndi anyani achisokonezo.

Kafka, kumbali ina, imagwira ntchito ngati nkhokwe yogawidwa. Chifukwa chake, mukamagwira ntchito, muyenera kuthana ndi boma, ndipo ndi lolemera kwambiri kuposa microservice. Kubernetes amathandizira katundu wochuluka, koma monga Kelsey Hightower akunenera mu ma tweets awiri, ayenera kusamaliridwa mosamala:

Anthu ena amaona kuti ngati mutayika Kubernetes pantchito yabwino, imakhala nkhokwe yoyendetsedwa bwino yomwe imapikisana ndi RDS. Izi ndi zolakwika. Mwina, ngati mutagwira ntchito molimbika mokwanira, onjezani zida zowonjezera ndikukopa gulu la akatswiri a SRE, mudzatha kupanga RDS pamwamba pa Kubernetes.

Nthawi zonse ndimalimbikitsa kuti aliyense azisamala kwambiri akamayendetsa ntchito zambiri pa Kubernetes. Anthu ambiri omwe amafunsa kuti "kodi nditha kuyendetsa ntchito zambiri pa Kubernetes" alibe chidziwitso chokwanira ndi Kubernetes, ndipo nthawi zambiri ndi ntchito yomwe amafunsa.

Ndiye, muyenera kuthamanga Kafka pa Kubernetes? Funso lotsutsana: kodi Kafka idzagwira ntchito bwino popanda Kubernetes? Ichi ndichifukwa chake ndikufuna kuwunikira m'nkhaniyi momwe Kafka ndi Kubernetes amayenderana, ndi misampha iti yomwe ingabwere ndikuwaphatikiza.

Nthawi yomaliza

Tiyeni tikambirane chinthu chofunikira - malo othamanga okha

ndondomeko

Kafka broker ndi ochezeka ndi CPU. TLS ikhoza kuwonetsa zina zowonjezera. Komabe, makasitomala a Kafka atha kukhala a CPU kwambiri ngati agwiritsa ntchito kubisa, koma izi sizikhudza ogulitsa.

chikumbukiro

Otsatsa a Kafka amadya kukumbukira. Kukula kwa mulu wa JVM nthawi zambiri kumakhala 4-5 GB, koma mudzafunikanso kukumbukira kwamakina ambiri popeza Kafka amagwiritsa ntchito cache yamasamba kwambiri. Ku Kubernetes, khazikitsani zotengera ndikufunsira malire moyenerera.

Malo ogulitsira

Kusungirako deta muzotengera ndi ephemeral - deta imatayika ikayatsidwanso. Kwa data ya Kafka mutha kugwiritsa ntchito voliyumu emptyDir, ndipo zotsatira zake zidzakhala zofanana: deta yanu ya broker idzatayika mukamaliza. Mauthenga anu atha kusungidwa kwa ma broker ena ngati zofananira. Chifukwa chake, mutatha kuyambiranso, wobwereketsa wolephera ayenera kubwereza deta yonse, ndipo izi zitha kutenga nthawi yambiri.

Ichi ndichifukwa chake muyenera kugwiritsa ntchito kusungirako kwanthawi yayitali. Zikhale zosungirako zanthawi yayitali ndi fayilo ya XFS kapena, ndendende, ext4. Osagwiritsa ntchito NFS. Ndinakuchenjezani. Mitundu ya NFS v3 kapena v4 sigwira ntchito. Mwachidule, broker wa Kafka adzagwa ngati sangathe kuchotsa chikwatu cha deta chifukwa cha vuto la "kupusa" mu NFS. Ngati sindinakutsimikizirenibe, mosamala kwambiri werengani nkhaniyi. Malo osungiramo data akuyenera kukhala osakhala amderali kuti Kubernetes athe kusankha mosavuta node yatsopano atayambiranso kapena kusamuka.

Mtanda

Mofanana ndi machitidwe ambiri omwe amagawidwa, machitidwe a Kafka amadalira kwambiri kusunga latency ya intaneti kuti ikhale yochepa komanso bandwidth mpaka pazipita. Osayesa kuchititsa ma broker onse pa node yomweyo, chifukwa izi zichepetsa kupezeka. Ngati node ya Kubernetes ilephera, gulu lonse la Kafka lidzalephera. Komanso, musabalalitse gulu la Kafka m'malo onse a data. Zomwezo zimapitanso ku gulu la Kubernetes. Kugwirizana kwabwino pankhaniyi ndikusankha madera osiyanasiyana opezeka.

Kukhazikika

Manifesto okhazikika

Webusaiti ya Kubernetes ili nayo chitsogozo chabwino kwambiri za momwe mungasinthire ZooKeeper pogwiritsa ntchito mawonetsero. Popeza ZooKeeper ndi gawo la Kafka, awa ndi malo abwino kuyamba kudziwa zomwe Kubernetes akugwiritsa ntchito pano. Mukamvetsetsa izi, mutha kugwiritsa ntchito malingaliro omwewo ndi gulu la Kafka.

• Ndi: Pod ndiye gawo laling'ono kwambiri lomwe lingagwiritsidwe ntchito ku Kubernetes. Pod ili ndi ntchito yanu, ndipo pod yokhayo imagwirizana ndi ndondomeko ya gulu lanu. Khodi limakhala ndi chidebe chimodzi kapena zingapo. Seva iliyonse ya ZooKeeper mu ensemble ndipo broker aliyense mu gulu la Kafka aziyenda mosiyanasiyana.
• StatefulSet: StatefulSet ndi chinthu cha Kubernetes chomwe chimagwira ntchito zingapo zodziwika bwino, ndipo ntchito zotere zimafunikira kulumikizana. StatefulSets imapereka zitsimikizo za kuyitanitsa ma pod ndi mawonekedwe ake apadera.
• Ntchito zopanda mutu: Ntchito zimakupatsani mwayi wochotsa makasitomala pogwiritsa ntchito dzina lomveka. Kubernetes pankhaniyi ali ndi udindo wowongolera katundu. Komabe, akamagwira ntchito zochulukirapo, monga ZooKeeper ndi Kafka, makasitomala amafunika kulumikizana ndi zochitika zinazake. Apa ndi pamene mautumiki opanda mutu amabwera bwino: pamenepa, kasitomala adzakhalabe ndi dzina lomveka, koma simudzasowa kulankhulana ndi pod mwachindunji.
• Kusungirako kwa nthawi yayitali: Ma voliyumu awa amafunikira kuti mukonze zosungira zomwe sizili mdera lanu zomwe zatchulidwa pamwambapa.

pa Yolean Imakupatsirani ziwonetsero zambiri zokuthandizani kuti muyambe ndi Kafka pa Kubernetes.

Ma chart a helm

Helm ndi woyang'anira phukusi la Kubernetes lomwe lingafanane ndi oyang'anira phukusi la OS monga yum, apt, Homebrew kapena Chocolatey. Zimapangitsa kuti zikhale zosavuta kukhazikitsa mapulogalamu omwe afotokozedwatu m'ma chart a Helm. Tchati cha Helm chosankhidwa bwino chimapangitsa ntchito yovuta ya momwe mungasinthire magawo onse kuti agwiritse ntchito Kafka pa Kubernetes mosavuta. Pali zithunzi zingapo za Kafka: yovomerezeka ili mu incubator condition, pali mmodzi wochokera Kusintha, winanso - kuchokera Bitnami.

Ogwira ntchito

Chifukwa Helm ili ndi zofooka zina, chida china chikutchuka kwambiri: Ogwiritsa ntchito a Kubernetes. Wogwira ntchitoyo samangoyika pulogalamu ya Kubernetes, komanso amakulolani kuyika mapulogalamuwa ndikuwongolera.

Pamndandanda ochita zodabwitsa Ogwiritsa ntchito awiri a Kafka amatchulidwa. Mmodzi wa iwo - Strimzi. Ndi Strimzi, ndikosavuta kuti gulu lanu la Kafka likhazikike m'mphindi zochepa. Pafupifupi palibe kasinthidwe komwe kumafunikira, kuwonjezera apo, woyendetsa yekha amapereka zinthu zabwino, mwachitsanzo, kubisa kwa point-to-point TLS mkati mwa tsango. Confluent imaperekanso mwiniwake.

Kukonzekera

Ndikofunikira kuyesa magwiridwe antchito poyesa chitsanzo chanu cha Kafka. Mayesero otere adzakuthandizani kupeza zolepheretsa zomwe zingatheke mavuto asanayambe. Mwamwayi, Kafka amapereka kale zida ziwiri zoyesera ntchito: kafka-producer-perf-test.sh и kafka-consumer-perf-test.sh. Agwiritseni ntchito mwachangu. Kuti mumve zambiri, mutha kulozera ku zotsatira zomwe zafotokozedwa mu positi iyi Jay Kreps, kapena yang'anani ndemanga iyi Amazon MSK ndi Stéphane Maarek.

Ntchito

Kuwunikira

Kuwonekera mu dongosolo ndikofunika kwambiri - mwinamwake simudzamvetsa zomwe zikuchitika mmenemo. Masiku ano pali zida zolimba zomwe zimapereka kuwunika motengera ma metrics mumayendedwe amtambo. Zida ziwiri zodziwika bwino pazifukwa izi ndi Prometheus ndi Grafana. Prometheus amatha kusonkhanitsa ma metrics kuchokera kumachitidwe onse a Java (Kafka, Zookeeper, Kafka Connect) pogwiritsa ntchito JMX kutumiza kunja - m'njira yosavuta. Mukawonjezera ma metric a cAdvisor, mutha kumvetsetsa bwino momwe zinthu zimagwiritsidwira ntchito ku Kubernetes.

Strimzi ali ndi chitsanzo chosavuta cha Grafana dashboard ya Kafka. Imawonera ma metrics ofunikira, mwachitsanzo, za magawo omwe sanatsatizidwe kapena omwe alibe intaneti. Zonse ndi zomveka bwino pamenepo. Ma metricswa amaphatikizidwa ndi kagwiritsidwe ntchito kazinthu ndi chidziwitso cha magwiridwe antchito, komanso zizindikiro zokhazikika. Chifukwa chake mumapeza kuwunika kwamagulu a Kafka pachabe!

Source: streamzi.io/docs/master/#kafka_dashboard

Zingakhale zabwino kuwonjezera zonsezi ndi kuwunika kwa kasitomala (ma metrics pa ogula ndi opanga), komanso kuyang'anira kuchedwa (pa izi pali Burrow) ndi kuwunika komaliza - pakugwiritsa ntchito izi Kafka Monitor.

Kudula mitengo

Kudula mitengo ndi ntchito ina yofunika kwambiri. Onetsetsani kuti zotengera zonse zomwe mumayika mu Kafka zalowetsedwa stdout и stderr, ndikuwonetsetsanso kuti gulu lanu la Kubernetes likuphatikiza zipika zonse kukhala malo apakati odula mitengo, mwachitsanzo. Elasticsearch.

Mayeso Ogwira Ntchito

Kubernetes amagwiritsa ntchito ma probes amoyo komanso okonzeka kuwona ngati ma pod anu akuyenda bwino. Ngati cheke chamoyo chikalephera, Kubernetes adzayimitsa chidebecho ndikuyambiranso ngati mfundo yoyambitsanso ikakhazikitsidwa moyenerera. Ngati cheke chokonzekera sichikanika, Kubernetes amalekanitsa pod kuti asagwiritse ntchito zopempha. Choncho, pazifukwa zotere, kulowererapo pamanja sikufunikiranso konse, zomwe ndizophatikiza zazikulu.

Kutulutsa zosintha

StatefulSets imathandizira zosintha zokha: mukasankha njira ya RollingUpdate, iliyonse pansi pa Kafka idzasinthidwanso. Mwa njira iyi, nthawi yopuma imatha kuchepetsedwa kukhala zero.

Kukulitsa

Kukulitsa gulu la Kafka si ntchito yophweka. Komabe, Kubernetes imapangitsa kukhala kosavuta kukulitsa ma pods kumitundu ina yofananira, zomwe zikutanthauza kuti mutha kufotokozera momveka bwino ma broker ambiri a Kafka momwe mungafunire. Chovuta kwambiri pankhaniyi ndikugawa magawo pambuyo pokweza kapena kutsitsa. Apanso, Kubernetes adzakuthandizani ndi ntchitoyi.

Ulamuliro

Ntchito zokhudzana ndi kuyang'anira gulu lanu la Kafka, monga kupanga mitu ndi magawo ogawa, zitha kuchitidwa pogwiritsa ntchito zolemba zomwe zilipo kale potsegula mawonekedwe a mzere wa malamulo mumapodi anu. Komabe, yankho ili silokongola kwambiri. Strimzi imathandizira kuyang'anira mitu pogwiritsa ntchito wogwiritsa ntchito wina. Pali malo oti muwongolere apa.

Sungani ndi kubwezeretsa

Tsopano kupezeka kwa Kafka kudzadaliranso kupezeka kwa Kubernetes. Ngati gulu lanu la Kubernetes lilephera, ndiye kuti pazovuta kwambiri, gulu lanu la Kafka lidzalepheranso. Malinga ndi lamulo la Murphy, izi zidzachitikadi, ndipo mudzataya deta. Kuti muchepetse chiopsezo chamtunduwu, khalani ndi lingaliro labwino losunga zobwezeretsera. Mukhoza kugwiritsa MirrorMaker, njira ina ndi ntchito S3 kwa izi, monga tafotokozera mu izi positi ku Zalando.

Pomaliza

Mukamagwira ntchito ndi magulu ang'onoang'ono mpaka apakatikati a Kafka, ndikofunikira kugwiritsa ntchito Kubernetes chifukwa imapereka kusinthasintha kowonjezera komanso kumathandizira wogwiritsa ntchito. Ngati muli ndi zofunikira kwambiri zosagwira ntchito komanso / kapena zomwe mukufunikira kuti mupitirize, ndiye kuti zingakhale bwino kuganizira njira ina yotumizira.

Source: www.habr.com