Kwaziwai, Habr!
Pane imwe nguva, isu takanga tiri vekutanga kuunza nyaya kumusika weRussia uye enderera nokuda kwekuvandudzwa kwayo. Kunyanya, takawana musoro wekudyidzana pakati peKafka uye . Zvinoonekwa (uye zvakanyatsonaka) chinyorwa ichi chakatsikiswa paConfluent blog kumashure muna Gumiguru gore rapfuura pasi pekunyora kwaGwen Shapira. Nhasi tinoda kukwevera pfungwa dzako kune imwe nyaya ichangoburwa kubva muna Kubvumbi naJohann Gyger, uyo, kunyangwe asina mucherechedzo wemubvunzo mumusoro, anoongorora musoro wenyaya nenzira yakadzikama, achiperekedza chinyorwa nezvinofadza zvinongedzo. Ndokumbira mutiregerere shandurudzo yemahara ye "chaos monkey" kana uchikwanisa!
Nhanganyaya
Kubernetes yakagadzirirwa kubata basa risingaverengeki. Kazhinji, mitoro yebasa yakadaro inounzwa muchimiro cheiyo microservice architecture, iwo akareruka, anoyera zvakanaka akatwasuka, anotevedzera misimboti ye12-factor application, uye anogona kushanda neanedunhu mabreak uye mhirizhonga tsoko.
Kafka, kune rumwe rutivi, inonyanya kuita sedhatabhesi rakagoverwa. Nokudaro, kana uchishanda, unofanirwa kutarisana nehurumende, uye inorema kudarika microservice. Kubernetes inotsigira mitoro inorema, asi sekunongedza kwaKelsey Hightower mumatweets maviri, anofanirwa kubatwa nehanya:
Vamwe vanhu vanonzwa kuti kana ukatenderedza Kubernetes kuita basa rakawanda, rinova dhatabhesi rakanyatsogadziriswa rinokwikwidza RDS. Izvi zvakaipa. Pamwe, kana ukashanda nesimba zvakakwana, wedzera zvimwe zvinhu uye kukwezva timu yeSRE mainjiniya, unozogona kuvaka RDS pamusoro peKubernetes.
Ini ndinogara ndichikurudzira kuti munhu wese angwarire zvakanyanya paanenge achimhanyisa basa rakawanda paKubernetes. Vanhu vazhinji vanobvunza kuti "ndingamhanyisa basa rakawanda paKubernetes" havana ruzivo rwakakwana neKubernetes, uye kazhinji nebasa ravari kubvunza nezvaro.
Saka, iwe unofanirwa kumhanya Kafka paKubernetes? Counter mubvunzo: Kafka ichashanda zvirinani pasina Kubernetes? Ndosaka ini ndichida kuburitsa muchinyorwa chino kuti Kafka naKubernetes vanozadzisana sei, uye ndeapi misungo inogona kuuya nekuisanganisa.
Nguva yekupedzisira
Ngatitaure nezve chinhu chakakosha - iyo yekumhanyisa nharaunda pachayo
gadzira
Kafka vatengesi vane hushamwari hweCPU. TLS inogona kuunza zvimwe pamusoro. Nekudaro, vatengi veKafka vanogona kunge vachiwedzera CPU yakasimba kana vakashandisa encryption, asi izvi hazvikanganisi vatengesi.
ndangariro
Kafka vatengesi vanodya ndangariro. Iyo JVM murwi saizi inowanzoganhurwa kune 4-5 GB, asi iwe zvakare uchada yakawanda system memory sezvo Kafka inoshandisa iyo peji cache zvakanyanya. MuKubernetes, isa mudziyo wemidziyo uye kukumbira miganho zvinoenderana.
Data store
Chengetedzo yedata mumidziyo ndeye ephemeral - data inorasika kana yatangwazve. Kune Kafka data unogona kushandisa vhoriyamu emptyDir, uye mhedzisiro ichave yakafanana: yako broker data icharasika kana yapera. Mameseji ako anogona kuramba achichengetwa kune mamwe mabroker se replicas. Nokudaro, mushure mekutangazve, mutengesi akakundikana anofanira kutanga adzokorora data rose, uye izvi zvinogona kutora nguva yakawanda.
Ichi ndicho chikonzero iwe uchifanira kushandisa kwenguva refu yekuchengetedza data. Ngaive isiri yenzvimbo yenguva refu yekuchengetedza neiyo XFS faira system kana, kunyanya, ext4. Usashandise NFS. Ndakakuyambira. NFS shanduro v3 kana v4 haishande. Muchidimbu, Kafka bhuroka ichaparara kana ikatadza kudzima dhairekitori yedata nekuda kwedambudziko re "stupid rename" muNFS. Kana ini ndisati ndakupwisa iwe, zvakanyatsonaka . Iyo dhata dhata inofanirwa kunge isiri yenzvimbo kuitira kuti Kubernetes akwanise kusarudza zviri nyore node nyowani mushure mekutangazve kana kutamiswa.
Network
Sezvinei nemasisitimu mazhinji akaparadzirwa, kuita kwaKafka kunonyanya kutsamira pakuchengeta network latency kusvika padiki uye bandwidth kusvika pakakwirira. Usayedze kugashira ese mabroker pane imwechete node, sezvo izvi zvichideredza kuwanikwa. Kana iyo Kubernetes node ikatadza, iyo yese Kafka cluster ichakundikana. Zvakare, usaparadzira sumbu reKafka munzvimbo dzese dzedata. Izvo zvakafanana zvinoenda kune Kubernetes cluster. Kubvumirana kwakanaka munyaya iyi ndiko kusarudza nzvimbo dzakasiyana dzekuwanikwa.
Kugadziriswa
Regular manifestos
Iyo Kubernetes webhusaiti ine nezve maitiro ekugadzirisa ZooKeeper uchishandisa manifesiti. Sezvo ZooKeeper iri chikamu cheKafka, ino inzvimbo yakanaka yekutanga kujairana neiyo Kubernetes pfungwa dzinoshanda pano. Kana uchinge wanzwisisa izvi, unogona kushandisa pfungwa dzakafanana neKafka cluster.
- Under: A pod ndiyo diki diki inogona kutumirwa muKubernetes. Iyo pod ine basa rako, uye iyo pod pachayo inoenderana nemaitiro musumbu rako. Podhi ine chigaba chimwe kana kupfuura. Imwe neimwe ZooKeeper sevha mune ensemble uye imwe neimwe broker muKafka cluster inomhanya mune imwe pod.
- StatefulSet: A StatefulSet chinhu cheKubernetes chinobata akawanda ane hunyanzvi mabasa, uye mabasa akadai anoda kurongeka. StatefulSets inopa vimbiso maererano nekurongeka kwemapods uye kusarudzika kwawo.
- Masevhisi asina musoro: Masevhisi anokutendera kuti utore mapodhi kubva kune vatengi uchishandisa zita rine musoro. Kubernetes mune iyi kesi ine basa rekuyera kuyera. Nekudaro, kana uchishanda zvine hunyanzvi basa, seZooKeeper neKafka, vatengi vanofanirwa kutaurirana neimwe muenzaniso. Apa ndipo panouya masevhisi asina musoro: mune iyi kesi, mutengi achange aine zita rine musoro, asi haufanirwe kubata pod zvakananga.
- Kureba kwekuchengetedza vhoriyamu: Aya mavhoriyamu anodiwa kugadzirisa iyo isiri yemunharaunda block inopfuurira kuchengetedza yataurwa pamusoro.
pamusoro Inopa yakazara seti yezviratidziro kuti ikubatsire kutanga neKafka paKubernetes.
Helm charts
Helm imaneja wepakeji yeKubernetes iyo inogona kufananidzwa neOS package mamaneja akadai yum, apt, Homebrew kana Chocolatey. Zvinoita kuti zvive nyore kuisa predefined software mapakeji anotsanangurwa mumachati eHelm. Yakanyatsosarudzwa Helm chati inoita kuti basa rakaoma rekugadzirisa nemazvo maparamita ese ekushandisa Kafka paKubernetes nyore. Kune akati wandei Kafka dhayagiramu: iyo yepamutemo iripo , pane imwe kubva , imwezve - kubva .
Operators
Nekuti Helm ine zvimwe zvikanganiso, chimwe chishandiso chiri kuwana mukurumbira wakakura: Kubernetes vanoshanda. Mushandi haango rongedza software yeKubernetes, asi zvakare inobvumidza iwe kuendesa software yakadaro uye kuibata.
Mune izvo zvinyorwa Vashandi vaviri veKafka vanotaurwa. Mumwe wavo - . NeStrimzi, zviri nyore kukwidza Kafka cluster uye kumhanya mumaminetsi. Zvinenge pasina kugadziridzwa kunodiwa, mukuwedzera, iyo opareta pachayo inopa mamwe maficha akanaka, semuenzaniso, point-to-point TLS encryption mukati mesumbu. Confluent inopawo .
Kubudirira
Izvo zvakakosha kuti uedze kuita nekuenzanisa yako Kafka muenzaniso. Miedzo yakadai ichakubatsira kuti uwane zvingangoita mabhodhoro matambudziko asati aitika. Sezvineiwo, Kafka inotopa maviri ekuita bvunzo maturusi: kafka-producer-perf-test.sh ΠΈ kafka-consumer-perf-test.sh. Zvishandise nesimba. Kuti uwane referensi, unogona kutarisa kune zvakatsanangurwa mu Jay Kreps, kana kutarisa Amazon MSK naStΓ©phane Maarek.
Mashandiro
Kuongorora
Kujeka muhurongwa kwakakosha zvikuru - kana zvisina kudaro haunganzwisisi zviri kuitika mairi. Nhasi kune yakasimba turusi kit inopa metrics-yakavakirwa kutarisa mune yegore rekare maitiro. Zvishandiso zviviri zvakakurumbira zvechinangwa ichi Prometheus uye Grafana. Prometheus inogona kuunganidza metrics kubva kune ese Java maitiro (Kafka, Zookeeper, Kafka Connect) uchishandisa JMX mutengesi kunze - nenzira yakapusa. Kana iwe ukawedzera cAdvisor metrics, unogona kunyatsonzwisisa kuti zviwanikwa zvinoshandiswa sei muKubernetes.
Strimzi ane muenzaniso wakanaka kwazvo weGrafana dashboard yeKafka. Inoona akakosha metrics, semuenzaniso, nezve pasi-yakadhindwa zvikamu kana izvo zvisiri pamhepo. Zvose zvakajeka ipapo. Aya ma metrics anowedzerwa nekushandisa zviwanikwa uye ruzivo rwekuita, pamwe nezviratidzo zvekugadzikana. Saka iwe unowana yakakosha Kafka cluster yekutarisa pasina!
Source:
Zvingave zvakanaka kuwedzera zvese izvi nekutarisa kwevatengi (metrics pavatengi nevagadziri), pamwe nekutarisa latency (nekuda kweizvi kune ) uye kupera-kusvika-kumagumo kutarisa - kwekushandisa uku .
Kutema miti
Kutema matanda nderimwe basa rinokosha. Ita shuwa kuti midziyo yese mune yako Kafka yekumisikidza yakanyorwa stdout ΠΈ stderr, uye zvakare ita shuwa kuti yako Kubernetes cluster inounganidza matanda ese mukati mepakati pekutema matanda, semuenzaniso. .
Rinoshanda cheki
Kubernetes inoshandisa hupenyu uye kugadzirira kuongorora kuti mapodhi ako ari kushanda sezvakangoita. Kana iyo yehupenyu cheki ikatadza, Kubernetes inomisa chigadziko ichocho uye obva atangazve otomatiki kana iyo restart policy ikaiswa saizvozvo. Kana cheki yekugadzirira ikatadza, Kubernetes inoparadzanisa pod kubva kune zvikumbiro zvekushandira. Nokudaro, mumamiriro ezvinhu akadaro, kupindira kwemaoko hakuchadiwi zvachose, iyo yakakura kuwedzera.
Kuburitsa zvigadziriso
StatefulSets inotsigira otomatiki zvigadziriso: kana ukasarudza iyo RollingUpdate zano, imwe neimwe iri pasi peKafka ichagadziridzwa zvakare. Nenzira iyi, nguva yekudzikisa inogona kuderedzwa kusvika zero.
Kuyera
Kuyera sumbu reKafka harisi basa riri nyore. Nekudaro, Kubernetes inoita kuti zvive nyore kwazvo kuyera mapodhi kune imwe nhamba yezvinyorwa, zvinoreva kuti iwe unogona kutsanangura uchitsanangura vazhinji veKafka broker sezvaunoda. Chinhu chakanyanya kuoma mune iyi kesi kugoverazve zvikamu mushure mekukwira kumusoro kana kusati kwadzika pasi. Zvekare, Kubernetes ichakubatsira iwe nebasa iri.
Utongi
Mabasa ane chekuita nekupa Kafka cluster yako, sekugadzira misoro uye kugoverazve zvikamu, zvinogona kuitwa uchishandisa magwaro aripo eganda nekuvhura iyo yekuraira mutsara interface mumapods ako. Nekudaro, iyi mhinduro haina kunaka kwazvo. Strimzi inotsigira kubata misoro uchishandisa akasiyana opareta. Pane imwe nzvimbo yekuvandudza pano.
Backup uye dzorera
Ikozvino kuwanikwa kweKafka kuchaenderanawo nekuwanikwa kweKubernetes. Kana yako Kubernetes cluster ikatadza, saka mune yakaipisisa mamiriro, yako Kafka cluster ichatadzawo. Zvinoenderana nemutemo waMurphy, izvi zvichaitika, uye iwe ucharasikirwa nedata. Kuti uderedze njodzi yerudzi urwu, iva nepfungwa yakanaka yekuchengetedza. Iwe unogona kushandisa MirrorMaker, imwe sarudzo ndeye kushandisa S3 yeizvi, sezvakatsanangurwa mune izvi kubva kuZalando.
mhedziso
Paunenge uchishanda nemasumbu madiki kusvika epakati nepakati eKafka, zvakakodzera kushandisa Kubernetes sezvo ichipa kumwe kuchinjika uye inorerutsa ruzivo rwemushandisi. Kana iwe uine yakakosha kwazvo isiri-inoshanda latency uye/kana throughput zvinodiwa, saka zvingave nani kufunga imwe sarudzo yekuendesa.
Source: www.habr.com