В chinyorwa chekupedzisira takatarisa kuRabbitMQ kuunganidza kwekutadza kushivirira uye kuwanikwa kwepamusoro. Zvino ngaticherei zvakadzika muApache Kafka.

Pano chikamu chekudzokorora ndicho chikamu. Musoro wega wega une chikamu chimwe kana kupfuura. Chikamu chimwe nechimwe chine mutungamiri ane kana asina vateveri. Paunenge uchigadzira musoro, unotsanangura nhamba yezvikamu uye kudzokorora coefficient. Iko kukosha kwechinyakare ndeye 3, iyo inoreva zvirevo zvitatu: mutungamiri mumwechete uye vateveri vaviri.

Mupunga. 1. Zvikamu zvina zvakagoverwa pakati pevatatu vatengesi

Zvose kuverenga nekunyora zvikumbiro zvinoenda kumutungamiri. Vateveri nguva nenguva vanotumira zvikumbiro kumutungamiriri kuti agamuchire mameseji ezvino. Vatengi havatombotendeukire kuvateveri; iyo yekupedzisira iripo chete yekuregererwa uye kukanganisa kushivirira.

Partition kukundikana

Kana broker atadza, vatungamiriri vezvikamu zvakati wandei vanowanzotadza. Mune imwe neimwe yavo, muteveri kubva kune imwe node anova mutungamiri. Muchokwadi, izvi hazvisi izvo nguva dzose, sezvo iyo yekuwiriranisa chinhu inokanganisawo: kana paine vateveri vakaenderana, uye kana zvisiri, saka kana kuchinjira kune unsynchronized replica kunobvumidzwa. Asi ngatisaomesa zvinhu parizvino.

Broker 3 anosiya network, uye mutungamiri mutsva anosarudzwa chikamu 2 kune broker 2.

Mupunga. 2. Broker 3 anofa uye muteveri wake pabroker 2 anosarudzwa semutungamiri mutsva wepartition 2.

Ipapo bhuroka 1 mashizha uye chikamu 1 zvakare inorasikirwa nemutungamiri wayo, ane basa rinopfuura kune broker 2.

Mupunga. 3. Kwasara broker mumwechete. Vatungamiri vese vari pane imwe broker ine zero redundancy

Kana broker 1 adzoka pamhepo, inowedzera vateveri vana, ichipa kumwe kudzikiswa kune yega yega chikamu. Asi vatungamiriri vese vakaramba vari pabroker 2.

Mupunga. 4. Vatungamiri vanoramba vari pabroker 2

Kana broker 3 auya, isu tadzoka kune matatu akadzokorora pachikamu. Asi vatungamiriri vese vachiri pabroker 2.

Mupunga. 5. Kuiswa kusina kuenzana kwevatungamiri mushure mekudzorerwa kwemabhuroka 1 ne3

Kafka ine chishandiso chemutungamiri ari nani kuenzanirana kupfuura RabbitMQ. Ikoko, iwe waifanira kushandisa yechitatu-bato plugin kana script yakashandura marongero ekufambisa iyo master node nekudzikisa redundancy panguva yekutama. Pamusoro pezvo, pamitsetse mikuru taifanira kubvuma kusawanikwa panguva yekuwiriranisa.

Kafka ine pfungwa ye "inofarirwa replicas" yebasa remutungamiri. Kana zvikamu zvemusoro zvagadzirwa, Kafka inoedza kugovera vatungamiriri zvakaenzana munzvimbo dzese uye inomaka avo vatungamiriri vekutanga sezvavanoda. Nekufamba kwenguva, nekuda kwesevha reboots, kutadza, uye kuparara kwekubatanidza, vatungamiriri vanogona kuguma pane dzimwe node, senge mune yakanyanyisa kesi inotsanangurwa pamusoro.

Kugadzirisa izvi, Kafka inopa sarudzo mbiri:

Sarudzo auto.leader.rebalance.enable=true inobvumira iyo controller node kuti idzorere otomatiki vatungamiriri kudzokera kune vaanofarira replicas uye nekudaro kudzoreredza kugovera kwakafanana.
Mutungamiri anogona kumhanyisa script kafka-preferred-replica-election.sh kuitira reassignment yemanyorero.

Mupunga. 6. Replicas mushure mekugadzirisa zvakare

Iyi yaive yakapfava vhezheni yekukundikana, asi chokwadi chakanyanya kuoma, kunyangwe pasina chinhu chakaomesesa apa. Izvo zvese zvinouya pasi kune yakawiriraniswa replicas (In-Sync Replicas, ISR).

Synchronized Replicas (ISR)

An ISR seti yezvakadzokororwa zvechikamu chinonzi "chakawiriraniswa" (in-sync). Pane mutungamiri, asi panogona kunge pasina vateveri. Muteveri anoonekwa seakawiriraniswa kana aita makopi chaiwo emashoko ese emutungamiri nguva isati yapera. replica.lag.time.max.ms.

Muteveri anobviswa kubva kuISR set kana:

haana kuita chikumbiro chekusarudza kwenguva replica.lag.time.max.ms (anofungidzirwa kuti akafa)
haina kukwanisa kugadzirisa panguva yekuzorora replica.lag.time.max.ms (inonzi inononoka)

Vateveri vanoita zvikumbiro zvesampling mukati menguva replica.fetch.wait.max.ms, iyo inosarudzika ku500ms.

Kuti titsanangure zvakajeka chinangwa cheISR, tinoda kutarisa zvisimbiso kubva kumugadziri uye zvimwe zviitiko zvekutadza. Vagadziri vanogona kusarudza kana broker atumira simbiso:

acks=0, simbiso haina kutumirwa
acks = 1, simbiso inotumirwa mushure mekunge mutungamiri anyora meseji kulogi yenzvimbo yake
acks = zvese, simbiso inotumirwa mushure mezvese replicas muISR yanyora meseji kumatanda enzvimbo.

MuKafka terminology, kana ISR yakachengeta meseji, "yakazvipira". Acks = zvese ndiyo yakachengeteka sarudzo, asi zvakare inowedzera kumwe kunonoka. Ngatitarisei mienzaniso miviri yekutadza uye kuti akasiyana 'acks' sarudzo anodyidzana sei neISR pfungwa.

Acks=1 uye ISR

Mumuenzaniso uyu, tichaona kuti kana mutungamiri akasamirira kuti meseji yese kubva kuvateveri iponeswe, ipapo kurasikirwa kwedata kunogoneka kana mutungamiri akatadza. Kufamba uchienda kumuteveri asina kuwiriraniswa kunogona kugoneswa kana kudzimwa nekuseta zvisina kuchena.mutungamiri.sarudzo.gonesa.

Mumuenzaniso uyu, mugadziri ane kukosha acks=1. Chikamu chinogoverwa kune ese matatu mabroker. Broker 3 iri kumashure, yakawiriraniswa nemutungamiri masekonzi masere apfuura uye ikozvino 7456 mameseji kumashure. Broker 1 yaingove sekondi imwe chete kumashure. Mugadziri wedu anotumira meseji uye nekukurumidza anogamuchira ack kumashure, pasina pamusoro pevanononoka kana vakafa vateveri izvo mutungamiri asiri kumirira.

Mupunga. 7. ISR ine replicas nhatu

Broker 2 inotadza uye mugadziri anogamuchira kukanganisa kwekubatanidza. Mushure mekunge hutungamiriri hwapfuura kune broker 1, tinorasikirwa ne123 mameseji. Muteveri pabroker 1 aive chikamu cheISR, asi haana kunyatsoenderana nemutungamiri payakadonha.

Mupunga. 8. Mharidzo dzinorasika painobondera

Mukugadzirisa bootstrap.servers Mugadziri ane akati wandei mabroker akanyorwa uye anogona kubvunza mumwe broker anova mutungamiri wechikamu chitsva. Inobva yagadzira chinongedzo kune broker 1 uye inoenderera nekutumira mameseji.

Mupunga. 9. Kutumira mameseji kunotangazve mushure mekuzorora kwenguva pfupi

Broker 3 yakatowedzera kumashure. Inoita zvikumbiro zvekutora asi haigone kuwiriranisa. Izvi zvingave zvichikonzerwa nekunonoka kwekubatana kwetiweki pakati pevatengesi, nyaya yekuchengetedza, nezvimwewo. Inobviswa kubva kuISR. Iye zvino ISR ine replica imwe - mutungamiri! Mugadziri anoenderera mberi nekutumira mameseji uye kugamuchira zvisimbiso.

Mupunga. 10. Muteveri pane broker 3 anobviswa kubva kuISR

Broker 1 inodzika uye basa rehutungamiriri rinoenda kune broker 3 nekurasikirwa kwemameseji 15286! Mugadziri anogamuchira meseji yekukanganisa yekubatanidza. Kuchinja kune mutungamiri kunze kweISR kwakagoneka chete nekuda kwekugadzirisa zvisina kuchena.mutungamiri.sarudzo.enable=chokwadi. Kana yakaiswa mukati venhema, ipapo shanduko yaisazoitika uye zvese zvikumbiro zvekuverenga nekunyora zvaizorambwa. Muchiitiko ichi, isu takamirira broker 1 kuti adzoke aine data rake chairo mune replica, inozotora zvakare hutungamiriri.

Mupunga. 11. Broker 1 inodonha. Kana kutadza kukaitika, nhamba huru yemameseji inorasika

Mugadziri anotanga hukama nemutengesi wekupedzisira uye anoona kuti iye zvino ndiye mutungamiri wechikamu. Anotanga kutumira mameseji kune broker 3.

Mupunga. 12. Mushure mekuzorora kwenguva pfupi, mameseji anotumirwa zvakare kuchikamu 0

Isu takaona kuti, kunze kwekukanganisa kupfupi kumisikidza mitsva yekubatanidza uye kutsvaga mutungamiriri mutsva, mugadziri aigara achitumira meseji. Iyi gadziriso inovimbisa kuwanikwa pamutengo wekusagadzikana (data chengetedzo). Kafka akarasikirwa nezviuru zvemashoko asi akaramba achigamuchira zvinyorwa zvitsva.

Acks = zvese uye ISR

Ngatidzokorore chiitiko ichi zvakare, asi ne acks=zvose. Broker 3 ine avhareji latency yemasekonzi mana. Mugadziri anotumira meseji ne acks=zvose, uye zvino haigamuchire mhinduro inokurumidza. Mutungamiri anomirira kuti meseji iponeswe neese replicas muISR.

Mupunga. 13. ISR ine replicas nhatu. Imwe inononoka, zvichikonzera kunonoka kurekodha

Mushure memasekonzi mana ekuwedzera kunonoka, broker 2 anotumira ack. Yese replicas ikozvino yakagadziridzwa zvizere.

Mupunga. 14. Zvese replicas chengetedza mameseji uye kutumira ack

Broker 3 ikozvino inowira kumashure kumashure uye inobviswa kubva kuISR. Latency yakaderedzwa zvakanyanya nekuti hapana anononoka replicas asara muISR. Broker 2 ikozvino inongomirira bhuroka 1 chete, uye ane avhareji lag ye500 ms.

Mupunga. 15. The replica on broker 3 inobviswa kubva kuISR

Ipapo broker 2 inodonha uye hutungamiriri hunoenda kune broker 1 pasina kurasikirwa kwemameseji.

Mupunga. 16. Broker 2 inowira

Mugadziri anowana mutungamiri mutsva uye anotanga kutumira mameseji kwaari. Iyo latency inowedzera kuderedzwa nekuti ISR ikozvino ine replica imwe! Naizvozvo sarudzo acks=zvose haina kuwedzera redundancy.

Mupunga. 17. Replica pane broker 1 inotungamira pasina kurasikirwa nemeseji

Ipapo bhuroka 1 inoputsika uye inotungamira inoenda kune bhuroka 3 nekurasikirwa kwe14238 meseji!

Mupunga. 18. Bhuroka 1 anofa uye shanduko yehutungamiri nekugadzirisa kusina kuchena kunoguma nekurasikirwa kukuru kwedata

Hatina kukwanisa kuisa sarudzo zvisina kuchena.mutungamiri.sarudzo.gonesa muchirevo zvechokwadi. By default yakaenzana venhema. Settings acks=zvose с zvisina kuchena.mutungamiri.sarudzo.enable=chokwadi inopa kusvikika neimwe yakawedzera data chengetedzo. Asi sezvauri kuona, tinogona kurasikirwa nemeseji.

Asi zvakadini kana tichida kuwedzera kuchengetedzwa kwemashoko? Unogona kuisa unclean.leader.election.enable = nhema, asi izvi hazvisati zvichizotidzivirira kubva pakurasikirwa nedata. Kana mutungamiri akadonha zvakasimba uye akatora data nayo, ipapo mameseji achiri kurasika, pamwe nekuwanikwa kunorasika kusvikira maneja adzoreredza mamiriro acho ezvinhu.

Zviri nani kuve nechokwadi chekuti mameseji ese haana basa, uye neimwe nzira kurasa kurekodha. Zvino, zvirinani kubva pakuona kwebroker, kurasikirwa kwedata kunogoneka chete kana paine kutadza kuviri kana kupfuura panguva imwe chete.

Acks=zvose, min.insync.replicas uye ISR

Nekugadzirisa nyaya min.insync.replicas Isu tiri kuwedzera mwero wekuchengetedza data. Ngatifambei nechikamu chekupedzisira chechiitiko chakapfuura zvakare, asi panguva ino min.insync.replicas=2.

Saka broker 2 ane replica mutungamiri uye muteveri pane bhuroka 3 anobviswa kubva kuISR.

Mupunga. 19. ISR kubva kune zvinyorwa zviviri

Broker 2 inodonha uye hutungamiriri hunoenda kune broker 1 pasina kurasikirwa kwemameseji. Asi ikozvino ISR inongova imwe chete yakafanana. Izvi hazvisviki nhamba shoma yekugamuchira marekodhi, uye naizvozvo mutengesi anopindura kuedza kwekunyora nechikanganiso NotEnoughReplicas.

Mupunga. 20. Nhamba yeISRs imwe yakaderera pane inotsanangurwa mu min.insync.replicas

Iyi gadziriso inosiya kuwanikwa kwekuenderana. Tisati tabvuma meseji, tinoona kuti yakanyorwa kune dzinenge mbiri replicas. Izvi zvinopa mugadziri kuvimba kwakanyanya. Pano, kurasikirwa kwemeseji kunogoneka chete kana ma replicas maviri akatadza panguva imwe chete munguva pfupi kusvika meseji yadzokororwa kune mumwe muteveri, izvo zvisingaite. Asi kana uri super paranoid, unogona kuseta replication factor kusvika 5, uye min.insync.replicas na 3. Pano vatatu vatengesi vanofanira kuwira panguva imwe chete kuti varasikirwe nerekodhi! Ehe, iwe unobhadhara kuvimbika uku mune yekuwedzera latency.

Kana kuwanikwa kunokosha pakuchengetedzwa kwemashoko

Sezvo mu nyaya ne RabbitMQ, dzimwe nguva kusvikika kunodiwa pakuchengetedza data. Hezvino zvaunofanira kufunga nezvazvo:

Ko muparidzi anogona kungodzosera chikanganiso oita kuti sevhisi yekumusoro kana mushandisi aedze zvakare gare gare?
Ko muparidzi anogona kuchengetedza meseji munharaunda kana mudhatabhesi kuti uedze zvakare gare gare?

Kana mhinduro iri kwete, saka kugadzirisa kuwanikwa kunovandudza kuchengetedzwa kwedata. Iwe ucharasikirwa nedata shoma kana ukasarudza kuwanikwa pane kusarekodha. Nokudaro, zvose zvinosvika pakutsvaga kuenzana, uye chisarudzo chinoenderana nemamiriro ezvinhu chaiwo.

Zvinoreva ISR

Iyo ISR suite inokutendera iwe kuti usarudze yakaringana chiyero pakati pekuchengetedza data uye latency. Semuyenzaniso, simbisai kuwanikwa muchiitiko chekutadza kwevazhinji replicas, kuderedza kukanganiswa kwevakafa kana kunonoka replicas maererano ne latency.

Isu tinosarudza zvinoreva isu pachedu replica.lag.time.max.ms maererano nezvaunoda. Chaizvoizvo, iyi parameter inoreva kuti kunonoka kwakadii kwatinoda kugamuchira rinhi acks=zvose. The default value is ten seconds. Kana izvi zvakakurebesa, unogona kuzvideredza. Zvadaro kuwanda kwekuchinja muISR kuchawedzera, sezvo vateveri vachabviswa uye vachiwedzerwa kakawanda.

RabbitMQ ingori seti yezvionioni inoda kudzokororwa. Magirazi anononoka anounza imwe latency, uye magirazi akafa anogona kumirira kusvika mapaketi anotarisa kuwanikwa kwenode imwe neimwe (net tick) kuti apindure. ISR inzira inonakidza yekudzivirira idzi latency nyaya. Asi isu tiri panjodzi yekurasikirwa redundancy sezvo ISR inogona kungodzikira kune mutungamiri. Kuti udzivise njodzi iyi, shandisa marongero min.insync.replicas.

Client connection garandi

Mune marongero bootstrap.servers mugadziri uye mutengi anogona kutsanangura akawanda mabroker ekubatanidza vatengi. Pfungwa ndeyokuti kana node imwe ichidzika pasi, kune akati wandei akasara akasara ayo mutengi anogona kuvhura chinongedzo. Ava havangori vatungamiriri vechikamu, asi chingori chitubu chekutanga kurodha. Mutengi anogona kuvabvunza kuti ndeipi node inobata mutungamiri wekuverenga / kunyora wechikamu.

MuRabbitMQ, vatengi vanogona kubatana kune chero node, uye nzira yemukati inotumira chikumbiro kwainoda kuenda. Izvi zvinoreva kuti iwe unogona kuisa chiyero chemutoro pamberi peRabbitMQ. Kafka inoda kuti vatengi vabatane kune node inobata mutungamiriri wechikamu anoenderana. Mumamiriro ezvinhu akadaro, haugone kuisa chiyero chemutoro. List bootstrap.servers Izvo zvakakosha kuti vatengi vagone kuwana uye nekuwana iyo chaiyo nodes mushure mekutadza.

Kafka Consensus Architecture

Kusvika ikozvino, isu hatisati tafunga kuti boka rinodzidza sei nezvekudonha kwebroker uye kuti mutungamiri mutsva anosarudzwa sei. Kuti unzwisise kuti Kafka inoshanda sei netiweki partitions, iwe unofanirwa kutanga wanzwisisa kubvumirana kwekuvaka.

Imwe neimwe Kafka cluster inoiswa pamwe chete neZookeeper cluster, inova yakagoverwa kubvumirana sevhisi inobvumira sisitimu kuti isvike kubvumirana pane imwe nyika yakapihwa, inotungamira kuenderana pamusoro pekuwanikwa. Mvumo yeruzhinji rweZookeeper node inodiwa kubvumidza kuverenga nekunyora mashandiro.

Zookeeper inochengetedza mamiriro eboka:

Rondedzero yemisoro, zvikamu, zvigadziriso, mutungamiriri wezvino replicas, anofarira replicas.
Nhengo dzeCluster. Mumwe nemumwe bhuroka anonongedza boka reZookeeper. Kana ikasagashira ping mukati menguva yakatarwa, Zookeeper inorekodha bhuroka seasipo.
Kusarudza huru uye spare nodes kune controller.

Iyo controller node ndeimwe yeKafka broker iyo ine basa rekusarudza replica vatungamiriri. Zookeeper inotumira zviziviso kune mutongi nezve nhengo yeboka uye shanduko yemusoro, uye mutongi anofanira kuita pane idzi shanduko.

Semuyenzaniso, ngatitorei musoro mutsva une mapartitions gumi uye replication factor ye 3. Mutongi anofanira kusarudza mutungamiriri wechikamu chega chega, achiedza kugovera vatungamiri pakati pemabroker.

Kune yega yega chikamu controller:

inovandudza ruzivo muZookeeper nezve ISR uye mutungamiri;
Inotumira Mutungamiri uyeISRCommand kune yega yega broker inobata replica yeichi chikamu, ichizivisa mabroker nezve ISR uye mutungamiri.

Kana broker ane mutungamiri akadonha, Zookeeper inotumira chiziviso kune controller, uye inosarudza mutungamiri mutsva. Zvekare, mutongi anotanga anogadziridza Zookeeper obva atumira rairo kune mumwe nemumwe broker achivazivisa nezve shanduko yehutungamiriri.

Mutungamiri wega wega ane basa rekutsvaga maISR. Settings replica.lag.time.max.ms inosarudza kuti ndiani achapinda imomo. Kana ISR yachinja, mutungamiri anoendesa ruzivo rutsva kuZookeeper.

Zookeeper inogara ichiziviswa nezve chero shanduko kuitira kuti kana paine kukundikana, manejimendi anochinja zvakanaka kuenda kumutungamiri mutsva.

Mupunga. 21. Kafka Consensus

Replication protocol

Kunzwisisa ruzivo rwekudzokorodza kunokubatsira kuti unzwisise zvirinani zvinogona kurasikirwa nedata.

Sampling mibvunzo, Log End Offset (LEO) uye Highwater Mark (HW)

Isu takafunga kuti vateveri nguva nenguva vanotumira zvikumbiro zvekutora kumutungamiri. The default interval ndeye 500ms. Izvi zvakasiyana neRabbitMQ pakuti muRabbitMQ kudzokorora hakutangwe negirazi remutsetse asi natenzi. Tenzi anosundidzira shanduko kumagirazi.

Mutungamiriri nevateveri vese chengetedza iyo Log End Offset (LEO) uye iyo Highwater (HW) label. Iyo LEO mucherechedzo inochengetedza iyo yekupedzisira meseji yekupedzisira mune replica yenzvimbo, uye HW inobata kudzoreredza kwekupedzisira kuita. Rangarira kuti kune chimiro chekuzvipira, meseji inofanirwa kuenderera kune ese ISR replicas. Izvi zvinoreva kuti LEO inowanzova pamberi zvishoma pane HW.

Kana mutungamiri agamuchira meseji, anoichengeta munharaunda. Muteveri anoita chikumbiro chekutora nekutumira LEO yake. Mutungamiri anobva atumira batch yemeseji kutanga kubva pane iyi LEO uye zvakare anotumira iyo yazvino HW. Kana mutungamiri agamuchira ruzivo rwekuti ese mareplicas akachengeta meseji pane yakapihwa offset, inofambisa chiratidzo cheHW. Mutungamiri chete ndiye anogona kufambisa HW, uye saka vateveri vese vachaziva kukosha kwazvino mumhinduro kuchikumbiro chavo. Izvi zvinoreva kuti vateveri vanogona kusarira kumashure kwemutungamiri mune zvese meseji uye ruzivo rweHW. Vatengi vanogamuchira mameseji chete kusvika kune yazvino HW.

Ziva kuti "kuramba" zvinoreva kunyorerwa kundangariro, kwete kudhisiki. Nekuita, Kafka inowiriranisa kune dhisiki pane imwe nguva. RabbitMQ inewo nguva yakadaro, asi ichatumira kubvuma kune muparidzi chete mushure mokunge tenzi uye magirazi ose anyora shoko ku diski. Vagadziri veKafka, nekuda kwezvikonzero zvekuita, vakafunga kutumira ack nekukurumidza kana meseji ichinyorwa kundangariro. Kafka inobheja iyo redundancy inobvisa njodzi yekuchengetedza muchidimbu mameseji anogamuchirwa mundangariro chete.

Mutungamiri akakundikana

Kana mutungamiri akadonha, Zookeeper inozivisa mutongi, uye inosarudza mutungamiriri mutsva replica. Mutungamiri mutsva anoisa mucherechedzo mutsva weHW zvinoenderana neLEO yake. Vateveri vanobva vagamuchira ruzivo nezvemutungamiri mutsva. Zvichienderana neshanduro yeKafka, muteveri achasarudza chimwe chezviitiko zviviri:

Ichadzikisa irogi yenzvimbo kune inozivikanwa HW uye kutumira chikumbiro kumutungamiri mutsva chemeseji mushure mechiratidzo ichi.
Achatumira chikumbiro kumutungamiri kuti azive HW panguva yaakasarudzwa kuve mutungamiri, uye obva adimburira irogi kune iyi. Zvinozotanga kuita zvikumbiro zvekutora nguva nenguva kutanga panguva ino.

Muteveri angangoda kudzikisa irogi nekuda kwezvikonzero zvinotevera:

Kana mutungamiri akundikana, muteveri wekutanga muISR set akanyoreswa neZookeeper anohwina sarudzo uye anova mutungamiri. Vese vateveri paISR, kunyangwe vachitariswa "mukuwiriranisa," vanogona kunge vasina kugamuchira makopi eese meseji kubva kune aimbove mutungamiriri. Zvinogoneka zvachose kuti muteveri anoratidzwa haana kopi yazvino-yezvino. Kafka inovimbisa kuti hapana mutsauko pakati pezvinyorwa. Saka, kudzivirira kusawirirana, muteveri wega wega anofanirwa kudzikisa irogi yake kune HW kukosha kwemutungamiri mutsva panguva yesarudzo yake. Ichi ndicho chimwe chikonzero nei kuseta acks=zvose saka zvakakosha pakuenderana.
Mameseji anonyorwa nguva nenguva kudhisiki. Kana ese cluster node akatadza panguva imwe chete, ipapo replicas ane akasiyana offset anozochengetwa pamadisiki. Zvinogoneka kuti kana mabroker adzoka pamhepo, mutungamiri mutsva anosarudzwa achange ari kumashure kwevateveri vake nekuti akachengeterwa disk pamberi pevamwe.

Kusanganazve neboka

Pakujoinha sumbu, zvidhori zvinoita zvimwe chete kana mutungamiri atadza: vanotarisa mufananidzo wemutungamiri uye vanodzikisa danda ravo kuHW yavo (panguva yesarudzo). Mukuenzanisa, RabbitMQ inobata zvakaenzana node dzakabatanidzwa sezvitsva zvachose. Muzviitiko zvese izvi, broker anorasa chero nyika iripo. Kana otomatiki synchronization ikashandiswa, saka tenzi anofanira kudzokorora zvachose zvese zviripo kugirazi nyowani nenzira "regai nyika yese imire". Tenzi haagamuchire chero kuverenga kana kunyora mashandiro panguva yekuvhiya uku. Iyi nzira inogadzira matambudziko mumitsetse mikuru.

Kafka ibhuku rakagoverwa uye kazhinji rinochengeta mamwe mameseji kupfuura RabbitMQ mutsara, uko data rinobviswa kubva pamutsetse mushure mekuverengwa. Mitsetse inoshanda inofanira kuramba iri diki. Asi Kafka igirogi ine yayo yekuchengeta mutemo, inogona kuseta nguva yemazuva kana mavhiki. Kuvharisa kwemutsara uye nzira yekuyananisa yakazara haigamuchirwi zvachose kune irogi rakagoverwa. Pane kudaro, vateveri veKafka vanongodzikisa danda ravo kune mutungamiri HW (panguva yekusarudzwa kwake) kana kopi yavo iri pamberi pemutungamiri. Muchiitiko chinowanzoitika, kana muteveri ari kumashure, anongotanga kuita zvikumbiro zvekutora kutanga ne LEO yazvino.

Vatsva kana vakajoinwazve vateveri vanotanga kunze kweISR uye havatore chikamu mukuita. Vanongoshanda pamwe chete neboka, vachigamuchira mameseji nekukurumidza sezvavanogona kusvikira vabata mutungamiri uye vapinda muISR. Iko hakuna kukiya-mukati uye hapana chikonzero chekurasa data rako rese.

Kurasikirwa kwekubatana

Kafka ine zvimwe zvikamu kupfuura RabbitMQ, saka ine yakaomesesa seti yemafambiro apo sumbu rinobviswa. Asi Kafka yakatanga kugadzirirwa masumbu, saka mhinduro dzakanyatso fungwa.

Pazasi pane akati wandei ekubatanidza kutadza zviitiko:

Mamiriro ekutanga: Muteveri haaoni mutungamiri, asi achiri kuona Muchengeti weZookeeper.
Mamiriro echipiri: Mutungamiriri haaoni vateveri, asi achiri kuona Zookeeper.
Chirevo chetatu: Muteveri anoona mutungamiri, asi haaone Muchengeti weZoo.
Mamiriro ezvinhu 4: Mutungamiri anoona vateveri, asi haaone Zookeeper.
Mamiriro ezvinhu 5: Muteveri wacho akaparadzana zvachose kubva kune mamwe maKafka node uye Zookeeper.
Mamiriro echitanhatu: Mutungamiriri akaparadzana zvachose kubva kune mamwe maKafka node uye Zookeeper.
Chimiro chechinomwe: Iyo Kafka controller node haigone kuona imwe Kafka node.
Chiitiko chechisere: Kafka controller haaone Zookeeper.

Chimwe nechimwe chiitiko chine maitiro aro.

Mamiriro ekutanga: Muteveri haaone mutungamiri, asi achiri kuona Zookeeper

Mupunga. 22. Chimiro chekutanga: ISR yezvinyorwa zvitatu

Iko kutadza kwekubatanidza kunoparadzanisa broker 3 kubva kune vatengesi 1 uye 2, asi kwete kubva kuZookeeper. Broker 3 haichakwanisi kutumira zvikumbiro zvekutora. Mushure menguva yapfuura replica.lag.time.max.ms inobviswa kubva kuISR uye haitore chikamu mukuita meseji. Kana kubatana kwadzorerwa, kunotangazve kutora zvikumbiro uye kujoina ISR kana yabata mutungamiri. Zookeeper acharamba achigamuchira pings uye achifunga kuti broker mupenyu uye akasimba.

Mupunga. 23. Scenario 1: Mutengesi anobviswa kubva kuISR kana pasina chikumbiro chekutora chakatambirwa kubva mairi mukati meiyo replica.lag.time.max.ms interval

Iko hakuna kupatsanurwa-uropi kana node kumiswa senge muRabbitMQ. Pane kudaro, redundancy inoderedzwa.

Mamiriro echipiri: Mutungamiriri haaoni vateveri, asi achiri kuona Zookeeper

Mupunga. 24. Chiitiko 2. Mutungamiri nevateveri vaviri

Kuparara kwekubatana kwetiweki kunoparadzanisa mutungamiri kubva kuvateveri, asi mutengesi anogona achiri kuona Zookeeper. Sezviri muchiitiko chekutanga, ISR inodzikira, asi panguva ino chete kune mutungamiri sezvo vateveri vese vanomira kutumira zvikumbiro zvekutora. Zvakare, hapana kupatsanurwa kune mufungo. Pane kudaro, pane kurasikirwa kwekuwedzerwa kwemameseji matsva kusvika kubatana kwadzorerwa. Zookeeper anoenderera mberi nekugashira pings uye anotenda kuti broker mupenyu uye akasimba.

Mupunga. 25. Mamiriro ezvinhu 2. ISR yakaderera kune mutungamiri chete

Chiitiko 3. Muteveri anoona mutungamiri, asi haaoni Muchengeti weZookeeper

Muteveri akaparadzaniswa kubva kuZookeeper, asi kwete kubva kumutengesi ane mutungamiri. Nekuda kweizvozvo, muteveri anoenderera mberi nekuita zvikumbiro zvekutora uye kuve nhengo yeISR. Zookeeper haichagamuchire pings uye inonyoresa kuparara kwebroker, asi sezvo ingori muteveri, hapana mhedzisiro mushure mekupora.

Mupunga. 26. Chiitiko chetatu: Muteveri anoenderera mberi nekutumira zvikumbiro zvekutora kumutungamiri

Chiitiko 4. Mutungamiri anoona vateveri, asi haaone Zookeeper

Mupunga. 27. Chiitiko 4. Mutungamiri nevateveri vaviri

Mutungamiriri akapatsanurwa kubva kuZookeeper, asi kwete kubva kune vatengesi vane vateveri.

Mupunga. 28. Chimiro chechina: Mutungamiri akaparadzaniswa neZookeeper

Mushure menguva yakati, Zookeeper inonyoresa kukundikana kwebroker uye kuzivisa mutongi nezvazvo. Achasarudza mutungamiri mutsva pakati pevateveri vake. Zvisinei, mutungamiri wepakutanga acharamba achifunga kuti ndiye mutungamiri uye acharamba achigamuchira zvinyorwa kubva acks=1. Vateveri havachamutumiri zvikumbiro zvekutora, saka anozovaona sevakafa uye oedza kudzikisira iyo ISR pachayo. Asi sezvo isina chinongedzo kuZookeeper, haizokwanise kuita izvi, uye panguva iyoyo icharamba kubvuma zvimwe zvinyorwa.

Mharidzo acks=zvose haigamuchire kubvumwa nekuti ISR inotanga kuvhura replicas dzese, uye mameseji haasvike kwavari. Kana mutungamiri wepakutanga akaedza kuvabvisa kubva kuISR, haizokwanisi kuita uye inomira kugamuchira chero meseji zvachose.

Vatengi vanokurumidza kuona shanduko yemutungamiri uye votanga kutumira marekodhi kune server nyowani. Kana mambure adzoreredzwa, mutungamiri wepakutanga anoona kuti haasisiri mutungamiri uye anoderedza danda rayo kune HW kukosha kwaive nemutungamiri mutsva panguva yekutadza kudzivirira kupatsanurwa kwelogi. Inobva yatanga kutumira zvikumbiro zvekutora kumutungamiri mutsva. Marekodhi ese kubva kumutungamiri wekutanga asina kudzokororwa kumutungamiri mutsva anorasika. Kureva kuti, mameseji asina kubvumwa nemutungamiri wekutanga mumasekonzi mashoma iwayo apo vatungamiri vaviri vaishanda acharasika.

Mupunga. 29. Scenario 4. Mutungamiriri pabroker 1 anova muteveri mushure mekunge network yadzorerwa.

Mamiriro ezvinhu 5: Muteveri wacho akaparadzana zvachose kubva kune mamwe maKafka node uye Zookeeper

Muteveri wacho akaparadzaniswa zvachose kubva kune dzimwe mbiri dzeKafka nodes uye Zookeeper. Anongozvibvisa kubva kuISR kusvikira mambure adzorerwa, uye obva abata nevamwe.

Mupunga. 30. Chiitiko chechishanu: Muteveri akasarudzika anobviswa kubva kuISR

Mamiriro echitanhatu: Mutungamiriri akaparadzana zvachose kubva kune mamwe maKafka node uye Zookeeper

Mupunga. 31. Chiitiko 6. Mutungamiri nevateveri vaviri

Mutungamiriri akaparadzaniswa zvachose nevateveri vake, mutongi uye Zookeeper. Kwenguva pfupi icharamba ichibvuma zvinyorwa kubva acks=1.

Mupunga. 32. Chiitiko chechitanhatu: Kuparadzanisa mutungamiri kubva kune dzimwe nzvimbo dzeKafka neZookeeper

Kusatambira zvikumbiro mushure mekupera replica.lag.time.max.ms, ichaedza kuderedza ISR pachayo, asi haizokwanisi kuzviita nokuti hapana kutaurirana neZookeeper, zvino icharega kugamuchira zvinyorwa.

Zvichakadaro, Zookeeper anozomaka mutengesi ari ega seafa uye mutongi achasarudza mutungamiri mutsva.

Mupunga. 33. Chiitiko 6. Vatungamiri vaviri

Mutungamiriri wepakutanga anogona kugamuchira zvinyorwa kwemasekonzi mashoma, asi obva amira kugamuchira chero meseji. Vatengi vanogadziridzwa ese makumi matanhatu masekonzi neazvino metadata. Vachaziviswa nezvekuchinja kwemutungamiri uye vachatanga kutumira zvinyorwa kumutungamiri mutsva.

Mupunga. 34. Chimiro chechitanhatu: Vagadziri vanochinjira kune mutungamiriri mutsva

Ese mapindiro akasimbiswa akaitwa nemutungamiri wepakutanga kubva pakarasikirwa nekubatana kucharasika. Kana network yadzoreredzwa, mutungamiri wekutanga achaona kuburikidza neZookeeper kuti haasisiri mutungamiri. Zvino ichadzikisa danda rayo kuHW yemutungamiri mutsva panguva yesarudzo uye yotanga kutumira zvikumbiro semuteveri.

Mupunga. 35. Chiitiko chechitanhatu: Mutungamiriri wepakutanga anova muteveri mushure mekunge network network yadzoreredzwa

Mumamiriro ezvinhu aya, kuparadzana kunonzwisisika kunogona kuitika kwenguva pfupi, asi chete kana acks=1 и min.insync.replicas uyewo 1. Kuparadzana kunonzwisisika kunopera kana mushure mokunge mambure adzorerwa, apo mutungamiri wepakutanga anoziva kuti haasisiri mutungamiri, kana apo vatengi vose vanoziva kuti mutungamiri achinja uye anotanga kunyora kumutungamiri mutsva - chero chii chinoitika pakutanga. Chero zvazvingava, mamwe mameseji anorasika, asi chete ne acks=1.

Pane imwe mutsauko weichi chiitiko apo, nguva isati yaparadzaniswa netiweki, vateveri vakawira kumashure uye mutungamiri akamanikidza ISR kuti iite iye. Inobva yave yega nekuda kwekurasikirwa kwekubatanidza. Mutungamiri mutsva anosarudzwa, asi mutungamiri wepakutanga anoramba achigamuchira zvinyorwa, kunyange acks=zvose, nokuti hakuna mumwe muISR kunze kwake. Aya marekodhi acharasika kana network yadzoreredzwa. Nzira chete yekudzivisa iyi sarudzo ndeye min.insync.replicas = 2.

Mamiriro 7: Kafka Controller Node Haikwanise Kuona Imwe Kafka Node

Kazhinji, kana kubatana neKafka node kwarasika, mutongi haazokwanisi kutumira chero mutungamiri anoshandura ruzivo kwairi. Mumamiriro ezvinhu akaipisisa, izvi zvinotungamirira kuparadzanisa kwenguva pfupi inonzwisisika, semamiriro ezvinhu 6. Kazhinji kazhinji, mutengesi haazove mumiriri wehutungamiri kana iyo yekupedzisira ikakundikana.

Chiitiko chechisere: Kafka controller haaone Zookeeper

Zookeeper haazogamuchire ping kubva kune akadonha controller uye achasarudza itsva Kafka node semutongi. Iyo yepakutanga controller inogona kuramba ichizviratidza seyakadaro, asi haigamuchire zviziviso kubva kuZookeeper, saka haizove nechero mabasa ekuita. Kamwe mambure akadzorerwa, achaziva kuti haasisiri mutongi, asi ave nguva dzose Kafka node.

Mhedziso kubva muzviitiko

Isu tinoona kuti kurasikirwa kwemuteveri yekubatanidza hakukonzeri kurasikirwa kwemeseji, asi kunongoderedza kwechinguva kudzoka kusvika network yadzorerwa. Izvi, hongu, zvinogona kutungamirira mukurasikirwa kwedata kana imwe kana imwe node yakarasika.

Kana mutungamiri akaparadzaniswa kubva kuZookeeper nekuda kwekurasikirwa kwekubatana, izvi zvinogona kukonzera kuti mameseji arasikirwe kubva. acks=1. Kushaikwa kwekutaurirana neZookeeper kunokonzeresa kupatsanuka kupfupi kune vatungamiriri vaviri. Dambudziko iri rinogadziriswa neparameter acks=zvose.

Parameter min.insync.replicas muzvikamu zviviri kana kupfuura zvinopa imwe vimbiso yekuti mamiriro enguva pfupi akadaro haazokonzeri mameseji akarasika sezviri muChikamu 6.

Pfupiso yeMashoko Akarasika

Ngatinyorei nzira dzese dzaungarasikirwa nadzo data muKafka:

Chero mutungamiri akakundikana kana mameseji akasimbiswa kushandiswa acks=1
Chero shanduko isina kuchena yehutungamiriri, kureva, kumuteveri ari kunze kweISR, kunyangwe ne acks=zvose
Kuparadzanisa mutungamiri kubva kuZookeeper kana mameseji akasimbiswa kushandiswa acks=1
Kuzviparadzanisa nevamwe kwemutungamiri anenge atodzikisa boka reISR kwaari. Mameseji ese acharasika, kunyangwe acks=zvose. Izvi ndezvechokwadi chete kana min.insync.replicas=1.
Kukundikana panguva imwe chete yezvikamu zvese zvezvikamu. Nekuti mameseji anobvumwa kubva mundangariro, mamwe anogona kunge asati anyorwa kudhisiki. Mushure mekutangazve maseva, mamwe mameseji anogona kunge asipo.

Shanduko dzehutungamiriri dzisina kuchena dzinogona kudziviswa nekuvarambidza kana kuona kuti kaviri kudzingwa basa. Iyo yakanyanya kusimba gadziriso ndeye musanganiswa acks=zvose и min.insync.replicas kudarika 1.

Kuenzanisa kwakananga kwekuvimbika kweRabbitMQ neKafka

Kuve nechokwadi chekuvimbika uye kuwanikwa kwepamusoro, ese ari maviri mapuratifomu anoisa yekutanga uye yechipiri kudzokorora system. Zvisinei, RabbitMQ ine Achilles chitsitsinho. Kana uchibatanidzazve mushure mekutadza, node dzinorasa data radzo uye kuwiriranisa kunovharwa. Iyi whammy kaviri inodaidzira kusahadzika hupenyu hurefu hwemitsara mikuru muRabbitMQ. Iwe unozofanirwa kugamuchira ingave yakaderedzwa redundancy kana kureba kuvharira nguva. Kuderedza redundancy kunowedzera njodzi yekurasikirwa kukuru kwedata. Asi kana mitsara iri duku, saka nekuda kwekuregererwa, nguva shomanana yekusavapo (masekonzi mashomanana) inogona kushandiswa nekushandisa kudzokorora kuedza kwekubatanidza.

Kafka haina dambudziko iri. Inorasa data chete kubva padanho rekusiyana pakati pemutungamiri nemuteveri. Yese data yakagovaniswa inochengetwa. Mukuwedzera, kudzokorora hakuvharidzi sisitimu. Mutungamiriri anoenderera mberi achigashira zvigaro apo muteveri mutsva achibata, saka kune ma devops, kujoinha kana kujoinha sumbu rinova basa diki. Ehe, pachine nyaya dzakadai setiweki bandwidth panguva yekudzokorora. Kana iwe ukawedzera vateveri vakawanda panguva imwe chete, unogona kusangana nemuganhu we bandwidth.

RabbitMQ inopfuura Kafka mukuvimbika kana maseva akawanda ari musumbu akakundikana panguva imwe chete. Sezvatatotaura, RabbitMQ inotumira chisimbiso kune muparidzi chete mushure mokunge meseji ichinyorwa ku diski natenzi uye magirazi ose. Asi izvi zvinowedzera kuwedzera latency nekuda kwezvikonzero zviviri:

fsync mazana mashoma milliseconds
Kukundikana kwegirazi kunogona kuonekwa chete mushure mehupenyu hwemapakiti anotarisa kuwanikwa kwenode imwe neimwe (net tick) yapera. Kana girazi rinononoka kana kudonha, izvi zvinowedzera kunonoka.

Kafka's bheti ndeyekuti kana meseji yakachengetwa munzvimbo dzakawanda, inogona kubvuma mameseji painongorova ndangariro. Nekuda kweizvi, pane njodzi yekurasikirwa nemameseji emhando ipi neipi (kunyangwe acks=zvose, min.insync.replicas=2) kana ukakundikana panguva imwe chete.

Pakazere, Kafka inoratidza zvirinani kuita kwesoftware uye yakagadzirirwa kubva pasi kuenda kumasumbu. Huwandu hwevateveri hunogona kuwedzerwa kusvika ku11 kana zvichidikanwa pakuvimbika. Replication factor 5 uye shoma nhamba ye replicas mukuyananisa min.insync.replicas=3 ichaita kuti kurasikirwa kwemeseji kuve chiitiko chisingawanzo. Kana zvivakwa zvako zvichikwanisa kutsigira iyi chiyero chekudzokorora uye nhanho yekudzokororwa, saka unogona kusarudza iyi sarudzo.

RabbitMQ clustering yakanaka kune mitsetse midiki. Asi nyangwe mitsetse midiki inogona kukura nekukurumidza kana paine traffic yakawanda. Kana mitsetse yakura, uchafanirwa kuita sarudzo dzakaoma pakati pekuwanikwa nekuvimbika. RabbitMQ clustering yakanyatsokodzera kune asiri-akajairika mamiriro apo mabhenefiti eRabbitMQ's flexibility anokunda chero zvipingamupinyi zvekubatana kwayo.

Mushonga mumwe wekusagadzikana kweRabbitMQ pamitsetse mikuru ndeyekuipatsanura kuita mitsetse midiki mizhinji. Kana iwe usingade kurongeka kwakazara kwemutsara wese, asi iwo chete mameseji akakodzera (semuenzaniso, mameseji kubva kune mumwe mutengi), kana kuti usaodha chero chinhu, saka iyi sarudzo inogamuchirwa: tarisa purojekiti yangu. Rebalancer kupatsanura mutsara (purojekiti ichiri padanho rekutanga).

Chekupedzisira, usakanganwa nezve akati wandei mabhugi mukubatanidza uye kudzokorora masisitimu ese ari maviri RabbitMQ uye Kafka. Nekufamba kwenguva, masisitimu ave akakura uye akagadzikana, asi hapana meseji ichazombove 100% yakachengeteka kubva mukurasikirwa! Mukuwedzera, tsaona dzakakura dzinoitika munzvimbo dze data!

Kana ini ndapotsa chimwe chinhu, ndakanganisa, kana iwe usingabvumirani neimwe yemapoinzi, inzwa wakasununguka kunyora mhinduro kana kundibata.

Ndinowanzobvunzwa: "Chii chekusarudza, Kafka kana RabbitMQ?", "Ndeipi puratifomu iri nani?". Chokwadi ndechekuti zvinoenderana nemamiriro ako ezvinhu, chiitiko chazvino, zvichingodaro. Ndinozengurira kupa maonero angu nekuti zvingave zvakanyanya kuwedzeredza kurumbidza imwe chikuva chese kesi dzekushandisa uye zvinogoneka zvisingakwanisi. Ndakanyora iyi nhevedzano yezvinyorwa kuitira kuti iwe ugone kuumba pfungwa dzako.

Ndinoda kutaura kuti masystem ese ari maviri vatungamiriri munzvimbo ino. Ini ndinogona kunge ndakarerekera zvishoma nekuti kubva muruzivo rwangu nemapurojekiti ini ndinowanzo kukoshesa zvinhu seyakavimbiswa meseji kurongeka uye kuvimbika.

Ini ndinoona mamwe matekinoroji asina kuvimbika uku uye kurongeka kwakavimbiswa, ndobva ndatarisa RabbitMQ neKafka ndoona kukosha kunoshamisa kweaya masystem.

Source: www.habr.com

RabbitMQ vs Kafka: Mhosva Kushivirira uye Kuwanikwa Kwepamusoro