В nkhani yomaliza tidayang'ana gulu la RabbitMQ la kulekerera zolakwika komanso kupezeka kwakukulu. Tsopano tiyeni tifufuze mozama mu Apache Kafka.

Apa gawo la kubwereza ndi kugawa. Mutu uliwonse uli ndi gawo limodzi kapena angapo. Gawo lirilonse liri ndi mtsogoleri wokhala ndi otsatira kapena opanda. Mukamapanga mutu, mumatchula chiwerengero cha magawo ndi coefficient yobwerezabwereza. Mtengo wanthawi zonse ndi 3, zomwe zikutanthauza kuti zofananira zitatu: mtsogoleri m'modzi ndi otsatira awiri.

Mpunga. 1. Magawo anayi amagawidwa pakati pa ma broker atatu

Zopempha zonse zowerenga ndi kulemba zimapita kwa mtsogoleri. Otsatira nthawi ndi nthawi amatumiza zopempha kwa mtsogoleri kuti alandire mauthenga aposachedwa. Ogwiritsa satembenukira kwa otsatira; zomalizazi zimakhalapo chifukwa cha kubwezeredwa komanso kulolerana zolakwa.

Kulephera kwa magawo

Pamene broker alephera, atsogoleri a zigawo zingapo nthawi zambiri amalephera. Mwa aliyense wa iwo, wotsatira kuchokera ku mfundo ina amakhala mtsogoleri. M'malo mwake, izi sizili choncho nthawi zonse, chifukwa cholumikizira chimakhudzanso: kaya pali otsatira olumikizidwa, ndipo ngati sichoncho, ndiye ngati kusinthana ndi chofananira chosasinthika kumaloledwa. Koma tisasokoneze zinthu pakadali pano.

Broker 3 amasiya netiweki, ndipo mtsogoleri watsopano amasankhidwa pagawo 2 pa broker 2.

Mpunga. 2. Broker 3 amwalira ndipo omutsatira pa broker 2 amasankhidwa kukhala mtsogoleri watsopano wa partition 2.

Kenako broker 1 masamba ndi gawo 1 amatayanso mtsogoleri wake, yemwe udindo wake umapita kwa broker 2.

Mpunga. 3. Kwatsala broker m'modzi. Atsogoleri onse ali pa broker m'modzi wopanda ziro redundancy

Pamene broker 1 abweranso pa intaneti, amawonjezera otsatira anayi, ndikupereka kuperewera kwa gawo lililonse. Koma atsogoleri onse adatsalirabe pa broker 2.

Mpunga. 4. Atsogoleri amakhalabe pa broker 2

Pamene broker 3 abwera, timabwereranso kuzinthu zitatu pagawo lililonse. Koma atsogoleri onse akadali pa broker 2.

Mpunga. 5. Kusayika bwino kwa atsogoleri pambuyo pa kubwezeretsedwa kwa ma broker 1 ndi 3

Kafka ili ndi chida chothandizira mtsogoleri wabwinoko kuyambiranso kuposa RabbitMQ. Kumeneko, munayenera kugwiritsa ntchito pulogalamu yowonjezera ya chipani chachitatu kapena script yomwe inasintha ndondomeko za kusamuka kwa master node mwa kuchepetsa redundancy panthawi yakusamuka. Kuphatikiza apo, pamizere yayikulu tidayenera kuvomereza kusapezeka panthawi yolumikizana.

Kafka ali ndi lingaliro la "zotengera zomwe amakonda" paudindo wa mtsogoleri. Magawo amitu akapangidwa, Kafka amayesa kugawa atsogoleri molingana m'malo onse ndikuyika atsogoleri oyambawo momwe akufunira. M'kupita kwa nthawi, chifukwa cha kuyambiranso kwa seva, kulephera, ndi kusokonezeka kwa maulumikizidwe, atsogoleri amatha kuthera pa mfundo zina, monga momwe zilili ndi zomwe tafotokozazi.

Kuti akonze izi, Kafka imapereka njira ziwiri:

• Yankho auto.leader.rebalance.enable=zoona imalola node yowongolera kuti igawanitsenso atsogoleri ku zofananira zomwe amakonda ndikubwezeretsanso kugawa kofanana.
• Woyang'anira akhoza kuyendetsa script kafka-preferred-replica-election.sh kwa reassignment pamanja.

Mpunga. 6. Replicas pambuyo rebancing

Uwu unali mtundu wosavuta wakulephera, koma zenizeni ndizovuta, ngakhale palibe chovuta kwambiri apa. Zonse zimatsikira ku zofananira (In-Sync Replicas, ISR).

Synchronized Replicas (ISR)

ISR ndi seti ya zofananira za gawo lomwe limatengedwa kuti "lolumikizidwa" (mu-sync). Pali mtsogoleri, koma mwina palibe otsatira. Wotsatira amaonedwa kuti ndi wolumikizidwa ngati wapanga zolemba zenizeni za mauthenga onse a mtsogoleri nthawiyo isanathe. replica.lag.time.max.ms.

Wotsatira amachotsedwa ku ISR set ngati:

• sanapemphe kuti asankhe nthawiyi replica.lag.time.max.ms (akuganiziridwa kuti wafa)
• sanakwanitse kusintha panthawiyi replica.lag.time.max.ms (amaganiziridwa kuti ndi ochedwa)

Otsatira amapempha zitsanzo panthawiyi replica.fetch.wait.max.ms, zomwe zimasintha kukhala 500ms.

Kuti tifotokoze momveka bwino cholinga cha ISR, tiyenera kuyang'ana zotsimikizira kuchokera kwa wopanga ndi zochitika zina zolephera. Opanga amatha kusankha pamene broker atumiza chitsimikiziro:

• acks=0, chitsimikizo sichinatumizidwe
• acks=1, chitsimikiziro chimatumizidwa pambuyo poti mtsogoleri walemba uthenga ku chipika chake chapafupi
• acks=onse, chitsimikiziro chatumizidwa pambuyo poti zofananira zonse mu ISR zalemba uthenga ku zipika zakomweko.

Mu terminology ya Kafka, ngati ISR ​​yasunga uthenga, "yadzipereka". Acks=onse ndiye njira yotetezeka, komanso imawonjezera kuchedwa. Tiyeni tiwone zitsanzo ziwiri zakulephera komanso momwe zosankha za 'acks' zimayenderana ndi lingaliro la ISR.

Acks = 1 ndi ISR

Mu chitsanzo ichi, tiwona kuti ngati mtsogoleri sadikira kuti uthenga uliwonse wochokera kwa otsatira onse upulumutsidwe, ndiye kuti kutayika kwa deta n'kotheka ngati mtsogoleri alephera. Kuyenda kwa wotsatira wosayanjanitsidwa kumatha kuyatsidwa kapena kuyimitsidwa pokhazikitsa chodetsedwa.chisankho.chachitsogoleri.yatsa.

Mu chitsanzo ichi, wopanga ali ndi mtengo acks=1. Gawoli limagawidwa kwa ma broker onse atatu. Broker 3 ali kumbuyo, adalumikizana ndi mtsogoleri masekondi asanu ndi atatu apitawo ndipo tsopano ali ndi mauthenga 7456 kumbuyo. Broker 1 anali ndi sekondi imodzi yokha kumbuyo. Wopanga wathu amatumiza uthenga ndipo amalandira mwachangu ack back, popanda kupitilira kwa otsatira pang'onopang'ono kapena akufa omwe mtsogoleri sakuyembekezera.

Mpunga. 7. ISR yokhala ndi zofananira zitatu

Broker 2 imalephera ndipo wopanga amalandira cholakwika cholumikizira. Utsogoleri ukadutsa kwa broker 1, timataya mauthenga a 123. Wotsatira pa broker 1 anali mbali ya ISR, koma sanagwirizane kwathunthu ndi mtsogoleri pamene adagwa.

Mpunga. 8. Mauthenga amatayika ikasweka

Mu kasinthidwe bootstrap.servers Wopangayo ali ndi ma broker angapo omwe adalembedwa ndipo atha kufunsa broker wina yemwe ali mtsogoleri wagawo latsopano. Kenako imakhazikitsa kulumikizana kwa broker 1 ndikupitiliza kutumiza mauthenga.

Mpunga. 9. Kutumiza mauthenga kumayambiranso pambuyo popuma pang'ono

Broker 3 ali kumbuyo kwambiri. Zimapanga zopempha koma sizitha kulunzanitsa. Izi zitha kukhala chifukwa cha kulumikizidwa kwapaintaneti pang'onopang'ono pakati pa ogulitsa, nkhani yosungirako, ndi zina zambiri. Zimachotsedwa ku ISR. Tsopano ISR ili ndi chithunzi chimodzi - mtsogoleri! Wopanga akupitiriza kutumiza mauthenga ndi kulandira zitsimikizo.

Mpunga. 10. Wotsatira pa broker 3 amachotsedwa ku ISR

Broker 1 amatsika ndipo udindo wa utsogoleri umapita kwa broker 3 ndikutayika kwa mauthenga a 15286! Wopanga amalandira uthenga wolakwika wolumikizana. Kusintha kwa mtsogoleri kunja kwa ISR kunali kotheka kokha chifukwa cha kukhazikitsidwa wosayera.mtsogoleri.kusankha.enable=zoona. Ngati imayikidwa mu zabodza, ndiye kuti kusintha sikungachitike ndipo zopempha zonse zowerenga ndi kulemba zidzakanidwa. Pamenepa, tikudikirira broker 1 kuti abwerere ndi deta yake mu replica, yomwe idzatengenso utsogoleri.

Mpunga. 11. Broker 1 akugwa. Kukanika kuchitika, mauthenga ambiri amatayika

Wopangayo amakhazikitsa mgwirizano ndi broker womaliza ndipo akuwona kuti tsopano ndi mtsogoleri wa gawolo. Amayamba kutumiza mauthenga kwa broker 3.

Mpunga. 12. Pambuyo popuma pang'ono, mauthenga amatumizidwanso ku gawo 0

Tidawona kuti, kupatula zododometsa zazifupi kukhazikitsa maulalo atsopano ndikufufuza mtsogoleri watsopano, wopanga anali kutumiza mauthenga nthawi zonse. Kukonzekera uku kumatsimikizira kupezeka pamtengo wokhazikika (chitetezo cha data). Kafka adataya mauthenga masauzande ambiri koma adapitiliza kuvomereza zolemba zatsopano.

Acks = onse ndi ISR

Tiyeni tibwerezenso izi, koma ndi zonse = zonse. Broker 3 amakhala ndi latency ya masekondi anayi. Wopanga amatumiza uthenga ndi zonse = zonse, ndipo tsopano sakulandira yankho lachangu. Mtsogoleri amadikirira kuti uthengawo upulumutsidwe ndi zolemba zonse mu ISR.

Mpunga. 13. ISR yokhala ndi zofananira zitatu. Imodzi imachedwa, zomwe zimapangitsa kuti kujambula kuchedwe

Pambuyo pa masekondi anayi akuchedwa kwina, broker 2 amatumiza ack. Zofananira zonse tsopano zasinthidwa kwathunthu.

Mpunga. 14. Zofananira zonse zimasunga mauthenga ndikutumiza ack

Broker 3 tsopano akugwera kumbuyo kwambiri ndikuchotsedwa ku ISR. Latency imachepetsedwa kwambiri chifukwa palibe zofananira pang'onopang'ono zomwe zatsala mu ISR. Broker 2 tsopano akudikirira broker 1 yekha, ndipo ali ndi kutsalira kwa 500 ms.

Mpunga. 15. Choyimira pa broker 3 chimachotsedwa ku ISR

Kenako broker 2 akugwa ndipo utsogoleri umadutsa kwa broker 1 popanda kutaya mauthenga.

Mpunga. 16. Broker 2 imagwa

Wopangayo amapeza mtsogoleri watsopano ndikuyamba kutumiza mauthenga kwa iye. Kuchedwa kumachepetsedwanso chifukwa ISR tsopano ili ndi chofanana chimodzi! Choncho njira zonse = zonse sichimawonjezera redundancy.

Mpunga. 17. Replica pa broker 1 imatsogolera popanda kutaya mauthenga

Kenako broker 1 akuwonongeka ndipo kutsogolera kumapita kwa broker 3 ndikutayika kwa mauthenga a 14238!

Mpunga. 18. Broker 1 amwalira ndipo kusintha kwa utsogoleri ndi malo osayera kumabweretsa kutayika kwakukulu kwa data

Sitinathe kukhazikitsa njira chodetsedwa.chisankho.chachitsogoleri.yatsa mu mtengo koona. Mwachikhazikitso ndizofanana ndi zabodza. Zokonda zonse = zonse с wosayera.mtsogoleri.kusankha.enable=zoona imapereka mwayi wopezeka ndi chitetezo chowonjezera cha data. Koma monga mukuonera, tikhoza kutaya mauthenga.

Koma bwanji ngati tikufuna kuwonjezera chitetezo deta? Inu mukhoza kuika unclean.leader.election.enable = zabodza, koma izi sizidzatiteteza ku kutaya deta. Ngati mtsogoleriyo adagwa mwamphamvu ndikutenga detayo, ndiye kuti mauthenga amatayikabe, kuphatikizapo kupezeka kumatayika mpaka woyang'anira abwezeretsanso zinthuzo.

Ndi bwino kuwonetsetsa kuti mauthenga onse ndi osafunikira, ndipo mwinamwake kutaya zojambulazo. Ndiye, osachepera kuchokera kumalingaliro a broker, kutayika kwa deta kumatheka pokhapokha ngati pali zolephera ziwiri kapena zingapo panthawi imodzi.

Acks=all, min.insync.replicas ndi ISR

Ndi kasinthidwe ka mutu min.insync.replicas Tikuwonjezera kuchuluka kwa chitetezo cha data. Tiyeni tidutsenso gawo lomaliza la zochitika zam'mbuyomu, koma nthawi ino ndi min.insync.replicas=2.

Chifukwa chake broker 2 ali ndi mtsogoleri wofananira ndipo wotsatira pa broker 3 amachotsedwa ku ISR.

Mpunga. 19. ISR kuchokera ku zojambula ziwiri

Broker 2 akugwa ndipo utsogoleri umadutsa kwa broker 1 popanda kutaya mauthenga. Koma tsopano ISR ili ndi chofanana chimodzi chokha. Izi sizikukwaniritsa chiwerengero chochepa kuti mulandire zolemba, choncho broker amayankha kuyesa kulemba ndi cholakwika NotEnoughReplicas.

Mpunga. 20. Chiwerengero cha ma ISR ndi chimodzi chotsika kuposa chomwe chafotokozedwa mu min.insync.replicas

Kukonzekera uku kumalepheretsa kupezeka kwa kusasinthasintha. Tisanavomere uthenga, timaonetsetsa kuti walembedwa zosachepera ziwiri. Izi zimapatsa wopanga chidaliro chochulukirapo. Apa, kutayika kwa uthenga kumatheka kokha ngati zolemba ziwiri zikulephera nthawi imodzi mu kanthawi kochepa mpaka uthengawo ubwerezedwa kwa wotsatira wina, zomwe sizingatheke. Koma ngati ndinu wodabwitsa kwambiri, mutha kukhazikitsa chobwerezabwereza ku 5, ndi min.insync.replicas ndi 3. Apa ma broker atatu ayenera kugwa nthawi imodzi kuti ataya mbiri! Inde, mumalipira kudalirika kumeneku mu latency yowonjezera.

Pamene kupezeka kuli kofunikira pachitetezo cha deta

Monga nkhani ndi RabbitMQ, nthawi zina kupezeka ndikofunikira pachitetezo cha data. Nazi zomwe muyenera kuganizira:

• Kodi wosindikizayo angangobweza cholakwika ndi kukhala ndi ntchito yakumtunda kapena wogwiritsa kuyesanso pambuyo pake?
• Kodi wosindikiza angasunge uthengawo kwanuko kapena m'nkhokwe kuti ayeserenso nthawi ina?

Ngati yankho liri ayi, ndiye kuti kukhathamiritsa kupezeka kumakulitsa chitetezo cha data. Mudzataya deta yocheperako ngati mutasankha kupezeka m'malo mopanda kujambula. Choncho, zonse zimabwera pakupeza malire, ndipo chisankho chimadalira pazochitika zenizeni.

Tanthauzo la ISR

ISR suite imakulolani kuti musankhe bwino pakati pa chitetezo cha data ndi latency. Mwachitsanzo, tsimikizirani kupezeka pakagwa kulephera kwa zofananira zambiri, kuchepetsa zotsatira zakufa kapena zofananira pang'onopang'ono potengera kuchedwa.

Timasankha tokha tanthauzo replica.lag.time.max.ms malinga ndi zosowa zanu. Kwenikweni, parameter iyi imatanthawuza kuchuluka kwa kuchedwa komwe timalolera kuvomereza zonse = zonse. Mtengo wokhazikika ndi masekondi khumi. Ngati izi zikutalika kwambiri kwa inu, mutha kuzichepetsa. Ndiye kusintha kwafupipafupi mu ISR kudzawonjezeka, popeza otsatira adzachotsedwa ndikuwonjezeredwa nthawi zambiri.

RabbitMQ ndi magalasi omwe amafunikira kufananizidwa. Magalasi oyenda pang'onopang'ono amayambitsa latency yowonjezera, ndipo magalasi akufa amatha kudikirira mpaka mapaketi omwe amayang'ana kupezeka kwa node iliyonse (tick tick) kuti ayankhe. ISR ndi njira yosangalatsa yopewera izi. Koma tili pachiwopsezo chotaya redundancy popeza ISR imatha kuchepera kwa mtsogoleri. Kuti mupewe ngoziyi, gwiritsani ntchito zoikamo min.insync.replicas.

Chitsimikizo cholumikizira kasitomala

Muzipangidwe bootstrap.servers wopanga ndi ogula amatha kufotokoza ma broker angapo kuti alumikizane ndi makasitomala. Lingaliro ndilakuti node imodzi ikatsikira, pali zingapo zotsalira zomwe kasitomala amatha kutsegula kulumikizana. Awa sikuti ndi atsogoleri azigawo, koma ndi njira yoyambira kutsitsa koyamba. Wothandizira atha kuwafunsa kuti ndi node iti yomwe imasunga mtsogoleri wowerengera / kulemba.

Mu RabbitMQ, makasitomala amatha kulumikizana ndi node iliyonse, ndipo njira zamkati zimatumiza pempho komwe likuyenera kupita. Izi zikutanthauza kuti mutha kukhazikitsa chosungira katundu kutsogolo kwa RabbitMQ. Kafka imafuna makasitomala kuti alumikizane ndi node yomwe imakhala ndi mtsogoleri wogawa. Zikatero, simungathe kuyika chojambulira katundu. Mndandanda bootstrap.servers Ndikofunikira kuti makasitomala athe kupeza ndikupeza ma node olondola atalephera.

Kafka Consensus Architecture

Mpaka pano, sitinaganizire momwe gululi limaphunzirira za kugwa kwa broker ndi momwe mtsogoleri watsopano amasankhidwira. Kuti mumvetsetse momwe Kafka amagwirira ntchito ndi magawo a netiweki, choyamba muyenera kumvetsetsa kamangidwe ka mgwirizano.

Gulu lililonse la Kafka limayikidwa limodzi ndi gulu la Zookeeper, lomwe ndi ntchito yogawana yomwe imalola kuti dongosololi likwaniritse mgwirizano pamayiko ena, ndikuyika patsogolo kusasinthika kuposa kupezeka. Chilolezo cha ambiri a Zookeeper node chikufunika kuti muvomereze kuwerenga ndi kulemba ntchito.

Zookeeper amasunga chikhalidwe cha gululi:

• Mndandanda wa mitu, magawo, kasinthidwe, zofananira za atsogoleri apano, zofananira zomwe amakonda.
• Mamembala a Cluster. Wogulitsa aliyense amawombera gulu la Zookeeper. Ngati sichilandira ping mkati mwa nthawi yodziwika, ndiye Zookeeper amalemba kuti brokeryo sakupezeka.
• Kusankha mfundo zazikulu ndi zotsalira za woyang'anira.

Node yowongolera ndi imodzi mwama broker a Kafka omwe ali ndi udindo wosankha atsogoleri ofananira. Zookeeper amatumiza zidziwitso kwa woyang'anira za umembala wamagulu ndi kusintha kwa mitu, ndipo wowongolera akuyenera kuchitapo kanthu pazosinthazi.

Mwachitsanzo, tiyeni titenge mutu watsopano wokhala ndi magawo khumi ndi chobwerezabwereza cha 3. Woyang'anira ayenera kusankha mtsogoleri pagawo lililonse, kuyesera kugawa bwino atsogoleri pakati pa ma broker.

Kwa wowongolera gawo lililonse:

• zosintha zambiri mu Zookeeper za ISR ndi mtsogoleri;
• Imatumiza MtsogoleriAndISRCommand kwa broker aliyense yemwe ali ndi chithunzi cha magawowa, kudziwitsa otsatsa za ISR ndi mtsogoleri.

Pamene broker ndi mtsogoleri agwa, Zookeeper amatumiza chidziwitso kwa wolamulira, ndipo amasankha mtsogoleri watsopano. Apanso, woyang'anira amasinthanso Zookeeper kenako amatumiza lamulo kwa broker aliyense kuwadziwitsa za kusintha kwa utsogoleri.

Mtsogoleri aliyense ali ndi udindo wolemba ma ISR. Zokonda replica.lag.time.max.ms amasankha amene adzalowa mmenemo. ISR ikasintha, mtsogoleriyo amatumiza zatsopano ku Zookeeper.

Zookeeper nthawi zonse amadziwitsidwa za kusintha kulikonse kotero kuti pakalephera, kasamalidwe kamasintha bwino kupita kwa mtsogoleri watsopano.

Mpunga. 21. Kafka Consensus

Replication protocol

Kumvetsetsa tsatanetsatane wa kubwereza kumakuthandizani kumvetsetsa bwino zomwe zingatayike deta.

Mafunso achitsanzo, Log End Offset (LEO) ndi Highwater Mark (HW)

Tidawona kuti otsatira nthawi ndi nthawi amatumiza zopempha kwa mtsogoleri. Nthawi yokhazikika ndi 500ms. Izi zimasiyana ndi RabbitMQ chifukwa kubwereza kwa RabbitMQ sikuyambika ndi galasi la pamzere koma ndi mbuye. Mbuye amakankhira kusintha kwa magalasi.

Mtsogoleri ndi otsatira onse amasunga chizindikiro cha Log End Offset (LEO) ndi Highwater (HW). Chizindikiro cha LEO chimasunga kuchotsera kwa uthenga womaliza pachifaniziro chakomweko, ndipo HW imagwira ntchito yomaliza. Kumbukirani kuti pakudzipereka, uthengawo uyenera kupitilizidwa pazithunzi zonse za ISR. Izi zikutanthauza kuti LEO nthawi zambiri imakhala patsogolo pang'ono kuposa HW.

Mtsogoleri akalandira uthenga, amausunga kumaloko. Wotsatira akupanga pempho lotenga potumiza LEO yake. Mtsogoleriyo ndiye amatumiza gulu la mauthenga kuyambira LEO iyi ndikutumizanso HW yomwe ilipo. Mtsogoleri akalandira zidziwitso kuti zofananira zonse zasunga uthengawo pamalo omwe wapatsidwa, zimasuntha chizindikiro cha HW. Mtsogoleri yekha ndi amene angasunthire HW, kotero kuti otsatira onse adzadziwa phindu lamakono mu mayankho a pempho lawo. Izi zikutanthauza kuti otsatira akhoza kutsalira kumbuyo kwa mtsogoleri mu uthenga ndi chidziwitso cha HW. Makasitomala amalandira mauthenga mpaka HW yomwe ilipo.

Dziwani kuti "kulimbikira" kumatanthauza kulembedwa kukumbukira, osati ku disk. Kuti mugwire ntchito, Kafka imagwirizanitsa ku diski panthawi inayake. RabbitMQ imakhalanso ndi nthawi yotereyi, koma idzatumiza kuvomereza kwa wofalitsa pokhapokha mbuye ndi magalasi onse alemba uthenga ku disk. Madivelopa a Kafka, pazifukwa zogwirira ntchito, adaganiza zotumiza ack uthengawo utangolembedwa kukumbukira. Kafka kubetcha kuti redundancy imathetsa chiopsezo chosunga mwachidule mauthenga ovomerezeka pamtima.

Kulephera kwa mtsogoleri

Mtsogoleri akagwa, Zookeeper amadziwitsa wowongolera, ndipo amasankha choyimira chatsopano. Mtsogoleri watsopano amayika chizindikiro chatsopano cha HW malinga ndi LEO yake. Otsatira amalandila zambiri za mtsogoleri watsopano. Kutengera mtundu wa Kafka, wotsatira adzasankha chimodzi mwazinthu ziwiri:

1. Idzachepetsa chipika chapafupi ku HW yodziwika ndikutumiza pempho kwa mtsogoleri watsopano kuti atumize mauthenga pambuyo pa chizindikirochi.
2. Adzatumiza pempho kwa mtsogoleri kuti adziwe HW panthawi yomwe adasankhidwa kukhala mtsogoleri, ndikuchepetsa chipikacho. Idzayambanso kupanga zopempha zokatenga nthawi ndi nthawi kuyambira pomwepa.

Wotsatira angafunike kuchepetsa chipika pazifukwa izi:

• Mtsogoleri akalephera, wotsatira woyamba mu ISR set yolembedwa ndi Zookeeper amapambana chisankho ndikukhala mtsogoleri. Otsatira onse pa ISR, ngakhale amaganiziridwa kuti "mu kulunzanitsa," mwina sanalandire makope a mauthenga onse kuchokera kwa mtsogoleri wakale. Ndizotheka kwathunthu kuti wotsatira yemwe wawonetsedwa alibe kope laposachedwa kwambiri. Kafka imawonetsetsa kuti palibe kusiyana pakati pa zofananira. Chifukwa chake, kuti apewe kusagwirizana, wotsatira aliyense ayenera kutsitsa chipika chake ku mtengo wa HW wa mtsogoleri watsopano pa nthawi yosankhidwa. Ichi ndi chifukwa china chokhazikitsa zonse = zonse kotero kuti kusasinthasintha.
• Mauthenga amalembedwa nthawi ndi nthawi ku disk. Ngati node zonse zamagulu zikulephera nthawi imodzi, ndiye kuti zofananira zokhala ndi zosinthika zosiyanasiyana zimasungidwa pa disks. Ndizotheka kuti ma broker akabweranso pa intaneti, mtsogoleri watsopano yemwe wasankhidwa adzakhala kumbuyo kwa otsatira ake chifukwa adapulumutsidwa ku disk pamaso pa ena.

Kuyanjananso ndi gulu

Akalowanso mgululi, zofananirazo zimagwiranso ntchito ngati mtsogoleri walephera: amawunika chithunzi cha mtsogoleri ndikudula chipika chawo ku HW yake (panthawi ya chisankho). Poyerekeza, RabbitMQ imagwiranso ntchito ngati ma node atsopano. Muzochitika zonsezi, broker amataya dziko lililonse lomwe alipo. Ngati kulunzanitsa kodziwikiratu kukugwiritsidwa ntchito, ndiye kuti mbuyeyo ayenera kubwereza zonse zomwe zilipo pagalasi latsopano munjira ya "dziko lonse lapansi lidikire". Mbuyeyo savomereza kuwerenga kapena kulemba ntchito iliyonse panthawiyi. Njira imeneyi imabweretsa mavuto m’mizere ikuluikulu.

Kafka ndi chipika chogawidwa ndipo kawirikawiri chimasunga mauthenga ambiri kuposa mzere wa RabbitMQ, kumene deta imachotsedwa pamzere pambuyo powerengedwa. Mizere ikuyenera kukhala yaying'ono. Koma Kafka ndi chipika chokhala ndi mfundo zake zosungira, zomwe zimatha kukhazikitsa masiku kapena masabata. Njira yotsekereza pamzere ndi kulunzanitsa kwathunthu ndizosavomerezeka pa chipika chogawidwa. M'malo mwake, otsatira a Kafka amangochepetsa chipika chawo ku HW ya mtsogoleri (panthawi yosankhidwa) ngati buku lawo lili patsogolo pa mtsogoleriyo. Nthawi zambiri, wotsatira akakhala kumbuyo, amangoyamba kupanga zopempha kuyambira ndi LEO yake yamakono.

Otsatira atsopano kapena olowa nawonso amayamba kunja kwa ISR ndipo satenga nawo mbali pazochita. Amangogwira ntchito limodzi ndi gulu, kulandira mauthenga mwamsanga momwe angathere mpaka atapeza mtsogoleri ndikulowa mu ISR. Palibe zotsekera ndipo palibe chifukwa chotaya deta yanu yonse.

Kutayika kwa kulumikizana

Kafka ili ndi zigawo zambiri kuposa RabbitMQ, kotero imakhala ndi machitidwe ovuta kwambiri pamene gululo likuchotsedwa. Koma Kafka poyamba adapangidwira masango, kotero mayankho amaganiziridwa bwino kwambiri.

M'munsimu muli zochitika zingapo zolephera kugwirizanitsa:

• Chitsanzo 1: Wotsatira samawona mtsogoleri, koma amawonabe Zookeeper.
• Chitsanzo 2: Mtsogoleri samawona otsatira, komabe amawona Zookeeper.
• Chitsanzo 3: Wotsatira amawona mtsogoleri, koma samamuwona Wosunga Zoo.
• Chitsanzo 4: Mtsogoleri amawona omutsatira, koma sakuwona Wosunga Zookeeper.
• Nkhani 5: Wotsatirayo ndi wosiyana kotheratu ndi ma Kafka ena onse ndi Zookeeper.
• Nkhani 6: Mtsogoleri ndi wosiyana kotheratu ndi ena onse a Kafka node ndi Zookeeper.
• Nkhani 7: Kafka controller node sangathe kuwona Kafka node ina.
• Nkhani 8: Wolamulira wa Kafka sakuwona Zookeeper.

Chochitika chilichonse chili ndi khalidwe lake.

Chitsanzo 1: Wotsatira samawona mtsogoleri, komabe amawona Zookeeper

Mpunga. 22. Chitsanzo 1: ISR ya zofananira zitatu

Kulephera kwamalumikizidwe kumalekanitsa broker 3 kuchokera kwa broker 1 ndi 2, koma osati ku Zookeeper. Broker 3 sangathenso kutumiza zopempha. Patapita nthawi replica.lag.time.max.ms imachotsedwa ku ISR ndipo sichitenga nawo mbali pazochita zauthenga. Kulumikizana kukabwezeretsedwa, kuyambiranso zopempha ndikujowina ISR ikakumana ndi mtsogoleri. Zookeeper apitilizabe kulandira ma pings ndikuganiza kuti broker ali moyo ndipo ali bwino.

Mpunga. 23. Chitsanzo 1: Wogulitsayo amachotsedwa ku ISR ngati palibe pempho lolandirira kuchokera kwa ilo mkati mwa nthawi ya replica.lag.time.max.ms

Palibe kugawanika-ubongo kapena node kuyimitsidwa monga mu RabbitMQ. M'malo mwake, redundancy imachepetsedwa.

Nkhani 2: Mtsogoleri samawona otsatira, koma amawonabe Zookeeper

Mpunga. 24. Nkhani 2. Mtsogoleri ndi omutsatira awiri

Kusokonekera kwa kulumikizana kwa netiweki kumalekanitsa mtsogoleri ndi omutsatira, koma broker amatha kuwona Zookeeper. Monga momwe zinalili poyamba, ISR ikucheperachepera, koma nthawi ino kwa mtsogoleri kokha pamene otsatira onse amasiya kutumiza zopempha. Apanso, palibe kugawanika koyenera. M'malo mwake, pali kutaya kwa redundancy kwa mauthenga atsopano mpaka kulumikizidwa kubwezeretsedwa. Zookeeper akupitilizabe kulandira ma pings ndipo amakhulupirira kuti brokeryo ali moyo ndipo ali bwino.

Mpunga. 25. Chitsanzo 2. ISR yachepa kwa mtsogoleri yekha

Chitsanzo 3. Wotsatira akuwona mtsogoleri, koma samawona Wosunga Zoo

Wotsatira amasiyanitsidwa ndi Zookeeper, koma osati kwa broker ndi mtsogoleri. Zotsatira zake, wotsatira akupitirizabe kuitanitsa zopempha ndikukhala membala wa ISR. Zookeeper salandiranso pings ndikulembetsa kuwonongeka kwa broker, koma popeza ndi wotsatira chabe, palibe zotsatira pambuyo pochira.

Mpunga. 26. Chitsanzo 3: Wotsatira akupitiriza kutumiza zopempha kwa mtsogoleri

Nkhani 4. Mtsogoleri amawona otsatira, koma samawona Zookeeper

Mpunga. 27. Nkhani 4. Mtsogoleri ndi omutsatira awiri

Mtsogoleriyo amasiyanitsidwa ndi Zookeeper, koma osati kwa ogulitsa omwe ali ndi otsatira.

Mpunga. 28. Nkhani 4: Mtsogoleri wolekanitsidwa ndi Zookeeper

Patapita kanthawi, Zookeeper adzalembetsa kulephera kwa broker ndikudziwitsa woyang'anira za izo. Iye adzasankha mtsogoleri watsopano pakati pa otsatira ake. Komabe, mtsogoleri woyambirira apitiliza kuganiza kuti ndiye mtsogoleri ndipo apitiliza kuvomera zolowa zolakwa=1. Otsatira sakumutumiziranso zopempha, kotero adzawaona ngati akufa ndikuyesera kudzichepetsera ISR yokha. Koma popeza ilibe kulumikizana ndi Zookeeper, siingathe kuchita izi, ndipo panthawiyo idzakana kuvomereza zolemba zina.

mauthenga zonse = zonse sangalandire chivomerezo chifukwa ISR imayatsa zofananira zonse, ndipo mauthenga samawafikira. Pamene mtsogoleri woyambirira ayesa kuwachotsa ku ISR, sangathe kutero ndipo adzasiya kuvomereza mauthenga aliwonse.

Makasitomala posakhalitsa amazindikira kusintha kwa mtsogoleri ndikuyamba kutumiza zolemba ku seva yatsopano. Netiweki ikabwezeretsedwa, mtsogoleri woyambirira amawona kuti salinso mtsogoleri ndipo amatsitsa chipika chake ku mtengo wa HW womwe mtsogoleri watsopanoyo anali nawo panthawi yomwe adalephera kupewa kusiyanasiyana kwamitengo. Kenako iyamba kutumiza zopempha kwa mtsogoleri watsopano. Zolemba zonse zochokera kwa mtsogoleri woyambirira zomwe sizinafotokozedwe kwa mtsogoleri watsopano zatayika. Ndiko kuti, mauthenga omwe sanavomerezedwe ndi mtsogoleri woyambirira mumasekondi ochepawo pamene atsogoleri awiri anali kugwira ntchito adzatayika.

Mpunga. 29. Zochitika 4. Mtsogoleri pa broker 1 amakhala wotsatira pambuyo pa kubwezeretsedwa kwa intaneti

Nkhani 5: Wotsatirayo ndi wosiyana kotheratu ndi ma Kafka ena onse ndi Zookeeper

Wotsatira ali wolekanitsidwa kwathunthu ndi ena onse a Kafka node ndi Zookeeper. Amangodzichotsa yekha ku ISR mpaka maukonde abwezeretsedwa, ndiyeno akugwirana ndi ena.

Mpunga. 30. Chitsanzo 5: Wotsatira wodzipatula amachotsedwa ku ISR

Nkhani 6: Mtsogoleri ndi wosiyana kotheratu ndi ena onse a Kafka node ndi Zookeeper

Mpunga. 31. Nkhani 6. Mtsogoleri ndi omutsatira awiri

Mtsogoleriyo ali wolekanitsidwa kwathunthu ndi otsatira ake, wolamulira ndi Zookeeper. Kwa kanthawi kochepa idzapitiriza kuvomereza zolemba kuchokera zolakwa=1.

Mpunga. 32. Nkhani 6: Kupatula mtsogoleri ku ma node ena a Kafka ndi Zookeeper

Popeza sanalandire zopempha pambuyo ntchito replica.lag.time.max.ms, idzayesa kuchepetsa ISR yokha, koma sichidzatha kutero chifukwa palibe kulankhulana ndi Zookeeper, ndiye idzasiya kuvomereza zolemba.

Pakadali pano, Zookeeper adzawonetsa kuti wobwereketsayo wamwalira ndipo wolamulirayo adzasankha mtsogoleri watsopano.

Mpunga. 33. Nkhani 6. Atsogoleri awiri

Mtsogoleri wapachiyambi akhoza kuvomereza zolembera kwa masekondi angapo, koma kenako amasiya kuvomereza mauthenga aliwonse. Makasitomala amasinthidwa masekondi 60 aliwonse ndi metadata yaposachedwa. Adzadziwitsidwa za kusintha kwa mtsogoleri ndipo ayamba kutumiza zolembera kwa mtsogoleri watsopano.

Mpunga. 34. Chitsanzo 6: Opanga asintha kukhala mtsogoleri watsopano

Zolemba zonse zotsimikiziridwa zopangidwa ndi mtsogoleri woyambirira kuyambira kutayika kwa kulumikizana zidzatayika. Netiweki ikabwezeretsedwa, mtsogoleri woyambirira apeza kudzera pa Zookeeper kuti salinso mtsogoleri. Kenako idzachepetsa chipika chake ku HW ya mtsogoleri watsopano panthawi ya chisankho ndikuyamba kutumiza zopempha ngati wotsatira.

Mpunga. 35. Nkhani 6: Mtsogoleri woyambirira amakhala wotsatira pambuyo poti kulumikizidwa kwa netiweki kubwezeretsedwa

Munthawi imeneyi, kulekana koyenera kumatha kuchitika kwakanthawi kochepa, koma pokhapokha ngati zolakwa=1 и min.insync.replicas komanso 1. Kupatukana koyenera kumathera mwina pambuyo pa kubwezeretsedwa kwa intaneti, pamene mtsogoleri wapachiyambi azindikira kuti salinso mtsogoleri, kapena pamene makasitomala onse azindikira kuti mtsogoleriyo wasintha ndikuyamba kulembera kwa mtsogoleri watsopano - chilichonse chimene chimachitika poyamba. Mulimonsemo, mauthenga ena adzatayika, koma ndi zolakwa=1.

Palinso mtundu wina wa zochitika izi pomwe, maukonde atangotsala pang'ono kugawanika, otsatira adagwa kumbuyo ndipo mtsogoleriyo adakakamiza ISR kuti ikhale yekha. Kenako imasiyanitsidwa chifukwa cha kutayika kwa kulumikizana. Mtsogoleri watsopano amasankhidwa, koma mtsogoleri woyambirira akupitiriza kuvomereza zolemba, ngakhale zonse = zonse, chifukwa palibe wina mu ISR kupatula iye. Zolemba izi zidzatayika pamene maukonde abwezeretsedwa. Njira yokhayo yopewera njirayi ndi min.insync.replicas = 2.

Nkhani 7: Kafka Controller Node Sitingawone Kafka Node Ina

Kawirikawiri, pamene kugwirizana ndi node ya Kafka kutayika, wolamulira sangathe kufalitsa chidziwitso cha kusintha kwa mtsogoleri aliyense. Pazifukwa zoipitsitsa, izi zidzatsogolera kupatukana kwanthawi yayitali, monga momwe zilili muzochitika 6. Nthawi zambiri, wobwereketsa sangakhale wosankhidwa kukhala mtsogoleri ngati womalizayo akulephera.

Nkhani 8: Wolamulira wa Kafka sakuwona Zookeeper

Zookeeper sadzalandira ping kuchokera kwa wolamulira wakugwa ndipo adzasankha node yatsopano ya Kafka monga wolamulira. Woyang'anira woyambirira amatha kupitiliza kudziwonetsa ngati choncho, koma samalandila zidziwitso kuchokera ku Zookeeper, chifukwa chake sadzakhala ndi ntchito iliyonse yoti achite. Maukondewo atabwezeretsedwa, adzazindikira kuti salinso wolamulira, koma wakhala kafka node wamba.

Mapeto a zochitika

Tikuwona kuti kutayika kwa kulumikizana kwa otsatira sikubweretsa kutayika kwa uthenga, koma kumangochepetsa kwakanthawi kochepa mpaka maukonde abwezeretsedwa. Izi, ndithudi, zingayambitse kutayika kwa deta ngati mfundo imodzi kapena zingapo zatayika.

Ngati mtsogoleri asiyanitsidwa ndi Zookeeper chifukwa chosowa kulumikizana, izi zitha kupangitsa kuti mauthenga atayike zolakwa=1. Kusalankhulana ndi Zookeeper kumayambitsa kugawanika kwachidule ndi atsogoleri awiriwa. Vutoli limathetsedwa ndi parameter zonse = zonse.

chizindikiro min.insync.replicas mu zofananira ziwiri kapena zingapo zimapereka chitsimikizo chowonjezera kuti zochitika zazifupi zotere sizingabweretse mauthenga otayika monga momwe zilili mu Scenario 6.

Chidule cha Mauthenga Otayika

Tiyeni titchule njira zonse zomwe mungataye data ku Kafka:

• Kulephera kwa mtsogoleri aliyense ngati mauthenga adatsimikiziridwa pogwiritsa ntchito zolakwa=1
• Kusintha kulikonse kodetsedwa kwa utsogoleri, ndiko kuti, kwa wotsatira kunja kwa ISR, ngakhale ndi zonse = zonse
• Kupatula mtsogoleri ku Zookeeper ngati mauthenga adatsimikiziridwa pogwiritsa ntchito zolakwa=1
• Kudzipatula kwathunthu kwa mtsogoleri yemwe wachepetsa kale gulu la ISR kukhala yekha. Mauthenga onse adzatayika, ngakhale zonse = zonse. Izi ndi zoona ngati min.insync.replicas=1.
• Zolephereka munthawi yomweyo magawo onse ogawa. Chifukwa chakuti mauthenga amavomerezedwa kuchokera pamtima, ena angakhale asanalembedwe ku disk. Pambuyo poyambitsanso ma seva, mauthenga ena angakhale akusowa.

Kusintha kosayenera kwa utsogoleri kutha kupewedwa powaletsa kapena kuwonetsetsa kuti achotsedwa ntchito kawiri. Kusintha kolimba kwambiri ndikophatikiza zonse = zonse и min.insync.replicas pa 1.

Kuyerekeza kwachindunji kwa kudalirika kwa RabbitMQ ndi Kafka

Kuonetsetsa kudalirika ndi kupezeka kwakukulu, nsanja zonse ziwiri zimagwiritsa ntchito njira yobwerezabwereza yoyamba ndi yachiwiri. Komabe, RabbitMQ ili ndi chidendene cha Achilles. Mukalumikizanso pambuyo polephera, ma node amataya deta yawo ndipo kulunzanitsa kumatsekedwa. Izi zikukayikitsa za kutalika kwa mizere yayikulu ku RabbitMQ. Muyenera kuvomereza kuchepetsedwa kwa redundancy kapena nthawi yayitali yotsekereza. Kuchepetsa redundancy kumawonjezera chiopsezo cha kutayika kwakukulu kwa data. Koma ngati mizereyo ndi yaying'ono, ndiye chifukwa cha redundancy, nthawi yochepa yosapezeka (masekondi angapo) ikhoza kuchitidwa pogwiritsa ntchito kuyesa mobwerezabwereza.

Kafka alibe vuto ili. Imataya deta pokhapokha pakusiyana pakati pa mtsogoleri ndi wotsatira. Zogawana zonse zasungidwa. Kuphatikiza apo, kubwereza sikulepheretsa dongosolo. Mtsogoleri akupitiriza kuvomereza zolemba pamene wotsatira watsopanoyo akugwira, kotero kwa ma devops, kujowina kapena kujowina gululi kumakhala ntchito yaing'ono. Zachidziwikire, palinso zovuta monga ma network bandwidth panthawi yobwereza. Ngati muwonjezera otsatira angapo nthawi imodzi, mutha kukumana ndi malire a bandwidth.

RabbitMQ ndi yapamwamba kuposa Kafka podalirika pamene ma seva angapo mumagulu amalephera nthawi imodzi. Monga tanenera kale, RabbitMQ imatumiza chitsimikiziro kwa wofalitsa pokhapokha uthenga utalembedwa ku disk ndi mbuye ndi magalasi onse. Koma izi zimawonjezera latency yowonjezera pazifukwa ziwiri:

• fsync mazana angapo milliseconds
• Kulephera kwa galasi kungawonekere pambuyo pa moyo wa mapaketi omwe amayang'ana kupezeka kwa node iliyonse (net tick) yatha. Ngati galasi likucheperachepera kapena kugwa, izi zimawonjezera kuchedwa.

Kubetcha kwa Kafka ndikuti ngati uthenga wasungidwa m'malo angapo, ukhoza kuvomereza mauthenga akangofika pamtima. Chifukwa cha izi, pali chiopsezo chotaya mauthenga amtundu uliwonse (ngakhale zonse = zonse, min.insync.replicas=2) pakalephera nthawi imodzi.

Ponseponse, Kafka ikuwonetsa magwiridwe antchito abwinoko apulogalamu ndipo idapangidwa kuchokera pansi mpaka magulu. Chiwerengero cha otsatira chikhoza kuwonjezeka kufika ku 11 ngati kuli kofunikira kuti mukhale odalirika. Replication factor 5 ndi chiwerengero chochepera cha zofananira pakulumikizana min.insync.replicas=3 zidzapangitsa kutaya uthenga kukhala chochitika chosowa kwambiri. Ngati maziko anu atha kuthandizira chiŵerengero chobwerezabwereza ichi ndi mlingo wa redundancy, ndiye kuti mukhoza kusankha njirayi.

RabbitMQ clustering ndi yabwino kwa mizere yaying'ono. Koma ngakhale mizere ing’onoing’ono imatha kukula msanga pakakhala anthu ambiri. Mizere ikakula, muyenera kupanga zisankho zovuta pakati pa kupezeka ndi kudalirika. Kuphatikizika kwa RabbitMQ ndikoyenera kunthawi zina pomwe phindu la kusinthasintha kwa RabbitMQ limaposa zovuta zilizonse zamagulu ake.

Njira imodzi yothanirana ndi chiopsezo cha RabbitMQ pamizere yayikulu ndikuigawa m'mizere yaying'ono yambiri. Ngati simukufuna kuyitanitsa kwathunthu pamzere wonse, koma mauthenga ofunikira (mwachitsanzo, mauthenga ochokera kwa kasitomala ena), kapena osayitanitsa kalikonse, ndiye kuti njirayi ndiyovomerezeka: yang'anani polojekiti yanga. Rebalancer kugawa mzere (ntchitoyo ikadali koyambirira).

Pomaliza, musaiwale za nsikidzi zingapo pamakina ophatikizira ndi kubwereza a RabbitMQ ndi Kafka. M'kupita kwa nthawi, machitidwe akhala okhwima komanso okhazikika, koma palibe uthenga umene udzakhala 100% wotetezeka ku imfa! Kuphatikiza apo, ngozi zazikuluzikulu zimachitika m'malo opangira data!

Ngati ndaphonya chinachake, ndalakwitsa, kapena simukugwirizana ndi mfundo iliyonse, khalani omasuka kulemba ndemanga kapena nditumizireni ine.

Nthawi zambiri ndimafunsidwa kuti: "Zosankha, Kafka kapena RabbitMQ?", "Ndi nsanja iti yomwe ili bwino?". Chowonadi ndi chakuti zimatengera momwe zinthu ziliri, zomwe zikukuchitikirani, ndi zina zambiri. Ndikuzengereza kupereka malingaliro anga chifukwa zingakhale zochulukira mochulukira kupangira nsanja imodzi pamilandu yonse yogwiritsira ntchito komanso zolephera zomwe zingatheke. Ndinalemba mndandanda wa nkhanizi kuti muthe kupanga maganizo anuanu.

Ndikufuna kunena kuti machitidwe onsewa ndi atsogoleri m'derali. Ndikhoza kukhala wokondera pang'ono chifukwa cha zomwe ndakumana nazo ndi ntchito ndimakonda kuyamikira zinthu monga kuyitanitsa uthenga wotsimikizika ndi kudalirika.

Ndikuwona matekinoloje ena omwe alibe kudalirika komanso kuyitanitsa kotsimikizika, ndiye ndimayang'ana RabbitMQ ndi Kafka ndikuzindikira kufunikira kodabwitsa kwa machitidwe onsewa.

Source: www.habr.com