Molweni nonke. Kule nqaku ndiza kukuxelela ukuba kutheni thina e-Avito sikhethe iKafka kwiinyanga ezilithoba ezidlulileyo kwaye yintoni na. Ndiza kubelana ngenye yeemeko zokusetyenziswa-umthengisi womyalezo. Kwaye ekugqibeleni, makhe sithethe malunga nokuba zeziphi izibonelelo esizifumanayo ekusebenziseni i-Kafka njengendlela yeNkonzo.

Ingxaki

Okokuqala, umxholo omncinci. Ngexesha elidlulileyo saqala ukusuka kwi-architecture ye-monolithic, kwaye ngoku i-Avito sele inamakhulu amaninzi eenkonzo ezahlukeneyo. Baneendawo zabo zokugcina, ezabo itekhnoloji istaki kwaye banoxanduva lwenxalenye yabo yengqiqo yeshishini.

Enye yeengxaki ngenani elikhulu leenkonzo lunxibelelwano. Inkonzo A ihlala ifuna ukwazi ulwazi olunayo iNkonzo B. Kule meko, iNkonzo A ifikelela kwiNkonzo B ngokusebenzisa i-synchronous API. Inkonzo B ifuna ukwazi ukuba kwenzekani ngeenkonzo ze-D kunye no-D, kwaye, nabo, banomdla kwiinkonzo ze-A kunye ne-B. Xa kukho ezininzi iinkonzo ezinjalo "ezinomdla", ukudibanisa phakathi kwabo kuguquke kwi-tangle tangle.

Kwangaxeshanye, inkonzo A inokungafumaneki nangaliphi na ixesha. Kwaye kufuneka zenze ntoni inkonzo B kunye nazo zonke ezinye iinkonzo eziqhagamshelwe kuyo kule meko? Kwaye ukuba ikhonkco leefowuni ezilandelelanayo ezilandelelanayo ziyafuneka ukuze kugqitywe umsebenzi weshishini, ukuba nokwenzeka kokusilela kwawo wonke umsebenzi kuba phezulu ngakumbi (kwaye okukhona ikhonkco lide, kokukhona liphezulu).

Ukukhetha iteknoloji

Kulungile, iingxaki zicacile. Zinokupheliswa ngokudala inkqubo yokuthumela imiyalezo phakathi kweenkonzo. Ngoku nganye yeenkonzo ifuna kuphela ukwazi malunga nale nkqubo yokuthumela imiyalezo. Ukongeza, inkqubo ngokwayo kufuneka ikwazi ukumelana neziphene kwaye ihlawulwe ngokuthe tye, kwaye, kwimeko yeengozi, iqokelele isithinteli sofikelelo ukulungiselela ukusetyenzwa okulandelayo.

Ngoku masikhethe itekhnoloji apho ukuhanjiswa komyalezo kuya kuphunyezwa. Ukwenza oku, masiqale siqonde ukuba silindele ntoni kuyo:

imiyalezo phakathi kweenkonzo akufuneki ilahleke;
imiyalezo inokuphinda-phindwa;
imiyalezo inokugcinwa kwaye ifundwe ubunzulu beentsuku ezininzi (isithinteli esizingileyo);
iinkonzo zinokurhuma kwidatha abanomdla kuyo;
iinkonzo ezininzi zinokufunda idatha efanayo;
imiyalezo inokuqulatha uxanduva oluneenkcukacha, umthamo omkhulu (udluliselo lwelizwe oluqhutywa sisiganeko);
Ngamanye amaxesha kufuneka uqinisekise ukulandelelana kwemiyalezo.

Kwakhona kwakubaluleke kakhulu ukuba sikhethe eyona nkqubo i-scalable nethembekileyo ene-throughput ephezulu (ubuncinci imiyalezo eyi-100k yeekhilobhayithi ezininzi ngesekhondi).

Ngeli xesha, sathi ndlela-ntle ku-RabbitMQ (kunzima ukugcina uzinzile kwi-rps ephezulu), i-PGQ esuka kwi-SkyTools (ayikhawulezi ngokwaneleyo kwaye ayilingani kakuhle) kunye ne-NSQ (engaqhubekiyo). Sisebenzisa zonke ezi teknoloji kwinkampani yethu, kodwa bezingafanelekanga ukuba ingxaki isonjululwe.

Emva koko, saqala ukujonga itekhnoloji eyayintsha kuthi - i-Apache Kafka, i-Apache Pulsar kunye ne-NATS Streaming.

UPulsar wayengowokuqala ukulahlwa. Sigqibe kwelokuba iKafka kunye nePulsar zizisombululo ezifanayo. Kwaye nangona i-Pulsar ivavanyiwe ziinkampani ezinkulu, intsha kwaye inikezela nge-latency ephantsi (kwithiyori), sagqiba ekubeni siyishiye iKafka yezi zimbini njengomgangatho we-de facto wemisebenzi enjalo. Mhlawumbi siya kubuyela kwi-Apache Pulsar kwixesha elizayo.

Kwaye ngoku kukho abaviwa ababini abasele: Ukusasazwa kwe-NATS kunye ne-Apache Kafka. Sazifunda zozibini izisombululo ngokweenkcukacha ezithile, kwaye zombini zaziwufanelekele umsebenzi. Kodwa ekugqibeleni, sasisoyika ulutsha olusondeleyo lwe-NATS Streaming (kunye nenyaniso yokuba omnye wabaphuhlisi abaphambili, uTyler Treat, wagqiba ekubeni ashiye iprojekthi kwaye aqale eyakhe - Liftbridge). Ngelo xesha, i-Clustering mode ye-NATS Streaming ayizange ibonelele ithuba lokulinganisa okuthe tye okuqinileyo (mhlawumbi oku akuseyiyo ingxaki emva kokongezwa kwendlela yokwahlula kwi-2017).

Nangona kunjalo, ukusasazwa kwe-NATS yitekhnoloji epholileyo ebhalwe kwi-Go kwaye ixhaswa yi-Cloud Native Computing Foundation. Ngokungafaniyo ne-Apache Kafka, ayifuni iZookeeper ukuba isebenze (mhlawumbi kungekudala kunokuthiwa okufanayo ngeKafka), kuba isebenzisa iRAFT ngaphakathi. Kwangaxeshanye, Ukusasazwa kwe-NATS kulula ukuyilawula. Asifuni ukuba sibuyele kule teknoloji kwixesha elizayo.

Kwaye okwangoku, namhlanje ophumeleleyo wethu nguApache Kafka. Kuvavanyo lwethu, ibonakalise ukuba iyakhawuleza kakhulu (ngaphezu kwesigidi semiyalezo ngomzuzwana yokufunda nokubhala ngevolumu yomyalezo oyikhilobhayithi e-1), ithembekile, inobungozi kakhulu kwaye ingqinwe ngamava emveliso yeenkampani ezinkulu. Ukongeza, iKafka ixhasa ubuncinci iinkampani ezininzi ezinkulu zorhwebo (thina, umzekelo, sisebenzisa inguqulelo yeConfluent), kwaye iKafka nayo inenkqubo yendalo ephuhlisiweyo.

Kafka amagqabantshintshi

Ngaphambi kokuba siqale, ndingathanda ukucebisa ngokukhawuleza incwadi ebalaseleyo - "Kafka: Isikhokelo esicacileyo" (kukwakho inguqulelo yesiRashiya, kodwa amagama ayancipha ingqondo). Iqulethe ulwazi oludingayo ukuze ufumane ukuqonda okusisiseko kweKafka kunye nangaphezulu. Amaxwebhu ka-Apache kunye nebhlog ye-Confluent nayo ibhalwe kakuhle kwaye kulula ukuyifunda.

Ke makhe sijonge iliso lentaka lendlela iKafka isebenza ngayo. I-topology esisiseko yeKafka ibandakanya umvelisi, umthengi, umthengisi kunye nomgcini wezilwanyana.

umrhwebi

Umthengisi unoxanduva lokugcina idatha yakho. Yonke idatha igcinwe kwifom yebhinari, kwaye umthengisi uyazi kancinci malunga nokuba yintoni kwaye yintoni isakhiwo sabo.

Uhlobo ngalunye lwesiganeko esinengqiqo ludla ngokufumaneka kwisihloko saso esahlukileyo. Ngokomzekelo, isiganeko sokudala isikhangiso sinokuwela kwisihloko esidaliweyo, kwaye isiganeko sokutshintsha kwayo sinokuwela kwinto.etshintshiweyo. Izihloko zinokuthathwa njengabahluli beziganeko. Kwinqanaba lesihloko, unokuseta iiparamitha zoqwalaselo ezifana:

ubungakanani bedatha egciniweyo kunye/okanye ubudala bayo (retention.bytes, retention.ms);
i-data redundancy factor (i-replication factor);
ubukhulu bobungakanani bomyalezo omnye (max.message.bytes);
inani elincinci leekopi ezihambelanayo apho idatha ingabhalwa kwisihloko (min.insync.replicas);
ukukwazi ukuphumeza i-failillover kwi-replica ye-non-synchronous lagging kunye nokulahleka kwedatha enokwenzeka (i-unclean.leader.election.enable);
kunye nokunye okuninzi (https://kafka.apache.org/documentation/#topicconfigs).

Ngokulandelayo, isihloko ngasinye sahlulwe sahlulo esinye okanye ngaphezulu. Kukumatheko apho iziganeko ekugqibeleni ziwa. Ukuba kukho ngaphezu kwe-broker enye kwiqela, ngoko izahlulelo ziya kusasazwa ngokulinganayo kuzo zonke ii-brokers (kangangoko kunokwenzeka), nto leyo eya kuvumela umthwalo ekubhaleni nasekufundeni kwisihloko esinye ukuba silinganiswe kwii-broker ezininzi ngaxeshanye.

Kwidisk, idatha yecandelo ngalinye ligcinwe ngendlela yeefayile zecandelo, ngokungagqibekanga elilingana negigabyte enye (elawulwa ngelog.segment.bytes). Into ebalulekileyo kukuba idatha iyacinywa kwizahlulo (xa ukugcinwa kuqalwa) kumacandelo (awukwazi ukucima isiganeko esinye kwisahlulelo, unokucima kuphela icandelo elipheleleyo, kwaye kuphela elingasebenziyo).

Umlondolozi

UZookeeper usebenza njengevenkile yemetadata kunye nomnxibelelanisi. Nguye okwaziyo ukuxelela ukuba ngaba abathengisi bayaphila (ungayijonga le ngeliso lomgcini wezilwanyana usebenzisa i-zookeeper-iqokobhe ngomyalelo. ls /brokers/ids), ngowuphi umthengisi ongumlawuli (get /controller), ingaba izahlulelo zikungqamaniso neekopi zazo (get /brokers/topics/topic_name/partitions/partition_number/state). Kwakhona, kumgcini wogcino-zilwanyana apho umvelisi kunye nomthengi baya kuqala ukuya kufumanisa ukuba yeyiphi i-broker ukuba zeziphi izihloko kunye nezahlulo ezigcinwe. Kwiimeko apho uphindaphindo olukhulu kuno-1 lucacisiwe kwisihloko, umgcini wogcino-zilwanyana uya kubonisa ukuba zeziphi izahlulo ezizinkokeli (ziya kubhalwa kwaye zifundwe ukusuka). Kwimeko yokungaphumeleli kwe-broker, ulwazi malunga nezahlulo zenkokeli entsha ziya kubhalwa kwi-zookeeper (ukusuka kwi-version 1.1.0 nge-asynchronously, kwaye oko kubalulekile).

Kwiinguqulelo ezindala ze-Kafka, umgcini-zilwanyana wayenoxanduva lokugcina izinto, kodwa ngoku zigcinwe kwisihloko esikhethekileyo. __consumer_offsets kumthengisi (nangona usengasebenzisa umgcini wezilwanyana ngezi njongo).

Eyona ndlela ilula yokuguqula idatha yakho ibe yithanga kukulahlekelwa lulwazi oluvela kumgcini wezilwanyana. Kwimeko enjalo, kuya kuba nzima kakhulu ukuqonda ukuba ufunde ntoni kwaye uvela phi.

producer

Umvelisi udla ngokuba yinkonzo ebhala ngokuthe ngqo idatha kwi-Apache Kafka. UMvelisi ukhetha isihloko aza kugcina kuso imiyalezo yesihloko kwaye aqalise ukubhala ulwazi kuso. Umzekelo, umvelisi unokuba yinkonzo yentengiso. Kule meko, iya kuthumela iziganeko ezifana "nentengiso eyenziwe", "intengiso ehlaziyiweyo", "i-ad icinyiwe", njl.njl kwizihloko ezinemixholo. Isiganeko ngasinye sisibini sexabiso eliphambili.

Ngokungagqibekanga, zonke iziganeko zisasazwa phakathi kwezahlulo zesihloko kusetyenziswa i-round-robin ukuba isitshixo asichazwanga (ukulahlekelwa ukucwangciswa), kunye ne-MurmurHash (isitshixo) ukuba isitshixo sikhona (ukuyalela ngaphakathi kwesahlulelo esinye).

Kuyafaneleka ukuba uqaphele kwangoko ukuba iKafka iqinisekisa ukulandelelana kweziganeko kuphela kwibhetshi enye. Kodwa enyanisweni oku akusoloko kuyingxaki. Umzekelo, ungaqiniseka ukuba wongeza zonke iinguqu kwisibhengezo esinye kwisahlulelo esinye (ngaloo ndlela ugcina ulungelelwaniso lolu tshintsho ngaphakathi kwesibhengezo). Ungathumela kwakhona inombolo yolandelelwano kwenye yemimandla yesiganeko.

Consumer

Umthengi unoxanduva lokulanda idatha kwi-Apache Kafka. Ukuba sibuyela kumzekelo ongentla, umthengi unokuba yinkonzo yokumodareyitha. Le nkonzo iya kubhaliswa kwisihloko senkonzo ye-ad, kwaye xa isikhangiso esitsha sivela, siya kuyifumana kwaye siyihlalutye ukuthotyelwa kwemigaqo-nkqubo ethile.

U-Apache Kafka ukhumbula ukuba zeziphi iziganeko zamva nje ezifunyenwe ngumthengi (isihloko senkonzo sisetyenziselwa oku __consumer__offsets), ngaloo ndlela iqinisekisa ukuba ukuba ukufunda kuphumelele, umthengi akafumani umyalezo ofanayo kabini. Nangona kunjalo, ukuba usebenzisa i-enable.auto.commit = ukhetho oluyinyani kwaye unikezele ngokupheleleyo umsebenzi wokulandelela indawo yomthengi kwisihloko ukuya eKafka, unako ulahlekelwe data. Kwikhowudi yemveliso, ngokuqhelekileyo indawo yomthengi ilawulwa ngesandla (umphuhlisi ulawula umzuzu xa ukuzinikela kwesiganeko esifundwayo kufuneka kwenzeke).

Kwiimeko apho umthengi omnye engonelanga (umzekelo, ukuhamba kweziganeko ezitsha kukhulu kakhulu), unokongeza abathengi abaninzi ngokudibanisa kunye kwiqela labathengi. Iqela labathengi ngokwengqiqo liyafana nomthengi, kodwa ngedatha esasazwa phakathi kwamalungu eqela. Oku kuvumela umthathi-nxaxheba ngamnye ukuba athathe isabelo sakhe semiyalezo, ngaloo ndlela enyusa isantya sokufunda.

Iziphumo zovavanyo

Andizukubhala isicatshulwa esichazayo apha, ndiza kwabelana ngeziphumo ezifunyenweyo. Uvavanyo lwenziwa koomatshini bomzimba aba-3 (12 CPU, 384GB RAM, 15k SAS DISK, 10GBit/s Net), iibroker kunye nomgcini wogcino-zilwanyana baye babekwa kwi-lxc.

Uvavanyo lokuSebenza

Ngexesha lovavanyo, ezi ziphumo zilandelayo zifunyenwe.

Isantya sokurekhoda imiyalezo ye-1KB ngaxeshanye ngabavelisi be-9 yiziganeko ze-1300000 ngomzuzwana.
Isantya sokufunda imiyalezo ye-1KB ngaxeshanye ngabathengi be-9 yiziganeko ze-1500000 ngomzuzwana.

Uvavanyo lokunyamezela impazamo

Ngexesha lovavanyo, ezi ziphumo zilandelayo zafunyanwa (3 brokers, 3 zookeepers).

Ukupheliswa ngokungaqhelekanga komnye wabarhwebi akubangeli ukuba iqela liyeke okanye lingafumaneki. Umsebenzi uyaqhubeka njengesiqhelo, kodwa abathengisi abaseleyo banomthwalo onzima.
Ukupheliswa okungaqhelekanga kwee-brokers ezimbini kwimeko ye-cluster of three brokers kunye ne-min.isr = 2 ikhokelela ekubeni i-cluster ingafumaneki ukuba ibhalwe, kodwa ifikeleleke ekufundeni. Ukuba i-min.isr = 1, i-cluster iyaqhubeka ifumaneka kokubili ukufunda nokubhala. Nangona kunjalo, le modi iyaphikisana nemfuno yokhuseleko oluphezulu lwedatha.
Ukuvalwa okungaqhelekanga kwenye yeeseva zeZookeeper akubangeli ukuba iqela liyeke okanye lingafumaneki. Umsebenzi uyaqhubeka njengesiqhelo.
Ukuvalwa okungaqhelekanga kweeseva ezimbini ze-Zookeeper kubangela ukuba iqela lingafumaneki de kube kanti enye yeeseva zeZookeeper ibuyiselwe. Le ngxelo iyinyani yeqela le-Zookeeper yeeseva ezi-3. Ngenxa yoko, emva kophando, kwagqitywa ekubeni kwandiswe iqela le-Zookeeper ukuya kwiiseva ezi-5 ukwandisa ukunyamezela iimpazamo.

Kafka njengenkonzo

Siqinisekile ukuba i-Kafka iteknoloji ebalaseleyo evumela ukuba sisombulule umsebenzi esiwunikiweyo (ukuphumeza umthengisi womyalezo). Nangona kunjalo, sagqiba ekubeni sithintele iinkonzo ekufikeleleni ngokuthe ngqo eKafka kwaye siyivale phezulu ngenkonzo yebhasi yedatha. Kwakutheni ukuze senze oku? Enyanisweni, kukho izizathu ezimbalwa.

Ibhasi yedatha ithathe yonke imisebenzi enxulumene nokudityaniswa neKafka (ukuphunyezwa kunye nokucwangciswa kwabathengi kunye nabavelisi, ukubeka iliso, ukulumkisa, ukugawulwa kwemithi, ukukala, njl.). Ngaloo ndlela, ukudibanisa kunye nomthengisi womyalezo kulula kangangoko kunokwenzeka.
Ibhasi yedatha isivumele ukuba sithathelwe kude kulwimi oluthile okanye kwithala leencwadi ukuze sisebenze neKafka.
Ibhasi yedatha ivumele ezinye iinkonzo ukuba zikhuphe umaleko wogcino. Mhlawumbi ngexesha elithile siya kutshintsha iKafka kwiPulsar, kwaye akukho mntu uya kuqaphela nantoni na (zonke iinkonzo ziyazi kuphela nge-data-bus API).
I-data-bus ithathele kuyo ukuqinisekiswa koyilo lwesiganeko.
Uqinisekiso luphunyezwa kusetyenziswa ibhasi yedatha.
Ngaphantsi koqweqwe lwedatha-yebhasi, sinokuhlaziya ngokuzolileyo iinguqulelo zeKafka ngaphandle kwexesha lokuphumla, silawule ulungelelwaniso lwabavelisi, abathengi, abarhwebi, njl.
Ibhasi yedatha yasivumela ukuba songeze iimpawu esizifunayo ezingekho eKafka (ezifana nezihloko zophicotho-zincwadi, ukubeka iliso okungahambi kakuhle kwiqela, ukudala i-DLQ, njl.).
Ibhasi yedatha ikuvumela ukuba usebenzise i-failover centrally kuzo zonke iinkonzo.

Okwangoku, ukuqala ukuthumela iziganeko kumthengisi womyalezo, kufuneka nje udibanise ilayibrari encinci kwikhowudi yakho yenkonzo. Konke oku. Unakho ukubhala, ukufunda kunye nokukala ngomgca omnye wekhowudi. Ukuphunyezwa okupheleleyo kufihliwe kuwe, kunye nezibambo ezimbalwa ze-batch eziphuma ngaphandle. Ngaphantsi kwe-hood, inkonzo yebhasi yedatha iphakamisa inani elifunekayo lomvelisi kunye neemeko zabathengi kwi-Kubernetes kwaye ibonelela ngokucwangciswa okufunekayo, kodwa konke oku kubonakala kwinkonzo yakho.

Ewe, akukho bullet yesilivere, kwaye le ndlela inemida yayo.

I-data-bus kufuneka ixhaswe ngaphakathi endlwini, ngokuchaseneyo namathala eencwadi angaphandle.
Ibhasi yedatha yonyusa inani lokusebenzisana phakathi kweenkonzo kunye nomthengisi womyalezo, okukhokelela ekusebenzeni okuphantsi xa kuthelekiswa neKafka engenanto.
Ayiyiyo yonke into enokufihlwa kwiinkonzo ngokulula; asifuni ukuphinda ukusebenza kwe-KSQL okanye iKafka Streams kwi-data-bus, ngamanye amaxesha kufuneka sivumele iinkonzo zihambe ngokuthe ngqo.

Kwimeko yethu, i-pros yagqithisa i-cons, kwaye isigqibo sokugubungela umthengisi womyalezo ngenkonzo eyahlukileyo sasifanelekile. Ebudeni bonyaka wokusebenza asizange sibe naziphi na iingozi ezimbi okanye iingxaki.

PS Ndiyabulela kwintombi yam, u-Ekaterina Obalyaeva, ngemifanekiso epholileyo yeli nqaku. Ukuba uyabathanda, apha kukho eminye imizekeliso ezayo.

umthombo: www.habr.com