Moni nonse. M'nkhaniyi ndikuuzani chifukwa chake ife ku Avito tinasankha Kafka miyezi isanu ndi inayi yapitayo ndi zomwe ziri. Ndigawana nawo imodzi mwazogwiritsa ntchito - meseji broker. Ndipo potsiriza, tiyeni tikambirane za ubwino umene tinapeza pogwiritsa ntchito Kafka monga njira ya Utumiki.

vuto

Choyamba, nkhani pang'ono. Kale tinayamba kuchoka ku zomangamanga za monolithic, ndipo tsopano Avito ali kale ndi mazana angapo a mautumiki osiyanasiyana. Ali ndi nkhokwe zawo, luso lawo laukadaulo ndipo ali ndi udindo pazantchito zawo zamabizinesi.

Imodzi mwa mavuto ndi chiwerengero chachikulu cha mautumiki ndi kulankhulana. Service A nthawi zambiri imafuna kudziwa zambiri zomwe Service B ili nazo. Pankhaniyi, Service A imapeza Service B kudzera mu synchronous API. Utumiki B umafuna kudziwa zomwe zikuchitika ndi mautumiki D ndi D, ndipo iwo, nawonso, ali ndi chidwi ndi mautumiki A ndi B. Pamene pali mautumiki ambiri "ofuna chidwi", kugwirizana pakati pawo kumasintha kukhala tangle yosokonezeka.

Nthawi yomweyo, ntchito A ikhoza kusapezeka nthawi iliyonse. Ndipo kodi utumiki B ndi mautumiki ena onse okhudzana nawo ayenera kuchita chiyani pamenepa? Ndipo ngati unyolo wa ma sequential synchronous call ukufunika kuti amalize bizinesi, mwayi wakulephera kwa ntchito yonseyo umakhala wokulirapo (ndipo unyolowo utalikirapo, umakhala wapamwamba).

Kusankhidwa kwaukadaulo

Chabwino, mavuto ndi omveka. Iwo akhoza kuthetsedwa mwa kupanga centralized mauthenga dongosolo pakati misonkhano. Tsopano aliyense wa misonkhano ayenera kudziwa za dongosolo mauthenga. Kuphatikiza apo, dongosolo lokha liyenera kukhala lololera zolakwika komanso mopingasa, komanso, pakachitika ngozi, sonkhanitsani chotchinga cholowera kuti chikonzenso.

Tiyeni tsopano tisankhe ukadaulo womwe kutumiza uthenga kudzakhazikitsidwa. Kuti tichite izi, choyamba timvetsetse zomwe tikuyembekezera kuchokera kwa izo:

• mauthenga pakati pa mautumiki sayenera kutayika;
• mauthenga akhoza kubwerezedwa;
• mauthenga amatha kusungidwa ndikuwerengedwa mozama kwa masiku angapo (buffer yosalekeza);
• mautumiki amatha kulembetsa ku data yomwe akufuna;
• mautumiki angapo amatha kuwerenga deta yomweyo;
• mauthenga amatha kukhala ndi malipiro atsatanetsatane, ochulukirapo (kutengerako kochitidwa ndi zochitika);
• Nthawi zina muyenera kutsimikizira dongosolo la mauthenga.

Zinalinso zofunika kwambiri kuti tisankhe njira yodalirika komanso yodalirika yokhala ndi zotulutsa zambiri (osachepera 100k mauthenga a kilobytes angapo pamphindi).

Panthawiyi, tidatsanzikana ndi RabbitMQ (zovuta kuti zikhazikike pamakwerero apamwamba), PGQ kuchokera ku SkyTools (yosathamanga mokwanira komanso siimakula bwino) ndi NSQ (osalimbikira). Timagwiritsa ntchito matekinoloje onsewa pakampani yathu, koma sizinali zoyenera kuti vutoli lithe.

Kenako, tinayamba kuyang'ana matekinoloje omwe anali atsopano kwa ife - Apache Kafka, Apache Pulsar ndi NATS Streaming.

Pulsar anali woyamba kutayidwa. Tidaganiza kuti Kafka ndi Pulsar ndi mayankho ofanana. Ndipo ngakhale kuti Pulsar yayesedwa ndi makampani akuluakulu, ndi atsopano ndipo amapereka latency yochepa (mwachidziwitso), tinaganiza zochoka ku Kafka mwa awiriwa monga de facto muyezo wa ntchito zoterezi. Tidzabwereranso ku Apache Pulsar mtsogolomo.

Ndipo tsopano kwatsala osankhidwa awiri: NATS Streaming ndi Apache Kafka. Tinaphunzira njira zonse ziwiri mwatsatanetsatane, ndipo zonse zinali zoyenera pa ntchitoyi. Koma pamapeto pake, tidachita mantha ndi wachibale wachinyamata wa NATS Streaming (komanso kuti m'modzi mwa omwe adayambitsa, Tyler Treat, adaganiza zosiya ntchitoyi ndikuyamba yake - Liftbridge). Nthawi yomweyo, Clustering mode ya NATS Streaming sinapereke mwayi wokweza mwamphamvu yopingasa (mwina izi sizilinso vuto pambuyo pa kuwonjezera kwa magawo ogawa mu 2017).

Komabe, NATS Streaming ndiukadaulo wabwino wolembedwa mu Go ndikuthandizidwa ndi Cloud Native Computing Foundation. Mosiyana ndi Apache Kafka, sizifunikira Zookeeper kuti agwire ntchito (mwina posachedwapa zomwezo zikhoza kunenedwa za Kafka), popeza imagwiritsa ntchito RAFT mkati. Nthawi yomweyo, kukhamukira kwa NATS ndikosavuta kuwongolera. Sitikuletsa kuti tidzabwereranso ku teknolojiyi m'tsogolomu.

Ndipo komabe, lero wopambana wathu ndi Apache Kafka. M'mayesero athu, idakhala yothamanga kwambiri (mauthenga opitilira miliyoni miliyoni pamphindikati kuti muwerenge ndi kulemba ndi uthenga wokwana 1 kilobyte), odalirika, owopsa kwambiri komanso otsimikiziridwa ndi luso lopanga makampani akulu. Kuphatikiza apo, Kafka imathandizira makampani akuluakulu angapo (ife, mwachitsanzo, timagwiritsa ntchito Confluent version), ndipo Kafka ilinso ndi chilengedwe chotukuka.

Kafka mwachidule

Tisanayambe, ndikufuna ndikupangira buku labwino kwambiri - "Kafka: The Definitive Guide" (palinso kumasulira kwa Chirasha, koma mawuwa ndi odabwitsa kwambiri). Lili ndi chidziwitso chomwe mukufunikira kuti mumvetsetse za Kafka komanso zinanso pang'ono. Zolemba za Apache ndi mabulogu a Confluent nawonso adalembedwa bwino komanso osavuta kuwerenga.

Ndiye tiyeni tiwone momwe Kafka amagwirira ntchito. Zoyambira zapamwamba za Kafka zimakhala ndi opanga, ogula, broker ndi wosunga zookeeper.

wogula

Wogulitsa ali ndi udindo wosunga deta yanu. Deta yonse imasungidwa mu mawonekedwe a binary, ndipo broker sadziwa pang'ono za zomwe iwo ali komanso zomwe zimapangidwira.

Mtundu uliwonse wa zochitika zomveka nthawi zambiri umakhala pamutu wake wosiyana. Mwachitsanzo, chochitika chopanga malonda chikhoza kugwera mumutu wopangidwa, ndipo kusintha kwake kungagwere mu chinthu.chasinthidwa. Mitu imatha kuonedwa ngati magulu a zochitika. Pamutu wamutu, mutha kukhazikitsa zosintha monga:

• kuchuluka kwa deta yosungidwa ndi/kapena zaka zake (retention.bytes, retention.ms);
• data redundancy factor (replication factor);
• kukula kwakukulu kwa uthenga umodzi (max.message.bytes);
• chiwerengero chochepa cha zofananira zosasinthasintha zomwe deta ingalembedwe kumutu (min.insync.replicas);
• Kutha kuchita kulephera pamtundu wosasinthika womwe ungathe kutaya deta (unclean.leader.election.enable);
• ndi zina zambiri (https://kafka.apache.org/documentation/#topicconfigs).

Kenako, mutu uliwonse umagawidwa mu gawo limodzi kapena angapo. Ndi m'maphwando omwe zochitika pamapeto pake zimagwa. Ngati pali otsatsa opitilira m'modzi mgululi, ndiye kuti magawowo azigawidwa mofanana kwa ma broker onse (momwe angathere), zomwe zidzalola kuti zolemba ndi kuwerenga zikhale mutu umodzi kuti ziwonjezedwe pa ma broker angapo nthawi imodzi.

Pa disk, deta ya gawo lililonse imasungidwa ngati mafayilo agawo, mwachisawawa chofanana ndi gigabyte imodzi (yolamulidwa kudzera pa log.segment.bytes). Chofunikira ndichakuti deta imachotsedwa m'magawo (pamene kusungirako kuyambika) m'magawo (simungathe kuchotsa chochitika chimodzi pagawo, mutha kungochotsa gawo lonse, ndi lokhalo losagwira ntchito).

Wosamalira Zookeeper

Zookeeper amagwira ntchito ngati sitolo ya metadata komanso wogwirizanitsa. Ndi iye amene amatha kudziwa ngati amalonda ali moyo (mutha kuyang'ana izi kudzera m'maso mwa osunga zookeeper pogwiritsa ntchito chipolopolo cha zookeeper ndi lamulo. ls /brokers/ids), broker ndi wolamulira (get /controller), kaya magawowa akugwirizana ndi zolemba zawo (get /brokers/topics/topic_name/partitions/partition_number/state). Komanso, ndikusungirako zookeeper kuti wopanga ndi wogula azipita kukafufuza kuti ndi madani ati omwe amasungidwa. Munthawi yomwe chinthu chobwerezabwereza choposa 1 chatchulidwa pamutu, woyang'anira malo osungira nyama adzawonetsa magawo omwe ali atsogoleri (adzalembedwa ndikuwerengedwa kuchokera). Kukalephera kwa broker, zambiri za magawo a atsogoleri atsopano zidzalembedwa mu zookeeper (kuchokera ku mtundu 1.1.0 motsatana, ndipo ndizofunika).

M'matembenuzidwe akale a Kafka, woyang'anira zookeeper analinso ndi udindo wosunga zochotsera, koma tsopano zasungidwa pamutu wapadera. __consumer_offsets pa broker (ngakhale mutha kugwiritsabe ntchito zookeeper pazolinga izi).

Njira yosavuta yosinthira deta yanu kukhala dzungu ndikutaya chidziwitso kuchokera kwa osunga nyama. Muzochitika zotere, zidzakhala zovuta kumvetsetsa zomwe mungawerenge komanso kuchokera kuti.

Producer

Wopanga nthawi zambiri amakhala ntchito yomwe imalemba mwachindunji ku Apache Kafka. Wopanga amasankha mutu woti asunge mauthenga ake amutu ndikuyamba kuulembera zambiri. Mwachitsanzo, wopanga akhoza kukhala ntchito yotsatsa. Pamenepa, idzatumiza zochitika monga "ad adapanga", "ad updated", "ad deleted", ndi zina zotero. Chochitika chilichonse ndi gulu lamtengo wapatali.

Mwachikhazikitso, zochitika zonse zimagawidwa pakati pa magawo a mitu pogwiritsa ntchito robin yozungulira ngati fungulo silinatchulidwe (kutaya kuyitanitsa), komanso kudzera mu MurmurHash (kiyi) ngati fungulo lilipo (kuyitanitsa mkati mwa gawo limodzi).

Ndikoyenera kudziwa nthawi yomweyo kuti Kafka imatsimikizira dongosolo la zochitika mkati mwa gulu limodzi. Koma zoona zake n’zakuti nthawi zambiri zimenezi si vuto. Mwachitsanzo, mutha kutsimikiza kuti mwawonjezera zosintha zonse pachidziwitso chomwecho mugawo limodzi (potero kusunga dongosolo la zosinthazi mkati mwa chilengezo). Mutha kutumizanso nambala yotsatizana mu gawo limodzi la zochitikazo.

ogula

Consumer ali ndi udindo wotengera deta kuchokera ku Apache Kafka. Ngati tibwerera ku chitsanzo pamwambapa, wogula akhoza kukhala ntchito yochepetsera. Ntchitoyi idzalembetsedwa kumutu wotsatsa malonda, ndipo malonda atsopano akawoneka, adzawalandira ndikuwusanthula kuti atsatire mfundo zina.

Apache Kafka amakumbukira zomwe ogula adalandira posachedwa (mutu wautumiki umagwiritsidwa ntchito pa izi __consumer__offsets), potero kuonetsetsa kuti ngati kuwerenga kukuyenda bwino, wogula salandira uthenga womwewo kawiri. Komabe, ngati mugwiritsa ntchito enable.auto.commit = njira yowona ndikugawiratu ntchito yotsata malo a ogula pamutuwu ku Kafka, mutha kutaya deta. M'makhodi opanga, nthawi zambiri malo a ogula amawongoleredwa pamanja (wopanga mapulogalamu amawongolera nthawi yomwe kuperekedwa kwa chochitika chowerengedwa kuyenera kuchitika).

Muzochitika zomwe wogula mmodzi sakwanira (mwachitsanzo, kutuluka kwa zochitika zatsopano ndi zazikulu kwambiri), mukhoza kuwonjezera ogula angapo powagwirizanitsa pamodzi mu gulu la ogula. Gulu la ogula ndilofanana ndendende ndi wogula, koma ndi deta yogawidwa pakati pa mamembala. Izi zimalola wophunzira aliyense kutenga nawo gawo la mauthenga, potero amakulitsa liwiro lowerenga.

Zotsatira za mayeso

Sindidzalemba zambiri zofotokozera pano, ndingogawana zotsatira zomwe ndapeza. Kuyesa kunachitika pamakina atatu (3 CPU, 12GB RAM, 384k SAS DISK, 15GBit/s Net), ma broker ndi oyang'anira malo osungira nyama adatumizidwa ku lxc.

Kuyesa magwiridwe antchito

Pakuyesedwa, zotsatira zotsatirazi zinapezedwa.

• Kuthamanga kwa mauthenga a 1KB nthawi imodzi ndi opanga 9 ndi zochitika 1300000 pamphindikati.
• Liwiro la kuwerenga mauthenga a 1KB nthawi imodzi ndi ogula 9 ndi zochitika 1500000 pamphindikati.

Kuyesa kulekerera zolakwika

Pakuyesedwa, zotsatirazi zidapezedwa (3 broker, 3 zookeeper).

• Kuyimitsa m'modzi mwama broker mosadziwika bwino sikupangitsa kuti gulu liyime kapena kusapezeka. Ntchito ikupitilirabe ngati yanthawi zonse, koma ma broker otsalawo ali ndi ntchito yayikulu.
• Kutha kwachilendo kwa ma broker awiri pakakhala gulu la ma broker atatu ndi min.isr = 2 kumapangitsa kuti gululi lisakhalepo kuti lilembedwe, koma likupezeka kuti liwerengedwe. Ngati min.isr = 1, gululo likupitiriza kupezeka powerenga ndi kulemba. Komabe, mawonekedwe awa amatsutsana ndi kufunikira kwa chitetezo chambiri.
• Kuyimitsidwa kwachilendo kwa seva imodzi ya Zookeeper sikupangitsa kuti gululo liyime kapena kusapezeka. Ntchito ikupitirirabe ngati yachibadwa.
• Kuyimitsidwa kwachilendo kwa ma seva awiri a Zookeeper kumapangitsa kuti gululi lisapezeke mpaka seva imodzi ya Zookeeper ibwezeretsedwa. Mawu awa ndi oona kwa gulu la Zookeeper la ma seva atatu. Zotsatira zake, pambuyo pa kafukufuku, adaganiza zoonjezera gulu la Zookeeper ku ma seva 3 kuti awonjezere kulolerana kwa zolakwika.
  

Kafka ngati ntchito

Tili otsimikiza kuti Kafka ndiukadaulo wabwino kwambiri womwe umatilola kuthana ndi ntchito yomwe tapatsidwa (kukhazikitsa uthenga broker). Komabe, tinaganiza zoletsa ntchito kuti zisalowe mwachindunji ku Kafka ndikuyitseka pamwamba ndi ntchito ya basi ya data. N’chifukwa chiyani tinachita zimenezi? Ndipotu pali zifukwa zingapo.

• Mabasi a data adatenga ntchito zonse zokhudzana ndi kuphatikizana ndi Kafka (kukhazikitsa ndikusintha ogula ndi opanga, kuyang'anira, kuchenjeza, kudula mitengo, kukulitsa, ndi zina). Chifukwa chake, kuphatikiza ndi meseji broker ndikosavuta momwe mungathere.

• Deta-bus idatilola kuti tichotse chilankhulo china kapena laibulale kuti tigwire ntchito ndi Kafka.

• Mabasi a data amalola kuti ntchito zina zichotseretu zosungirako. Mwinamwake panthawi ina tidzasintha Kafka ku Pulsar, ndipo palibe amene angazindikire kalikonse (ntchito zonse zimangodziwa za API ya data-bus).

• Data-bus idatenga kutsimikizika kwa ziwembu zazochitika.

• Kutsimikizika kumayendetsedwa pogwiritsa ntchito data-bus.

• Pansi pa chivundikiro cha mabasi a data, titha kusinthira mwakachetechete mitundu ya Kafka popanda nthawi yopumira, kuyang'anira makonzedwe a opanga, ogula, ogulitsa, ndi zina zambiri.

• Data-bus idatilola kuwonjezera zomwe timafunikira zomwe sizili ku Kafka (monga mitu yowerengera, kuyang'anira zolakwika mgulu, kupanga DLQ, ndi zina).

• Data-bus imakulolani kuti mugwiritse ntchito failover pakati pa mautumiki onse.

Pakalipano, kuti muyambe kutumiza zochitika kwa wothandizira mauthenga, mumangofunika kulumikiza laibulale yaing'ono ku code yanu yautumiki. Izi ndizo zonse. Muli ndi luso lolemba, kuwerenga ndi kukula ndi mzere umodzi wa code. Kukhazikitsa konseko kukubisidwa kwa inu, ndikungogwira zogwirira ntchito zochepa. Pansi pa hood, ntchito yamabasi a data imakweza chiwerengero chofunikira cha opanga ndi ogula ku Kubernetes ndikuwapatsa masinthidwe ofunikira, koma zonsezi ndizowonekera kuntchito yanu.

Inde, palibe chipolopolo cha siliva, ndipo njira iyi ili ndi malire ake.

• Mabasi a data akuyenera kuthandizidwa mkati, mosiyana ndi malaibulale apagulu.
• Mabasi a data amawonjezera kuchuluka kwa mayanjano pakati pa mautumiki ndi otumizira mauthenga, zomwe zimabweretsa kuchepa kwa magwiridwe antchito poyerekeza ndi Kafka wopanda kanthu.
• Sizinthu zonse zomwe zingabisike ku mautumiki mosavuta; sitikufuna kubwereza magwiridwe antchito a KSQL kapena Kafka Mitsinje mumabasi a data, chifukwa chake nthawi zina timayenera kulola kuti ntchito zipite mwachindunji.

Kwa ife, zopindulitsa zinaposa zolakwa, ndipo chigamulo chophimba uthenga wa broker ndi ntchito yosiyana chinali choyenera. M’chaka cha opareshoni sitinachite ngozi kapena mavuto aakulu.

PS Zikomo bwenzi langa, Ekaterina Obalyaeva, chifukwa cha zithunzi zabwino za nkhaniyi. Ngati mumawakonda, apa pali mafanizo enanso omwe akubwera.

Source: www.habr.com