🥇Kumvetsetsa ma broker. Kuphunzira zimango zotumizirana mauthenga ndi ActiveMQ ndi Kafka. Mutu 3. Kafka

Kupitiliza kumasulira kabukhu kakang'ono:
Kumvetsetsa Message Brokers
wolemba: Jakub Korab, wosindikiza: O'Reilly Media, Inc., tsiku lofalitsidwa: June 2017, ISBN: 9781492049296.

Gawo lomasuliridwa m'mbuyomu: Kumvetsetsa ma broker. Kuphunzira zimango zotumizirana mauthenga ndi ActiveMQ ndi Kafka. Mutu 1 Mau oyamba

MUTU 3

Kafka

Kafka adapangidwa ndi LinkedIn kuti azitha kuthana ndi zolephera za otsatsa mauthenga achikhalidwe ndikupewa kukhazikitsa ma broker angapo kuti athe kulumikizana mosiyanasiyana, zomwe zafotokozedwa m'bukuli pansi pa "Kukweza ndi Kutuluka" patsamba 28. .Kugwiritsa ntchito milandu LinkedIn yadalira kwambiri njira imodzi yolowera deta yochuluka kwambiri, monga kusindikiza masamba ndi zolemba zolowera, pomwe amalola kuti detayi igwiritsidwe ntchito ndi machitidwe angapo popanda kukhudza zokolola za opanga kapena ogula ena. M'malo mwake, chifukwa chomwe Kafka alipo ndikupeza mtundu wamamangidwe a mauthenga omwe Universal Data Pipeline akufotokoza.

Poganizira cholinga chomaliza ichi, zofunikira zina zidabuka mwachibadwa. Kafka ayenera:

Khalani othamanga kwambiri
Perekani bandwidth yochulukirapo mukamagwira ntchito ndi mauthenga
Thandizani Ofalitsa-Olembetsa ndi Zitsanzo za Point-to-Point
Osachedwetsa powonjezera ogula. Mwachitsanzo, magwiridwe antchito a pamzere ndi mutu wa ActiveMQ amatsika pomwe kuchuluka kwa ogula komwe akupita kukukulira.
Khalani horizontally scalable; ngati broker m'modzi yemwe amalimbikira mauthenga atha kutero pa liwiro lalikulu la disk, ndiye kuti ndizomveka kupitilira gawo limodzi la broker kuti muwonjezere magwiridwe antchito.
Chepetsani mwayi wosunga ndi kubwezeretsanso mauthenga

Kuti akwaniritse zonsezi, Kafka adatengera zomanga zomwe zidafotokozanso maudindo ndi maudindo amakasitomala ndi otumizirana mauthenga. Mtundu wa JMS ndi wokonda kwambiri broker, komwe broker ali ndi udindo wogawa mauthenga ndipo makasitomala amangodandaula za kutumiza ndi kulandira mauthenga. Komano, Kafka ndi wokhazikika pamakasitomala, ndipo kasitomala amatenga zinthu zambiri za broker wamba, monga kugawa bwino mauthenga ofunikira kwa ogula, posinthanitsa ndi broker wothamanga kwambiri komanso wowopsa. Kwa anthu omwe agwira ntchito ndi mauthenga achikhalidwe, kugwira ntchito ndi Kafka kumafuna kusintha kwakukulu kwa malingaliro.
Mayendedwe a uinjiniyawa apangitsa kuti pakhale njira yolumikizirana mauthenga yomwe ingathe kuchulukirachulukira ndi maoda ambiri poyerekeza ndi broker wamba. Monga momwe tidzaonera, njirayi imabwera ndi malonda, zomwe zikutanthauza kuti Kafka si yoyenera kwa mitundu ina ya ntchito ndi mapulogalamu oikidwa.

Mtundu Wogwirizana wa Kopita

Kuti akwaniritse zofunikira zomwe tafotokozazi, Kafka yaphatikiza kusindikiza-kulembetsa ndi kutumizirana mameseji kumalo amodzi - mutu. Izi ndi zosokoneza kwa anthu omwe agwira ntchito ndi makina otumizira mauthenga, pomwe mawu oti "mutu" amatanthauza njira yowulutsira yomwe (kuchokera pamutuwo) kuwerenga sikungatheke. Mitu ya Kafka iyenera kuganiziridwa ngati mtundu wa kopita wosakanizidwa, monga tafotokozera kumayambiriro kwa bukuli.

Kwa gawo lotsala la mutu uno, pokhapokha titanena momveka bwino, mawu oti "mutu" atanthauza mutu wa Kafka.

Kuti timvetse bwino momwe mitu ikuchitira komanso zomwe amapereka, tiyenera kuyang'ana kaye momwe ikugwiritsidwira ntchito ku Kafka.
Mutu uliwonse ku Kafka uli ndi chipika chake.
Opanga omwe amatumiza mauthenga ku Kafka amalembera chipikachi, ndipo ogula amawerenga kuchokera pachipikacho pogwiritsa ntchito zolozera zomwe zikupita patsogolo nthawi zonse. Nthawi ndi nthawi, Kafka amachotsa mbali zakale kwambiri za chipikacho, kaya mauthenga omwe ali m'zigawozo awerengedwa kapena ayi. Mbali yapakati pa mapangidwe a Kafka ndikuti broker samasamala ngati mauthenga awerengedwa kapena ayi - ndilo udindo wa kasitomala.

Mawu akuti "log" ndi "pointer" samawoneka mkati Kafka zolemba. Mawu odziwika bwinowa amagwiritsidwa ntchito pano kuthandizira kumvetsetsa.

Mtunduwu ndi wosiyana kwambiri ndi ActiveMQ, pomwe mauthenga ochokera pamizere yonse amasungidwa mu chipika chimodzi, ndipo wotsatsa amalemba kuti mauthengawo achotsedwa atawerengedwa.
Tsopano tiyeni tifufuze mozama ndikuyang'ana pa chipika cha mutu mwatsatanetsatane.
Lolemba la Kafka lili ndi magawo angapo (Chithunzi 3-1). Kafka imatsimikizira kuyitanitsa mosamalitsa pagawo lililonse. Izi zikutanthauza kuti mauthenga olembedwa ku magawo mu dongosolo linalake adzawerengedwa mu dongosolo lomwelo. Gawo lirilonse likugwiritsidwa ntchito ngati fayilo yopukutira yomwe ili ndi kagawo kakang'ono (gawo) la mauthenga onse otumizidwa pamutu ndi opanga ake. Mutu wopangidwa uli ndi, mwachisawawa, gawo limodzi. Lingaliro la magawo ndi lingaliro lapakati la Kafka pakukweza kopingasa.

Chithunzi 3-1. Kafka Partitions

Wopanga akatumiza uthenga kumutu wa Kafka, amasankha gawo lomwe angatumizeko. Tidzawona izi mwatsatanetsatane pambuyo pake.

Kuwerenga mauthenga

Makasitomala amene akufuna kuwerenga mauthenga amayang'anira cholozera chotchedwa gulu la ogula, zomwe zimaloza kuchepetsa mauthenga mu gawo. Kuchotsera ndi malo owonjezera omwe amayamba pa 0 kumayambiriro kwa magawo. Gulu la ogula ili, lomwe limatchulidwa mu API kudzera pa gulu lofotokozedwa ndi wosuta_id, likufanana ndi wogula m'modzi womveka kapena dongosolo.

Makina ambiri otumizirana mameseji amawerenga zomwe akupita pogwiritsa ntchito maulendo angapo ndi ulusi kuti akonze mauthenga mofanana. Chifukwa chake, nthawi zambiri pamakhala ogula ambiri omwe amagawana gulu limodzi la ogula.

Vuto la kuwerenga likhoza kuimiridwa motere:

Mutu uli ndi magawo angapo
Magulu angapo a ogula amatha kugwiritsa ntchito mutu nthawi imodzi
Gulu la ogula likhoza kukhala ndi zochitika zingapo zosiyana

Ili ndi vuto laling'ono mpaka lambiri. Kuti timvetsetse momwe Kafka amachitira maubwenzi pakati pa magulu ogula, zochitika za ogula, ndi magawo, tiyeni tiwone mndandanda wa zochitika zovuta kwambiri zowerengera.

Ogula ndi magulu ogula

Tiyeni titenge ngati poyambira mutu wokhala ndi gawo limodzi (Chithunzi 3-2).

Chithunzi 3-2. Wogula amawerenga kuchokera kugawo

Nthawi ya ogula ikalumikizana ndi gulu lake_id kumutuwu, amapatsidwa gawo lowerengera ndi kuchotsera pagawolo. Malo ochotsera izi amatha kusinthidwa mwa kasitomala ngati cholozera chaposachedwa kwambiri (uthenga waposachedwa) kapena malo akale kwambiri (uthenga wakale kwambiri). Ogula amapempha (zosankha) mauthenga kuchokera pamutu, zomwe zimawapangitsa kuti aziwerengedwa motsatizana kuchokera pa chipikacho.
Udindo wa offset umabwereranso ku Kafka ndikusungidwa ngati mauthenga pamutu wamkati _consumer_offsets. Mauthenga owerengeka sanachotsedwe, mosiyana ndi broker wamba, ndipo kasitomala amatha kubwezeretsanso zomwe adaziwona kale.

Wogula wachiwiri womveka akalumikizana pogwiritsa ntchito gulu losiyana_id, amawongolera cholozera chachiwiri chomwe chili chodziyimira pawokha choyamba (Chithunzi 3-3). Chifukwa chake, mutu wa Kafka umakhala ngati mzere pomwe pali wogula m'modzi komanso ngati mutu wamba wosindikiza (pub-sub) womwe ogula ambiri amaulembera, ndi phindu lowonjezera lomwe mauthenga onse amasungidwa ndipo amatha kusinthidwa kangapo.

Chithunzi 3-3. Ogula awiri m'magulu osiyanasiyana ogula amawerenga kuchokera kugawo lomwelo

Ogula mu gulu la ogula

Nthawi imodzi ya ogula ikawerenga deta kuchokera kugawo, imakhala ndi mphamvu zonse za pointer ndikusintha mauthenga monga momwe tafotokozera m'gawo lapitalo.
Ngati nthawi zingapo za ogula zidalumikizidwa ndi gulu lomwelo_id kumutu womwe uli ndi gawo limodzi, ndiye kuti cholumikizira chomaliza chidzapatsidwa ulamuliro pa cholozera ndipo kuyambira nthawi imeneyo ilandila mauthenga onse (Chithunzi 3-4).

Chithunzi 3-4. Ogula awiri mu gulu limodzi la ogula amawerenga kuchokera kugawo lomwelo

Njira yosinthira iyi, momwe kuchuluka kwa ogula kumapitilira kuchuluka kwa magawo, kumatha kuganiziridwa ngati mtundu wa ogula okha. Izi zitha kukhala zothandiza ngati mukufuna "active-passive" (kapena "hot-worm") kuphatikiza kwa ogula, ngakhale kuyendetsa ogula angapo mofananira ("active-active" kapena "hot-hot") ndikofanana kwambiri kuposa ogula.

Mchitidwe wogawa uthenga uwu womwe wafotokozedwa pamwambapa ungakhale wodabwitsa poyerekeza ndi momwe mzere wamba wa JMS umachita. Muchitsanzo ichi, mauthenga otumizidwa pamzere adzagawidwa mofanana pakati pa ogula awiri.

Nthawi zambiri, pamene tipanga maulendo angapo a ogula, timachita izi kuti tigwiritse ntchito mauthenga mofanana, kapena kuwonjezera liwiro la kuwerenga, kapena kuwonjezera kukhazikika kwa kuwerenga. Popeza kuti wogula mmodzi yekha amatha kuwerenga deta kuchokera kugawo limodzi, kodi izi zimatheka bwanji ku Kafka?

Njira imodzi yochitira izi ndikugwiritsa ntchito chitsanzo chimodzi cha ogula kuti muwerenge mauthenga onse ndikuwapereka ku dziwe la ulusi. Ngakhale njira iyi imawonjezera kupititsa patsogolo, imawonjezera zovuta zamalingaliro a ogula ndipo sizichita chilichonse kuti ziwonjezere kulimba kwa pulogalamu yowerengera. Ngati kope limodzi la ogula likutsika chifukwa cha kulephera kwa mphamvu kapena zochitika zofanana, ndiye kuti kuchotsako kumasiya.

Njira yovomerezeka yothetsera vutoli ku Kafka ndikugwiritsa ntchito bОmagawo ambiri.

Kugawa

Magawo ndiye njira yayikulu yofananira kuwerenga ndikukweza mutu kupitilira bandwidth yamtundu wa broker m'modzi. Kuti timvetsetse bwino izi, tiyeni tilingalire za nthawi yomwe pali mutu wokhala ndi magawo awiri ndipo wogula m'modzi amalembetsa pamutuwu (Chithunzi 3-5).

Chithunzi 3-5. Wogula m'modzi amawerenga kuchokera ku magawo angapo

Muzochitika izi, wogula amapatsidwa ulamuliro pa zolozera zogwirizana ndi gulu_id yake m'magawo onse awiri ndikuyamba kuwerenga mauthenga kuchokera ku magawo onse awiri.
Pamene wogula wowonjezera wa gulu lomwelo_id awonjezeredwa pamutuwu, Kafka imagawiranso gawo limodzi kuchokera koyamba mpaka wogula wachiwiri. Pambuyo pake, chitsanzo chilichonse cha wogula chidzawerenga kuchokera ku gawo limodzi la mutuwo (Chithunzi 3-6).

Kuonetsetsa kuti mauthenga asinthidwa mofanana mu ulusi 20, muyenera magawo 20 osachepera. Ngati pali magawo ochepa, mudzasiyidwa ndi ogula omwe alibe chochita, monga tafotokozera kale pokambirana za ogula okha.

Chithunzi 3-6. Ogula awiri omwe ali mugulu limodzi la ogula amawerenga kuchokera ku magawo osiyanasiyana

Dongosololi limachepetsa kwambiri zovuta za broker wa Kafka poyerekeza ndi kugawa uthenga wofunikira kuti asunge mzere wa JMS. Apa simuyenera kuda nkhawa ndi mfundo zotsatirazi:

Ndi wogula uti yemwe akuyenera kulandira uthenga wotsatira, kutengera kugawika kwa robin, kuchuluka kwa ma buffers, kapena mauthenga am'mbuyomu (monga magulu a mauthenga a JMS).
Ndi mauthenga ati omwe amatumizidwa kwa ogula komanso ngati akuyenera kutumizidwanso ngati atalephera.

Zomwe broker wa Kafka ayenera kuchita ndikutumiza mauthenga motsatizana kwa ogula akawapempha.

Komabe, zofunikira zofananira ndi kuwerengera ndi kutumizanso mauthenga olephera sizichoka - udindo wawo umangochoka kwa broker kupita kwa kasitomala. Izi zikutanthauza kuti ziyenera kuganiziridwa mu code yanu.

Kutumiza mauthenga

Ndi udindo wa wopanga uthengawo kusankha gawo lomwe angatumizeko uthengawo. Kuti timvetsetse momwe izi zimachitikira, choyamba tiyenera kulingalira zomwe kwenikweni tikutumiza.

Pomwe mu JMS timagwiritsa ntchito mawonekedwe a uthenga okhala ndi metadata (mitu ndi katundu) ndi thupi lomwe lili ndi zolipira (zolipira), ku Kafka uthengawo ndi awiri "key-value". Malipiro a uthenga amatumizidwa ngati mtengo. Mfungulo, kumbali ina, imagwiritsidwa ntchito kwambiri pogawa ndipo iyenera kukhala Business logic enieni kiyikuyika mauthenga okhudzana ndi magawo omwewo.

Mu Chaputala 2, tidakambirana za kubetcha pa intaneti komwe kumayenera kukonzedwa ndi wogula m'modzi:

Akaunti ya ogwiritsa ntchito idakonzedwa.
Ndalama zimayikidwa ku akaunti.
Kubetcha kumapangidwa komwe kumachotsa ndalama ku akaunti.

Ngati chochitika chilichonse chili uthenga wotumizidwa kumutu, ndiye kuti kiyi yachilengedwe ingakhale ID ya akaunti.
Uthenga ukatumizidwa pogwiritsa ntchito Kafka Producer API, umaperekedwa ku gawo la magawo omwe, atapatsidwa uthenga ndi momwe gulu la Kafka lilili panopa, amabwezeretsanso ID ya gawo lomwe uthengawo uyenera kutumizidwa. Izi zimachitika mu Java kudzera mu mawonekedwe a Partitioner.

Mawonekedwe awa akuwoneka motere:

interface Partitioner {
    int partition(String topic,
        Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster);
}

Kukhazikitsa kwa Partitioner kumagwiritsa ntchito algorithm yokhazikika-cholinga chambiri pa kiyiyo kuti idziwe gawo, kapena kuzungulira ngati palibe kiyi yotchulidwa. Mtengo wosasinthawu umagwira ntchito bwino nthawi zambiri. Komabe, mtsogolomu mudzafuna kulemba zanu.

Kulemba njira yanu yogawanitsa

Tiyeni tiwone chitsanzo chomwe mukufuna kutumiza metadata limodzi ndi kuchuluka kwa uthenga. Kulipira muchitsanzo chathu ndi malangizo oti mupange ndalama ku akaunti yamasewera. Langizo ndi chinthu chomwe tikufuna kuti chitsimikizidwe kuti sichisinthidwa pakutumiza ndipo tikufuna kutsimikiza kuti ndi njira yodalirika yokhayo yomwe ingayambitse malangizowo. Pankhaniyi, machitidwe otumizira ndi kulandira amavomereza kugwiritsa ntchito siginecha kutsimikizira uthengawo.
Mu JMS wamba, timangofotokozera za "siginecha ya uthenga" ndikuwonjezera ku uthengawo. Komabe, Kafka simatipatsa njira yodutsira metadata, makiyi okha ndi mtengo.

Popeza mtengo wake ndi ndalama zotumizira banki zomwe tikufuna kusunga umphumphu, tilibe chochita koma kufotokozera dongosolo la data lomwe lingagwiritsidwe ntchito pakiyi. Pongoganiza kuti tikufuna ID ya akaunti yogawa, popeza mauthenga onse okhudzana ndi akaunti ayenera kusinthidwa mwadongosolo, tidzabwera ndi mawonekedwe awa a JSON:

{
  "signature": "541661622185851c248b41bf0cea7ad0",
  "accountId": "10007865234"
}

Chifukwa kufunikira kwa siginecha kumasiyana malinga ndi kuchuluka kwa malipiro, njira yokhazikika ya hashing ya mawonekedwe a Partitioner sichingadalire mauthenga okhudzana ndi gulu. Chifukwa chake, tifunika kulemba njira yathu yomwe ingafotokozere funguloli ndikugawa mtengo wa accountId.

Kafka imaphatikizapo ma checksums kuti azindikire ziphuphu za mauthenga mu sitolo ndipo ali ndi zida zonse zachitetezo. Ngakhale zili choncho, nthawi zina zimawoneka zofunikira pamakampani, monga zomwe zili pamwambapa.

Njira yogawanitsa ya wogwiritsayo iyenera kuwonetsetsa kuti mauthenga onse okhudzana ndi gawo limodzi. Ngakhale izi zikuwoneka zophweka, chofunikiracho chikhoza kukhala chovuta chifukwa cha kufunikira koyitanitsa mauthenga okhudzana ndi momwe chiwerengero cha magawo pamutu chiliri.

Chiwerengero cha magawo pamutuwu chimatha kusintha pakapita nthawi, chifukwa amatha kuwonjezeredwa ngati kuchuluka kwa magalimoto kupitilira zomwe tikuyembekezera. Chifukwa chake, makiyi a mauthenga amatha kulumikizidwa ndi magawo omwe adatumizidwako koyambirira, kutanthauza gawo lachigawo lomwe liyenera kugawidwa pakati pa opanga.

Mfundo ina yofunika kuiganizira ndi kugaŵidwa kofanana kwa mauthenga m’magawo onse. Nthawi zambiri, makiyi samagawidwa mofanana pa mauthenga onse, ndipo ntchito za hashi sizimatsimikizira kugawidwa koyenera kwa mauthenga a makiyi ang'onoang'ono.
Ndikofunika kuzindikira kuti ngakhale mutasankha kugawa mauthenga, cholekanitsacho chiyenera kugwiritsidwanso ntchito.

Ganizirani kufunikira kobwereza deta pakati pa magulu a Kafka m'malo osiyanasiyana. Pachifukwa ichi, Kafka amabwera ndi chida cha mzere cholamula chotchedwa MirrorMaker, chomwe chimagwiritsidwa ntchito powerenga mauthenga kuchokera kumagulu amodzi ndikuwatengera ku gulu lina.

MirrorMaker iyenera kumvetsetsa makiyi a mutu womwe wabwerezedwa kuti asunge dongosolo pakati pa mauthenga akamafanana pakati pa magulu, popeza kuchuluka kwa magawo a mutuwo sikungakhale kofanana m'magulu awiri.

Njira zogawanitsa mwamakonda ndizosowa, chifukwa hashing kapena robin yozungulira imagwira ntchito bwino nthawi zambiri. Komabe, ngati mukufuna zitsimikizo zamphamvu zoyitanitsa kapena mukufunika kuchotsa metadata kuchokera pamalipira, ndiye kuti kugawa ndichinthu chomwe muyenera kuyang'anitsitsa.

Kuchulukirachulukira ndi mapindu a Kafka amabwera chifukwa chosintha maudindo ena a broker kupita kwa kasitomala. Pachifukwa ichi, chigamulo chapangidwa kuti chigawidwe mauthenga omwe angakhale okhudzana ndi ogula angapo omwe amagwira ntchito mofanana.

Otsatsa a JMS amafunikanso kuthana ndi izi. Chochititsa chidwi n'chakuti, njira yotumizira mauthenga okhudzana ndi wogula yemweyo, yomwe ikugwiritsidwa ntchito kudzera m'magulu a mauthenga a JMS (kusiyana kwa njira ya Sticky load balancing (SLB)), imafunanso kuti wotumizayo alembe mauthenga ngati okhudzana. Pankhani ya JMS, broker ali ndi udindo wotumiza gulu ili la mauthenga okhudzana ndi ogula m'modzi mwa ambiri, ndikusamutsa umwini wa gulu ngati wogula akugwa.

Mapangano Opanga

Kugawa si chinthu chokhacho choyenera kuganizira potumiza mauthenga. Tiyeni tiwone njira zotumizira () za gulu la Producer mu Java API:

Future < RecordMetadata > send(ProducerRecord < K, V > record);
Future < RecordMetadata > send(ProducerRecord < K, V > record, Callback callback);

Tiyenera kuzindikira kuti njira zonse ziwirizi zimabwerera ku Future, zomwe zimasonyeza kuti ntchito yotumizayo sichitika nthawi yomweyo. Chotsatira chake ndikuti uthenga (ProducerRecord) umalembedwa ku buffer yotumizira gawo lililonse logwira ntchito ndikutumizidwa kwa broker ngati ulusi wakumbuyo mu library yamakasitomala a Kafka. Ngakhale izi zimapangitsa zinthu kukhala zofulumira kwambiri, zikutanthauza kuti pulogalamu yosadziwa imatha kutaya mauthenga ngati njira yake yayimitsidwa.

Monga nthawi zonse, pali njira yopangira ntchito yotumiza yodalirika pamtengo wa ntchito. Kukula kwa buffer iyi kutha kukhazikitsidwa ku 0, ndipo ulusi wotumizira ukakamizidwa kudikirira mpaka kutumiza uthenga kwa broker kukamalizidwa, motere:

RecordMetadata metadata = producer.send(record).get();

Zambiri pakuwerenga mauthenga

Kuwerenga mauthenga kuli ndi zovuta zina zomwe ziyenera kuganiziridwa. Mosiyana ndi JMS API, yomwe imatha kuyendetsa omvera uthenga poyankha uthenga, the ogula Kafka mavoti okha. Tiyeni tione mwatsatanetsatane njira voti ()amagwiritsidwa ntchito pazifukwa izi:

ConsumerRecords < K, V > poll(long timeout);

Mtengo wobwezera wa njirayo ndi kapangidwe ka chidebe chokhala ndi zinthu zingapo mbiri ya ogula kuchokera ku magawo angapo. mbiri ya ogula palokha ndi chinthu chogwirizira pamtengo wamtengo wapatali wokhala ndi metadata yogwirizana, monga kugawa komwe kwachokera.

Monga tafotokozera m'Mutu 2, tiyenera kukumbukira zomwe zimachitika mauthenga atatha kukonzedwa bwino kapena osachita bwino, mwachitsanzo, ngati wofuna chithandizo akulephera kukonza uthengawo kapena ngati wachotsa mimba. Mu JMS, izi zidayendetsedwa kudzera munjira yovomerezeka. Wogulitsayo amachotsa uthenga womwe wakonzedwa bwino, kapena adzaperekanso uthenga wosaphika kapena wabodza (poganiza kuti zogulitsa zidagwiritsidwa ntchito).
Kafka imagwira ntchito mosiyana kwambiri. Mauthenga sachotsedwa mu broker pambuyo powerengera, ndipo zomwe zimachitika pakulephera ndi udindo wa code yowerengera yokha.

Monga tanenera, gulu la ogula limagwirizana ndi zomwe zili mu chipika. Malo a chipika okhudzana ndi kuchotsera uku akufanana ndi uthenga wotsatira womwe uyenera kuperekedwa poyankha voti (). Nthawi yomwe kuchotsera uku kumawonjezeka ndiyofunikira pakuwerenga.

Kubwerera ku mtundu wowerengera womwe takambirana kale, kukonza uthenga kumakhala ndi magawo atatu:

Pezani uthenga woti muwerenge.
Yambitsani uthenga.
Tsimikizirani uthenga.

Wogula wa Kafka amabwera ndi njira yosinthira enable.auto.commit. Izi ndizomwe zimagwiritsidwa ntchito pafupipafupi, monga momwe zimakhalira ndi zokonda zomwe zimakhala ndi mawu oti "auto".

Kafka 0.10 isanachitike, kasitomala wogwiritsa ntchito njirayi amatha kutumiza uthenga womaliza womwe uwerengedwa pa foni yotsatira. voti () pambuyo processing. Izi zikutanthauza kuti mauthenga aliwonse omwe adalandidwa kale akhoza kukonzedwanso ngati kasitomala anali atawakonza kale koma adawonongeka mosayembekezereka asanayimbire. voti (). Chifukwa broker sasunga boma lililonse la kuchuluka kwa uthenga womwe wawerengedwa, wogula wotsatira yemwe amapeza uthengawo sadziwa chilichonse chomwe chidachitika. Khalidweli linali lachinyengo. Chotsitsacho chinangoperekedwa ngati uthengawo udakonzedwa bwino, koma ngati kasitomala achotsa, broker amatumizanso uthenga womwewo kwa kasitomala wina. Khalidweli linali logwirizana ndi chitsimikizo chotumiza uthenga "kamodzi".

Ku Kafka 0.10, nambala yamakasitomala yasinthidwa kotero kuti kudzipereka kumayambika nthawi ndi nthawi ndi laibulale yamakasitomala, monga idakonzedwa. auto.commit.interval.ms. Izi zili penapake pakati pa mitundu ya JMS AUTO_ACKNOWLEDGE ndi DUPS_OK_ACKNOWLEDGE. Mukamagwiritsa ntchito autocommit, mauthenga amatha kuperekedwa mosasamala kanthu kuti adakonzedwadi - izi zitha kuchitika ngati wogula pang'onopang'ono. Ngati wogula ataya, mauthenga amatengedwa ndi wogula wina, kuyambira pamalo odzipereka, zomwe zingayambitse uthenga wophonya. Pachifukwa ichi, Kafka sanataye mauthengawo, nambala yowerengera sinawagwiritse ntchito.

Njirayi ili ndi lonjezo lofanana ndi la 0.9: mauthenga akhoza kusinthidwa, koma ngati alephera, kuchotseratu sikungapangidwe, zomwe zingathe kuchititsa kuti kutumiza kuwirikize kawiri. Mauthenga ochulukirachulukira omwe mumatenga mukamaliza voti (), m'pamenenso vuto ili.

Monga tafotokozera mu "Kuwerenga Mauthenga Kuchokera Pamzere" patsamba 21, palibe chinthu chonga kutumiza uthenga kamodzi kokha mumayendedwe a mauthenga pamene njira zolephereka zimaganiziridwa.

Ku Kafka, pali njira ziwiri zochitira (kuchita) zochotsera (zochotsa): zokha komanso pamanja. Muzochitika zonsezi, mauthenga amatha kusinthidwa kangapo ngati uthengawo udasinthidwa koma walephera musanapereke. Mukhozanso kusankha kusakonza uthengawo ngati ntchitoyo idachitika kumbuyo ndipo nambala yanu idamalizidwa isanathe kukonzedwa (mwina ku Kafka 0.9 ndi kale).

Mutha kuwongolera njira yoyendetsera ntchito mu Kafka Consumer API pokhazikitsa parameter enable.auto.commit kunama ndikutchula imodzi mwa njira izi:

void commitSync();
void commitAsync();

Ngati mukufuna kukonza uthengawo "kamodzi", muyenera kuchitapo kanthu pamanja ndi commitSync ()pochita lamuloli mutangomaliza kukonza mauthengawo.

Njirazi sizimalola kuti mauthenga avomerezedwe asanasinthidwe, koma sachita chilichonse kuti athetse kuchedwa komwe kungachitike pomwe akupereka mawonekedwe ngati akugulitsidwa. Palibe zochitika ku Kafka. Wothandizira alibe kuthekera kochita izi:

Bwezerani uthenga wabodza basi. Makasitomala pawokha amayenera kuthana ndi zopatula zomwe zimadza chifukwa chazolipira zovuta komanso kuzimitsa kwapambuyo, chifukwa sangathe kudalira broker kuti atumizenso mauthenga.
Tumizani mauthenga kumitu ingapo pakugwiritsa ntchito atomiki imodzi. Monga tiwona posachedwa, kuwongolera mitu ndi magawo osiyanasiyana kumatha kukhala pamakina osiyanasiyana mumagulu a Kafka omwe samagwirizanitsa zochitika zikatumizidwa. Panthawi yolemba izi, ntchito ina yachitika kuti izi zitheke ndi KIP-98.
Gwirizanitsani kuwerenga uthenga umodzi kuchokera pamutu wina ndikutumiza uthenga wina kumutu wina. Apanso, zomangamanga za Kafka zimadalira makina ambiri odziimira omwe akuyenda ngati basi imodzi ndipo palibe kuyesa kubisa izi. Mwachitsanzo, palibe zigawo za API zomwe zingakulolezeni kulumikiza wogula и Wopanga mu malonda. Mu JMS, izi zimaperekedwa ndi chinthucho Gawokuchokera komwe kulengedwa MessageProducers и MessageConsumers.

Ngati sitingadalire zochita, tingapereke bwanji semantics pafupi ndi zomwe zimaperekedwa ndi mauthenga achikhalidwe?

Ngati pali zotheka kuti kuchotsera kwa ogula kukhoza kuwonjezeka uthenga usanathe kusinthidwa, monga panthawi ya kuwonongeka kwa ogula, ndiye kuti wogula alibe njira yodziwira ngati gulu lake la ogula linaphonya uthengawo pamene apatsidwa magawo. Chifukwa chake njira imodzi ndikubweza zomwe zidalipo kale. Kafka Consumer API imapereka njira izi:

void seek(TopicPartition partition, long offset);
void seekToBeginning(Collection < TopicPartition > partitions);

Njira kufuna () angagwiritsidwe ntchito ndi njira
offsetsForTimes (Mapu timestampsToSearch) kubwereranso kudera linalake lakale.

Kunena zoona, kugwiritsa ntchito njirayi kumatanthauza kuti ndizotheka kuti mauthenga ena omwe adasinthidwa kale awerengedwe ndikusinthidwanso. Kuti tipewe izi, titha kugwiritsa ntchito kuwerenga mopanda nzeru, monga momwe tafotokozera m'Mutu 4, kuti tizisunga mauthenga omwe adawonedwa kale ndikuchotsa zobwereza.

Kapenanso, nambala yanu ya ogula ikhoza kukhala yosavuta, bola ngati kutaya uthenga kapena kubwereza ndikovomerezeka. Tikamaganizira za kagwiritsidwe ntchito kamene Kafka amagwiritsidwa ntchito kwambiri, monga kusamalira zochitika za chipika, ma metrics, kutsatira kudina, ndi zina zambiri, timamvetsetsa kuti kutayika kwa mauthenga amodzi sikungakhudze kwambiri mapulogalamu ozungulira. Zikatero, zikhalidwe zosasinthika ndizovomerezeka. Kumbali ina, ngati pulogalamu yanu ikufunika kutumiza ndalama, muyenera kusamalira uthenga uliwonse. Zonse zimatengera nkhani.

Zowona zaumwini zikuwonetsa kuti kukula kwa mauthenga kumawonjezeka, mtengo wa uthenga uliwonse umachepa. Mauthenga akuluakulu amakhala amtengo wapatali akamawonedwa mwaphatikizidwe.

Kupezeka Kwambiri

Njira ya Kafka yopezeka kwambiri ndiyosiyana kwambiri ndi njira ya ActiveMQ. Kafka idapangidwa mozungulira magulu akulu momwe ma broker onse amalandila ndikugawa mauthenga nthawi imodzi.

Gulu la Kafka limakhala ndi ma broker angapo omwe akuyenda pamaseva osiyanasiyana. Kafka idapangidwa kuti iziyenda pazida zodziyimira zokha, pomwe node iliyonse ili ndi malo ake osungira. Kugwiritsa ntchito network attached storage (SAN) sikuvomerezeka chifukwa ma compute node angapo amatha kupikisana nthawi.Ыe nthawi zosungira ndikupanga mikangano.

Kafka pa nthawi zonse dongosolo. Ogwiritsa ntchito ambiri a Kafka samatseka masango awo ndipo pulogalamuyo imasinthidwa nthawi zonse ndikuyambiranso motsatizana. Izi zimatheka potsimikizira kuti zimagwirizana ndi mtundu wakale wa mauthenga ndi mayanjano pakati pa ogulitsa.

Ma broker olumikizidwa ku gulu la seva Zojambula Zoo, yomwe imagwira ntchito ngati registry yosinthira deta ndipo imagwiritsidwa ntchito kugwirizanitsa maudindo a broker aliyense. ZooKeeper palokha ndi njira yogawidwa yomwe imapereka kupezeka kwakukulu kudzera mukubwereza kwa chidziwitso pakukhazikitsa chiwerengero.

Pachiyambi, mutu umapangidwa mu gulu la Kafka lomwe lili ndi zotsatirazi:

Chiwerengero cha magawo. Monga tafotokozera kale, phindu lenileni lomwe lagwiritsidwa ntchito pano limadalira mlingo womwe ukufunidwa wa kuwerengera kofanana.
Replication factor (factor) imatsimikizira kuchuluka kwa ma broker mu cluster ayenera kukhala ndi zipika za magawowa.

Pogwiritsa ntchito ZooKeepers kuti agwirizane, Kafka amayesa kugawa magawo atsopano pakati pa ogulitsa mgululi. Izi zimachitika ndi chitsanzo chimodzi chomwe chimagwira ntchito ngati Controller.

Pa nthawi yake pagawo lililonse la mutu Wolamulira perekani maudindo kwa broker mtsogoleri (mtsogoleri, master, presenter) ndi otsatira (otsatira, akapolo, omvera). Wogulitsa, yemwe ali mtsogoleri wa gawoli, ali ndi udindo wolandira mauthenga onse omwe amatumizidwa kwa iwo ndi opanga ndikugawa mauthenga kwa ogula. Mauthenga akatumizidwa kugawo la mutu, amatsatiridwa ku ma broker onse omwe amakhala ngati otsatira gawolo. Node iliyonse yomwe ili ndi zipika za magawo amatchedwa chofanizira. Wogulitsa akhoza kukhala mtsogoleri wa magawo ena komanso ngati otsatira ena.

Wotsatira yemwe ali ndi mauthenga onse omwe ali ndi mtsogoleri amatchedwa synchronized replica (chifaniziro chomwe chili mumkhalidwe wolunzanitsidwa, mu-sync replica). Ngati broker yemwe ali ngati mtsogoleri wagawo atsika, broker aliyense yemwe ali ndi nthawi kapena wolumikizidwa pagawolo akhoza kutenga udindo wotsogolera. Ndi amazipanga zisathe kamangidwe.

Gawo la kasinthidwe ka wopanga ndi parameter acks, zomwe zimatsimikizira kuti ndi ma replicas angati omwe akuyenera kuvomereza (kuvomereza) kulandira uthenga ulusi wofunsira usanapitirire kutumiza: 0, 1, kapena zonse. Ngati akhazikitsidwa onse, ndiye pamene uthenga walandiridwa, mtsogoleriyo adzatumiza chitsimikiziro kwa wopangayo atangolandira zitsimikizo (zovomerezeka) za mbiriyo kuchokera kuzinthu zingapo (kuphatikizapo yokha) zomwe zimatanthauzidwa ndi mutu wa mutu. min.insync.replicas (chofikira 1). Ngati uthengawo sungathe kufotokozedwanso bwino, ndiye kuti wopanga ataya pulogalamuyo (NotEnoughReplicas kapena NotEnoughReplicasAfterAppend).

Kusintha kwanthawi zonse kumapanga mutu wokhala ndi chobwerezabwereza cha 3 (mtsogoleri 1, otsatira 2 pagawo lililonse) ndi parameter min.insync.replicas yakhazikitsidwa ku 2. Pachifukwa ichi, gululo lidzalola mmodzi wa ma broker omwe akuyang'anira magawo a mutu kuti apite pansi popanda kukhudza mapulogalamu a kasitomala.

Izi zimatibweretsanso ku malonda omwe amadziwika kale pakati pa ntchito ndi kudalirika. Kubwereza kumachitika ndikuwononga nthawi yodikirira kuti zitsimikizidwe (kuvomereza) kuchokera kwa otsatira. Ngakhale, chifukwa zimayendera limodzi, kubwereza kwa ma node osachepera atatu kumakhala ndi ntchito yofanana ndi ziwiri (kunyalanyaza kuwonjezeka kwa kugwiritsidwa ntchito kwa bandwidth).

Pogwiritsa ntchito ndondomeko yobwereza iyi, Kafka amapewa mwanzeru kufunika kolemba uthenga uliwonse ku disk ndi ntchitoyo. kulunzanitsa (). Uthenga uliwonse wotumizidwa ndi wopanga udzalembedwa ku chipika chogawa, koma monga momwe tafotokozera mu Chaputala 2, kulembera fayilo kumayamba kuchitidwa mu buffer ya opareshoni. Ngati uthengawu ubwerezedwanso ku chitsanzo china cha Kafka ndipo uli m'chikumbukiro chake, kutayika kwa mtsogoleri sizikutanthauza kuti uthengawo unatayika - ukhoza kutengedwa ndi chofanana chofanana.
Kukana kuchita opareshoni kulunzanitsa () zikutanthauza kuti Kafka akhoza kulandira mauthenga mofulumira monga momwe angalembere pamtima. Kumbali ina, mukatalikirako mungapewe kusuntha kukumbukira ku diski, ndibwino. Pachifukwa ichi, si zachilendo kuti Kafka broker agawidwe 64 GB kapena kukumbukira zambiri. Kugwiritsa ntchito kukumbukira kumeneku kumatanthauza kuti chitsanzo chimodzi cha Kafka chimatha kuthamanga mosavuta maulendo masauzande ambiri kuposa momwe amachitira mauthenga achikhalidwe.

Kafka imathanso kukhazikitsidwa kuti igwiritse ntchito ntchitoyi kulunzanitsa () ku mauthenga phukusi. Popeza zonse ku Kafka ndizokhazikika pamaphukusi, zimagwira ntchito bwino pamagwiritsidwe ambiri ndipo ndi chida chothandiza kwa ogwiritsa ntchito omwe amafunikira zitsimikizo zamphamvu kwambiri. Zambiri mwazochita zoyera za Kafka zimachokera ku mauthenga omwe amatumizidwa kwa broker ngati mapaketi komanso kuti mauthengawa amawerengedwa kuchokera kwa broker muzitsulo zotsatizana pogwiritsa ntchito. zero kopi ntchito (ntchito zomwe ntchito yokopera deta kuchokera kumalo a kukumbukira kupita ku ina sikuchitika). Chotsatirachi ndi ntchito yaikulu komanso kupindula kwazinthu ndipo ndizotheka pokhapokha pogwiritsa ntchito ndondomeko ya deta yomwe imatanthauzira ndondomeko yogawa.

Kuchita bwinoko kumatheka pagulu la Kafka kuposa ndi broker m'modzi wa Kafka, chifukwa magawo a mitu amatha kufalikira pamakina ambiri osiyanasiyana.

Zotsatira

M'mutu uno, tidawona momwe Kafka architecture amaganiziranso ubale wamakasitomala ndi ma broker kuti apereke payipi yolimba kwambiri yotumizirana mauthenga, yomwe imadutsa nthawi zambiri kuposa ya broker wamba. Takambirana momwe zimagwiritsidwira ntchito kuti tikwaniritse izi ndipo tayang'ana mwachidule mamangidwe a mapulogalamu omwe amapereka ntchitoyi. M'mutu wotsatira, tiwona zovuta zomwe zimagwiritsidwa ntchito potumizirana mameseji zomwe zimafunikira kuthetsa ndikukambirana njira zothanirana nazo. Timaliza mutuwu pofotokoza momwe mungalankhulire za matekinoloje otumizirana mameseji ambiri kuti muwunikire kuyenerera kwawo pakugwiritsa ntchito.

Gawo lomasuliridwa m'mbuyomu: Kumvetsetsa ma broker. Kuphunzira zimango zotumizirana mauthenga ndi ActiveMQ ndi Kafka. Mutu 1

Kumasulira kwatheka: tele.gg/middle_java

Zipitilizidwa…

Ogwiritsa ntchito olembetsedwa okha ndi omwe angatenge nawo gawo pa kafukufukuyu. Lowani muakauntichonde.

Kodi Kafka amagwiritsidwa ntchito m'bungwe lanu?

kuti
No
Kale ntchito, tsopano ayi
Tikukonzekera kugwiritsa ntchito

Ogwiritsa 38 adavota. Ogwiritsa ntchito 8 adakana.

Source: www.habr.com

Kumvetsetsa ma broker. Kuphunzira zimango zotumizirana mauthenga ndi ActiveMQ ndi Kafka. Mutu 3. Kafka