Perekani bandwidth yochulukirapo mukamagwira ntchito ndi mauthenga
Thandizani Ofalitsa-Olembetsa ndi Zitsanzo za Point-to-Point
Osachedwetsa powonjezera ogula. Mwachitsanzo, magwiridwe antchito a pamzere ndi mutu wa ActiveMQ amatsika pomwe kuchuluka kwa ogula komwe akupita kukukulira.
Khalani horizontally scalable; ngati broker m'modzi yemwe amalimbikira mauthenga atha kutero pa liwiro lalikulu la disk, ndiye kuti ndizomveka kupitilira gawo limodzi la broker kuti muwonjezere magwiridwe antchito.
Chepetsani mwayi wosunga ndi kubwezeretsanso mauthenga
Kuti akwaniritse zonsezi, Kafka adatengera zomanga zomwe zidafotokozanso maudindo ndi maudindo amakasitomala ndi otumizirana mauthenga. Mtundu wa JMS ndi wokonda kwambiri broker, komwe broker ali ndi udindo wogawa mauthenga ndipo makasitomala amangodandaula za kutumiza ndi kulandira mauthenga. Komano, Kafka ndi wokhazikika pamakasitomala, ndipo kasitomala amatenga zinthu zambiri za broker wamba, monga kugawa bwino mauthenga ofunikira kwa ogula, posinthanitsa ndi broker wothamanga kwambiri komanso wowopsa. Kwa anthu omwe agwira ntchito ndi mauthenga achikhalidwe, kugwira ntchito ndi Kafka kumafuna kusintha kwakukulu kwa malingaliro.
Mayendedwe a uinjiniyawa apangitsa kuti pakhale njira yolumikizirana mauthenga yomwe ingathe kuchulukirachulukira ndi maoda ambiri poyerekeza ndi broker wamba. Monga momwe tidzaonera, njirayi imabwera ndi malonda, zomwe zikutanthauza kuti Kafka si yoyenera kwa mitundu ina ya ntchito ndi mapulogalamu oikidwa.
Mtundu Wogwirizana wa Kopita
Kuti akwaniritse zofunikira zomwe tafotokozazi, Kafka yaphatikiza kusindikiza-kulembetsa ndi kutumizirana mameseji kumalo amodzi - mutu. Izi ndi zosokoneza kwa anthu omwe agwira ntchito ndi makina otumizira mauthenga, pomwe mawu oti "mutu" amatanthauza njira yowulutsira yomwe (kuchokera pamutuwo) kuwerenga sikungatheke. Mitu ya Kafka iyenera kuganiziridwa ngati mtundu wa kopita wosakanizidwa, monga tafotokozera kumayambiriro kwa bukuli.
Kwa gawo lotsala la mutu uno, pokhapokha titanena momveka bwino, mawu oti "mutu" atanthauza mutu wa Kafka.
Kuti timvetse bwino momwe mitu ikuchitira komanso zomwe amapereka, tiyenera kuyang'ana kaye momwe ikugwiritsidwira ntchito ku Kafka. Mutu uliwonse ku Kafka uli ndi chipika chake.
Opanga omwe amatumiza mauthenga ku Kafka amalembera chipikachi, ndipo ogula amawerenga kuchokera pachipikacho pogwiritsa ntchito zolozera zomwe zikupita patsogolo nthawi zonse. Nthawi ndi nthawi, Kafka amachotsa mbali zakale kwambiri za chipikacho, kaya mauthenga omwe ali m'zigawozo awerengedwa kapena ayi. Mbali yapakati pa mapangidwe a Kafka ndikuti broker samasamala ngati mauthenga awerengedwa kapena ayi - ndilo udindo wa kasitomala.
Makasitomala amene akufuna kuwerenga mauthenga amayang'anira cholozera chotchedwa gulu la ogula, zomwe zimaloza kuchepetsa mauthenga mu gawo. Kuchotsera ndi malo owonjezera omwe amayamba pa 0 kumayambiriro kwa magawo. Gulu la ogula ili, lomwe limatchulidwa mu API kudzera pa gulu lofotokozedwa ndi wosuta_id, likufanana ndi wogula m'modzi womveka kapena dongosolo.
Makina ambiri otumizirana mameseji amawerenga zomwe akupita pogwiritsa ntchito maulendo angapo ndi ulusi kuti akonze mauthenga mofanana. Chifukwa chake, nthawi zambiri pamakhala ogula ambiri omwe amagawana gulu limodzi la ogula.
Vuto la kuwerenga likhoza kuimiridwa motere:
Mutu uli ndi magawo angapo
Magulu angapo a ogula amatha kugwiritsa ntchito mutu nthawi imodzi
Gulu la ogula likhoza kukhala ndi zochitika zingapo zosiyana
Ili ndi vuto laling'ono mpaka lambiri. Kuti timvetsetse momwe Kafka amachitira maubwenzi pakati pa magulu ogula, zochitika za ogula, ndi magawo, tiyeni tiwone mndandanda wa zochitika zovuta kwambiri zowerengera.
Chithunzi 3-5. Wogula m'modzi amawerenga kuchokera ku magawo angapo
Muzochitika izi, wogula amapatsidwa ulamuliro pa zolozera zogwirizana ndi gulu_id yake m'magawo onse awiri ndikuyamba kuwerenga mauthenga kuchokera ku magawo onse awiri.
Pamene wogula wowonjezera wa gulu lomwelo_id awonjezeredwa pamutuwu, Kafka imagawiranso gawo limodzi kuchokera koyamba mpaka wogula wachiwiri. Pambuyo pake, chitsanzo chilichonse cha wogula chidzawerenga kuchokera ku gawo limodzi la mutuwo (Chithunzi 3-6).
Kuonetsetsa kuti mauthenga asinthidwa mofanana mu ulusi 20, muyenera magawo 20 osachepera. Ngati pali magawo ochepa, mudzasiyidwa ndi ogula omwe alibe chochita, monga tafotokozera kale pokambirana za ogula okha.
Chithunzi 3-6. Ogula awiri omwe ali mugulu limodzi la ogula amawerenga kuchokera ku magawo osiyanasiyana
Dongosololi limachepetsa kwambiri zovuta za broker wa Kafka poyerekeza ndi kugawa uthenga wofunikira kuti asunge mzere wa JMS. Apa simuyenera kuda nkhawa ndi mfundo zotsatirazi:
Ndi wogula uti yemwe akuyenera kulandira uthenga wotsatira, kutengera kugawika kwa robin, kuchuluka kwa ma buffers, kapena mauthenga am'mbuyomu (monga magulu a mauthenga a JMS).
Ndi mauthenga ati omwe amatumizidwa kwa ogula komanso ngati akuyenera kutumizidwanso ngati atalephera.
Zomwe broker wa Kafka ayenera kuchita ndikutumiza mauthenga motsatizana kwa ogula akawapempha.
Komabe, zofunikira zofananira ndi kuwerengera ndi kutumizanso mauthenga olephera sizichoka - udindo wawo umangochoka kwa broker kupita kwa kasitomala. Izi zikutanthauza kuti ziyenera kuganiziridwa mu code yanu.
Pomwe mu JMS timagwiritsa ntchito mawonekedwe a uthenga okhala ndi metadata (mitu ndi katundu) ndi thupi lomwe lili ndi zolipira (zolipira), ku Kafka uthengawo ndi awiri "key-value". Malipiro a uthenga amatumizidwa ngati mtengo. Mfungulo, kumbali ina, imagwiritsidwa ntchito kwambiri pogawa ndipo iyenera kukhala Business logic enieni kiyikuyika mauthenga okhudzana ndi magawo omwewo.
Mu Chaputala 2, tidakambirana za kubetcha pa intaneti komwe kumayenera kukonzedwa ndi wogula m'modzi:
Akaunti ya ogwiritsa ntchito idakonzedwa.
Ndalama zimayikidwa ku akaunti.
Kubetcha kumapangidwa komwe kumachotsa ndalama ku akaunti.
Ngati chochitika chilichonse chili uthenga wotumizidwa kumutu, ndiye kuti kiyi yachilengedwe ingakhale ID ya akaunti.
Uthenga ukatumizidwa pogwiritsa ntchito Kafka Producer API, umaperekedwa ku gawo la magawo omwe, atapatsidwa uthenga ndi momwe gulu la Kafka lilili panopa, amabwezeretsanso ID ya gawo lomwe uthengawo uyenera kutumizidwa. Izi zimachitika mu Java kudzera mu mawonekedwe a Partitioner.
Tiyeni tiwone chitsanzo chomwe mukufuna kutumiza metadata limodzi ndi kuchuluka kwa uthenga. Kulipira muchitsanzo chathu ndi malangizo oti mupange ndalama ku akaunti yamasewera. Langizo ndi chinthu chomwe tikufuna kuti chitsimikizidwe kuti sichisinthidwa pakutumiza ndipo tikufuna kutsimikiza kuti ndi njira yodalirika yokhayo yomwe ingayambitse malangizowo. Pankhaniyi, machitidwe otumizira ndi kulandira amavomereza kugwiritsa ntchito siginecha kutsimikizira uthengawo.
Mu JMS wamba, timangofotokozera za "siginecha ya uthenga" ndikuwonjezera ku uthengawo. Komabe, Kafka simatipatsa njira yodutsira metadata, makiyi okha ndi mtengo.
Popeza mtengo wake ndi ndalama zotumizira banki zomwe tikufuna kusunga umphumphu, tilibe chochita koma kufotokozera dongosolo la data lomwe lingagwiritsidwe ntchito pakiyi. Pongoganiza kuti tikufuna ID ya akaunti yogawa, popeza mauthenga onse okhudzana ndi akaunti ayenera kusinthidwa mwadongosolo, tidzabwera ndi mawonekedwe awa a JSON:
Chifukwa kufunikira kwa siginecha kumasiyana malinga ndi kuchuluka kwa malipiro, njira yokhazikika ya hashing ya mawonekedwe a Partitioner sichingadalire mauthenga okhudzana ndi gulu. Chifukwa chake, tifunika kulemba njira yathu yomwe ingafotokozere funguloli ndikugawa mtengo wa accountId.
Kafka imaphatikizapo ma checksums kuti azindikire ziphuphu za mauthenga mu sitolo ndipo ali ndi zida zonse zachitetezo. Ngakhale zili choncho, nthawi zina zimawoneka zofunikira pamakampani, monga zomwe zili pamwambapa.
Njira yogawanitsa ya wogwiritsayo iyenera kuwonetsetsa kuti mauthenga onse okhudzana ndi gawo limodzi. Ngakhale izi zikuwoneka zophweka, chofunikiracho chikhoza kukhala chovuta chifukwa cha kufunikira koyitanitsa mauthenga okhudzana ndi momwe chiwerengero cha magawo pamutu chiliri.
Chiwerengero cha magawo pamutuwu chimatha kusintha pakapita nthawi, chifukwa amatha kuwonjezeredwa ngati kuchuluka kwa magalimoto kupitilira zomwe tikuyembekezera. Chifukwa chake, makiyi a mauthenga amatha kulumikizidwa ndi magawo omwe adatumizidwako koyambirira, kutanthauza gawo lachigawo lomwe liyenera kugawidwa pakati pa opanga.
Mfundo ina yofunika kuiganizira ndi kugaŵidwa kofanana kwa mauthenga m’magawo onse. Nthawi zambiri, makiyi samagawidwa mofanana pa mauthenga onse, ndipo ntchito za hashi sizimatsimikizira kugawidwa koyenera kwa mauthenga a makiyi ang'onoang'ono.
Ndikofunika kuzindikira kuti ngakhale mutasankha kugawa mauthenga, cholekanitsacho chiyenera kugwiritsidwanso ntchito.
Ganizirani kufunikira kobwereza deta pakati pa magulu a Kafka m'malo osiyanasiyana. Pachifukwa ichi, Kafka amabwera ndi chida cha mzere cholamula chotchedwa MirrorMaker, chomwe chimagwiritsidwa ntchito powerenga mauthenga kuchokera kumagulu amodzi ndikuwatengera ku gulu lina.
MirrorMaker iyenera kumvetsetsa makiyi a mutu womwe wabwerezedwa kuti asunge dongosolo pakati pa mauthenga akamafanana pakati pa magulu, popeza kuchuluka kwa magawo a mutuwo sikungakhale kofanana m'magulu awiri.
Kuchulukirachulukira ndi mapindu a Kafka amabwera chifukwa chosintha maudindo ena a broker kupita kwa kasitomala. Pachifukwa ichi, chigamulo chapangidwa kuti chigawidwe mauthenga omwe angakhale okhudzana ndi ogula angapo omwe amagwira ntchito mofanana.
Otsatsa a JMS amafunikanso kuthana ndi izi. Chochititsa chidwi n'chakuti, njira yotumizira mauthenga okhudzana ndi wogula yemweyo, yomwe ikugwiritsidwa ntchito kudzera m'magulu a mauthenga a JMS (kusiyana kwa njira ya Sticky load balancing (SLB)), imafunanso kuti wotumizayo alembe mauthenga ngati okhudzana. Pankhani ya JMS, broker ali ndi udindo wotumiza gulu ili la mauthenga okhudzana ndi ogula m'modzi mwa ambiri, ndikusamutsa umwini wa gulu ngati wogula akugwa.
Mapangano Opanga
Kugawa si chinthu chokhacho choyenera kuganizira potumiza mauthenga. Tiyeni tiwone njira zotumizira () za gulu la Producer mu Java API:
Tiyenera kuzindikira kuti njira zonse ziwirizi zimabwerera ku Future, zomwe zimasonyeza kuti ntchito yotumizayo sichitika nthawi yomweyo. Chotsatira chake ndikuti uthenga (ProducerRecord) umalembedwa ku buffer yotumizira gawo lililonse logwira ntchito ndikutumizidwa kwa broker ngati ulusi wakumbuyo mu library yamakasitomala a Kafka. Ngakhale izi zimapangitsa zinthu kukhala zofulumira kwambiri, zikutanthauza kuti pulogalamu yosadziwa imatha kutaya mauthenga ngati njira yake yayimitsidwa.
Monga nthawi zonse, pali njira yopangira ntchito yotumiza yodalirika pamtengo wa ntchito. Kukula kwa buffer iyi kutha kukhazikitsidwa ku 0, ndipo ulusi wotumizira ukakamizidwa kudikirira mpaka kutumiza uthenga kwa broker kukamalizidwa, motere:
Kuwerenga mauthenga kuli ndi zovuta zina zomwe ziyenera kuganiziridwa. Mosiyana ndi JMS API, yomwe imatha kuyendetsa omvera uthenga poyankha uthenga, the ogula Kafka mavoti okha. Tiyeni tione mwatsatanetsatane njira voti ()amagwiritsidwa ntchito pazifukwa izi:
ConsumerRecords < K, V > poll(long timeout);
Mtengo wobwezera wa njirayo ndi kapangidwe ka chidebe chokhala ndi zinthu zingapo mbiri ya ogula kuchokera ku magawo angapo. mbiri ya ogula palokha ndi chinthu chogwirizira pamtengo wamtengo wapatali wokhala ndi metadata yogwirizana, monga kugawa komwe kwachokera.
Monga tanenera, gulu la ogula limagwirizana ndi zomwe zili mu chipika. Malo a chipika okhudzana ndi kuchotsera uku akufanana ndi uthenga wotsatira womwe uyenera kuperekedwa poyankha voti (). Nthawi yomwe kuchotsera uku kumawonjezeka ndiyofunikira pakuwerenga.
Kubwerera ku mtundu wowerengera womwe takambirana kale, kukonza uthenga kumakhala ndi magawo atatu:
Pezani uthenga woti muwerenge.
Yambitsani uthenga.
Tsimikizirani uthenga.
Wogula wa Kafka amabwera ndi njira yosinthira enable.auto.commit. Izi ndizomwe zimagwiritsidwa ntchito pafupipafupi, monga momwe zimakhalira ndi zokonda zomwe zimakhala ndi mawu oti "auto".
Kafka 0.10 isanachitike, kasitomala wogwiritsa ntchito njirayi amatha kutumiza uthenga womaliza womwe uwerengedwa pa foni yotsatira. voti () pambuyo processing. Izi zikutanthauza kuti mauthenga aliwonse omwe adalandidwa kale akhoza kukonzedwanso ngati kasitomala anali atawakonza kale koma adawonongeka mosayembekezereka asanayimbire. voti (). Chifukwa broker sasunga boma lililonse la kuchuluka kwa uthenga womwe wawerengedwa, wogula wotsatira yemwe amapeza uthengawo sadziwa chilichonse chomwe chidachitika. Khalidweli linali lachinyengo. Chotsitsacho chinangoperekedwa ngati uthengawo udakonzedwa bwino, koma ngati kasitomala achotsa, broker amatumizanso uthenga womwewo kwa kasitomala wina. Khalidweli linali logwirizana ndi chitsimikizo chotumiza uthenga "kamodzi".
Pa nthawi yake pagawo lililonse la mutuWolamulira perekani maudindo kwa broker mtsogoleri (mtsogoleri, master, presenter) ndi otsatira (otsatira, akapolo, omvera). Wogulitsa, yemwe ali mtsogoleri wa gawoli, ali ndi udindo wolandira mauthenga onse omwe amatumizidwa kwa iwo ndi opanga ndikugawa mauthenga kwa ogula. Mauthenga akatumizidwa kugawo la mutu, amatsatiridwa ku ma broker onse omwe amakhala ngati otsatira gawolo. Node iliyonse yomwe ili ndi zipika za magawo amatchedwa chofanizira. Wogulitsa akhoza kukhala mtsogoleri wa magawo ena komanso ngati otsatira ena.
Wotsatira yemwe ali ndi mauthenga onse omwe ali ndi mtsogoleri amatchedwa synchronized replica (chifaniziro chomwe chili mumkhalidwe wolunzanitsidwa, mu-sync replica). Ngati broker yemwe ali ngati mtsogoleri wagawo atsika, broker aliyense yemwe ali ndi nthawi kapena wolumikizidwa pagawolo akhoza kutenga udindo wotsogolera. Ndi amazipanga zisathe kamangidwe.
Gawo la kasinthidwe ka wopanga ndi parameter acks, zomwe zimatsimikizira kuti ndi ma replicas angati omwe akuyenera kuvomereza (kuvomereza) kulandira uthenga ulusi wofunsira usanapitirire kutumiza: 0, 1, kapena zonse. Ngati akhazikitsidwa onse, ndiye pamene uthenga walandiridwa, mtsogoleriyo adzatumiza chitsimikiziro kwa wopangayo atangolandira zitsimikizo (zovomerezeka) za mbiriyo kuchokera kuzinthu zingapo (kuphatikizapo yokha) zomwe zimatanthauzidwa ndi mutu wa mutu. min.insync.replicas (chofikira 1). Ngati uthengawo sungathe kufotokozedwanso bwino, ndiye kuti wopanga ataya pulogalamuyo (NotEnoughReplicas kapena NotEnoughReplicasAfterAppend).
Kusintha kwanthawi zonse kumapanga mutu wokhala ndi chobwerezabwereza cha 3 (mtsogoleri 1, otsatira 2 pagawo lililonse) ndi parameter min.insync.replicas yakhazikitsidwa ku 2. Pachifukwa ichi, gululo lidzalola mmodzi wa ma broker omwe akuyang'anira magawo a mutu kuti apite pansi popanda kukhudza mapulogalamu a kasitomala.
Izi zimatibweretsanso ku malonda omwe amadziwika kale pakati pa ntchito ndi kudalirika. Kubwereza kumachitika ndikuwononga nthawi yodikirira kuti zitsimikizidwe (kuvomereza) kuchokera kwa otsatira. Ngakhale, chifukwa zimayendera limodzi, kubwereza kwa ma node osachepera atatu kumakhala ndi ntchito yofanana ndi ziwiri (kunyalanyaza kuwonjezeka kwa kugwiritsidwa ntchito kwa bandwidth).
Pogwiritsa ntchito ndondomeko yobwereza iyi, Kafka amapewa mwanzeru kufunika kolemba uthenga uliwonse ku disk ndi ntchitoyo. kulunzanitsa (). Uthenga uliwonse wotumizidwa ndi wopanga udzalembedwa ku chipika chogawa, koma monga momwe tafotokozera mu Chaputala 2, kulembera fayilo kumayamba kuchitidwa mu buffer ya opareshoni. Ngati uthengawu ubwerezedwanso ku chitsanzo china cha Kafka ndipo uli m'chikumbukiro chake, kutayika kwa mtsogoleri sizikutanthauza kuti uthengawo unatayika - ukhoza kutengedwa ndi chofanana chofanana.
Kukana kuchita opareshoni kulunzanitsa () zikutanthauza kuti Kafka akhoza kulandira mauthenga mofulumira monga momwe angalembere pamtima. Kumbali ina, mukatalikirako mungapewe kusuntha kukumbukira ku diski, ndibwino. Pachifukwa ichi, si zachilendo kuti Kafka broker agawidwe 64 GB kapena kukumbukira zambiri. Kugwiritsa ntchito kukumbukira kumeneku kumatanthauza kuti chitsanzo chimodzi cha Kafka chimatha kuthamanga mosavuta maulendo masauzande ambiri kuposa momwe amachitira mauthenga achikhalidwe.
Kafka imathanso kukhazikitsidwa kuti igwiritse ntchito ntchitoyi kulunzanitsa () ku mauthenga phukusi. Popeza zonse ku Kafka ndizokhazikika pamaphukusi, zimagwira ntchito bwino pamagwiritsidwe ambiri ndipo ndi chida chothandiza kwa ogwiritsa ntchito omwe amafunikira zitsimikizo zamphamvu kwambiri. Zambiri mwazochita zoyera za Kafka zimachokera ku mauthenga omwe amatumizidwa kwa broker ngati mapaketi komanso kuti mauthengawa amawerengedwa kuchokera kwa broker muzitsulo zotsatizana pogwiritsa ntchito. zero kopi ntchito (ntchito zomwe ntchito yokopera deta kuchokera kumalo a kukumbukira kupita ku ina sikuchitika). Chotsatirachi ndi ntchito yaikulu komanso kupindula kwazinthu ndipo ndizotheka pokhapokha pogwiritsa ntchito ndondomeko ya deta yomwe imatanthauzira ndondomeko yogawa.
Kuchita bwinoko kumatheka pagulu la Kafka kuposa ndi broker m'modzi wa Kafka, chifukwa magawo a mitu amatha kufalikira pamakina ambiri osiyanasiyana.