Ukuqhubekeka koguqulelo lwencwadi encinci:
Ukuqonda iiBroker zoMyalezo
umbhali: Jakub Korab, umshicileli: O'Reilly Media, Inc., umhla wokupapashwa: Juni 2017, ISBN: 9781492049296.
Inxalenye eguqulelwe ngaphambili:
ISAHLUKO 3
Kafka
I-Kafka yaphuhliswa yi-LinkedIn ukuze ijikeleze eminye imida yabathengisi bemiyalezo yemveli kwaye igweme ukuseta imiyalezo emininzi yemiyalezo eyahlukeneyo yokudibana, echazwe kule ncwadi phantsi ko "Scaling up and out" kwiphepha lama-28. .Ukusetyenziswa kwamatyala i-LinkedIn ixhomekeke kakhulu kwindlela enye yokungeniswa kwedatha eninzi kakhulu, njengokucofa iphepha kunye nokufikelela kwiilogi, ngelixa uvumela ukuba loo datha isetyenziswe kwiinkqubo ezininzi ngaphandle kokuchaphazela imveliso yabavelisi okanye abanye abathengi. Ngapha koko, isizathu sokuba iKafka ikhona kukufumana uhlobo loyilo lwemiyalezo oluchazwa yi-Universal Data Pipeline.
Ngokunikwa le njongo iphambili, ezinye iimfuno zavela ngokwendalo. Kafka kufuneka:
- Khawuleza kakhulu
- Nika i-bandwidth engaphezulu xa usebenza ngemiyalezo
- Inkxaso yoMshicileli-Umrhumo kunye neemodeli zeNqaku leNqaku
- Musa ukucotha ngokongeza abathengi. Umzekelo, ukusebenza kwemigca yomibini kunye nesihloko kwi-ActiveMQ kwehla njengoko inani labathengi kwindawo yokufikela likhula.
- Yiba nokulinganisa ngokuthe tye; ukuba umthengisi omnye oqhubeleka nemiyalezo angenza njalo ngesantya esiphezulu sedisk, ngoko iyavakala ukuya ngaphaya kohlobo lomthengisi omnye ukwandisa ukusebenza.
- Nciphisa ufikelelo ekugcineni kunye nokuphinda ufumane imiyalezo
Ukufezekisa konke oku, iKafka yamkele uyilo oluchaza ngokutsha iindima kunye noxanduva lwabathengi kunye nabathengisi bemiyalezo. Imodeli ye-JMS ijolise kakhulu kwi-broker, apho i-broker inoxanduva lokuhambisa imiyalezo kunye nabathengi kufuneka bakhathazeke kuphela ngokuthumela nokufumana imiyalezo. I-Kafka, kwelinye icala, i-client-centric, kunye nomxhasi othatha iimpawu ezininzi ze-broker yendabuko, njengokusasazwa ngokufanelekileyo kwemiyalezo efanelekileyo kubathengi, ngokutshintshiselana nomthengisi okhawuleza kakhulu kunye ne-scalable broker. Kubantu abasebenze ngeenkqubo zemiyalezo yemveli, ukusebenza neKafka kufuna utshintsho olusisiseko lwengqondo.
Olu lwathiso lobunjineli lukhokelele ekudalweni kwesiseko semiyalezo esikwaziyo ukunyusa umthamo wemveliso ngemiyalelo emininzi yobukhulu xa kuthelekiswa nomthengisi oqhelekileyo. Njengoko siza kubona, le ndlela iza kunye nokurhweba, oku kuthetha ukuba i-Kafka ayifanelekanga kwiintlobo ezithile zemisebenzi kunye nesofthiwe efakwe.
Imodeli Yendawo Edityanisiweyo
Ukuzalisekisa iimfuno ezichazwe ngasentla, i-Kafka idibanise ukupapasha-ukurhuma kunye ne-point-to-point yemiyalezo phantsi kolunye uhlobo lwendawo - isihloko. Oku kuyababhida abantu abasebenze ngeenkqubo zokuthumela imiyalezo, apho igama elithi "isihloko" libhekisa kwindlela yosasazo apho (ukusuka kwisihloko) ukufundwa kunganyamezelekiyo. Izihloko ze-Kafka kufuneka zithathwe njengohlobo lwendawo ekusingwa kuyo, njengoko kuchaziwe kwintshayelelo yale ncwadi.
Kwintsalela yesi sahluko, ngaphandle kokuba sichaza ngenye indlela ngokucacileyo, igama elithi "isihloko" liza kubhekisa kwisihloko seKafka.
Ukuqonda ngokupheleleyo indlela izihloko eziziphatha ngayo kunye neziphi iziqinisekiso ezibonelela ngazo, kufuneka siqale sijonge indlela eziphunyezwa ngayo eKafka.
Isihloko ngasinye eKafka sinelogi yaso.
Abavelisi abathumela imiyalezo kwi-Kafka babhala kule log, kwaye abathengi bafunda kwilogi besebenzisa izikhombisi ezihlala ziqhubela phambili. Ngamaxesha athile, iKafka isusa ezona ndawo zindala zelog, nokuba imiyalezo ekula malungu ifundiwe okanye hayi. Inxalenye esembindini yoyilo lweKafka kukuba umthengisi akakhathali ukuba imiyalezo ifundwe okanye hayi - luxanduva lomxhasi.
Amagama athi "log" kunye "ne-pointer" akabonakali ngaphakathi . La magama aziwayo asetyenziswa apha ukunceda ukuqonda.
Le modeli yahluke ngokupheleleyo kwi-ActiveMQ, apho imiyalezo evela kuyo yonke imigca igcinwa kwilogi enye, kwaye umthengisi uphawula imiyalezo njengecinyiweyo emva kokuba ifundiwe.
Ngoku makhe simbe nzulu ngakumbi kwaye sijonge kwilog yesihloko ngokweenkcukacha ngakumbi.
Ilog yeKafka iqulathe izahlulo ezininzi (). I-Kafka iqinisekisa uku-odola okungqongqo kwisahlulo ngasinye. Oku kuthetha ukuba imiyalezo ebhalelwe isahlulelo ngokolandelelwano oluthile iya kufundwa ngendlela efanayo. Isahlulo ngasinye siphunyezwa njengefayile yelog eqengqelekayo equlathe iseti engaphantsi (i-subset) yayo yonke imiyalezo ethunyelwe kwisihloko ngabavelisi bayo. Isihloko esenziweyo siqulathe, ngokungagqibekanga, isahlulelo esinye. Umbono wezahlulo ngumbono ophambili weKafka wokukala okuthe tye.
Umfanekiso 3-1. Izahlulo zeKafka
Xa umvelisi ethumela umyalezo kwisihloko se-Kafka, uthatha isigqibo sokuba yeyiphi isahlulelo sokuthumela umyalezo. Siza kujonga oku ngakumbi kamva.
Ukufunda imiyalezo
Umxhasi ofuna ukufunda imiyalezo ulawula isalathisi esibizwa ngokuba yi iqela labathengi, esalatha kuyo i-offset imiyalezo kwisahlulelo. I-offset yindawo eyongezelekayo eqala ku-0 ekuqaleni kwesahlulelo. Eli qela labathengi, elibhekiselwe kwi-API ngeqela elichaziweyo lomsebenzisi_id, lihambelana umthengi omnye okanye inkqubo.
Uninzi lweenkqubo zokuthumela imiyalezo zifunda idatha ukusuka kwindawo ekusingwa kuyo kusetyenziswa iimeko ezininzi kunye nemisonto ukucubungula imiyalezo ngokunxuseneyo. Ke, kuya kubakho iimeko ezininzi zabathengi ababelana ngeqela elinye labathengi.
Ingxaki yokufunda inokubonakaliswa ngolu hlobo lulandelayo:
- Isihloko sinezahlulo ezininzi
- Amaqela amaninzi abathengi angasebenzisa isihloko ngexesha elinye
- Iqela labathengi linokuba neemeko ezininzi ezahlukeneyo
Le yingxaki engenamsebenzi kaninzi ukuya kwabaninzi. Ukuqonda indlela i-Kafka ephatha ngayo ubudlelwane phakathi kwamaqela abathengi, iimeko zabathengi, kunye nezahlulo, makhe sijonge kuthotho lweemeko zokufunda ezintsonkothileyo.
Abathengi kunye namaqela abathengi
Masithathe njengesiqalo isihloko ngesahlulelo esinye ().
Umfanekiso 3-2. Umthengi ufunda ukusuka kwisahlulelo
Xa umzekelo womthengi eqhagamshela neyakhe group_id kwesi sihloko, yabelwa isahlulelo sokufunda kunye ne-offset kweso sahlulelo. Indawo yolu hlaselo iqwalaselwe kumxhasi njengesalathisi kweyona ndawo yakutsha nje (umyalezo omtsha) okanye eyona ndawo yokuqala (umyalezo omdala). Umthengi ucela (i-polls) imiyalezo evela kwisihloko, ebangela ukuba ifundwe ngokulandelelana kwilogi.
Isikhundla se-offset sinikezelwa rhoqo kwi-Kafka kwaye igcinwe njengemiyalezo kwisihloko sangaphakathi _umthengi_ukunciphisa. Ukufunda imiyalezo ayikacinywa, ngokungafaniyo nomthengisi oqhelekileyo, kwaye umxhasi unokubuyisela umva i-offset ukuze aphinde aqhubekisele phambili imiyalezo esele ijongiwe.
Xa umsebenzisi wesibini onengqiqo eqhagamshela usebenzisa iqela_id elahlukileyo, lilawula isalathisi sesibini esizimeleyo kwesokuqala (). Ngaloo ndlela, isihloko se-Kafka sisebenza njengomgca apho kukho umthengi omnye kwaye njengesiqhelo sokushicilela-ukubhalisa (i-pub-sub) isihloko apho abathengi abaninzi babhalisela, kunye nenzuzo eyongeziweyo yokuba yonke imilayezo igcinwe kwaye inokucutshungulwa ngamaxesha amaninzi.
Umfanekiso 3-3. Abathengi ababini kumaqela ahlukeneyo abathengi bafunda kwisahlulo esifanayo
Abathengi kwiqela labathengi
Xa umzekelo omnye wabathengi ufunda idatha ukusuka kwisahlulelo, inolawulo olupheleleyo lwesalathisi kwaye iqhube imiyalezo njengoko ichazwe kwicandelo langaphambili.
Ukuba iimeko ezininzi zabathengi bezidityaniswe neqela elifanayo_id kwisihloko ngesahlulelo esinye, ngoko umzekelo oqhagamsheleneyo wokugqibela uya kunikwa ulawulo phezu kwesalathisi kwaye ukusuka kuloo mzuzu ukuya phambili iya kufumana yonke imiyalezo ().
Umfanekiso 3-4. Abathengi ababini kwiqela elifanayo labathengi bafunde kwisahlulo esifanayo
Le ndlela yokucubungula, apho inani leemeko zabathengi lidlula inani lezahlulo, lingacingelwa njengohlobo lomthengi okhethekileyo. Oku kunokuba luncedo ukuba ufuna "i-active-passive" (okanye "ishushu-eshushu") idibanise iimeko zabathengi, nangona usebenzisa abathengi abaninzi ngokuhambelanayo ("iyasebenza-esebenzayo" okanye "ishushu-shushu") iqhelekile kakhulu kunoko. abasebenzisi abakwimo yokulinda.
Oku kuziphatha kokuhanjiswa komyalezo kuchazwe ngasentla kunokumangalisa xa kuthelekiswa nendlela umgca oqhelekileyo we-JMS oziphatha ngayo. Kulo mzekelo, imiyalezo ethunyelwe kumgca iya kusasazwa ngokulinganayo phakathi kwabathengi ababini.
Ngokuqhelekileyo, xa sidala iimeko ezininzi zabathengi, senza oku mhlawumbi ukucubungula imiyalezo ngokuhambelana, okanye ukwandisa isantya sokufunda, okanye ukwandisa ukuzinza kwenkqubo yokufunda. Ekubeni umzekelo omnye womthengi unokufunda idatha ukusuka kwisahlulelo ngexesha, le nto iphunyezwa njani eKafka?
Enye indlela yokwenza oku kukusebenzisa umzekelo omnye womthengi ukufunda yonke imiyalezo kwaye uyidlulisele kumsonto womsonto. Ngelixa le ndlela yokwandisa ukuqhubela phambili kwenkqubo, yonyusa ubunzima bengqiqo yabathengi kwaye ayenzi nto ukunyusa ukomelela kwenkqubo yokufunda. Ukuba ikopi enye yomthengi ihla ngenxa yokungaphumeleli kwamandla okanye isiganeko esifanayo, ngoko ukuthabatha kuyeka.
Indlela esemthethweni yokusombulula le ngxaki eKafka kukusebenzisa bΠizahlulo ezininzi.
Ukwahlulahlula
Izahlulo yeyona ndlela iphambili yokuthelekisa ukufunda kunye nokulinganisa isihloko ngaphaya komda wendlela yomthengisi omnye. Ukuqonda ngcono oku, makhe siqwalasele imeko apho kukho isihloko esinezahlulo ezibini kunye nomthengi omnye ubhalisela kwesi sihloko ().
Umfanekiso 3-5. Umthengi omnye ufunda kwizahlulo ezininzi
Kule meko, umthengi unikwe ulawulo phezu kwezikhombisi ezihambelana neqela layo_id kuzo zombini izahlulo kwaye iqala ukufunda imiyalezo evela kuzo zombini izahlulo.
Xa umthengi owongezelelweyo kwiqela_id efanayo yongezwa kwesi sihloko, i-Kafka iphinda iphinde ihlukanise enye yezahlulo ukusuka kwelokuqala ukuya kumthengi wesibini. Emva koko, umzekelo ngamnye womthengi uya kufunda kwisahlulo esinye sesihloko ().
Ukuqinisekisa ukuba imiyalezo icutshungulwa ngokuhambelanayo kwimisonto engama-20, udinga ubuncinci izahlulo ezingama-20. Ukuba kukho izahlulo ezimbalwa, uya kusala kunye nabathengi abangenanto yokusebenza, njengoko kuchaziwe ngaphambili kwingxoxo yabathengi abakhethekileyo.
Umfanekiso 3-6. Abathengi ababini kwiqela elinye labathengi bafunde kwiindawo ezahlukeneyo
Esi sikimu sinciphisa kakhulu ubunzima be-Kafka broker xa kuthelekiswa nokuhanjiswa komyalezo ofunekayo ukuze kugcinwe umgca we-JMS. Apha awudingi kuzikhathaza ngezi ngongoma zilandelayo:
- Ngowuphi umsebenzisi ofanele ukufumana umyalezo olandelayo, ngokusekwe kulwabiwo lwerobin engqukuva, umthamo wangoku webuffers yokulanda, okanye imiyalezo yangaphambili (njengamaqela omyalezo weJMS).
- Yeyiphi imiyalezo ethunyelwa kubaphi abathengi kwaye nokuba kufuneka inikezelwe kwakhona kwimeko yokusilela.
Yonke into eyenziwa ngumthengisi weKafka kukudlulisa imiyalezo ngokulandelelanayo kumthengi xa umthengi ebacela.
Nangona kunjalo, iimfuno zokuthelekisa uvavanyo kunye nokuthumela kwakhona imiyalezo engaphumeleliyo azipheli - uxanduva lwazo lusuka kumthengisi ukuya kumxhasi. Oku kuthetha ukuba kufuneka zithathelwe ingqalelo kwikhowudi yakho.
Ukuthumela imiyalezo
Luxanduva lomvelisi waloo myalezo ukwenza isigqibo sokuba yeyiphi na isahlulelo omawuthumele kuso umyalezo. Ukuqonda indlela okwenziwa ngayo oku, kufuneka kuqala siqwalasele ukuba yintoni kanye kanye esiyithumelayo.
Ngelixa kwi-JMS sisebenzisa ulwakhiwo lomyalezo olunemetadata (imibhalo engasentla kwentloko kunye neempawu) kunye nomzimba oqulathe umthwalo (umthwalo wokuhlawula), eKafka umyalezo ngu. iperi "key-value". Umthwalo womyalezo uthunyelwa njengexabiso. Isitshixo, kwelinye icala, sisetyenziswa ikakhulu ukwahlula kwaye kufuneka siqulathe isitshixo esicacileyo sengqiqo yeshishiniukubeka imiyalezo enxulumeneyo kwisahlulelo esifanayo.
KwiSahluko sesi-2, sixoxe ngemeko yokubheja kwi-intanethi apho iziganeko ezinxulumeneyo kufuneka ziqwalaselwe ngokulandelelana ngumthengi omnye:
- Iakhawunti yomsebenzisi iqwalaselwe.
- Imali ifakwa kwiakhawunti.
- Ubhejo lwenziwa olutsala imali kwiakhawunti.
Ukuba isiganeko ngasinye ngumyalezo othunyelwe kwisihloko, ngoko isitshixo sendalo siya kuba yi-ID ye-akhawunti.
Xa umyalezo uthunyelwa kusetyenziswa i-Kafka Producer API, idluliselwe kumsebenzi wokwahlula, onikwe umyalezo kunye nemeko yangoku yeqela le-Kafka, ibuyisela i-ID yesahlulelo apho umyalezo kufuneka uthunyelwe. Olu phawu luphunyeziwe kwiJava ngojongano lweSahlulo.
Olu jongano lujongeka ngolu hlobo:
interface Partitioner {
int partition(String topic,
Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster);
}Ukuphunyezwa kweSahlulelo kusebenzisa ialgorithm ye-hashing engagqibekanga yenjongo-jikelele ngaphezulu kwesitshixo ukumisela ulwahlulo, okanye i-robin ejikelezileyo ukuba akukho sitshixo sichaziweyo. Eli xabiso lingagqibekanga lisebenza kakuhle kwiimeko ezininzi. Nangona kunjalo, kwixesha elizayo uya kufuna ukubhala eyakho.
Ukubhala iqhinga lakho lokwahlulahlula
Makhe sijonge umzekelo apho ufuna ukuthumela khona imethadatha kunye nomthwalo womyalezo. Umvuzo kumzekelo wethu ngumyalelo wokwenza idiphozithi kwi-akhawunti yomdlalo. Umyalelo yinto esingathanda ukuqinisekiswa ukuba ayilungiswanga kuthumelo kwaye sifuna ukuqiniseka ukuba kuphela yinkqubo ethembekileyo enyukayo enokuqalisa loo myalelo. Kule meko, ukuthumela kunye nokwamkela iinkqubo ziyavumelana ngokusetyenziswa kwesiginitsha ukungqinisisa umyalezo.
Kwi-JMS eqhelekileyo, sichaza ngokulula ipropathi "yotyikityo lomyalezo" kwaye songeze kumyalezo. Nangona kunjalo, i-Kafka ayisiboneleli ngendlela yokudlulisa imethadatha, kuphela isitshixo kunye nexabiso.
Ekubeni ixabiso liyintlawulo yokudluliselwa kwebhanki esifuna ukuyigcina ingqibelelo yayo, asinakho ukukhetha ngaphandle kokuchaza isakhiwo sedatha esisetyenziswe kwisitshixo. Ukucinga ukuba sifuna i-ID ye-akhawunti yokwahlula, kuba yonke imilayezo enxulumene ne-akhawunti kufuneka iqhutywe ngokulandelelana, siya kuza nesakhiwo se-JSON esilandelayo:
{
"signature": "541661622185851c248b41bf0cea7ad0",
"accountId": "10007865234"
}Ngenxa yokuba ixabiso lomsayino lizakwahluka ngokuxhomekeke kumthwalo ohlawulwayo, isicwangciso-qhinga se-hashing esingagqibekanga sojongano lwe-Partitioner aluyi kuba nemiyalezo ehambelana neqela ngokuthembekileyo. Ngoko ke, kuya kufuneka sibhale isicwangciso sethu esiya kucazulula esi sitshixo kunye nokwahlula ixabiso le-akhawuntiId.
I-Kafka ibandakanya ii-checksums zokubona urhwaphilizo lwemiyalezo evenkileni kwaye ineseti epheleleyo yeempawu zokhuseleko. Nangona kunjalo, iimfuno ezithe ngqo kwishishini, njengale ingasentla, ngamanye amaxesha ziyavela.
Isicwangciso somsebenzisi sokwahlula kufuneka siqinisekise ukuba yonke imiyalezo enxulumeneyo iphelela kwisahlulelo esifanayo. Nangona oku kubonakala kulula, imfuno inokudidaniswa ngokubaluleka kokuyalela imiyalezo ehambelanayo kunye nendlela elizinziswe ngayo inani lezahlulo kwisihloko.
Inani lezahlulo kwisihloko zinokutshintsha ngokuhamba kwexesha, njengoko zinokongezwa ukuba i-traffic ihamba ngaphaya kokulindela kwangaphambili. Ngoko ke, izitshixo zomyalezo zinokunxulunyaniswa nesahlulelo ebesithunyelwe kuso ekuqaleni, oko kuthetha ukuba inxalenye yelizwe ekwabelwana ngayo phakathi kwemizekelo yomvelisi.
Enye into ekufuneka iqwalaselwe kukusasazwa ngokulinganayo kwemiyalezo kuzo zonke izahlulo. Ngokwesiqhelo, izitshixo azisasazwanga ngokulinganayo kuyo yonke imiyalezo, kwaye imisebenzi ye-hash ayiqinisekisi unikezelo olufanelekileyo lwemiyalezo kwiseti encinci yezitshixo.
Kubalulekile ukuqaphela ukuba nangona ukhetha ukwahlula imiyalezo, umahluli ngokwaso unokufuna ukusetyenziswa kwakhona.
Qwalasela imfuno yokuphinda idatha phakathi kwamaqela e-Kafka kwiindawo ezahlukeneyo zejografi. Ngenxa yale njongo, i-Kafka iza nesixhobo somgca womyalelo esibizwa ngokuba yi-MirrorMaker, esetyenziselwa ukufunda imiyalezo evela kwiqela elinye kwaye idlulisele kwelinye.
I-MirrorMaker kufuneka iqonde izitshixo zesihloko esiphindiweyo ukuze kugcinwe ulungelelwaniso olunxulumeneyo phakathi kwemiyalezo xa iphinda-phinda phakathi kwamaqela, ekubeni inani lezahlulo zeso sihloko lisenokungafani kumaqela amabini.
Izicwangciso zokwahlulahlula ngokwesiko zinqabile, njengoko i-hashing engagqibekanga okanye i-robin engqukuva isebenza kakuhle kwiimeko ezininzi. Nangona kunjalo, ukuba ufuna iziqinisekiso eziqinileyo zokuodola okanye ufuna ukukhupha imethadatha kwi-payloads, ngoko ukwahlula yinto omele uyijonge ngakumbi.
I-scalability kunye neenzuzo zokusebenza ze-Kafka zivela ekutshintsheni ezinye zeemfanelo zomthengisi wendabuko kumthengi. Kule meko, isigqibo senziwe ukusasaza imiyalezo enokuthi inxulumene phakathi kwabathengi abaninzi abasebenza ngokufanayo.
Abathengisi be-JMS nabo kufuneka bajongane neemfuno ezinjalo. Okubangela umdla kukuba, indlela yokuthumela imiyalezo enxulumeneyo kumthengi omnye, iphunyezwe ngamaQela oMyalezo we-JMS (ukwahluka kwisicwangciso sokulinganisa okuncamathelayo komthwalo (SLB), ikwafuna ukuba umthumeli amakishe imiyalezo njengenxulumene. Kwimeko ye-JMS, umthengisi unoxanduva lokuthumela eli qela lemiyalezo ehambelanayo kumthengi omnye kwabaninzi, kunye nokudlulisa ubunini beqela ukuba umthengi uyawa.
Izivumelwano zoMvelisi
Ukwahlula akukuphela kwento ekufuneka uyiqwalasele xa uthumela imiyalezo. Makhe sijonge indlela yokuthumela () yeklasi yoMvelisi kwiJava API:
Future < RecordMetadata > send(ProducerRecord < K, V > record);
Future < RecordMetadata > send(ProducerRecord < K, V > record, Callback callback);Kufuneka kuqatshelwe ngokukhawuleza ukuba zombini iindlela zibuyisela i-Future, ebonisa ukuba umsebenzi wokuthumela awenziwanga ngokukhawuleza. Isiphumo sesokuba umyalezo (ProducerRecord) ubhalwe kwi-buffer yokuthumela kwisahlulelo ngasinye esisebenzayo kwaye ithunyelwe kumthengisi njengentambo yangasemva kwilayibrari yomxhasi weKafka. Ngelixa oku kwenza izinto zikhawuleze ngokumangalisayo, oko kuthetha ukuba isicelo esingenamava sinokuphulukana nemiyalezo ukuba inkqubo yaso iyekile.
Njengamaxesha onke, kukho indlela yokwenza umsebenzi wokuthumela uthembeke ngakumbi kwiindleko zokusebenza. Ubungakanani bale buffer bunokusetwa ku-0, kwaye umsonto wesicelo sokuthumela uya kunyanzeleka ukuba ulinde de ugqithiso lomyalezo kumthengisi lugqitywe, ngolu hlobo lulandelayo:
RecordMetadata metadata = producer.send(record).get();Okunye malunga nokufunda imiyalezo
Ukufunda imiyalezo kuneengxaki ezongezelelweyo ekufuneka kuqikelelwe ngazo. Ngokungafaniyo ne-JMS API, enokuqhuba umphulaphuli womyalezo ekuphenduleni umyalezo, i Consumer Kafka kuphela iivoti. Makhe sijonge ngakumbi indlela Uvoto()isetyenziselwa le njongo:
ConsumerRecords < K, V > poll(long timeout);Ixabiso lembuyekezo yendlela yisakhiwo sesikhongozeli esinezinto ezininzi irekhodi yabathengi ukusuka kwizahlulo ezininzi ezinokubakho. irekhodi yabathengi yona ngokwayo iyinto yomphathi wexabiso eliphambili-ixabiso elidityanisiweyo kunye nemetadata enxulumeneyo, njengesahlulelo esivela kuso.
Njengoko kuxutyushwa kwiSahluko sesi-2, kufuneka sikhumbule okwenzekayo kwimiyalezo emva kokuba iqhutywe ngempumelelo okanye ayiphumelelanga, umzekelo, ukuba umxhasi akakwazi ukucubungula umyalezo okanye ukuba uyalahla. Kwi-JMS, oku kwasingathwa ngendlela yokuvuma. Umthengisi usenokucima umyalezo oqhutyelwe ngempumelelo, okanye athumele kwakhona umyalezo okrwada okanye ongeyonyani (ecinga ukuba kusetyenziswe iintengiselwano).
I-Kafka isebenza ngokwahlukileyo kakhulu. Imiyalezo ayicinywanga kwi-broker emva kokuhlolwa kovavanyo, kwaye kwenzeka ntoni ekungaphumeleliyo luxanduva lwekhowudi yokuvavanya ngokwayo.
Njengoko sele sitshilo, iqela labathengi lidibene ne-offset kwilog. Indawo yelog edityaniswe nale offset ihambelana nomyalezo olandelayo oza kukhutshwa ngempendulo kuwo Uvoto(). Inqaku lexesha xa le offset inyuka lisigqibo sokufunda.
Ukubuyela kwimodeli yokufunda ebekuxoxwe ngayo ngaphambili, ukusetyenzwa komyalezo kukho izigaba ezithathu:
- Fumana umyalezo wokufunda.
- Lungiselela umyalezo.
- Qinisekisa umyalezo.
Umthengi weKafka uza kunye nenketho yoqwalaselo vumela.auto.commit. Olu lulungiselelo oluhlala lusetyenziswa rhoqo, njengoko luqhelekile ngoseto oluqulathe igama elithi "auto".
Phambi kwe Kafka 0.10, umxhasi osebenzisa olu khetho angathumela i offset yomyalezo wokugqibela ofundwe kumnxeba olandelayo. Uvoto() emva kokucubungula. Oku kuthetha ukuba nayiphi na imiyalezo esele ilandwa inokuphinda iphinde iqhutyelwe phambili ukuba umxhasi sele eqhubekile kodwa yatshatyalaliswa ngequbuliso phambi kokufowuna. Uvoto(). Ngenxa yokuba umthengisi akagcini naliphi na ilizwe malunga nokuba mangaphi amaxesha umyalezo ufundwe, umthengi olandelayo ofumana loo myalezo akayi kuyazi into embi eyenzekileyo. Le ndlela yokuziphatha yayiyintengiselwano. I-offset yenziwe kuphela ukuba umyalezo uqhutywe ngempumelelo, kodwa ukuba umxhasi uyekile, umthengisi angathumela umyalezo ofanayo kwakhona komnye umxhasi. Le ndlela yokuziphatha yayihambelana nesiqinisekiso sokuhanjiswa komyalezo "nokuba kube kanyeΒ«.
Kwi-Kafka 0.10, ikhowudi yomxhasi itshintshiwe ukuze ukuzinikela kuqaliswe ngamaxesha athile lithala leencwadi lomxhasi, njengoko kuqwalaselwe. auto.commit.interval.ms. Le ndlela yokuziphatha iphakathi kwe-JMS AUTO_ACKNOWLEDGE kunye ne-DUPS_OK_ACKNOWLEDGE iindlela. Xa usebenzisa i-autocommit, imiyalezo inokuzibophelela nokuba iqhutywe ngokwenene - oku kunokwenzeka kwimeko yomthengi ocothayo. Ukuba umthengi ulahlile, imiyalezo iya kulandwa ngumthengi olandelayo, iqala kwindawo ezinikeleyo, enokubangela umyalezo ophosiweyo. Kule meko, uKafka akazange alahlekelwe yimiyalezo, ikhowudi yokufunda ayizange iyenze.
Le ndlela inesithembiso esifanayo nesohlobo lwe-0.9: imiyalezo inokuqhutyelwa phambili, kodwa ukuba ayiphumelelanga, i-offset ayinakwenziwa, enokubangela ukuba ukuhanjiswa kuphindwe kabini. Imiyalezo emininzi oyilandayo xa uphumeza Uvoto(), ngakumbi le ngxaki.
Njengoko kuxoxwe ngokuthi "Ukufunda iMilayezo evela kuMgceni" kwiphepha le-21, akukho nto efana nokuhanjiswa kwexesha elilodwa lomyalezo kwinkqubo yokuthumela imiyalezo xa iindlela zokungaphumeleli zithathelwa ingqalelo.
E-Kafka, kukho iindlela ezimbini zokwenza (ukuzibophelela) i-offset (i-offset): ngokuzenzekelayo kwaye ngesandla. Kuzo zombini ezi meko, imiyalezo inokusetyenzwa kaninzi ukuba umyalezo uqhutyelwe phambili kodwa awuphumelelanga phambi kokuzinikela. Unokukhetha kwakhona ukungawuqhubeki umyalezo konke konke ukuba ukuzinikela kwenzeke ngasemva kwaye ikhowudi yakho yagqitywa ngaphambi kokuba iqwalaselwe (mhlawumbi eKafka 0.9 nangaphambili).
Ungalawula inkqubo yokwenza i-offset ye-manual kwi-API yabathengi be-Kafka ngokucwangcisa ipharamitha vumela.auto.commit ubuxoki kunye nokubiza ngokucacileyo enye yezi ndlela zilandelayo:
void commitSync();
void commitAsync();Ukuba ufuna ukuqhubela phambili umyalezo "ubuncinci kube kanye", kufuneka ubophelele i-offset ngesandla nge commitSync()ngokwenza lo myalelo ngokukhawuleza emva kokucubungula imiyalezo.
Ezi ndlela azivumeli ukuba imiyalezo yamkelwe phambi kokuba iqwalaselwe, kodwa ayenzi nto ukuphelisa ukulibaziseka okunokwenzeka ngelixa inika ukubonakala kokuthengiselana. Akukho ntengiselwano eKafka. Umxhasi akakwazi ukwenza oku kulandelayo:
- Ukubuyisela umva umyalezo womgunyathi ngokuzenzekelayo. Abathengi ngokwabo kufuneka baphathe izinto ezingaqhelekanga ezivela kumthwalo oyingxaki kunye nokungabikho kwe-backend, njengoko bengenako ukuxhomekeka kumthengisi ukuba athumele kwakhona imiyalezo.
- Thumela imiyalezo kwizihloko ezininzi ekusebenzeni kweathom enye. Njengoko siza kubona kungekudala, ukulawula izihloko ezahlukeneyo kunye nokwahlulahlula kunokuhlala koomatshini abahlukeneyo kwiqela leKafka elingalungelelanisi ukuthengiselana xa kuthunyelwa. Ngexesha lokubhala, omnye umsebenzi sele wenziwe ukwenza oku kwenzeke nge-KIP-98.
- Nxulumanisa ukufunda umyalezo omnye osuka kwesinye isihloko kunye nokuthumela omnye umyalezo kwesinye isihloko. Kwakhona, ukwakhiwa kweKafka kuxhomekeke koomatshini abaninzi abazimeleyo abaqhuba njengebhasi enye kwaye akukho mzamo wenziwayo ukufihla oku. Ngokomzekelo, akukho zixhobo ze-API ezinokukuvumela ukuba udibanise umthengi ΠΈ Umvelisi kwintengiselwano. Kwi-JMS, oku kubonelelwa yinto iKhusiezidalwe ngazo MessageProducers ΠΈ MessageConsumers.
Ukuba asikwazi ukuxhomekeka kwintengiselwano, singabonelela njani ngesemantics ngokusondeleyo kwezo zibonelelwa ziinkqubo zemiyalezo yemveli?
Ukuba kukho ithuba lokuba i-offset yomthengi inganyuka ngaphambi kokuba umyalezo uqhutywe, njengokuba ngexesha lokuphazamiseka kwabathengi, ngoko umthengi akanayo indlela yokwazi ukuba iqela lakhe lomthengi lilahlekelwe ngumyalezo xa linikwa isahlulo. Ke iqhinga elinye kukubuyisela umva ioffset kwindawo yangaphambili. I-API yabathengi baseKafka ibonelela ngezi ndlela zilandelayo:
void seek(TopicPartition partition, long offset);
void seekToBeginning(Collection < TopicPartition > partitions);
Indlela funa () inokusetyenziswa kunye nendlela
offsetsForTimes(Imaphu izitampu zexeshaUkukhangela) ukubuyisela umva kwisimo kwindawo ethile kwixesha elidlulileyo.
Ngokucacileyo, ukusebenzisa le ndlela kuthetha ukuba kusenokwenzeka ukuba eminye imiyalezo ebesetyenzwe ngaphambili iya kufundwa kwaye iqwalaselwe kwakhona. Ukunqanda oku, sinokusebenzisa ukufunda ngokungenangqondo, njengoko kuchazwe kwiSahluko sesi-4, ukugcina umkhondo wemiyalezo ebikhe yajongwa ngaphambili kunye nokuphelisa ukuphinda-phinda.
Kungenjalo, ikhowudi yakho yomthengi inokugcinwa ilula, nje ukuba ukulahleka komyalezo okanye ukuphindaphinda kwamkelekile. Xa siqwalasela iimeko zokusetyenziswa apho i-Kafka isetyenziswa ngokuqhelekileyo, njengokuphatha imicimbi yelog, i-metrics, ukucofa ukulandelela, njl., siyaqonda ukuba ukulahlekelwa kwemiyalezo nganye akunakwenzeka ukuba kube nempembelelo ebalulekileyo kwizicelo ezingqongileyo. Kwiimeko ezinjalo, amaxabiso angagqibekanga amkelekile ngokugqibeleleyo. Kwelinye icala, ukuba isicelo sakho sifuna ukuthumela iintlawulo, kufuneka unakekele ngononophelo umyalezo ngamnye. Konke kuhla kumxholo.
Ukuqwalaselwa komntu ngamnye kubonisa ukuba njengoko ubuninzi bemiyalezo bukhula, ixabiso lomyalezo ngamnye liyancipha. Imiyalezo emikhulu ithande ukuba luncedo xa ijongwe ngendlela edityanisiweyo.
Ukufumaneka okuphezulu
Indlela ye-Kafka yokufumaneka okuphezulu yahluke kakhulu kwindlela ye-ActiveMQ. I-Kafka iyilwe malunga neqela eliphumayo apho zonke iimeko zomthengisi zifumana kwaye zisasaze imiyalezo ngaxeshanye.
Iqela le-Kafka liquka iimeko ezininzi ze-broker ezisebenza kwiiseva ezahlukeneyo. I-Kafka yenzelwe ukuba isebenze kwi-hardware eqhelekileyo, apho i-node nganye inokugcinwa kwayo okunikezelweyo. Ukusetyenziswa kwe-network attached storage (SAN) akukhuthazwa kuba i-computing nodes ezininzi zinokukhuphisana ngexesha.Π«e ngamaxesha okugcina kunye nokudala iingxabano.
Kafka nguye rhoqo inkqubo. Abasebenzisi abaninzi be-Kafka abakhulu abazange bavale amaqela abo kwaye isofthiwe ihlala ihlaziya ngokuqala kwakhona ngokulandelelana. Oku kuphunyezwa ngokuqinisekisa ukuhambelana noguqulelo lwangaphambili lwemiyalezo kunye nokusebenzisana phakathi kwabarhwebi.
Iibroker eziqhagamshelwe kwiqela leseva , esebenza njengobhaliso lwedatha yoqwalaselo kwaye isetyenziselwa ukulungelelanisa iindima zomthengisi ngamnye. I-ZooKeeper ngokwayo yinkqubo esasazwayo ebonelela ngokufumaneka okuphezulu ngokuphinda-phindwa kolwazi ngokuseka ikhoram.
Kwimeko yesiseko, isihloko senziwe kwi-cluster ye-Kafka eneempawu ezilandelayo:
- Inani lezahlulo. Njengoko kuxoxiwe ngaphambili, ixabiso elichanekileyo elisetyenziswe apha lixhomekeke kwinqanaba elifunekayo lokufunda ngokuhambelanayo.
- I-replication factor (i-factor) imisela ukuba zingaphi iimeko ze-broker kwiqela ekumele ziqulathe iilog zesi sahlulelo.
Isebenzisa iZooKeepers ukulungelelanisa, iKafka izama ukusasaza ngokufanelekileyo izahlulo ezintsha phakathi kwabarhwebi kwiqela. Oku kwenziwa ngomzekelo omnye osebenza njengoMlawuli.
Ngexesha lokusebenza kwisahlulelo sesihloko ngasinye Umlawuli yabela iindima kumthengisi inkokeli (inkokeli, inkosi, umsasazi) kunye abalandeli (abalandeli, amakhoboka, abaphantsi). Umthengisi, osebenza njengenkokeli yolu lwahlulo, unoxanduva lokufumana yonke imiyalezo ethunyelwe kuyo ngabavelisi kunye nokusabalalisa imiyalezo kubathengi. Xa imiyalezo ithunyelwa kulwahlulo lwesihloko, iphinda-phinda kuzo zonke iindawo zokuthengisa ezisebenza njengabalandeli kweso sahlulelo. Indawo nganye equlethe iilog zesahlulelo ibizwa ngokuba umfanekiso. Umrhwebi unokusebenza njengenkokeli yezahlulo ezithile kwaye njengomlandeli wabanye.
Umlandeli ophethe yonke imiyalezo ebanjwe yinkokeli uyabizwa ikopi edityanisiweyo (iReplica ekwimo yongqamaniso, in-sync replica). Ukuba i-broker esebenza njengenkokeli yolwahlulo iyehla, nayiphi na i-broker esexesheni okanye elungelelaniswe kweso sahlulelo inokuthatha indima yobunkokeli. Luyilo oluzinzileyo ngokumangalisayo.
Inxalenye yoqwalaselo lomvelisi yiparameter ii acks, emisela ukuba zingaphi iikopi ekufuneka zivume (zivume) ukufumana umyalezo phambi kokuba umsonto wesicelo uqhubeke nokuthumela: 0, 1, okanye zonke. Ukuba imiselwe ku zonke, ngoko xa umyalezo ufunyenwe, inkokeli iya kuthumela isiqinisekiso kumvelisi ngokukhawuleza xa ifumana iziqinisekiso (iziqinisekiso) zerekhodi ukusuka kwiimpawu ezininzi (kubandakanywa ngokwazo) ezichazwe ngokucwangcisa isihloko. min.insync.replicas (ehlala ikho 1). Ukuba umyalezo awunakuphindwa ngempumelelo, ngoko umvelisi uya kuphosa isicelo ngaphandle (NotEnoughReplicas okanye NotEnoughReplicasAfterAppend).
Uqwalaselo oluqhelekileyo ludala isihloko kunye nento ephindaphindwayo ye-3 (inkokeli e-1, abalandeli aba-2 ngesahlulelo ngasinye) kunye neparameter. min.insync.replicas ibekwe ku 2. Kulo mzekelo, iqela liya kuvumela omnye wabarhwebi abalawula ulwahlulo lwesihloko ukuba baye ezantsi ngaphandle kokuchaphazela izicelo zabaxumi.
Oku kusibuyisela kurhwebo olusele luqhelekile phakathi kokusebenza nokuthembeka. Ukuphindaphinda kwenzeka ngeendleko zexesha elongezelelweyo lokulinda lokuqinisekisa (umbulelo) kubalandeli. Nangona, ngenxa yokuba iqhuba ngokuhambelanayo, ukuphindaphinda ubuncinane kwiinodi ezintathu zinentsebenzo efanayo neyesibini (ukungahoywa kokunyuka kokusetyenziswa kwe-bandwidth yenethiwekhi).
Ngokusebenzisa olu dweliso lokuphindaphinda, iKafka inqanda ngobuchule imfuneko yokubhala umyalezo ngamnye kwidiski ngomsebenzi. Ungqamaniso (). Umyalezo ngamnye othunyelwe ngumvelisi uya kubhalwa kwilog yesahlulelo, kodwa njengoko kuxoxiwe kwiSahluko sesi-2, ukubhala kwifayile kuqala kwenziwa kwisithinteli senkqubo yokusebenza. Ukuba lo myalezo uphindwa komnye umzekelo we-Kafka kwaye ukwinkumbulo yayo, ukulahlekelwa yinkokeli akuthethi ukuba umyalezo ngokwawo ulahlekile - unokuthatyathwa yi-replica ehambelanayo.
Ukwala ukwenza utyando Ungqamaniso () kuthetha ukuba iKafka inokufumana imiyalezo ngokukhawuleza njengoko inokubhalela kwimemori. Ngakolunye uhlangothi, ixesha elide unokunqanda ukugungxula imemori kwidiski, kungcono. Ngenxa yesi sizathu, akuqhelekanga ukuba i-Kafka brokers inikwe i-64 GB okanye ngaphezulu kwememori. Oku kusetyenziswa kwememori kuthetha ukuba umzekelo omnye weKafka unokubaleka ngokulula kwisantya samawakawaka amaxesha ngokukhawuleza kunomthengisi womyalezo wesintu.
I-Kafka inokuqwalaselwa kwakhona ukuba isebenzise ukusebenza Ungqamaniso () kwiipakethe zomyalezo. Kuba yonke into eKafka ijolise kwiphakheji, eneneni isebenza kakuhle kwiimeko ezininzi zokusetyenziswa kwaye sisixhobo esiluncedo kubasebenzisi abafuna iziqinisekiso ezinamandla kakhulu. Uninzi lwentsebenzo ecocekileyo yeKafka ivela kwimiyalezo ethunyelwa kumthengisi njengepakethi kwaye le miyalezo ifundwa kumthengisi kwiibhloko ezilandelelanayo kusetyenziswa. imisebenzi (imisebenzi apho umsebenzi wokukopa idatha ukusuka kwindawo yememori ukuya kwenye ungenziwanga). Le yokugqibela yintsebenzo enkulu kunye nenzuzo yobutyebi kwaye inokwenzeka kuphela ngokusetyenziswa kwesakhiwo sedatha yelog esisiseko esichaza iskimu sokwahlula.
Ukusebenza okungcono kakhulu kunokwenzeka kwiqela le-Kafka kunomthengisi omnye we-Kafka, kuba izahlulo zesihloko zinokukhula koomatshini abaninzi abahlukeneyo.
Iziphumo
Kwesi sahluko, sijonge indlela i-architecture ye-Kafka iphinda icinge kwakhona ubudlelwane phakathi kwabathengi kunye nee-brokers ukubonelela ngombhobho wemiyalezo owomeleleyo ngokumangalisayo, kunye ne-output ephindwe kaninzi kune-broker yemiyalezo eqhelekileyo. Siye saxoxa ngomsebenzi osetyenziswayo ukufezekisa oku kwaye sijonge ngokufutshane kwi-architecture yezicelo ezibonelela ngalo msebenzi. Kwisahluko esilandelayo, siza kujonga iingxaki eziqhelekileyo izicelo ezisekelwe kwimiyalezo ekufuneka zisonjululwe kwaye sixoxe ngobuchule bokujongana nazo. Siza kugqiba isahluko ngokuchaza indlela yokuthetha ngetekhnoloji yemiyalezo ngokubanzi ukuze ukwazi ukuvavanya ukufaneleka kwazo kwiimeko zakho zokusebenzisa.
Inxalenye eguqulelwe ngaphambili:
Inguqulelo yenziwe:
Iza kuqhubekaβ¦
Ngabasebenzisi ababhalisiweyo kuphela abanokuthatha inxaxheba kuphando. , ndiyacela.
Ngaba iKafka isetyenziswa kumbutho wakho?
-
ukuba
-
akukho
-
Ngaphambili, hayi ngoku
-
Siceba ukusebenzisa
Bali-38 abasebenzisi abavotileyo. Abasebenzisi abasi-8 abakhange.
umthombo: www.habr.com