Ukuqhutshwa kokuhumusha kwencwadi encane:
Ukuqonda Abadayisi Bemiyalezo
umbhali: Jakub Korab, umshicileli: O'Reilly Media, Inc., usuku lokushicilelwa: Juni 2017, ISBN: 9781492049296.
Ingxenye ehunyushwe ngaphambilini:
ISAHLUKO 3
KAFKA
I-Kafka yathuthukiswa kwa-LinkedIn ukuze ikwazi ukubhekana nemikhawulo ethile yabadayisi bemiyalezo bendabuko futhi igweme ukumisa ama-broker amaningi emiyalezo ukuze kube nokuhlangana okuhlukile kwephuzu nephuzu, okuchazwe kule ncwadi ngaphansi kokuthi "Ukukhuphula nokukhipha" ekhasini 28. .Izimo zokusebenzisa i-LinkedIn ithembele kakhulu ekufakweni kwendlela eyodwa kwenani elikhulu kakhulu ledatha, njengokuchofozwa kwekhasi kanye namalogi wokufinyelela, kuyilapho isavumela leyo datha ukuthi isetshenziswe amasistimu amaningi ngaphandle kokuba nomthelela ekukhiqizeni kwabakhiqizi noma abanye abathengi. Eqinisweni, isizathu sokuba khona kuka-Kafka ukuthola uhlobo lwesakhiwo semiyalezo esichazwa yi-Universal Data Pipeline.
Ngokunikezwa kwalo mgomo omkhulu, ezinye izimfuneko zavela ngokwemvelo. I-Kafka kufanele:
- Shesha kakhulu
- Nikeza ngomkhawulokudonsa owengeziwe lapho usebenza ngemilayezo
- Sekela umshicileli-Obhalisile kanye namamodeli akhomba-kuya-ephuzwini
- Ungehli ijubane ngokungeza abathengi. Isibonelo, ukusebenza kwakho kokubili ulayini nesihloko ku-ActiveMQ kuyehla njengoba inani labathengi endaweni liya likhula.
- Yiba ne-scalable evundlile; uma umthengisi oyedwa ophikelela nemiyalezo angakwenza lokho kuphela ngesivinini esiphezulu sediski, khona-ke kunengqondo ukweqa isenzakalo somthengisi oyedwa ukwandisa ukusebenza.
- Khawulela ukufinyelela ekulondolozeni nasekubuyiseni imilayezo
Ukufeza konke lokhu, i-Kafka yamukela isakhiwo esichaza kabusha izindima nezibopho zamakhasimende kanye nabathengi bemiyalezo. Imodeli ye-JMS igxile kakhulu kumthengisi, lapho umdayisi enesibopho sokusabalalisa imilayezo futhi amakhasimende kufanele akhathazeke ngokuthumela nokwamukela imilayezo kuphela. I-Kafka, ngakolunye uhlangothi, igxile kumakhasimende, iklayenti lithatha izici eziningi zomdayisi wendabuko, njengokusabalalisa okulungile kwemilayezo efanelekile kubathengi, ukuze uthole umthengisi oshesha kakhulu futhi osheshayo. Kubantu abasebenze ngezinhlelo zemiyalezo yendabuko, ukusebenza ne-Kafka kudinga ukuguqulwa kwengqondo okubalulekile.
Lesi siqondiso sobunjiniyela siholele ekudalweni kwengqalasizinda yemiyalezo ekwazi ukukhulisa ukuphuma ngama-oda amaningi wobukhulu uma kuqhathaniswa nomthengisi ojwayelekile. Njengoba sizobona, le ndlela ihambisana nokuhwebelana, okusho ukuthi i-Kafka ayifanele izinhlobo ezithile zemithwalo yemisebenzi kanye nesofthiwe efakiwe.
Imodeli Yendawo Ehlanganisiwe
Ukuze kufezeke izimfuneko ezichazwe ngenhla, i-Kafka ihlanganise ukushicilela nokuthumela imilayezo iphuzu nephuzu ngaphansi kohlobo olulodwa lwendawo β isihloko. Lokhu kuyabadida abantu abasebenze nezinhlelo zemiyalezo, lapho igama elithi "isihloko" lisho indlela yokusakaza lapho (kusuka esihlokweni) ukufundwa okungabekezeleleki kuyo. Izihloko ze-Kafka kufanele zithathwe njengohlobo lwendawo eyingxube, njengoba kuchazwe esethulweni sale ncwadi.
Esikhathini esisele salesi sahluko, ngaphandle uma sisho ngokuhlukile, igama elithi "isihloko" lizobhekisela esihlokweni se-Kafka.
Ukuze siqonde ngokugcwele ukuthi izihloko ziziphatha kanjani nokuthi yiziphi iziqinisekiso ezinikezayo, sidinga kuqala sibheke ukuthi zisetshenziswa kanjani e-Kafka.
Isihloko ngasinye e-Kafka sinelogi yaso.
Abakhiqizi abathumela imilayezo ku-Kafka babhala kuleli logi, futhi abathengi bafunda kusuka kulogi besebenzisa izikhombisi ezihlala ziqhubekela phambili. Ngezikhathi ezithile, i-Kafka isusa izingxenye ezindala zelogi, kungakhathaliseki ukuthi imilayezo ekulezo zingxenye ifundiwe noma cha. Ingxenye emaphakathi yedizayini ye-Kafka ukuthi umthengisi akanandaba ukuthi imilayezo ifundiwe noma cha - lowo umthwalo wemfanelo weklayenti.
Amagama "log" kanye "pointer" awaveli phakathi . La magama aziwayo asetshenziswa lapha ukuze asize ukuqonda.
Le modeli ihluke ngokuphelele ku-ActiveMQ, lapho imilayezo evela kuyo yonke imigqa igcinwa kulogi efanayo, futhi umthengisi umaka imilayezo njengesusiwe ngemva kokuba isifundiwe.
Manje ake sijule kancane futhi sibheke ilogi yesihloko ngokuningiliziwe.
Ilogi ye-Kafka iqukethe izingxenye eziningana (). I-Kafka iqinisekisa uku-oda okuqinile engxenyeni ngayinye. Lokhu kusho ukuthi imilayezo ebhalwe ekuhlukaniseni ngokulandelana kwayo izofundwa ngokulandelana okufanayo. Ingxenye ngayinye isetshenziswa njengefayela lokungena eligoqayo eliqukethe isethi engaphansi (i-subset) yayo yonke imilayezo ethunyelwe esihlokweni ngabadidiyeli bayo. Isihloko esidaliwe siqukethe, ngokuzenzakalelayo, ukwahlukanisa okukodwa. Umqondo wokuhlukanisa umqondo omaphakathi we-Kafka wokukala okuvundlile.
Umfanekiso 3-1. I-Kafka Partitions
Uma umkhiqizi ethumela umlayezo esihlokweni se-Kafka, uyanquma ukuthi iyiphi ingxenye ezothumela umlayezo kuyo. Sizobheka lokhu ngokuningiliziwe ngokuhamba kwesikhathi.
Ukufunda imiyalezo
Iklayenti elifuna ukufunda imilayezo liphatha isikhombisi esiqanjwe igama iqembu labathengi, elikhomba offset imiyalezo ekuhlukaniseni. I-offset iyindawo ekhuphukayo eqala ku-0 ekuqaleni kokuhlukanisa. Leli qembu labathengi, elikhonjwe ku-API nge-group_id echazwe ngumsebenzisi, elihambisana nalo umthengi oyedwa onengqondo noma uhlelo.
Amasistimu amaningi okuthumela imiyalezo afunda idatha evela endaweni esebenzisa izimo eziningi nochungechunge ukucubungula imilayezo ngokuhambisana. Ngakho, ngokuvamile kuzoba nezimo eziningi zabathengi ababelana ngeqembu elifanayo labathengi.
Inkinga yokufunda ingamelwa kanje:
- Isihloko sinokuhlukaniswa okuningi
- Amaqembu amaningi abathengi angasebenzisa isihloko ngesikhathi esisodwa
- Iqembu labathengi lingaba nezimo eziningi ezihlukene
Lena inkinga engeyona into encane-kuya-kwabaningi. Ukuze uqonde ukuthi i-Kafka ibuphatha kanjani ubudlelwano phakathi kwamaqembu abathengi, izimo zabathengi, nokuhlukaniswa, ake sibheke uchungechunge lwezimo zokufunda eziyinkimbinkimbi kancane kancane.
Abathengi namaqembu abathengi
Ake sithathe njengesiqalo isihloko esinengxenye eyodwa ().
Umfanekiso 3-2. Umthengi ufunda kusukela ekuhlukaniseni
Uma isibonelo somthengi sixhuma ne-group_id yalo kulesi sihloko, sinikezwa isigaba sokufunda kanye ne-offset kuleso sahlukaniso. Indawo yalokhu kususwa iyalungiseka kuklayenti njengesikhombi sendawo yakamuva (umlayezo omusha) noma indawo yakuqala (umlayezo omdala kakhulu). Umthengi ucela (ukuvota) imilayezo esihlokweni, okubangela ukuthi ifundwe ngokulandelana ukusuka kulogi.
Isimo se-offset sihlala sibuyiselwa e-Kafka futhi sigcinwa njengemilayezo esihlokweni sangaphakathi _i-consumer_offsets. Imilayezo efundiwe ayikasuswa, ngokungafani nomdayiseli ovamile, futhi iklayenti lingahlehlisa i-offset ukuze liphinde licubungule imilayezo esivele ibukiwe.
Lapho umthengi wesibili onengqondo exhuma esebenzisa i-id yeqembu_ehlukile, uphatha isikhombi sesibili esizimele kwesokuqala (). Ngakho, isihloko se-Kafka sisebenza njengolayini lapho kunomthengi oyedwa futhi njengesihloko esivamile sokushicilela-ukubhalisa (i-pub-sub) abathengi abaningi ababhalisela kuso, ngenzuzo eyengeziwe yokuthi yonke imilayezo igcinwa futhi ingacutshungulwa izikhathi eziningi.
Umfanekiso 3-3. Abathengi ababili emaqenjini ahlukene abathengi bafunda ekwahlukaniseni okufanayo
Abathengi eqenjini labathengi
Uma isenzakalo somthengi esisodwa sifunda idatha kusukela ku-partition, sinokulawula okugcwele kwesikhombi futhi sicubungule imilayezo njengoba kuchazwe esigabeni sangaphambilini.
Uma izimo ezimbalwa zabathengi bezixhunywe ne-group_id efanayo esihlokweni esinengxenye eyodwa, isibonelo esixhume ekugcineni sizonikezwa ukulawula phezu kwesikhombi futhi kusukela ngaleso sikhathi kuqhubeke sizothola yonke imilayezo ().
Umfanekiso 3-4. Abathengi ababili eqenjini elifanayo labathengi bafunde kusukela ekuhlukaniseni okufanayo
Le ndlela yokucubungula, lapho inani lezimo zomthengi lidlula inani lama-partitions, ingacatshangwa njengohlobo lomthengi okhethekile. Lokhu kungaba wusizo uma udinga "i-active-passive" (noma "i-hot-warm") yokuhlanganisa izimo zakho zabathengi, nakuba ukusebenzisa abathengi abaningi ngokuhambisana ("active-active" noma "hot-hot") kuvame kakhulu kune abathengi. Kokubekwe eceleni.
Lokhu kuziphatha kokusabalalisa umlayezo okuchazwe ngenhla kungamangaza uma kuqhathaniswa nendlela ulayini ojwayelekile we-JMS oziphatha ngayo. Kule modeli, imilayezo ethunyelwe kulayini izosatshalaliswa ngokulinganayo phakathi kwabathengi ababili.
Ngokuvamile, lapho sidala izimo eziningi zabathengi, lokhu sikwenza ukucubungula imilayezo ngokuhambisana, noma ukukhulisa isivinini sokufunda, noma ukukhulisa ukuzinza kwenqubo yokufunda. Njengoba isenzakalo somthengi esisodwa kuphela esingakwazi ukufunda idatha kusukela esakhiweni ngesikhathi, lokhu kufinyelelwa kanjani e-Kafka?
Enye indlela yokwenza lokhu ukusebenzisa isibonelo somthengi oyedwa ukuze ufunde yonke imilayezo futhi uyidlulisele ku-pool pool. Ngenkathi le ndlela ikhulisa ukucubungula okudlulele, inyusa ubunkimbinkimbi bengqondo yomthengi futhi ayenzi lutho ukukhulisa ukuqina kwesistimu yokufunda. Uma ikhophi eyodwa yomthengi yehla ngenxa yokwehluleka kwamandla noma isehlakalo esifanayo, ukususa kuyama.
Indlela esemthethweni yokuxazulula le nkinga e-Kafka ukusebenzisa i-bΠama-partitions amaningi.
Ukwahlukanisa
Izahluko ziyindlela eyinhloko yokuqhathanisa ukufunda nokukala isihloko ngale komkhawulokudonsa wesibonelo somthengisi oyedwa. Ukuze sikuqonde kangcono lokhu, ake sicabangele isimo lapho kunesihloko esinezingxenye ezimbili futhi umthengi oyedwa obhalisele lesi sihloko ().
Umfanekiso 3-5. Umthengi oyedwa ufunda kuma-partitions amaningi
Kulesi simo, umthengi unikezwa ukulawula izikhombi ezihambisana ne-group_id yayo kuzo zombili izingxenye futhi uqala ukufunda imilayezo kuzo zombili izingxenye.
Uma umthengi owengeziwe weqembu_id elifanayo engezwa kulesi sihloko, i-Kafka iphinda ihlukanise ingxenye eyodwa ukusuka kweyokuqala ukuya kumthengi wesibili. Ngemva kwalokho, isibonelo ngasinye somthengi sizofunda engxenyeni eyodwa yesihloko ().
Ukuqinisekisa ukuthi imilayezo icutshungulwa ngokufana emiculweni engu-20, udinga okungenani ama-partitions angama-20. Uma kunokuhlukaniswa okumbalwa, uzosala nabathengi abangenakho ukusebenza, njengoba kuchazwe ekuqaleni engxoxweni yabathengi abakhethekile.
Umfanekiso 3-6. Abathengi ababili eqenjini elifanayo labathengi bafunde kuma-partitions ahlukene
Lolu hlelo lunciphisa kakhulu inkimbinkimbi yomthengisi we-Kafka uma kuqhathaniswa nokusabalalisa umlayezo odingekayo ukuze kugcinwe ulayini we-JMS. Lapha awudingi ukukhathazeka ngamaphuzu alandelayo:
- Imuphi umthengi okufanele athole umlayezo olandelayo, ngokusekelwe ekwabeni kwe-robin eyindilinga, umthamo wamanje wamabhafa okulanda kuqala, noma imilayezo yangaphambilini (njengamaqembu emilayezo ye-JMS).
- Imiphi imilayezo ethunyelwa kubaphi abathengi nokuthi kufanele yini iphinde ilethwe uma kwenzeka yehluleka.
Okufanele umthengisi we-Kafka akwenze ukudlulisa imilayezo ngokulandelana kumthengi lapho owakamuva eyicela.
Kodwa-ke, izimfuneko zokufanisa ukuhlolwa kokuhlola nokuthumela kabusha imilayezo ehlulekile azisuki - umthwalo wazo usuka kumdayisi uye kwiklayenti. Lokhu kusho ukuthi kufanele zicatshangelwe kukhodi yakho.
Ithumela imilayezo
Kuyisibopho somkhiqizi walowo mlayezo ukunquma ukuthi iyiphi ukwahlukanisa ozothumela kuyo umlayezo. Ukuze siqonde indlela lokhu okwenziwa ngayo, okokuqala sidinga ukucabangela ukuthi yini ngempela esiyithumelayo.
Nakuba ku-JMS sisebenzisa isakhiwo somlayezo esinemethadatha (izihloko nezakhiwo) kanye nendikimba equkethe umthwalo okhokhelwayo (inkokhelo), e-Kafka umlayezo uthi bhanqa "inani elingukhiye". Ukulayishwa komlayezo kuthunyelwa njengenani. Ukhiye, ngakolunye uhlangothi, usetshenziselwa ikakhulukazi ukwahlukanisa futhi kufanele uqukathe ukhiye othize we-logic yebhizinisiukubeka imilayezo ehlobene engxenyeni efanayo.
Esahlukweni sesi-2, sixoxe ngesimo sokubheja ku-inthanethi lapho imicimbi ehlobene idinga ukucutshungulwa ngokulandelana ngumthengi oyedwa:
- I-akhawunti yomsebenzisi imisiwe.
- Imali ifakwa ku-akhawunti.
- Kwenziwa ukubheja okukhipha imali ku-akhawunti.
Uma umcimbi ngamunye ungumlayezo othunyelwe esihlokweni, ukhiye wemvelo uzoba i-ID ye-akhawunti.
Uma umlayezo uthunyelwa kusetshenziswa i-API Yomkhiqizi we-Kafka, udluliselwa kumsebenzi wokuhlukanisa okuthi, uma kunikezwe umlayezo kanye nesimo samanje seqoqo le-Kafka, ubuyisele i-ID yengxenye lapho umlayezo kufanele uthunyelwe khona. Lesi sici sisetshenziswa ku-Java ngokusebenzisa isixhumi esibonakalayo se-Partitioner.
Lesi sixhumanisi sibukeka kanje:
interface Partitioner {
int partition(String topic,
Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster);
}Ukusetshenziswa kwe-Partitioner kusebenzisa i-algorithm ye-hashing yenhloso evamile ezenzakalelayo phezu kokhiye ukuze kunqunywe ukwahlukanisa, noma i-round-robin uma kungekho khiye oshiwo. Leli nani elizenzakalelayo lisebenza kahle ezimweni eziningi. Nokho, esikhathini esizayo uzofuna ukubhala owakho.
Ukubhala isu lakho lokuhlukanisa
Ake sibheke isibonelo lapho ufuna ukuthumela khona imethadatha kanye nomthwalo okhokhelwayo womlayezo. Umthwalo okhokhelwayo esibonelweni sethu uwumyalelo wokwenza idiphozi ku-akhawunti yegeyimu. Umyalelo yinto esingathanda ukuqinisekiswa ukuthi ngeke ishintshwe ekudluliselweni futhi sifuna ukuqiniseka ukuthi isistimu ethembekile kuphela ekhuphuka nomfula engaqalisa lowo myalelo. Kulesi simo, amasistimu okuthumela nokwamukela ayavumelana ngokusetshenziswa kwesiginesha ukuze kuqinisekiswe umlayezo.
Ku-JMS evamile, sivele sichaze "isiginesha yomlayezo" futhi siyengeze emlayezo. Nokho, i-Kafka ayisinikezi ngendlela yokudlulisa imethadatha, ukhiye kanye nenani kuphela.
Njengoba inani liwumthwalo okhokhelwayo wokudluliswa kwebhange esifuna ukukulondoloza ubuqotho bawo, asinakho ukukhetha ngaphandle kokuchaza isakhiwo sedatha esizosetshenziswa kukhiye. Ngokucabangela ukuthi sidinga i-akhawunti ye-ID yokuhlukanisa, njengoba yonke imilayezo ehlobene ne-akhawunti kufanele icutshungulwe ngokulandelana kwayo, sizoqhamuka nesakhiwo esilandelayo se-JSON:
{
"signature": "541661622185851c248b41bf0cea7ad0",
"accountId": "10007865234"
}Ngenxa yokuthi inani lesiginesha lizohluka kuye ngomthwalo okhokhelwayo, isu elizenzakalelayo le-hashing lesixhumi esibonakalayo se-Partitioner ngeke likwazi ukwethenjwa imilayezo ehlobene neqembu. Ngakho-ke, sizodinga ukubhala isu lethu elizoncozulula lo khiye futhi lihlukanise inani le-akhawuntiId.
I-Kafka ihlanganisa amasheke ukuze kutholwe inkohlakalo yemilayezo esitolo futhi inesethi egcwele yezici zokuphepha. Noma kunjalo, izimfuneko eziqondene nomkhakha othile, njengalena engenhla, ngezinye izikhathi ziyavela.
Isu lokuhlukanisa lomsebenzisi kufanele liqinisekise ukuthi yonke imilayezo ehlobene igcina ekuhlukaniseni okufanayo. Nakuba lokhu kubonakala kulula, imfuneko ingaba yinkimbinkimbi ngokubaluleka koku-oda okuthunyelwe okuhlobene nokuthi inani lokuhlukaniswa kwesihloko liqine kangakanani.
Inani lama-partitions esihlokweni lingashintsha ngokuhamba kwesikhathi, njengoba lingangezwa uma ithrafikhi idlula obekulindelwe ekuqaleni. Ngakho-ke, okhiye bomlayezo bangahlotshaniswa nokwahlukanisa abathunyelwe kukho ekuqaleni, okusho ucezu lwesimo okufanele kwabelwane ngalo phakathi kwezimo zomkhiqizi.
Esinye isici okufanele sicatshangelwe ukusatshalaliswa ngokulinganayo kwemilayezo kuzo zonke izingxenyekazi. Ngokuvamile, okhiye abasatshalaliswa ngokulinganayo kuyo yonke imilayezo, futhi imisebenzi ye-hashi ayiqinisekisi ukusatshalaliswa okufanele kwemilayezo kusethi encane yokhiye.
Kubalulekile ukuqaphela ukuthi noma ngabe ukhetha ukuhlukanisa imilayezo, isihlukanisi ngokwaso singadinga ukuphinda sisetshenziswe.
Cabangela imfuneko yokuphindaphinda idatha phakathi kwamaqoqo e-Kafka ezindaweni ezihlukene zendawo. Ngale njongo, i-Kafka iza nethuluzi lomugqa womyalo elibizwa ngokuthi i-MirrorMaker, elisetshenziselwa ukufunda imilayezo evela kuqoqo elilodwa bese uyidlulisela kwelinye.
I-MirrorMaker kufanele iqonde okhiye besihloko esiphindiwe ukuze kugcinwe ukuhleleka okuhlobene phakathi kwemilayezo lapho iphindaphinda phakathi kwamaqoqo, njengoba inani lezingxenye zaleso sihloko lingase lingafani kumaqoqo amabili.
Amasu okuhlukanisa ngokwezifiso awavamile, njengoba i-hashing ezenzakalelayo noma i-round robin isebenza kahle ezimeni eziningi. Kodwa-ke, uma udinga iziqinisekiso zoku-oda eziqinile noma udinga ukukhipha imethadatha kumthwalo okhokhelwayo, khona-ke ukwahlukanisa kuyinto okufanele uyibhekisise.
Ukukhula kanye nezinzuzo zokusebenza kwe-Kafka zivela ekuguquleni eminye yezibopho zomthengisi wendabuko iye kuklayenti. Kulokhu, kwenziwa isinqumo sokusabalalisa imilayezo okungenzeka ihlobene phakathi kwabathengi abambalwa abasebenza ngokufana.
Abathengi be-JMS nabo badinga ukubhekana nezidingo ezinjalo. Kuyathakazelisa ukuthi indlela yokuthumela imilayezo ehlobene kumthengi ofanayo, esetshenziswa ngamaQembu Omlayezo we-JMS (okuhlukile kuqhinga le-Sticky load balancing (SLB)), idinga nokuthi umthumeli amake imilayezo njengehlobene. Endabeni ye-JMS, umthengisi unesibopho sokuthumela leli qembu lemilayezo ehlobene kumthengi oyedwa kwabaningi, kanye nokudlulisa ubunikazi beqembu uma umthengi ewa.
Izivumelwano Zomkhiqizi
Ukwahlukanisa akuyona ukuphela kwento okufanele icatshangelwe lapho uthumela imilayezo. Ake sibheke izindlela zokuthumela () zekilasi Lomkhiqizi ku-Java API:
Future < RecordMetadata > send(ProducerRecord < K, V > record);
Future < RecordMetadata > send(ProducerRecord < K, V > record, Callback callback);Kufanele kuqashelwe ngokushesha ukuthi zombili izindlela zibuyisela i-Future, okubonisa ukuthi umsebenzi wokuthumela awenziwa ngokushesha. Umphumela uba ukuthi umlayezo (ProducerRecord) ubhalwa kusigcinalwazi sokuthumela sengxenye ngayinye esebenzayo futhi uthunyelwe kumthengisi njengochungechunge lwangemuva kulabhulali yeklayenti le-Kafka. Nakuba lokhu kwenza izinto zisheshe ngendlela emangalisayo, kusho ukuthi uhlelo lokusebenza olungenalwazi lungalahlekelwa imilayezo uma inqubo yalo imiswa.
Njengenjwayelo, kunendlela yokwenza umsebenzi wokuthumela uthembeke kakhulu ngezindleko zokusebenza. Usayizi wale buffer ungasethwa ku-0, futhi intambo yesicelo sokuthumela izophoqeleka ukuthi ilinde kuze kuqedwe ukudluliselwa komlayezo kumthengisi, ngendlela elandelayo:
RecordMetadata metadata = producer.send(record).get();Okuningi mayelana nokufunda imilayezo
Ukufunda imilayezo kunezinye izinto eziyinkimbinkimbi okudingeka kucatshangwe ngazo. Ngokungafani ne-JMS API, engasebenzisa isilaleli somlayezo ekuphenduleni umlayezo, i Consumer Ama-polls e-Kafka kuphela. Ake sihlolisise indlela ipholi()esetshenziselwa le njongo:
ConsumerRecords < K, V > poll(long timeout);Inani elibuyayo lendlela isakhiwo sesitsha esiqukethe izinto eziningi irekhodi lomthengi kusuka kuma-partitions ambalwa okungenzeka. irekhodi lomthengi yona ngokwayo iyinto yesibambi yokubhanqwa kwenani elingukhiye enemethadatha ehlotshaniswayo, njengengxenye ethathwe kuyo.
Njengoba kuxoxiwe eSahlukweni sesi-2, kufanele sihlale sikhumbula ukuthi kwenzekani emilayezweni ngemva kokuba isicutshungulwe ngempumelelo noma engaphumelelanga, isibonelo, uma iklayenti lingakwazi ukucubungula umlayezo noma uma lihoxisa. Ku-JMS, lokhu kwasingathwa ngemodi yokuvuma. Umdayiseli uzosusa umlayezo ocutshungulwe ngempumelelo, noma aphinde alethe umlayezo ongahluziwe noma ongelona iqiniso (kucatshangwa ukuthi kusetshenziswe okwenziwayo).
I-Kafka isebenza ngokuhluke kakhulu. Imilayezo ayisuswa kumdayisi ngemva kokuhlolwa kokuhlolwa, futhi okwenzekayo ekuhlulekeni kuwumthwalo wekhodi yokuhlola iphutha ngokwayo.
Njengoba sesishilo, iqembu labathengi lihlotshaniswa ne-offset kulogi. Indawo yelogi ehlotshaniswa nalokhu kususwa ihambisana nomlayezo olandelayo ozokhishwa kuphendulwe kuwo ipholi(). Iphuzu ngesikhathi lapho lokhu kukhuphuka kwe-offset kuyisinqumo sokufunda.
Uma sibuyela kumodeli yokufunda okukhulunywe ngayo ngaphambili, ukucubungula umlayezo kunezigaba ezintathu:
- Buyisa umlayezo ozofundwa.
- Sicubungula umlayezo.
- Qinisekisa umlayezo.
Umthengi we-Kafka uza nenketho yokumisa vumela.auto.commit. Lesi isilungiselelo esizenzakalelayo esisetshenziswa njalo, njengoba kuvamile ngezilungiselelo eziqukethe igama elithi "okuzenzakalelayo".
Ngaphambi kwe-Kafka 0.10, iklayenti elisebenzisa le nketho lingathumela umlayezo wokugcina ofundwe ocingweni olulandelayo. ipholi() ngemva kokucubungula. Lokhu bekusho ukuthi noma yimiphi imilayezo esivele ilandiwe ingacutshungulwa futhi uma iklayenti seliyicubungulile kodwa yashabalala kungazelelwe ngaphambi kokushaya ucingo. ipholi(). Njengoba umthengisi engagcini noma yisiphi isimo mayelana nokuthi umlayezo ufundwe kangaki, umthengi olandelayo obuyisa lowo mlayezo ngeke azi ukuthi kukhona okubi okwenzekile. Lokhu kuziphatha bekungumsebenzi mbumbulu. I-offset yenziwa kuphela uma umlayezo ucutshungulwe ngempumelelo, kodwa uma iklayenti lihoxisiwe, umthengisi uzophinde athumele umlayezo ofanayo kwelinye iklayenti. Lokhu kuziphatha bekuhambisana nesiqinisekiso sokulethwa komlayezo "okungenani kanye".
Ku-Kafka 0.10, ikhodi yeklayenti ishintshiwe ukuze ukuzinikela kuqalwe ngezikhathi ezithile umtapo wezincwadi weklayenti, njengoba kumisiwe. auto.commit.interval.ms. Lokhu kuziphatha kuphakathi kwamamodi e-JMS AUTO_ACKNOWLEDGE kanye ne-DUPS_OK_ACKNOWLEDGE. Uma usebenzisa ukuzibophezela okuzenzakalelayo, imilayezo ingase izinikele kungakhathaliseki ukuthi icutshungulwe ngempela - lokhu kungenzeka esimweni somthengi ongasheshi. Uma umthengi ehoxisiwe, imilayezo izolandwa umthengi olandelayo, iqale endaweni yokuzibophezela, okungaholela kumlayezo ogejiwe. Kulokhu, i-Kafka ayizange ilahlekelwe yimilayezo, ikhodi yokufunda ayizange iyicubungule.
Le modi inesithembiso esifanayo nesenguqulo engu-0.9: imilayezo ingacutshungulwa, kodwa uma ihluleka, i-offset ingase ingazibophezeli, okungabangela ukulethwa kuphindwe kabili. Imilayezo eminingi oyilandayo uma usayinda ipholi(), iyanda le nkinga.
Njengoba kuxoxiwe kokuthi βUkufunda Imilayezo Elayiniβ ekhasini 21, ayikho into efana nokulethwa kanye kanye komlayezo ohlelweni lwemiyalezo lapho izindlela zokuhluleka zicatshangelwa.
E-Kafka, kunezindlela ezimbili zokwenza (ukwenza) i-offset (offset): ngokuzenzakalelayo futhi ngesandla. Kuzo zombili izimo, imilayezo ingacutshungulwa izikhathi eziningi uma umlayezo ucutshunguliwe kodwa wehluleka ngaphambi kokuzibophezela. Ungaphinda ukhethe ukungawucubunguli nhlobo umlayezo uma ukuzibophezela kwenzeke ngemuva futhi ikhodi yakho yaqedwa ngaphambi kokuthi icutshungulwe (mhlawumbe ku-Kafka 0.9 nangaphambilini).
Ungakwazi ukulawula inqubo yokuzibophezela okwenziwa ngesandla ku-API yomthengi we-Kafka ngokusetha ipharamitha vumela.auto.commit ukuqamba amanga kanye nokubiza ngokusobala enye yezindlela ezilandelayo:
void commitSync();
void commitAsync();Uma ufuna ukucubungula umlayezo "okungenani kanye", kufanele wenze iphutha ngesandla commitSync()ngokwenza lo myalo ngokushesha ngemva kokucubungula imilayezo.
Lezi zindlela azikuvumeli ukuthi imilayezo yamukelwe ngaphambi kokuthi icutshungulwe, kodwa azenzi lutho ukuqeda ukubambezeleka kokucubungula okungaba khona kuyilapho kunikeza ukubonakala kokuthi iyenziwa. Akukho ukuthenga e-Kafka. Iklayenti alinawo amandla okwenza lokhu okulandelayo:
- Buyisela emuva ngokuzenzakalelayo umlayezo omgunyathi. Abathengi ngokwabo kufanele baphathe okuhlukile okuvela ekulayishweni okuyinkinga okuyinkinga kanye nokuphelelwa yisikhathi, njengoba bengenakuthembela kumthengisi ukuthi aphinde alethe imilayezo.
- Thumela imilayezo ezihlokweni eziningi ekusebenzeni okukodwa kwe-athomu. Njengoba sizobona maduzane, ukulawula izihloko ezihlukene kanye nokuhlukaniswa kungahlala emishinini ehlukene kuqoqo le-Kafka elingahlanganisi ukuthengiswa uma kuthunyelwa. Ngesikhathi sokubhala lokhu, omunye umsebenzi wenziwe ukwenza lokhu kwenzeke nge-KIP-98.
- Hlobanisa ukufunda umlayezo owodwa usuka kwesinye isihloko nokuthumela omunye umlayezo kwesinye isihloko. Futhi, ukwakheka kweKafka kuncike emishinini eminingi ezimele esebenza njengebhasi elilodwa futhi akukho mzamo owenziwayo wokufihla lokhu. Isibonelo, azikho izingxenye ze-API ezingakuvumela ukuthi uxhume umthengi ΠΈ Umkhiqizi ekwenziweni. Ku-JMS, lokhu kunikezwa into Sessionokuvela kuzo MessageProducers ΠΈ MessageConsumers.
Uma singakwazi ukuthembela emisebenzini, singanikeza kanjani ama-semantics eduze kwalawo ahlinzekwa amasistimu emiyalezo evamile?
Uma kunethuba lokuthi i-offset yomthengi ingase ikhule ngaphambi kokuthi umlayezo ucutshungulwe, njengalapho umthengi ephahlazeka, khona-ke umthengi akanayo indlela yokwazi ukuthi iqembu lakhe labathengi liphuthelwe yini umlayezo lapho abelwe ukwahlukanisa. Ngakho elinye isu liwukubuyisela emuva i-offset endaweni yangaphambilini. I-Kafka Consumer API inikeza izindlela ezilandelayo zalokhu:
void seek(TopicPartition partition, long offset);
void seekToBeginning(Collection < TopicPartition > partitions);
Indlela funa () ingasetshenziswa ngendlela
offsetsForTimes(Imephu izitembu zesikhathiUsesho) ukuhlehlisela esimweni endaweni ethile esikhathini esidlule.
Ngokusobala, ukusebenzisa le ndlela kusho ukuthi maningi amathuba okuthi eminye imilayezo eke yacutshungulwa ngaphambilini izofundwa futhi isetshenzwe futhi. Ukuze sigweme lokhu, singasebenzisa ukufunda ngokungenangqondo, njengoba kuchazwe eSahlukweni 4, ukuze silandelele imilayezo ebukwe ngaphambilini futhi sisuse izimpinda.
Kungenjalo, ikhodi yakho yomthengi ingagcinwa ilula, inqobo nje uma ukulahlekelwa umlayezo noma ukuphindwaphindwa kwamukelekile. Uma sicabangela izimo zokusebenzisa i-Kafka evame ukusetshenziswa kuzo, njengokuphatha imicimbi yefayela lokungena, amamethrikhi, ukulandelela ngokuchofoza, njll., siyaqonda ukuthi ukulahlekelwa kwemilayezo ngayinye mancane amathuba okuba kube nomthelela omkhulu kuzinhlelo zokusebenza ezizungezile. Ezimweni ezinjalo, amanani azenzakalelayo amukeleka ngokuphelele. Ngakolunye uhlangothi, uma isicelo sakho sidinga ukuthumela izinkokhelo, kufanele unakekele ngokucophelela umlayezo ngamunye. Konke kuncike kumongo.
Ukuqaphela komuntu siqu kubonisa ukuthi njengoba ukushuba kwemilayezo kukhula, inani lomlayezo ngamunye liyancipha. Imilayezo emikhulu ivama ukuba yigugu uma ibukwa ngefomu elihlanganisiwe.
Ukutholakala Okuphezulu
Indlela ye-Kafka yokutholakala okuphezulu ihluke kakhulu endleleni ye-ActiveMQ. I-Kafka yakhelwe ukuzungeza amaqoqo okuphuma lapho zonke izimo zabathengi zithola futhi zisabalalisa imilayezo ngesikhathi esisodwa.
Iqoqo le-Kafka liqukethe izimo zomthengisi eziningi ezisebenza kumaseva ahlukene. I-Kafka yakhelwe ukuthi isebenze ku-hardware evamile ezimele, lapho inodi ngayinye inesitoreji sayo esizinikele. Ukusetshenziswa kwesitoreji esinamathiselwe kunethiwekhi (i-SAN) akunconyiwe ngoba amanodi ekhomputha amaningi angaqhudelana ngesikhathi.Π«e izikhawu zokugcina futhi adale ukungqubuzana.
I-Kafka njalo uhlelo. Abasebenzisi abaningi be-Kafka abakhulu abakaze bavale amaqoqo abo futhi isofthiwe ihlala ibuyekeza ngokuqalisa kabusha okulandelanayo. Lokhu kufinyelelwa ngokuqinisekisa ukuhambisana nenguqulo yangaphambilini yemilayezo nokusebenzisana phakathi kwabathengi.
Abathengisi baxhumeke kuqoqo leseva , esebenza njengerejista yedatha yokumisa futhi isetshenziselwa ukuxhumanisa izindima zomthengisi ngamunye. I-ZooKeeper ngokwayo iwuhlelo olusabalalisiwe olunikeza ukutholakala okuphezulu ngokuphindaphinda kolwazi ngokusungula ikhoramu.
Esimeni esiyisisekelo, isihloko sidalwa kuqoqo le-Kafka elinezici ezilandelayo:
- Inani lama-partitions. Njengoba kuxoxiwe ngaphambili, inani eliqondile elisetshenziswe lapha lincike ezingeni elifunwayo lokufunda okufanayo.
- Isici esiphindaphindayo (isici) sinquma ukuthi zingaki izimo zomthengisi kuqoqo okufanele ziqukathe amalogi alokhu kuhlukaniswa.
Isebenzisa i-ZooKeepers ukuze ihlanganise, i-Kafka izama ukusabalalisa ngokufanelekile izingxenye ezintsha phakathi kwabathengi kuqoqo. Lokhu kwenziwa ngesenzakalo esisodwa esisebenza njengoMlawuli.
Ngesikhathi sokusebenza ngokwehlukana kwesihloko ngasinye Isilawuli yabela umthengisi izindima umholi (umholi, inkosi, umethuli) kanye abalandeli (abalandeli, izigqila, abangaphansi). Umthengisi, osebenza njengomholi walesi sigaba, unesibopho sokwamukela yonke imilayezo ethunyelwe kuyo ngabakhiqizi futhi asabalalise imilayezo kubathengi. Uma imilayezo ithunyelwa ekuhlukaniseni isihloko, iphindaphindwa kuwo wonke ama-broker nodes asebenza njengabalandeli baleyo ngxenye. I-node ngayinye equkethe izingodo zokuhlukanisa ibizwa ikhophi. Umdayisi angasebenza njengomholi kwezinye izingxenye futhi njengomlandeli kwabanye.
Kubizwa umlandeli oqukethe yonke imiyalezo ephethwe umholi ikhophi evumelanisiwe (isithombe esifana nesesimeni sokuvumelanisa, isifaniso esivunyelanisiwe). Uma umthengisi osebenza njengomholi wesigaba ehla, noma yimuphi umthengisi osesikhathini samanje noma ovumelaniswe kulokho kuhlukaniswa angathatha indima yomholi. Kuyidizayini esimeme ngendlela emangalisayo.
Ingxenye yokucushwa komkhiqizi ipharamitha ama-acks, enquma ukuthi zingaki izifaniso okufanele zivume (zivume) ukuthi ziwatholile umlayezo ngaphambi kokuthi uchungechunge lwesicelo luqhubeke nokuthumela: 0, 1, noma konke. Uma kusethelwe ku konke, khona-ke uma umlayezo wamukelwe, umholi uzothumela isiqinisekiso kumkhiqizi ngokushesha nje lapho ethola iziqinisekiso (iziqinisekiso) zerekhodi ezivela kuzimpawu ezimbalwa (kuhlanganise nalo) ezichazwe ukulungiselelwa kwesihloko. min.insync.replicas (okuzenzakalelayo 1). Uma umlayezo ungeke ukwazi ukuphindwaphindwa ngempumelelo, umkhiqizi uzophonsa okuhlukile kwesicelo (NotEnoughReplicas noma NotEnoughReplicasAfterAppend).
Ukucushwa okujwayelekile kudala isihloko esinesici sokuphindaphinda esingu-3 (umholi ongu-1, abalandeli abangu-2 ngengxenye ngayinye) kanye nepharamitha min.insync.replicas Kusethwe ku-2. Kulesi simo, iqoqo lizovumela omunye wabathengisi abaphethe ukuhlukaniswa kwesihloko ukuthi ehle ngaphandle kokuthikameza izinhlelo zokusebenza zeklayenti.
Lokhu kusibuyisela ekuhwebeni osekujwayelekile phakathi kokusebenza nokuthembeka. Ukuphindaphinda kwenzeka ngezindleko zesikhathi esengeziwe sokulinda sokuqinisekisa (ukubonga) okuvela kubalandeli. Nakuba, ngenxa yokuthi isebenza ngokufana, ukuphindaphinda okungenani ama-node amathathu kunokusebenza okufanayo nokubili (ukuziba ukwanda kokusetshenziswa komkhawulokudonsa wenethiwekhi).
Ngokusebenzisa lolu hlelo lokuphindaphinda, i-Kafka igwema ngobuhlakani isidingo sokubhala umlayezo ngamunye kudiski ngokusebenza. vumelanisa(). Umlayezo ngamunye othunyelwe umkhiqizi uzobhalwa kulogi yokuhlukanisa, kodwa njengoba kuxoxwe ngakho eSahlukweni sesi-2, ukubhala ifayela ekuqaleni kwenziwa kusigcinalwazi sesistimu yokusebenza. Uma lo mlayezo uphindwa kwesinye isenzakalo se-Kafka futhi usenkumbulweni yawo, ukulahlekelwa umholi akusho ukuthi umlayezo ngokwawo ulahlekile - ungathathwa ikhophi evumelanisiwe.
Ukwenqaba ukwenza umsebenzi vumelanisa() kusho ukuthi i-Kafka ingathola imilayezo ngokushesha njengoba ingayibhala ngekhanda. Ngakolunye uhlangothi, uma isikhathi eside ungagwema ukufaka inkumbulo kudiski, kuba ngcono. Ngalesi sizathu, akuvamile ukuthi abathengisi be-Kafka banikezwe inkumbulo engu-64 GB noma ngaphezulu. Lokhu kusetshenziswa kwenkumbulo kusho ukuthi isenzakalo esisodwa se-Kafka singasebenza kalula ngesivinini esiphindwe kazinkulungwane eziningi kunomthengisi wemiyalezo ovamile.
I-Kafka nayo ingalungiselelwa ukusebenzisa ukusebenza vumelanisa() kumaphakheji womlayezo. Njengoba yonke into e-Kafka igxile kumaphakheji, empeleni isebenza kahle ezimweni eziningi zokusetshenziswa futhi iyithuluzi eliwusizo kubasebenzisi abadinga iziqinisekiso eziqine kakhulu. Ukusebenza okuningi okumsulwa kwe-Kafka kuvela emilayezweni ethunyelwa kumdayisi njengamaphakethe nokuthi le milayezo ifundwa kumthengisi ngamabhulokhi alandelanayo kusetshenziswa. imisebenzi (imisebenzi lapho umsebenzi wokukopisha idatha usuka endaweni ethile yememori uye kwenye ungenziwanga). Lokhu kokugcina kuwumsebenzi omkhulu kanye nenzuzo yensiza futhi kungenzeka kuphela ngokusebenzisa isakhiwo sedatha yelogi eyisisekelo esichaza uhlelo lokuhlukanisa.
Ukusebenza okungcono kakhulu kungenzeka kuqoqo le-Kafka kunomthengisi oyedwa we-Kafka, ngoba izingxenye zesihloko zingafinyelela kumishini eminingi ehlukene.
Imiphumela
Kulesi sahluko, sibheke ukuthi i-architecture ye-Kafka ibucabanga kanjani kabusha ubudlelwano phakathi kwamakhasimende nabathengi ukuze kuhlinzekwe ngepayipi lemiyalezo eliqine ngendlela emangalisayo, elinomphumela ophindwe kaningi kunowomthengisi wemiyalezo ojwayelekile. Sixoxile ngokusebenza ekusebenzisayo ukufeza lokhu futhi sabheka kafushane ukwakheka kwezinhlelo zokusebenza ezinikeza lokhu kusebenza. Esahlukweni esilandelayo, sizobheka izinkinga ezivamile izinhlelo zokusebenza ezisuselwe emilayezweni okudingeka zixazululwe futhi sixoxe ngamasu okubhekana nazo. Sizophetha isahluko ngokuchaza indlela yokukhuluma ngobuchwepheshe bemiyalezo ngokuvamile ukuze uhlole ukufaneleka kwabo ezimweni zakho zokusebenzisa.
Ingxenye ehunyushwe ngaphambilini:
Ukuhumusha kwenziwe:
Kuzoqhubeka ...
Abasebenzisi ababhalisiwe kuphela abangabamba iqhaza kuhlolovo. , wamukelekile.
Ingabe i-Kafka isetshenziswa enhlanganweni yakho?
-
Yebo
-
No
-
Kusetshenziswe ngaphambilini, hhayi manje
-
Sihlela ukusebenzisa
Bangu-38 abasebenzisi abavotile. Abasebenzisi abangu-8 bagobile.
Source: www.habr.com