🥇Isipiliyoni sokuthuthukisa isevisi Yethuluzi Lokubuyisela Imali nge-API engavumelanisi ku-Kafka

Yini engaphoqa inkampani enkulu kangaka njengeLamoda, enenqubo ehlelekile kanye nenqwaba yezinsizakalo ezixhumene, ukuthi iguqule ngokuphawulekayo indlela yayo? Ukugqugquzeleka kungahluka ngokuphelele: ukusuka emthethweni kuya esifisweni sokuhlola okukhona kubo bonke abahleli bohlelo.

Kodwa lokhu akusho ukuthi awukwazi ukuthembela ezinzuzweni ezengeziwe. U-Sergey Zaika uzokutshela ukuthi yini ngempela ongayinqoba uma usebenzisa i-API eqhutshwa imicimbi ku-Kafka (ambalwa). Kuzophinde kukhulunywe ngezithombe ezinkulu kanye nokutholwe okuthakazelisayo - ukuhlolwa ngeke kwenzeke ngaphandle kwakho.

Umshwana wokuzihlangula: Lesi sihloko sisekelwe ezintweni ezivela kumhlangano u-Sergey awubamba ngoNovemba 2018 ku-HighLoad++. Ulwazi olubukhoma lukaLamoda lokusebenza no-Kafka luhehe abalaleli ngaphezu kweminye imibiko ohlelweni. Sicabanga ukuthi lesi isibonelo esihle kakhulu sokuthi ungakwazi futhi kufanele uhlale uthola abantu abanomqondo ofanayo, futhi abahleli be-HighLoad++ bazoqhubeka nokuzama ukudala isimo esivumela lokhu.

Mayelana nenqubo

I-Lamoda iyinkundla enkulu ye-e-commerce enesikhungo sayo sokuxhumana, insizakalo yokulethwa kwezidingo (kanye namanxusa amaningi), isitudiyo sezithombe, indawo yokugcina impahla enkulu, futhi konke lokhu kusebenza ngesoftware yayo. Kukhona inqwaba yezindlela zokukhokha, ozakwethu be-b2b abangasebenzisa ezinye noma zonke lezi zinsizakalo futhi bafuna ukwazi ulwazi lwakamuva ngemikhiqizo yabo. Ngaphezu kwalokho, iLamoda isebenza emazweni amathathu ngaphandle kweRussian Federation futhi yonke into ihluke kancane lapho. Sekukonke, cishe kunezindlela ezingaphezu kwekhulu zokumisa i-oda elisha, okumele licutshungulwe ngendlela yalo. Konke lokhu kusebenza ngosizo lwezinsizakalo eziningi kwesinye isikhathi ezixhumana ngezindlela ezingabonakali. Kukhona nesistimu emaphakathi umsebenzi wayo oyinhloko izimo zokuhleleka. Simbiza ngoBOB, ngisebenza naye.

Ithuluzi Lokubuyisela Nge-API eqhutshwa imicimbi

Igama elithi ukuqhutshwa kwezehlakalo liyi-hackneyed; ngokuqhubekayo sizochaza kabanzi ukuthi kuqondweni ngalokhu. Ngizoqala ngomongo lapho sinqume ukuzama indlela ye-API eqhutshwa imicimbi e-Kafka.

Kunoma isiphi isitolo, ngaphezu kwama-oda abathengi abawakhokhayo, kunezikhathi lapho isitolo sidinga ukubuyisela imali ngenxa yokuthi umkhiqizo awuhambisani nekhasimende. Lena inqubo emfushane uma kuqhathaniswa: sicacisa ulwazi, uma kunesidingo, bese sidlulisela imali.

Kodwa ukubuyisela kuye kwaba nzima kakhulu ngenxa yezinguquko emthethweni, futhi kudingeke ukuthi sisebenzise i-microservice ehlukile kuyo.

Ukugqugquzela kwethu:

Umthetho FZ-54 - ngamafuphi, umthetho udinga ukuthi kubikwe ehhovisi lentela mayelana nakho konke ukuthengiselana ngemali, kungaba imbuyiselo noma irisidi, ngaphakathi kwe-SLA emfushane kakhulu yemizuzu embalwa. Thina, njengenkampani ye-e-commerce, senza imisebenzi eminingi. Ngobuchwepheshe, lokhu kusho isibopho esisha (ngakho-ke isevisi entsha) kanye nokuthuthukiswa kwazo zonke izinhlelo ezihilelekile.
I-BOB ihlukaniswe iphrojekthi yangaphakathi yenkampani yokukhulula i-BOB enanini elikhulu lemithwalo yemfanelo engeyona eyinhloko futhi inciphise ubunzima bayo jikelele.

Lo mdwebo ukhombisa izinhlelo ezinkulu zeLamoda. Manje iningi labo selingaphezulu umlaza wama-microservices angu-5-10 azungeze i-monolith enciphayo. Zikhula kancane kancane, kodwa sizama ukuwenza abe mancane, ngoba ukuhambisa isiqeshana esikhethiwe phakathi kuyethusa - asikwazi ukusivumela ukuba siwe. Siphoqeleka ukuthi sigcine konke ukushintshana (imicibisholo) futhi sicabangele iqiniso lokuthi noma iyiphi yazo ingase ingatholakali.

I-BOB iphinde ibe nokuhwebelana okuningi: izinhlelo zokukhokha, izinhlelo zokuletha, izinhlelo zezaziso, njll.

Ngokwezobuchwepheshe i-BOB ithi:

~150k imigqa yekhodi + ~100k imigqa yokuhlola;
php7.2 + Zend 1 & Symfony Izingxenye 3;
>100 APIs & ~50 ukuhlanganiswa okuphumayo;
Amazwe ama-4 anomqondo wawo webhizinisi.

Ukuthumela i-BOB kuyabiza futhi kubuhlungu, inani lekhodi nezinkinga elizixazululayo liwukuthi akekho ongabeka konke emakhanda abo. Ngokuvamile, kunezizathu eziningi zokukwenza kube lula.

Inqubo Yokubuyisela

Ekuqaleni, izinhlelo ezimbili zihilelekile kule nqubo: I-BOB kanye Nokukhokha. Manje kuvela ezinye ezimbili:

Isevisi yeFiscalization, ezobhekana nezinkinga ngokufakwa kwezimali kanye nokuxhumana nezinsizakalo zangaphandle.
Ithuluzi Lokubuyisela, eliqukethe nje ukuhwebelana okusha ukuze lingakhulisi i-BOB.

Manje inqubo ibukeka kanje:

I-BOB ithola isicelo sokubuyiselwa imali.
U-BOB ukhuluma ngaleli Thuluzi Lokubuyisela Imali.
Ithuluzi Lokubuyisela litshela Inkokhelo: “Buyisela imali.”
Inkokhelo ibuyisela imali.
Ithuluzi lokubuyisela imali kanye ne-BOB zivumelanisa izimo, ngoba okwamanje bobabili bayayidinga. Asikakalungeli ukushintshela ngokuphelele Ethuluzini Lokubuyisela Imali, njengoba i-BOB ine-UI, ibika ukubalwa kwezimali, futhi ngokuvamile idatha eminingi engakwazi ukudluliselwa kalula. Kufanele uhlale ezihlalweni ezimbili.
Isicelo sokuxhaswa kwezimali siyaphela.

Ngenxa yalokho, senze uhlobo lwebhasi lomcimbi e-Kafka - ibhasi lomcimbi, lapho konke kwaqala khona. Hurray, manje sinephuzu elilodwa lokwehluleka (ukubhuqa).

Izinzuzo nezingozi zisobala kakhulu. Senze ibhasi, okusho ukuthi manje zonke izinkonzo zincike kulo. Lokhu kwenza idizayini ibe lula, kodwa yethula iphuzu elilodwa lokwehluleka ohlelweni. I-Kafka izophahlazeka, inqubo izoma.

Iyini i-API eqhutshwa imicimbi

Impendulo enhle yalo mbuzo isembikweni kaMartin Fowler (GOTO 2017) "Izincazelo Eziningi Zezakhiwo Eziqhutshwa Umcimbi".

Kafushane esikwenzile:

Qeda konke ukuhwebelana okungavumelani nge isitoreji semicimbi. Esikhundleni sokwazisa wonke umthengi onentshisekelo mayelana nokushintsha kwesimo kunethiwekhi, sibhala umcimbi mayelana nokushintsha kwesimo endaweni yokulondoloza emaphakathi, futhi abathengi abanentshisekelo esihlokweni bafunda yonke into evela lapho.
Umcimbi kuleli cala uyisaziso (izaziso) ukuthi kukhona okushintshile ndawana thize. Isibonelo, isimo se-oda sishintshile. Umthengi onentshisekelo kwenye idatha ehambisana noshintsho lwesimo olungafakiwe kusaziso angathola isimo saso ngokwakhe.
Inketho enkulu wukuthola umcimbi ogcwele, ukudluliswa kombuso, lapho umcimbi uqukethe lonke ulwazi oludingekayo ukuze lucutshungulwe: lapho luvela khona nokuthi lwaya kusiphi isimo, ukuthi idatha yashintsha kanjani, njll. Umbuzo kuphela ukuthi kungenzeka yini kanye nenani lolwazi ongakwazi ukuligcina.

Njengengxenye yokwethulwa kweThuluzi Lokubuyisela Imali, sisebenzise inketho yesithathu. Lokhu kucutshungulwa komcimbi okwenziwe lula njengoba besingekho isidingo sokukhipha imininingwane enemininingwane, futhi kususe isimo lapho umcimbi omusha ngamunye udala ukucacisa kokuthola izicelo ezivela kubathengi.

Isevisi Yethuluzi Lokubuyisela ayilayishiwe, ngakho-ke i-Kafka kukhona ukunambitheka kwepeni ngaphezu kwesidingo. Angicabangi ukuthi uma isevisi yokubuyisela imali iba iphrojekthi enomthwalo omkhulu, ibhizinisi lingajabula.

I-Async exchange AS IS

Ekushintshanisweni kwe-asynchronous, umnyango we-PHP uvamise ukusebenzisa i-RabbitMQ. Siqoqe idatha yesicelo, sayibeka emgqeni, futhi umthengi wesevisi efanayo wayifunda wayithumela (noma akazange ayithumele). Ku-API ngokwayo, iLamoda isebenzisa i-Swagger ngenkuthalo. Sakha i-API, siyichaza nge-Swagger, futhi sikhiqize iklayenti nekhodi yeseva. Siphinde sisebenzisa i-JSON RPC 2.0 ethuthukisiwe kancane.

Kwezinye izindawo kusetshenziswa amabhasi e-ESB, amanye ahlala ku-activeMQ, kodwa, ngokuvamile, RabbitMQ - ejwayelekile.

I-Async exchange TO BE

Uma uklama ukushintshana ngebhasi lemicimbi, isifaniso singalandelelwa. Sichaza ngokufanayo ukushintshana kwedatha kwesikhathi esizayo ngezincazelo zesakhiwo somcimbi. Ifomethi ye-yaml, bekufanele sizenzele thina ukukhiqiza amakhodi, ijeneretha idala ama-DTO ngokuya ngemininingwane futhi ifundisa amakhasimende namaseva ukusebenza nawo. Isizukulwane siya ezilimini ezimbili - golang kanye php. Lokhu kusiza ukugcina imitapo yolwazi ingashintshi. Ijeneretha ibhalwe nge-golang, yingakho yathola igama elithi gogi.

Ukuthola umcimbi ku-Kafka kuyinto ejwayelekile. Kukhona isisombululo esivela kunguqulo yebhizinisi eyinhloko ye-Kafka Confluent, sikhona nakadi, isixazululo esiphuma esizindeni sethu abafowethu Zalando. Eyethu ugqozi lokuqala nge-vanilla Kafka - lokhu kusho ukushiya isisombululo sikhululekile kuze kube yilapho ekugcineni sinquma ukuthi sizosisebenzisa yonke indawo, futhi sizishiyele indawo yokuhlela nokwenza ngcono: sifuna ukwesekwa kwethu. I-JSON RPC 2.0, amajeneretha ezilimi ezimbili futhi ake sibone ukuthi yini enye.

Kuyaxaka ukuthi ngisho esimweni esinjalo esijabulisayo, lapho kunebhizinisi elicishe lifane, iZalando, elenza isisombululo esicishe sifane, ngeke sikwazi ukulisebenzisa ngempumelelo.

Iphethini yezakhiwo ekuqalisweni imi kanje: sifunda ngokuqondile e-Kafka, kodwa sibhala kuphela ngemicimbi-ibhasi. Kuningi okulungele ukufundwa e-Kafka: abathengisi, ababhalansi, futhi isilungele ukukalwa okuvundlile, bengifuna ukukugcina lokhu. Besifuna ukuqedela ukuqopha ngebhasi elilodwa le-Gateway aka Events-bus, yingakho.

Imicimbi-ibhasi

Noma ibhasi lomcimbi. Lokhu kumane kuyisango elingenasisekelo le-http, elithatha izindima ezimbalwa ezibalulekile:

Ukukhiqiza Ukuqinisekisa - sihlola ukuthi izehlakalo ziyahlangabezana yini nemibandela yethu.
Uhlelo oluyinhloko lomcimbi, okungukuthi, lena yisistimu eyinhloko futhi kuphela enkampanini ephendula umbuzo wokuthi yiziphi izenzakalo lapho izakhiwo zibhekwa njengezivumelekile. Ukuqinisekisa kufaka phakathi izinhlobo zedatha kanye nama-enum ukuze ucacise okuqukethwe.
Umsebenzi we-Hash ukuze uhlukanise - isakhiwo somlayezo we-Kafka singukhiye-value futhi kusetshenziswa i-hashi yokhiye kubalwa ukuthi ibekwe kuphi.

Kungani

Sisebenza enkampanini enkulu enenqubo ehlelekile. Kungani ushintsha noma yini? Lokhu ukuhlola, futhi silindele ukuvuna izinzuzo ezimbalwa.

1:n+1 ukushintshanisa (okukodwa kuya kokuningi)

I-Kafka yenza kube lula kakhulu ukuxhuma abathengi abasha ku-API.

Ake sithi unohla lwemibhalo okudingeka uhlale unolwazi lwakamuva kumasistimu amaningana ngesikhathi esisodwa (nakwamanye amasha). Ngaphambilini, sisungule inqwaba esebenzisa i-set-API, futhi isistimu eyinhloko yaziswa ngamakheli omthengi. Manje uhlelo oluyinhloko luthumela izibuyekezo esihlokweni, futhi wonke umuntu onentshisekelo uyayifunda. Kuvele isistimu entsha - siyibhalisele isihloko. Yebo, futhi inqwaba, kodwa ilula.

Endabeni yethuluzi lokubuyisela imali, okuwucezu lwe-BOB, kulula ngathi ukuzigcina zivumelanisiwe nge-Kafka. Inkokhelo ithi imali ibuyisiwe: BOB, RT bathola ngalokhu, bashintsha isimo sabo, I-Fiscalization Service yathola ngalokhu futhi yakhipha isheke.

Sinezinhlelo zokudala Isevisi Yezaziso Ehlanganisiwe engazisa iklayenti mayelana nezindaba eziphathelene ne-oda/izimbuyiselo zalo. Manje lesi sibopho sesabalele phakathi kwezinhlelo. Kuyokwanela ukuthi sifundise Isevisi Yezaziso ukuze sibambe ulwazi olufanele oluvela e-Kafka futhi siphendule kulo (futhi sikhubaze lezi zaziso kwamanye amasistimu). Akukho ukushintshaniswa okusha okuqondile okuzodingeka.

Kuqhutshwa idatha

Ulwazi phakathi kwamasistimu luba sobala - noma ngabe unaliphi “ibhizinisi elinegazi” futhi kungakhathaliseki ukuthi ukusilela emuva kwakho kukhulu kangakanani. I-Lamoda inomnyango we-Data Analytics oqoqa idatha kumasistimu futhi uyifake efomini elisebenziseka kabusha, kokubili ibhizinisi kanye nezinhlelo ezihlakaniphile. I-Kafka ikuvumela ukuthi ubanikeze ngokushesha idatha eningi futhi ugcine lolo lwazi lugeleza lusesikhathini samanje.

Ilogi yokuphindaphinda

Imilayezo ayinyamalala ngemva kokufundwa, njengakuRabbitMQ. Uma umcimbi uqukethe ulwazi olwanele lokucutshungulwa, sinomlando wezinguquko zakamuva entweni, futhi, uma sifisa, nekhono lokusebenzisa lezi zinguquko.

Isikhathi sokulondoloza selogi yokuphindaphinda sincike ekushubeni kokubhala kulesi sihloko; I-Kafka ikuvumela ukuthi ubeke imikhawulo kalula ngesikhathi sokugcina kanye nevolumu yedatha. Ezihlokweni ezijulile, kubalulekile ukuthi bonke abathengi babe nesikhathi sokufunda ulwazi ngaphambi kokuthi lunyamalale, ngisho nalapho kungasebenzi isikhathi esifushane. Ngokuvamile kungenzeka ukugcina idatha amayunithi wezinsuku, okwanele ukusekela.

Okulandelayo, ukulandisa okuncane kwemibhalo, kulabo abangajwayelene ne-Kafka (isithombe sivela emibhalweni)

I-AMQP inolayini: sibhalela umthengi imilayezo emgqeni. Ngokuvamile, umugqa owodwa ucutshungulwa isistimu eyodwa enomqondo ofanayo webhizinisi. Uma udinga ukwazisa amasistimu ambalwa, ungafundisa uhlelo lokusebenza ukubhalela imigqa eminingana noma ulungiselele ukushintshana ngomshini we-fanout, ozikopisha wona ngokwawo.

I-Kafka ine-abstraction efanayo isihloko, lapho ubhala khona imiyalezo, kodwa ayipheli ngemva kokuyifunda. Ngokuzenzakalelayo, uma uxhuma ku-Kafka, uthola yonke imilayezo futhi unenketho yokulondoloza lapho ugcine khona. Okusho ukuthi, ufunda ngokulandelana, awukwazi ukumaka umlayezo njengofundiwe, kodwa gcina i-id ongaqhubeka nokufunda kuyo. I-Id olungise kuyo ibizwa nge-offset, futhi indlela yokusebenza i-offset.

Ngokuvumelana nalokho, i-logic ehlukene ingasetshenziswa. Isibonelo, sine-BOB ezimweni ezi-4 emazweni ahlukene - iLamoda iseRussia, Kazakhstan, Ukraine, Belarus. Njengoba zisetshenziswe ngokuhlukana, zinokucushwa okuhluke kancane kanye nengqondo yebhizinisi yazo. Sikhombisa emlayezweni ukuthi ibhekisa kuliphi izwe. Umthengi ngamunye we-BOB ezweni ngalinye ufunda ne-groupId ehlukile, futhi uma umlayezo ungasebenzi kubo, bayaweqa, i.e. ngokushesha yenza offset +1. Uma isihloko esifanayo sifundwa Isevisi yethu Yokukhokha, khona-ke ikwenza ngeqembu elihlukile, ngakho-ke ama-offset awaphambani.

Izimfuneko zomcimbi:

Ukuphelela kwedatha. Ngingathanda ukuthi umcimbi ube nedatha eyanele ukuze ukwazi ukucutshungulwa.

Ubuqotho. Sithumela Emicimbini-ngebhasi ukuqinisekiswa kokuthi umcimbi uyahambisana futhi ungawucubungula.
I-oda libalulekile. Endabeni yokubuya, siyaphoqeleka ukuthi sisebenze ngomlando. Ngezaziso, i-oda alibalulekile, uma kuyizaziso ezifanayo, i-imeyili izofana kungakhathaliseki ukuthi iyiphi i-oda efike kuqala. Endabeni yokubuyiselwa kwemali, kunenqubo ecacile; uma sishintsha i-oda, okuhlukile kuzovela, ukubuyiselwa kwemali ngeke kudalwe noma kucutshungulwe - sizogcina sisesimweni esihlukile.
Ukuvumelana. Sinesitolo, futhi manje sidala imicimbi esikhundleni se-API. Sidinga indlela yokudlulisa ngokushesha futhi ngokushibhile ulwazi mayelana nemicimbi emisha nezinguquko kwezikhona kakade kumasevisi ethu. Lokhu kufinyelelwa kusetshenziswa ukucaciswa okujwayelekile endaweni ehlukile ye-git kanye namakhodi akhiqiza amakhodi. Ngakho-ke, amaklayenti namaseva kumasevisi ahlukene ayahlanganiswa.

I-Kafka eLamoda

Sinokufakwa kwe-Kafka okuthathu:

Izingodo;
I-R&D;
Imicimbi-ibhasi.

Namuhla sikhuluma ngephuzu lokugcina kuphela. Emicimbini-ibhasi, asinakho ukufaka okukhulu kakhulu - ama-broker ama-3 (amaseva) nezihloko ezingama-27 kuphela. Njengomthetho, isihloko esisodwa inqubo eyodwa. Kodwa leli yiphuzu elicashile, futhi sizolithinta manje.

Ngenhla igrafu ye-rps. Inqubo yokubuyisela imakwe ngomugqa oluhlaza (yebo, lowo oseksisi X), futhi ulayini ophinki uyinqubo yokubuyekeza okuqukethwe.

Ikhathalogi ye-Lamoda iqukethe izigidi zemikhiqizo, futhi idatha ivuselelwa ngaso sonke isikhathi. Amanye amaqoqo ayaphuma emfashinini, akhishwe amasha ukuze athathelwe indawo, futhi amamodeli amasha ahlala evela kukhathalogi. Sizama ukubikezela ukuthi yini ezothakazelisa kumakhasimende ethu kusasa, ngakho sihlala sithenga izinto ezintsha, sizithwebula futhi sibuyekeza ikesi lokubonisa.

Iziqongo ezibomvana ziyizibuyekezo zomkhiqizo, okungukuthi, izinguquko emikhiqizweni. Kuyabonakala ukuthi abafana bathatha izithombe, bathatha izithombe, baphinde futhi! - Layisha iphakethe lemicimbi.

Imicimbi yeLamoda isebenzisa amacala

Sisebenzisa i-architecture eyakhiwe kule misebenzi elandelayo:

Buyisela ukulandelela isimo: ukubizelwa esenzweni nokulandelela isimo kuwo wonke amasistimu ahililekile. Inkokhelo, izimo, ukufakwa kwezimali, izaziso. Lapha sahlola indlela, senza amathuluzi, saqoqa zonke izimbungulu, sabhala imibhalo futhi satshela ozakwethu ukuthi isetshenziswa kanjani.
Ibuyekeza amakhadi omkhiqizo: ukumisa, imethadatha, izici. Isistimu eyodwa ifundeka (ebonisa), kanye nokubhala okuningana.
I-imeyili, i-push kanye ne-sms: i-oda seliqoqiwe, i-oda selifikile, ukubuya kwamukelwe, njll., ziningi.
Isitoko, ukuvuselelwa kweshede - Ukuvuselelwa kwenani lezinto, izinombolo nje: ukufika endaweni yokugcina impahla, ukubuya. Kudingeka ukuthi wonke amasistimu ahlotshaniswa nokugcinwa kwezimpahla asebenze nedatha yamanje kakhulu. Njengamanje, uhlelo lokuvuselela isitoko luyinkimbinkimbi kakhulu; i-Kafka izokwenza lula.
Ukuhlaziya Data (Umnyango we-R&D), amathuluzi e-ML, izibalo, izibalo. Sifuna ulwazi lube sobala - i-Kafka ifaneleka kahle kulokhu.

Manje ingxenye ethakaseleka kakhulu mayelana namaqhubu amakhulu kanye nokutholwa okuthakazelisayo okwenzeke ezinyangeni eziyisithupha ezedlule.

Izinkinga zokuklama

Ake sithi sifuna ukwenza into entsha - isibonelo, ukudlulisa yonke inqubo yokulethwa e-Kafka. Manje ingxenye yenqubo isetshenziswa ku-Order Processing ku-BOB. Kukhona imodeli yesimo ngemuva kokudluliselwa kwe-oda kusevisi yokudiliva, ukuhambisa endaweni yokugcina impahla ephakathi, nokunye. Kukhona i-monolith ephelele, ngisho nambili, kanye nenqwaba yama-API anikezelwe ukulethwa. Bazi okuningi ngokudiliva.

Lezi zibonakala ziyizindawo ezifanayo, kodwa i-Order Processing ku-BOB kanye ne-Shipping System inezimo ezihlukene. Isibonelo, ezinye izinsiza ze-courier azithumeli izimo ezimaphakathi, kodwa kuphela ezokugcina: "kulethiwe" noma "kulahlekile". Abanye, ngokuphambene nalokho, babika ngokuningiliziwe mayelana nokuhamba kwezimpahla. Wonke umuntu unemithetho yakhe yokuqinisekisa: kwabanye, i-imeyili ivumelekile, okusho ukuthi izocutshungulwa; kwabanye ayisebenzi, kodwa i-oda lisazocutshungulwa ngoba kunenombolo yocingo yokuxhumana, futhi omunye uzothi lowo myalelo ngeke ucutshungulwe nhlobo.

Ukusakaza kwedatha

Endabeni ye-Kafka, umbuzo wokuhlela ukugeleza kwedatha uphakama. Lo msebenzi ubandakanya ukukhetha isu elisuselwa kumaphuzu amaningana; ake sidlule kuwo wonke.

Esihlokweni esisodwa noma kwezihlukene?

Sinencazelo yomcimbi. Ku-BOB sibhala ukuthi i-oda enjalo naleyo idinga ukulethwa, futhi ibonise: inombolo ye-oda, ukwakheka kwayo, amanye ama-SKU namakhodi ebha, njll. Lapho izimpahla zifika endaweni yokugcina izimpahla, ukulethwa kuzokwazi ukuthola izimo, izitembu zesikhathi nakho konke okudingekayo. Kodwa-ke sifuna ukuthola izibuyekezo kule datha ku-BOB. Sinenqubo ehlehlayo yokwamukela idatha ekulethweni. Ingabe lesi yisenzakalo esifanayo? Noma ingabe lokhu ukushintshaniswa okuhlukile okufanele isihloko sakho?

Cishe, zizofana kakhulu, futhi isilingo sokwenza isihloko esisodwa asinasisekelo, ngoba isihloko esihlukile sisho abathengi abahlukene, ama-configs ahlukene, isizukulwane esihlukile sakho konke lokhu. Kodwa akulona iqiniso.

Inkambu entsha noma umcimbi omusha?

Kodwa uma usebenzisa izenzakalo ezifanayo, khona-ke enye inkinga ivela. Isibonelo, akuwona wonke amasistimu okulethwa angakhiqiza uhlobo lwe-DTO engakhiqizwa yi-BOB. Sibathumelela i-id, kodwa abayilondolozi ngoba abayidingi, futhi ngokombono wokuqala inqubo yebhasi lomcimbi, le nkambu iyadingeka.

Uma sethula umthetho webhasi lomcimbi wokuthi le nkambu iyadingeka, lapho-ke siphoqeleka ukuthi sibeke imithetho yokuqinisekisa eyengeziwe ku-BOB noma kusiphathi somcimbi wokuqala. Ukuqinisekisa kuqala ukusabalala kuyo yonke isevisi - lokhu akulula kakhulu.

Enye inkinga yisilingo sokukhula okukhulayo. Sitshelwa ukuthi kukhona okumele kwengezwe emcimbini, futhi mhlawumbe, uma sicabanga ngakho, bekufanele kube umcimbi ohlukile. Kodwa ohlelweni lwethu, umcimbi ohlukile uyisihloko esihlukile. Isihloko esihlukile iyonke inqubo engiyichaze ngenhla. Umthuthukisi ulingeka ukuthi avele engeze enye inkambu ku-schema se-JSON futhi asenze kabusha.

Endabeni yokubuyiselwa kwemali, safika emcimbini wemicimbi phakathi nonyaka. Sibe ne-meta-mcimbi eyodwa ebizwa ngesibuyekezo sokubuyiselwa imali, esinenkambu yohlobo echaza ukuthi lesi sibuyekezo besiyini ngempela. Ngenxa yalokhu, sibe noshintsho “olumangalisayo” olunabaqinisekisi abasitshele ukuthi singawuqinisekisa kanjani lo mcimbi ngalolu hlobo.

Inguqulo yomcimbi

Ukuqinisekisa imilayezo e-Kafka ungasebenzisa euro, kodwa kwakudingeka ukuthi ulale phezu kwayo ngokushesha futhi usebenzise i-Confluent. Esimweni sethu, kufanele siqaphele ekwenzeni inguqulo. Ngeke kuhlale kwenzeka ukufunda kabusha imilayezo evela kulogi yokuphindaphinda ngoba imodeli "isele". Ngokuyisisekelo, kuvela ukuthi kwakhiwe izinguqulo ukuze imodeli ihambisane nokubuyela emuva: isibonelo, yenza inkambu ibe ngokuzikhethela okwesikhashana. Uma umehluko unamandla kakhulu, siqala ukubhala esihlokweni esisha, bese sidlulisela amaklayenti lapho eqeda ukufunda esidala.

I-oda lokufunda eliqinisekisiwe lama-partitions

Izihloko ngaphakathi kwe-Kafka zihlukaniswe ngama-partitions. Lokhu akubalulekile kakhulu ngenkathi siklama amabhizinisi nokuhwebelana, kodwa kubalulekile lapho sinquma ukuthi singakusebenzisa kanjani futhi sikukale kanjani.

Esimweni esivamile, ubhala isihloko esisodwa ku-Kafka. Ngokuzenzakalelayo, ingxenye eyodwa iyasetshenziswa, futhi yonke imilayezo kulesi sihloko iya kuyo. Futhi ngenxa yalokho umthengi ufunda le milayezo ngokulandelana. Ake sithi manje sidinga ukwandisa uhlelo ukuze imilayezo ifundwe ngabathengi ababili abahlukene. Uma, ngokwesibonelo, uthumela i-SMS, ungatshela u-Kafka ukuthi enze ukwahlukanisa okwengeziwe, futhi i-Kafka izoqala ukuhlukanisa imilayezo ibe izingxenye ezimbili - ingxenye lapha, ingxenye lapha.

Kafka ubahlukanisa kanjani? Umlayezo ngamunye unomzimba (lapho sigcina khona i-JSON) kanye nokhiye. Ungakwazi ukunamathisela umsebenzi we-hash kulo khiye, ozocacisa ukuthi iyiphi ukwahlukanisa umlayezo ozongena kukho.

Esimweni sethu ngokubuyiselwa kwemali, lokhu kubalulekile, uma sithatha izingxenye ezimbili, khona-ke kukhona ithuba lokuthi umthengi ofanayo uzocubungula umcimbi wesibili ngaphambi kokuqala futhi kuzoba nenkinga. Umsebenzi we-hashi uqinisekisa ukuthi imilayezo enokhiye ofanayo igcina isesigabeni esifanayo.

Imicimbi vs imiyalo

Lena enye inkinga esihlangabezane nayo. Umcimbi umcimbi othize: sithi okuthile kwenzeke ndawana thize (okuthile_kwenzekile), isibonelo, into ethile ikhanseliwe noma imbuyiselo yenzekile. Uma othile elalela le micimbi, ngokusho "kwento ikhanseliwe," ibhizinisi lembuyiselo lizodalwa, futhi "ukubuyiselwa kwemali kwenzeke" kuzobhalwa endaweni ethile ekusetheni.

Kodwa ngokuvamile, lapho uklama imicimbi, awufuni ukuyibhala ngeze - uthembele eqinisweni lokuthi othile uzoyifunda. Kunesilingo esikhulu sokubhala ukuthi kungenzeki okuthize (into_ikhanseliwe, ibuyiselwe_ibuyiselwe), kodwa okuthize_okufanele_kwenziwe. Isibonelo, into isilungele ukubuyiselwa.

Ngakolunye uhlangothi, iphakamisa indlela umcimbi ozosetshenziswa ngayo. Ngakolunye uhlangothi, kuzwakala kuncane kakhulu njengegama lomcimbi ojwayelekile. Ngaphandle kwalokho, akukude ukusuka lapha ukuya kumyalo othi do_something. Kodwa awunaso isiqiniseko sokuthi othile ufunde lesi senzakalo; futhi uma uyifunda, khona-ke uyifunda ngempumelelo; futhi uma ukufunde ngempumelelo, khona-ke wenze okuthile, nokuthi okuthile kwaba yimpumelelo. Lapho umcimbi uba do_something, impendulo iba yimfuneko, futhi lokho kuyinkinga.

Ngokushintshana okungahambisani ku-RabbitMQ, uma ufunda umlayezo, hamba ku-http, unempendulo - okungenani ukuthi umlayezo wamukelwe. Uma ubhalela i-Kafka, kunomlayezo owubhalele i-Kafka, kodwa awazi lutho ngendlela ocutshungulwe ngayo.

Ngakho-ke, esimweni sethu, kwakudingeka sethule umcimbi wokuphendula futhi simise ukuqapha ukuze uma izenzakalo eziningi zithunyelwe, emva kwalesi sikhathi kanye nesikhathi esinjalo inani elifanayo lezenzakalo zokuphendula kufanele lifike. Uma lokhu kungenzeki, khona-ke kubonakala sengathi kukhona okungahambanga kahle. Isibonelo, uma sithumele umcimbi othi “item_ready_to_refund”, silindele ukuthi imbuyiselo izodalwa, imali izobuyiselwa kuklayenti, futhi umcimbi othi “money_refund” uzothunyelwa kithi. Kodwa lokhu akuqinisekile, ngakho ukuqapha kuyadingeka.

Ama-Nuances

Kunenkinga esobala: uma ufunda esihlokweni ngokulandelana, futhi unomyalezo omubi, umthengi uzowa, futhi ngeke uqhubeke. Udinga misa bonke abathengi, yenza i-offset ngokuqhubekayo ukuze uqhubeke nokufunda.

Sasazi ngakho, sathembela kukho, nokho kwenzeka. Futhi lokhu kwenzeka ngoba umcimbi wawuvumelekile ngokombono wezenzakalo-ibhasi, umcimbi wawuvumelekile ngokombono wokuqinisekisa isicelo, kodwa wawungavumelekile ngokombono we-PostgreSQL, ngoba ohlelweni lwethu olulodwa. I-MySQL ene-UNSIGNED INT, futhi ohlelweni olusanda kubhalwa uhlelo lwalune-PostgreSQL ene-INT. Usayizi wakhe mncane, futhi i-Id ayizange ilingane. U-Symfony washona ngaphandle kwalokho. Thina, kunjalo, sibambe okuhlukile ngoba sithembele kukho, futhi besizokwenza lokhu, kodwa ngaphambi kwalokho besifuna ukukhulisa isibali senkinga, njengoba umlayezo ucutshungulwe ngempumelelo. Izibali kule phrojekthi nazo zikusizindalwazi, futhi i-Symfony isivele ivalile ukuxhumana nesizindalwazi, futhi okwesibili kwabulala yonke inqubo ngaphandle nethuba lokuzibophezela.

Inkonzo yalala isikhathi esithile - ngenhlanhla, nge-Kafka lokhu akukubi kakhulu, ngoba imilayezo isala. Lapho umsebenzi usubuyile, ungaqeda ukuzifunda. Ikhululekile.

I-Kafka inamandla okusetha i-offset engafanele ngokusebenzisa ithuluzi. Kodwa ukwenza lokhu, udinga ukumisa bonke abathengi - kithi, lungiselela ukukhululwa okuhlukile lapho kungeke kube khona abathengi, ukuthunyelwa kabusha. Khona-ke e-Kafka ungakwazi ukushintsha i-offset ngokusebenzisa ithuluzi, futhi umlayezo uzodlula.

Enye i-nuance - ilogi yokuphindaphinda vs rdkafka.so - ihlobene nemininingwane yephrojekthi yethu. Sisebenzisa i-PHP, futhi ku-PHP, njengomthetho, yonke imitapo yolwazi ixhumana ne-Kafka nge-rdkafka.so repository, bese kuba khona uhlobo oluthile lwe-wrapper. Mhlawumbe lobu ubunzima bethu bomuntu siqu, kodwa kwavela ukuthi ukumane sifunde kabusha isiqeshana salokho ebesivele sikufundile akulula kangako. Ngokuvamile, kube nezinkinga zesofthiwe.

Ukubuyela emininingwaneni yokusebenza ngama-partitions, kubhalwe khona kanye kumadokhumenti abathengi >= ukuhlukaniswa kwesihloko. Kodwa ngathola ngalokhu kamuva kakhulu kunalokho engangingathanda. Uma ufuna ukukala futhi ube nabathengi ababili, udinga okungenani ama-partitions amabili. Okusho ukuthi, uma unengxenye eyodwa lapho kuqoqwe khona imilayezo eyizinkulungwane ezingama-20, futhi wenza entsha, inani lemiyalezo ngeke lilingane maduze. Ngakho-ke, ukuze ube nabathengi ababili abahambisanayo, udinga ukubhekana nokuhlukaniswa.

Ukuqapha

Ngicabanga ukuthi indlela esiqapha ngayo izocaca nakakhulu ukuthi yiziphi izinkinga ezikhona endleleni ekhona.

Isibonelo, sibala ukuthi mingaki imikhiqizo ekusizindalwazi esanda kushintsha isimo sayo, futhi, ngokufanelekile, izehlakalo kufanele ngabe zenzekile ngokusekelwe kulezi zinguquko, futhi sithumela le nombolo ohlelweni lwethu lokuqapha. Khona-ke kusukela e-Kafka sithola inombolo yesibili, ukuthi zingaki imicimbi eyaqoshwa ngempela. Ngokusobala, umehluko phakathi kwalezi zinombolo ezimbili kufanele uhlale uziro.

Ukwengeza, udinga ukuqapha ukuthi umkhiqizi wenza kanjani, ukuthi imicimbi-imiyalezo etholwe yibhasi, nokuthi umthengi wenza kanjani. Isibonelo, kumashadi angezansi, Ithuluzi Lokubuyisela Imali lenza kahle, kodwa i-BOB ngokusobala inezinkinga ezithile (iziqongo eziluhlaza okwesibhakabhaka).

Sengikushilo kakade ukuleleka kweqembu labathengi. Uma sikhuluma nje, leli inani lemilayezo engafundiwe. Ngokuvamile, abathengi bethu basebenza ngokushesha, ngakho-ke i-lag ngokuvamile i-0, kodwa ngezinye izikhathi kungase kube khona ukuphakama kwesikhashana. I-Kafka ingakwenza lokhu ngaphandle kwebhokisi, kodwa udinga ukusetha isikhawu esithile.

Kukhona iphrojekthi Burrowokuzokunikeza ulwazi olwengeziwe nge-Kafka. Imane isebenzisa i-API yeqembu labathengi ukunikeza isimo sokuthi leli qembu lenza kanjani. Ngaphezu kokuthi KULUNGILE futhi Kwehlulekile, kunesexwayiso, futhi ungathola ukuthi abathengi bakho abakwazi ukubhekana nesivinini sokukhiqiza - abanaso isikhathi sokuhlola okubhaliwe. Isistimu ihlakaniphile futhi kulula ukuyisebenzisa.

Lena yindlela impendulo ye-API ebukeka ngayo. Nali iqembu bob-live-fifa, partition refund.update.v1, isimo SILUNGILE, lag 0 - okokugcina kokugcina okunje nokunye.

Ukuqapha ibuyekeziwe_ku-SLA (ibambekile) Sengishilo. Isibonelo, umkhiqizo ushintshele esimweni sokuthi ulungele ukubuyiselwa. Sifaka i-Cron, ethi uma emaminithini angu-5 le nto ingakabuyi ukubuyiselwa imali (sibuyisela imali ngezinhlelo zokukhokha ngokushesha kakhulu), khona-ke kukhona okungahambanga kahle, futhi lokhu kuyicala lokusekela. Ngakho-ke, simane sithatha i-Cron, efunda izinto ezinjalo, futhi uma zikhulu kuno-0, bese ithumela isexwayiso.

Ukufingqa, ukusebenzisa imicimbi kulula uma:

ulwazi ludingwa amasistimu amaningana;
umphumela wokucubungula awubalulekile;
kunemicimbi embalwa noma izehlakalo ezincane.

Kungase kubonakale sengathi lesi sihloko sinesihloko esicacile kakhulu - i-API engavumelanisi ku-Kafka, kodwa mayelana nayo ngithanda ukuncoma izinto eziningi ngesikhathi esisodwa.
Okokuqala, okulandelayo Ukulayisha Okuphezulu++ kudingeka silinde kuze kube uNovemba, ngo-April kuzoba nenguqulo yaseSt. Petersburg, futhi ngoJuni sizokhuluma ngemithwalo ephezulu eNovosibirsk.
Okwesibili, umbhali wombiko, uSergei Zaika, uyilungu leKomidi Lohlelo lwenkomfa yethu entsha yokuphathwa kolwazi. UlwaziConf. Le ngqungquthela ingeyosuku olulodwa, izoba ngo-Ephreli 26, kodwa uhlelo lwayo lushubile.
Futhi kuzoba ngoMeyi PHP Russia и RIT++ (kufakwe i-DevOpsConf) - ungaphakamisa nesihloko sakho lapho, ukhulume ngolwazi lwakho futhi ukhononde ngamakhoni akho agxishiwe.

Source: www.habr.com

Isipiliyoni sokuthuthukisa isevisi Yethuluzi Lokubuyisela Imali nge-API engavumelanisi ku-Kafka