Ukusukela ngo-2019, iRussia ibinomthetho wokuleyibhile okunyanzelekileyo. Umthetho awusebenzi kuwo onke amaqela eempahla, kwaye imihla yokungeniswa kokunyanzeliswa kokubhaliswa kwamaqela emveliso yahlukile. Icuba, izihlangu, kunye namayeza aya kuba ngawokuqala ukuba phantsi kophawu olunyanzelekileyo; ezinye iimveliso ziya kongezwa kamva, umzekelo, iziqholo, iingubo kunye nobisi. Olu tshintsho lwezomthetho luye lwabangela ukuphuhliswa kwezisombululo ezintsha ze-IT eziza kwenza ukuba kube lula ukulandelela yonke ikhonkco lobomi bemveliso ukusuka kwimveliso ukuya kumthengi wokugqibela, kubo bonke abathathi-nxaxheba kwinkqubo: zombini urhulumente ngokwawo kunye nayo yonke imibutho ethengisa iimpahla kunye. ukuleyibhelishwa okunyanzelekileyo.
Kwi-X5, inkqubo eya kulandelela iimpahla ezibhaliweyo kunye notshintshiselwano lwedatha kunye nombuso kunye nabanikezeli kuthiwa "Marcus". Makhe sikuxelele ngokulandelelana ukuba iphuhliswe njani kwaye ngubani, yintoni isitaki sayo setekhnoloji, kwaye kutheni sinento yokuzingca ngayo.
Umthwalo oPhezulu wokwenyani
"I-Marcus" isombulula iingxaki ezininzi, eyona nto iphambili yintsebenziswano phakathi kweenkqubo zolwazi ze-X5 kunye nenkqubo yolwazi yelizwe kwiimveliso ezibhalwe amagama (GIS MP) ukulandelela ukuhamba kweemveliso ezibhaliweyo. Iqonga likwagcina zonke iikhowudi zokuleyibhile ezifunyenwe kuthi kunye nembali yonke yokuhamba kwezi khowudi kwizinto zonke, kwaye inceda ukuphelisa ukuhlelwa kwakhona kweemveliso ezibhaliweyo. Usebenzisa umzekelo weemveliso zecuba, ezaqukwa kumaqela okuqala eempahla ezibhalwe amagama, ilori enye nje yecuba ineepakethi ezimalunga nama-600 000, nganye inekhowudi yayo eyahlukileyo. Kwaye umsebenzi wenkqubo yethu kukulandelela kunye nokuqinisekisa ukuba semthethweni kweentshukumo zepakethi nganye enjalo phakathi kweendawo zokugcina iimpahla kunye neevenkile, kwaye ekugqibeleni uqinisekise ukuvunywa kokuthengiswa kwazo kumthengi wokugqibela. Kwaye sirekhoda malunga ne-125 yentengiselwano yemali ngeyure, kwaye kufuneka sibhale ukuba ipakethi nganye ingene njani evenkileni. Ngaloo ndlela, ngokuqwalasela zonke iintshukumo phakathi kwezinto, silindele amashumi eebhiliyoni zeerekhodi ngonyaka.
Iqela uM
Ngaphandle kwento yokuba iMarcus ithathwa njengeprojekthi ngaphakathi kweX5, iphunyezwa kusetyenziswa indlela yemveliso. Iqela lisebenza ngokweScrum. Iprojekthi yaqala ihlobo elidlulileyo, kodwa iziphumo zokuqala zafika kuphela ngo-Oktobha - iqela lethu lahlanganiswa ngokupheleleyo, i-architecture yenkqubo yaphuhliswa kwaye izixhobo zathengwa. Ngoku iqela linabantu abayi-16, abathandathu kubo babandakanyeka ekuphuhliseni i-backend kunye ne-frontend, abathathu kubo ababandakanyekayo kuhlalutyo lwenkqubo. Abanye abantu abathandathu babandakanyeka kuvavanyo lwezandla, umthwalo, uvavanyo oluzenzekelayo, kunye nokugcinwa kwemveliso. Ukongeza, sinengcali ye-SRE.
Ayingobaphuhlisi kuphela ababhala ikhowudi kwiqela lethu; phantse bonke abafana bayayazi indlela yokucwangcisa kunye nokubhala ii-autotests, ukulayisha izikripthi kunye nezikripthi ezizenzekelayo. Sihlawula ingqalelo ekhethekileyo kule nto, kuba nenkxaso yemveliso ifuna umgangatho ophezulu wokuzenzekelayo. Sihlala sizama ukucebisa kunye nokunceda oogxa bethu abangazange bacwangcise ngaphambili, kwaye sibanike imisebenzi emincinci ekufuneka basebenze kuyo.
Ngenxa yobhubhani we-coronavirus, sitshintshele lonke iqela kumsebenzi okude; ukufumaneka kwazo zonke izixhobo zolawulo lophuhliso, ukuhamba komsebenzi okwakhiwe eJira naseGitLab kwenze ukuba kudlule ngokulula kweli nqanaba. Iinyanga ezichithwe kude zibonise ukuba imveliso yeqela ayizange ihlupheke ngenxa yoko; kwabaninzi, intuthuzelo emsebenzini yanda, ekuphela kwento engekhoyo lunxibelelwano oluphilayo.
Intlanganiso yeqela elikude
Iintlanganiso ngexesha lomsebenzi okude
Isitaki seTekhnoloji yesisombululo
Uvimba osemgangathweni kunye nesixhobo se-CI/CD se-X5 yi-GitLab. Siyisebenzisela ukugcinwa kwekhowudi, ukuvavanywa okuqhubekayo, kunye nokuthunyelwa ukuvavanya kunye neeseva zokuvelisa. Sikwasebenzisa umkhuba wokuhlaziywa kwekhowudi, xa ubuncinane abalingane be-2 kufuneka bavume utshintsho olwenziwa ngumphuhlisi kwikhowudi. Abahlalutyi bekhowudi ye-Static i-SonarQube kunye ne-JaCoCo zisinceda sigcine ikhowudi yethu icocekile kwaye siqinisekise inqanaba elifunekayo leyunithi yovavanyo. Zonke iinguqu kwikhowudi kufuneka zidlule kwezi zitsheki. Zonke izikripthi zovavanyo eziqhutywa ngesandla ziyazenzekela.
Ukuphunyezwa ngempumelelo kweenkqubo zoshishino ngu "Marcus", kwafuneka sisombulule inani leengxaki zobuchwepheshe, malunga neyodwa ngokulandelelana.
Umsebenzi 1. Imfuno yokulinganisa okuthe tye kwenkqubo
Ukusombulula le ngxaki, sikhethe indlela ye-microservice yolwakhiwo. Kwangaxeshanye, kwakubaluleke kakhulu ukuqonda iinkalo zoxanduva lweenkonzo. Sazama ukuzahlula zibe yimisebenzi yoshishino, sithathela ingqalelo iinkcukacha zeenkqubo. Ngokomzekelo, ukwamkelwa kwindawo yokugcina izinto akuqhelekanga, kodwa kukhulu kakhulu, ngexesha apho kuyimfuneko ukufumana ngokukhawuleza ulwazi lomlawuli welizwe malunga neeyunithi zeempahla ezamkelweyo, inani lazo ekuhanjisweni kwelinye lifikelela kuma-600000. , khangela ukuvunywa kokuvuma le mveliso kwindawo yokugcina kwaye ubuyisele yonke ingcaciso efunekayo kwisistim somatshini wokuzenzekelayo. Kodwa ukuthunyelwa kwiindawo zokugcina iimpahla kunomlinganiselo omkhulu kakhulu, kodwa kwangaxeshanye kusebenza ngemithamo emincinci yedatha.
Sisebenzisa zonke iinkonzo ngokungenammiselo kwaye sizame ukwahlula imisebenzi yangaphakathi ibe ngamanyathelo, sisebenzisa into esiyibiza ngokuba yi-Kafka self-topics. Oku kuxa i-microservice ithumela umyalezo kuyo, ekuvumela ukuba ulungelelanise umthwalo kwimisebenzi enzulu yobutyebi kunye nokwenza lula ukugcinwa kwemveliso, kodwa ngakumbi emva koko.
Sagqiba ekubeni sahlule iimodyuli zokunxibelelana neenkqubo zangaphandle kwiinkonzo ezahlukeneyo. Oku kwenza kube lula ukusombulula ingxaki yokutshintsha rhoqo ii-APIs zeenkqubo zangaphandle, kwaye akukho mpembelelo kwiinkonzo kunye nokusebenza kweshishini.
Zonke ii-microservices zifakwe kwiqela le-OpenShift, elisombulula zombini ingxaki yokulinganisa i-microservice nganye kwaye isivumela ukuba singasebenzisi izixhobo zokuFumana iiNkonzo zomntu wesithathu.
Umsebenzi 2. Isidingo sokugcina umthwalo ophezulu kunye notshintshiselwano olunzulu kakhulu lwedatha phakathi kweenkonzo zeqonga: Ngexesha lesigaba sokuphehlelelwa kweprojekthi kuphela, malunga nemisebenzi engama-600 ngomzuzwana yenziwa. Silindele ukuba eli xabiso linyuke liye kwi-5000 ops / sec njengoko iindawo zokuthengisa zidibanisa kwiqonga lethu.
Le ngxaki yasonjululwa ngokuthumela iqela le-Kafka kwaye phantse ukulahla ngokupheleleyo ukusebenzisana okuhambelanayo phakathi kwee-microservices zeqonga. Oku kufuna uhlalutyo olunenyameko kakhulu lweemfuno zenkqubo, kuba ingeyiyo yonke imisebenzi enokuba yi-asynchronous. Kwangaxeshanye, asidluliseli kuphela iziganeko nge-broker, kodwa sikwahambisa zonke iinkcukacha ezifunekayo zeshishini kumyalezo. Ke, ubungakanani bomyalezo bunokufikelela kumakhulu eekhilobhayithi. Umda wobungakanani bomyalezo eKafka ufuna ukuba siqikelele ngokuchanekileyo ubungakanani bomyalezo, kwaye ukuba kuyimfuneko, siyabahlula, kodwa ulwahlulo lunengqiqo, lunxulumene nemisebenzi yezoshishino.
Umzekelo, sahlulahlula iimpahla ezifika ngemoto kwiibhokisi. Kwimisebenzi ehambelanayo, ii-microservices ezahlukeneyo zabelwe kwaye uvavanyo olucokisekileyo lwemithwalo luyenziwa. Ukusebenzisa i-Kafka kusinike omnye umngeni-ukuvavanya ukusebenza kwenkonzo yethu kuthathelwa ingqalelo ukudityaniswa kweKafka kwenza zonke iimvavanyo zethu zeyunithi zihambelane. Siyisombulule le ngxaki ngokubhala ezethu iindlela eziluncedo sisebenzisa i-Embedded Kafka Broker. Oku akuphelisi imfuneko yokubhala iimvavanyo zeyunithi kwiindlela zomntu ngamnye, kodwa sikhetha ukuvavanya iimeko ezinzima usebenzisa i-Kafka.
Ingqwalasela eninzi ihlawulwe ekulandeleni iinkuni ukwenzela ukuba iTraceId yabo ingalahleki xa iimeko ezingaqhelekanga zenzeka ngexesha lokusebenza kweenkonzo okanye xa usebenza nebhetshi yeKafka. Kwaye ukuba bekungekho miba ikhethekileyo kunye neyokuqala, ngoko kwimeko yesibini siphoqeleka ukuba sibhale zonke ii-TraceIds ezaza nazo ibhetshi kwaye ukhethe enye ukuqhubeka nokulandela umkhondo. Emva koko, xa ukhangela ngeTraceId yoqobo, umsebenzisi uya kufumanisa ngokulula ukuba ukulandelela kuqhubeka ntoni.
Umsebenzi 3. Isidingo sokugcina isixa esikhulu sedatha: Ngaphezulu kwebhiliyoni yeebhiliyoni zamagama ngonyaka kwicuba kuphela zifika kwi-X1. Bafuna ukufikelela rhoqo kwaye ngokukhawuleza. Iyonke, inkqubo kufuneka iqhube malunga ne-5 yeebhiliyoni zeerekhodi zembali yokunyakaza kwezi mpahla zibhalwe.
Ukusombulula ingxaki yesithathu, i-database ye-NoSQL i-MongoDB yakhethwa. Siye sakha i-shard yee-nodes ezi-5 kwaye i-node nganye ine-Replica Seti yeeseva ezi-3. Oku kukuvumela ukuba ulinganise inkqubo ngokuthe tye, ukongeza iiseva ezintsha kwiqela, kwaye uqinisekise ukunyamezela impazamo yayo. Apha siye sadibana nenye ingxaki-ukuqinisekisa ukuthengiselana kwi-mongo cluster, kuthathelwa ingqalelo ukusetyenziswa kwee-microservices ezithe tye. Umzekelo, omnye wemisebenzi yenkqubo yethu kukuchonga iinzame zokuthengisa kwakhona iimveliso ezineekhowudi zokuleyibheli ezifanayo. Apha, ukugqunyelelwa kubonakala ngokuskena okuphosakeleyo okanye imisebenzi enempazamo eyenziwa ngabantu abathatha imali. Sifumanise ukuba uphinda-phindo olunjalo lunokwenzeka zombini ngaphakathi kwibhetshi enye ye-Kafka elungiswayo, kwaye kwiibhetshi ezimbini zisetyenzwa ngokufanayo. Ngaloo ndlela, ukukhangela iimpinda ngokubuza kwisiseko sedatha akunikanga nto. Kwi-microservice nganye, sisombulule ingxaki ngokwahlukileyo ngokusekelwe kwingqiqo yeshishini le nkonzo. Umzekelo, kwiitshekhi, songeze itshekhi ngaphakathi kwibhetshi kunye nokwahlula ukusetyenzwa kokubonakala kokuphindaphinda xa ufaka.
Ukuqinisekisa ukuba umsebenzi wabasebenzisi kunye nembali yemisebenzi ayichaphazeli nangayiphi na indlela eyona nto ibalulekileyo - ukusebenza kweenkqubo zethu zoshishino, siye sahlula yonke idatha yembali kwinkonzo eyahlukileyo kunye nedathabheyisi eyahlukileyo, ekwafumana ulwazi ngeKafka. . Ngale ndlela, abasebenzisi basebenza ngenkonzo eyodwa ngaphandle kokuchaphazela iinkonzo ezenza idatha kwimisebenzi eqhubekayo.
Umsebenzi 4: Ukusetyenzwa kwakhona kunye nokubekwa kweliso emgceni:
Kwiinkqubo ezisasazwayo, iingxaki kunye neempazamo zivela ngokungaphephekiyo ekufumanekeni kogcino-lwazi, imigca, kunye nemithombo yedatha yangaphandle. Kwimeko kaMarcus, umthombo weempazamo ezinjalo kukudibanisa neenkqubo zangaphandle. Kwakuyimfuneko ukufumana isisombululo esiza kuvumela izicelo eziphindaphindiweyo zeempendulo eziphosakeleyo kunye nexesha elithile elichaziweyo, kodwa kwangaxeshanye ungayeki ukucubungula izicelo eziyimpumelelo kumgca oyintloko. Ukwenzela le njongo, into ebizwa ngokuba yi-"topic based try" yakhethwa. Kwisihloko ngasinye esiphambili, enye okanye ngaphezulu izihloko zama kwakhona zenziwe apho imiyalezo ephosakeleyo ithunyelwa kwaye kwangaxeshanye ukulibaziseka kokucubungula imiyalezo evela kwisihloko esiphambili kupheliswa. Iskimu sentsebenziswano -
Ukuphumeza isikimu esinjalo, sifuna oku kulandelayo: ukudibanisa esi sisombululo kunye neNtwasahlobo kwaye sigweme ukuphindaphinda kwekhowudi. Ngelixa sisefa iwebhu, siye safumana isisombululo esifanayo esekwe kwi-Spring BeanPostProccessor, kodwa kwakubonakala ngathi kunzima ngokungeyomfuneko kuthi. Iqela lethu lenze isisombululo esilula esisivumela ukuba sidibanise kumjikelo waseNtwasahlobo wokudala abathengi kwaye songeze kwakhona ukuZama abathengi. Sinike iprototype yesisombululo sethu kwiqela le-Spring, ungayibona . Inani lokuzama kwakhona abathengi kunye nenani lemizamo yomthengi ngamnye iqwalaselwe ngeeparamitha, ngokuxhomekeke kwiimfuno zenkqubo yeshishini, kwaye yonke into isebenze, konke okuseleyo kukongeza isichasiselo org.springframework.kafka.annotation.KafkaListener , eyaziwayo kubo bonke abaphuhlisi baseNtwasahlobo.
Ukuba umyalezo awukwazanga ukuqhutyelwa phambili emva kwayo yonke imizamo yokuzama kwakhona, uya kwi-DLT (isihloko seleta efileyo) usebenzisa i-Spring DeadLetterPublishingRecoverer. Ngesicelo senkxaso, sandise lo msebenzi kwaye senza inkonzo eyahlukileyo evumela ukuba ujonge imiyalezo efakwe kwi-DLT, stackTrace, traceId kunye nolunye ulwazi oluluncedo malunga nabo. Ukongezelela, ukubeka iliso kunye nezilumkiso zongezwa kuzo zonke izihloko ze-DLT, kwaye ngoku, ngokwenene, ukubonakala komyalezo kwisihloko se-DLT sisizathu sokuhlalutya kunye nokulungisa isiphene. Oku kulungeleka kakhulu - ngegama lesihloko, siyaqonda ngokukhawuleza ukuba yiyiphi inyathelo lenkqubo eyavelayo ingxaki, ekhawuleza kakhulu ukukhangela unobangela wayo.
Kutshanje, siphumeze i-interface evumela ukuba sithumele kwakhona imiyalezo ngokusebenzisa inkxaso yethu emva kokuphelisa izizathu zabo (umzekelo, ukubuyisela ukusebenza kwenkqubo yangaphandle) kwaye, ngokuqinisekileyo, ukuseka isiphene esihambelanayo sohlalutyo. Kulapho ii-self-topics zethu ziza luncedo: ukuze ungaqalisi kwakhona ikhonkco elide lokucubungula, unokuliqala kwakhona ukusuka kwinqanaba elifunekayo.
Ukusebenza kweQonga
Iqonga sele lisebenza ngokuvelisa, yonke imihla siqhuba ukuhanjiswa kunye nokuthunyelwa, ukudibanisa amaziko amatsha okusasaza kunye neevenkile. Njengenxalenye yomqhubi, inkqubo isebenza kunye namaqela emveliso "Icuba" kunye ne "Shoes".
Iqela lethu lonke lithatha inxaxheba ekuqhubeni abaqhubi, ukuhlalutya iingxaki ezivelayo kwaye lenze iingcebiso zokuphucula imveliso yethu, ukusuka ekuphuculeni iilogi ukuya kwiinkqubo eziguqukayo.
Ukuze singaphindi iimpazamo zethu, zonke iimeko ezifunyenwe ngexesha lokulinga zibonakaliswa kwiimvavanyo ezizenzekelayo. Ubukho benani elikhulu le-autotests kunye novavanyo lweyunithi ikuvumela ukuba uqhube uvavanyo lokubuyisela kwaye ufake i-hotfix ngokoqobo kwiiyure ezimbalwa.
Ngoku siyaqhubeka nokuphuhlisa nokuphucula iqonga lethu, kwaye sihlala sijongana nemingeni emitsha. Ukuba unomdla, siya kuthetha ngezicombululo zethu kumanqaku alandelayo.
umthombo: www.habr.com