Kusukela ngo-2019, iRussia ibinomthetho wokulebula okuyimpoqo. Umthetho awusebenzi kuwo wonke amaqembu ezimpahla, futhi izinsuku zokuqalisa ukusebenza kokulebula okuyisibopho kwamaqembu omkhiqizo zihlukile. Ugwayi, izicathulo, nemithi kuzoba ngeyokuqala ukulebula okuyisibopho; eminye imikhiqizo izongezwa kamuva, isibonelo, iziqholo, izindwangu, nobisi. Lokhu okusha okusemthethweni kugqugquzele ukuthuthukiswa kwezixazululo ezintsha ze-IT ezizokwenza kube nokwenzeka ukulandelela lonke iketango lempilo lomkhiqizo kusukela ekukhiqizweni kuya kumthengi wokugcina, kubo bonke ababambiqhaza kule nqubo: kokubili umbuso ngokwawo nazo zonke izinhlangano ezithengisa izimpahla ukulebula okuyisibopho.
Ku-X5, isistimu ezolandelela izimpahla ezinelebula futhi ishintshisane ngedatha nezwe nabahlinzeki ibizwa ngokuthi “Marcus”. Ake sikutshele ngokulandelana ukuthi ithuthukiswe kanjani futhi ubani, siyini isitaki sayo sobuchwepheshe, nokuthi kungani sinokuthile esingaziqhenya ngakho.
Ukulayisha Okuphezulu Kwangempela
I-“Marcus” ixazulula izinkinga eziningi, eyinhloko ukuxhumana phakathi kwezinhlelo zolwazi ze-X5 kanye nesistimu yolwazi yezwe yemikhiqizo enelebula (GIS MP) ukuze ilandelele ukunyakaza kwemikhiqizo enelebula. Inkundla iphinde igcine wonke amakhodi amalebula atholwe yithi nawo wonke umlando wokuhamba kwalawa makhodi ezintweni zonke, futhi isiza ukuqeda ukugredwa kabusha kwemikhiqizo enelebula. Kusetshenziswa isibonelo semikhiqizo kagwayi, eyafakwa emaqenjini okuqala ezimpahla ezinelebula, iloli elilodwa nje likagwayi liqukethe amaphakethe angaba ngu-600 000, ngalinye linekhodi yalo ehlukile. Futhi umsebenzi wohlelo lwethu ukulandelela kanye nokuqinisekisa ukuba semthethweni kokunyakaza kwephakethe ngalinye elinjalo phakathi kwezindawo zokugcina izimpahla nezitolo, futhi ekugcineni kuqinisekiswe ukuvunyelwa kokuthengiswa kwazo kumthengi wokugcina. Futhi siqopha imali elinganiselwa ku-125 yemali ngehora, futhi sidinga ukuqopha ukuthi iphekhi ngalinye elinjalo lingene kanjani esitolo. Ngakho, ngokucabangela konke ukunyakaza phakathi kwezinto, silindele amashumi ezigidigidi zamarekhodi ngonyaka.
Ithimba M
Ngaphandle kweqiniso lokuthi i-Marcus ithathwa njengephrojekthi ngaphakathi kwe-X5, isetshenziswa kusetshenziswa indlela yomkhiqizo. Ithimba lisebenza ngokweScrum. Le phrojekthi yaqala ehlobo eledlule, kodwa imiphumela yokuqala yafika kuphela ngo-Okthoba - ithimba lethu lahlanganiswa ngokugcwele, isakhiwo sesistimu sathuthukiswa futhi imishini yathengwa. Manje ithimba selinabantu abangu-16, abayisithupha kubo abathintekayo ekuthuthukiseni i-backend kanye ne-frontend, abathathu kubo abathintekayo ekuhlaziyeni uhlelo. Abanye abantu abayisithupha babambe iqhaza ekuhlolweni okwenziwa ngesandla, ekulayisheni, ekuhloleni okuzenzakalelayo, nasekunakekeleni umkhiqizo. Ngaphezu kwalokho, sinochwepheshe we-SRE.
Akubona onjiniyela kuphela ababhala ikhodi eqenjini lethu; cishe bonke abafana bayakwazi ukuhlela nokubhala izivivinyo ezizenzakalelayo, ukulayisha imibhalo kanye nemibhalo ezenzakalelayo. Sinaka ngokukhethekile lokhu, njengoba ngisho nokusekelwa komkhiqizo kudinga izinga eliphezulu lokuzenzekelayo. Sihlala sizama ukweluleka nokusiza ozakwethu abangazange bahlele ngaphambilini, futhi sibanikeze imisebenzi emincane okufanele basebenze kuyo.
Ngenxa yobhubhane lwe-coronavirus, sidlulisele lonke ithimba emsebenzini wokude; ukutholakala kwawo wonke amathuluzi okuphatha intuthuko, ukugeleza komsebenzi okwakhiwe e-Jira nase-GitLab kwenze kwaba nokwenzeka ukudlula lesi sigaba kalula. Izinyanga ezichithwe kude zabonisa ukuthi ukukhiqiza kweqembu akuzange kuphazamiseke ngenxa yalokho; kwabaningi, ukunethezeka emsebenzini kwanda, okuwukuphela kwento eyayishoda kwakuwukuxhumana okubukhoma.
Umhlangano wethimba elikude
Imihlangano ngesikhathi sokusebenza kude
Isitaki sobuchwepheshe sesixazululo
Inqolobane evamile kanye nethuluzi le-CI/CD le-X5 yi-GitLab. Siyisebenzisela ukugcinwa kwekhodi, ukuhlola okuqhubekayo, kanye nokusatshalaliswa kokuhlola nokukhiqiza amaseva. Siphinde sisebenzisa umkhuba wokubuyekeza ikhodi, lapho okungenani ozakwethu abangu-2 kudingeka bagunyaze izinguquko ezenziwe unjiniyela kukhodi. Abahlaziyi bekhodi emile i-SonarQube ne-JaCoCo bayasisiza ukuthi sigcine ikhodi yethu ihlanzekile futhi siqinisekise izinga elidingekayo lokufakwa kokuhlolwa kweyunithi. Zonke izinguquko kukhodi kufanele zidlule kulawa masheke. Zonke izikripthi zokuhlola ezisebenza mathupha ziyazenzekela.
Ukuze kusetshenziswe ngempumelelo izinqubo zebhizinisi ngu "Marcus", kwakudingeka sixazulule izinkinga eziningi zobuchwepheshe, mayelana ngayinye ngokulandelana.
Umsebenzi 1. Isidingo sokukala okuvundlile kwesistimu
Ukuxazulula le nkinga, sikhethe indlela ye-microservice yezokwakha. Ngesikhathi esifanayo, kwakubaluleke kakhulu ukuqonda izindawo zomthwalo wemfanelo wezinsizakalo. Sizamile ukuwahlukanisa ngokwemisebenzi yebhizinisi, sicabangela imininingwane yezinqubo. Isibonelo, ukwamukelwa endaweni yokugcina impahla akuyona into evamile, kodwa enkulu kakhulu, lapho kudingekile ukuthola ngokushesha ulwazi kumlawuli kahulumeni mayelana namayunithi ezimpahla ezamukelwayo, inani lawo ekulethweni okulodwa lifinyelela ku-600000. , hlola ukuvunyelwa kokwamukela lo mkhiqizo esitolo bese ubuyisela lonke ulwazi oludingekayo lwesistimu ye-automation system. Kodwa ukuthunyelwa okuvela ezindlini zokugcina izimpahla kunomfutho omkhulu kakhulu, kodwa ngesikhathi esifanayo kusebenza ngenani elincane ledatha.
Sisebenzisa wonke amasevisi ngendlela engapheli futhi sizame nokuhlukanisa imisebenzi yangaphakathi ibe yizinyathelo, sisebenzisa lokho esikubiza ngokuthi izihloko ezizimele ze-Kafka. Kulapho i-microservice izithumelela khona umlayezo, okuvumela ukuthi ulinganisele umthwalo ekusebenzeni okudinga izinsiza futhi wenze ukunakekelwa komkhiqizo kube lula, kodwa okwengeziwe ngalokho kamuva.
Sinqume ukuhlukanisa amamojula okusebenzelana namasistimu angaphandle sibe ngamasevisi ahlukene. Lokhu kwenze kwaba nokwenzeka ukuxazulula inkinga yokushintsha njalo ama-API wezinhlelo zangaphandle, ngaphandle komthelela kumasevisi anomsebenzi webhizinisi.
Wonke ama-microservices asetshenziswa kuqoqo le-OpenShift, elixazulula kokubili inkinga yokukala i-microservice ngayinye futhi lisivumela ukuthi singasebenzisi amathuluzi e-OpenShift Discovery.
Umsebenzi 2. Isidingo sokugcina umthwalo ophezulu kanye nokushintshisana kwedatha okujulile phakathi kwamasevisi enkundla: Phakathi nesigaba sokwethulwa kwephrojekthi kuphela, cishe imisebenzi engama-600 ngomzuzwana yenziwa. Silindele ukuthi leli nani likhuphuke libe ngu-5000 ops/isekhondi njengoba izitolo zixhuma kunkundla yethu.
Le nkinga yaxazululwa ngokuthumela iqoqo le-Kafka futhi icishe ishiye ngokuphelele ukusebenzisana okuvumelanayo phakathi kwama-microservices epulatifomu. Lokhu kudinga ukuhlaziya okucophelelayo kwezidingo zesistimu, njengoba kungeyona yonke imisebenzi engaba yi-asynchronous. Ngesikhathi esifanayo, asidluliseli imicimbi kuphela ngomthengisi, kodwa futhi sidlulisa lonke ulwazi lwebhizinisi oludingekayo emlayezo. Ngakho, usayizi womlayezo ungafinyelela amakhilobhayithi angamakhulu amaningana. Umkhawulo kasayizi womlayezo e-Kafka udinga ukuthi sibikezele ngokunembile usayizi womlayezo, futhi uma kunesidingo, siyawahlukanisa, kodwa ukuhlukaniswa kunengqondo, kuhlobene nokusebenza kwebhizinisi.
Isibonelo, sihlukanisa izimpahla ezifika ngemoto emabhokisini. Ngokusebenza okuvumelanayo, ama-microservices ahlukene anikezwa futhi ukuhlolwa okuphelele komthwalo kuyenziwa. Ukusebenzisa i-Kafka kusinikeze enye inselele - ukuhlola ukusebenza kwensizakalo yethu kucatshangelwa ukuhlanganiswa kwe-Kafka kwenza zonke izivivinyo zeyunithi yethu zivumelane. Sixazulule le nkinga ngokubhala izindlela zethu ezisetshenziswayo sisebenzisa i-Embedded Kafka Broker. Lokhu akusisusi isidingo sokubhala ukuhlolwa kweyunithi yezindlela ngazinye, kodwa sincamela ukuhlola amacala ayinkimbinkimbi sisebenzisa i-Kafka.
Kwagxilwa kakhulu ekulandeleni izingodo ukuze i-TraceId yabo ingalahleki lapho okuhlukile kwenzeka ngesikhathi sokusebenza kwezinsizakalo noma lapho kusebenza nenqwaba ye-Kafka. Futhi uma bekungekho zinkinga ezikhethekile ngeyokuqala, khona-ke esimweni sesibili siyaphoqeleka ukuthi sibhale wonke ama-TraceIds iqoqo elize nawo bese sikhetha eyodwa ukuze siqhubeke nokulandela umkhondo. Bese, lapho usesha nge-TraceId yoqobo, umsebenzisi uzothola kalula ukuthi ukulandelela kuqhutshwe ngani.
Umsebenzi 3. Isidingo sokugcina inani elikhulu ledatha: Amalebula angaphezu kwebhiliyoni e-1 ngonyaka kagwayi kuphela afika ku-X5. Zidinga ukufinyelela okungaguquki nokusheshayo. Sekukonke, uhlelo kumele lucubungule amarekhodi acishe abe yizigidi eziyizinkulungwane eziyi-10 omlando wokunyakaza walezi zimpahla ezinelebula.
Ukuxazulula inkinga yesithathu, kukhethwe isizindalwazi se-NoSQL i-MongoDB. Sakhe ucezu lwamanodi angu-5 futhi inodi ngayinye inesethi eyi-Replica yamaseva angu-3. Lokhu kukuvumela ukuthi ulinganise isistimu ngokuvundlile, wengeze amaseva amasha kuqoqo, futhi uqinisekise ukubekezelela kwayo iphutha. Lapha sihlangabezane nenye inkinga - ukuqinisekisa ukuthengiselana kuqoqo le-mongo, kucatshangelwa ukusetshenziswa kwama-microservices avundlile angakaleka. Isibonelo, omunye wemisebenzi yesistimu yethu ukukhomba imizamo yokuthengisa kabusha imikhiqizo enamakhodi okulebula afanayo. Lapha, izimbondela zivela nezikena eziyiphutha noma imisebenzi eyiphutha eyenziwa okheshi. Sithole ukuthi izimpinda ezinjalo zingenzeka kokubili phakathi kwenqwaba eyodwa ye-Kafka ecutshungulwayo, futhi phakathi kwamaqoqo amabili acutshungulwa ngokufana. Ngakho, ukuhlola izimpinda ngokubuza ku-database akunikanga lutho. Kusevisi encane ngayinye, sixazulule inkinga ngokwehlukana ngokusekelwe kumqondo webhizinisi wale sevisi. Isibonelo, ukuze sihlole, sengeze isheke ngaphakathi kwenqwaba kanye nokucubungula okuhlukile kokubukeka kwezimpinda uma ufaka.
Ukuqinisekisa ukuthi umsebenzi wabasebenzisi ngomlando wokusebenza akuthinti nganoma iyiphi indlela into ebaluleke kakhulu - ukusebenza kwezinqubo zebhizinisi lethu, sihlukanise yonke idatha yomlando saba yinkonzo ehlukile enesizindalwazi esihlukile, esiphinde sithole ulwazi nge-Kafka. . Ngale ndlela, abasebenzisi basebenza nesevisi engayodwa ngaphandle kokuthikameza izinsizakalo ezicubungula idatha ukuze kwenziwe imisebenzi eqhubekayo.
Umsebenzi 4: Umugqa wokucubungula kabusha nokuqapha:
Kuzinhlelo ezisabalalisiwe, izinkinga namaphutha avela ngokungagwemeki ekutholakaleni kolwazi, imigqa, nemithombo yedatha yangaphandle. Endabeni ka-Marcus, umthombo wamaphutha anjalo ukuhlanganiswa nezinhlelo zangaphandle. Bekudingekile ukuthola isisombululo esizovumela izicelo eziphindaphindiwe zezimpendulo eziyiphutha ngesikhathi esithile esimisiwe, kodwa ngesikhathi esifanayo singayeki ukucubungula izicelo eziyimpumelelo kulayini oyinhloko. Ngale njongo, kwakhethwa umqondo obizwa ngokuthi “ukuzama kabusha okusekelwe esihlokweni”. Esihlokweni ngasinye esiyinhloko, isihloko esisodwa noma eziningi zokuphinda uzame kwakhiwa lapho imilayezo eyiphutha ithunyelwa khona futhi ngesikhathi esifanayo ukubambezeleka kokucubungula imilayezo evela esihlokweni esiyinhloko kuyasuswa. Isikimu sokusebenzisana -
Ukuze sisebenzise uhlelo olunjalo, besidinga okulandelayo: ukuhlanganisa lesi sixazululo neNtwasahlobo futhi sigweme ukuphindaphinda kwekhodi. Ngenkathi sihlola iwebhu, sihlangabezane nesixazululo esifanayo esisekelwe ku-Spring BeanPostProccessor, kodwa sibonakale sinzima ngokungadingekile kithi. Ithimba lethu lenze isixazululo esilula esisivumela ukuthi sihlanganise nomjikelezo weNtwasahlobo ukuze sidale abathengi futhi singeze okuthi Zama kabusha Abathengi. Sinikeze isibonelo sesixazululo sethu eqenjini leNtwasahlobo, ungasibona . Inombolo Yabasebenzisi Bazama Kabusha kanye nenani lemizamo yomthengi ngamunye ilungiswa ngemingcele, kuye ngezidingo zenqubo yebhizinisi, futhi ukuze yonke into isebenze, okusele nje ukwengeza isichasiselo org.springframework.kafka.annotation.KafkaListener , eyaziwa yibo bonke abathuthukisi baseNtwasahlobo.
Uma umlayezo ungakwazanga ukucutshungulwa ngemva kwayo yonke imizamo yokuphinda uzame, uya ku-DLT (isihloko sencwadi efile) kusetshenziswa i-Spring DeadLetterPublishingRecoverer. Ngesicelo sokusekelwa, sandise lokhu kusebenza futhi sakha isevisi ehlukile ekuvumela ukuthi ubuke imilayezo efakwe ku-DLT, stackTrace, traceId nolunye ulwazi oluwusizo olumayelana nayo. Ngaphezu kwalokho, ukuqapha kanye nezixwayiso zengezwe kuzo zonke izihloko ze-DLT, futhi manje, empeleni, ukubonakala komlayezo esihlokweni se-DLT kuyisizathu sokuhlaziya nokulungisa iphutha. Lokhu kulula kakhulu - ngegama lesihloko, siyaqonda ngokushesha ukuthi yisiphi isinyathelo senqubo inkinga eyavela, okusheshisa kakhulu ukucinga imbangela yayo.
Muva nje, sisebenzise isixhumi esibonakalayo esisivumela ukuthi sithumele kabusha imilayezo sisebenzisa ukusekela kwethu ngemva kokuqeda izimbangela zayo (isibonelo, ukubuyisela ukusebenza kwesistimu yangaphandle) futhi, vele, ukusungula iphutha elihambisanayo ukuze lihlaziywe. Kulapho izihloko zethu ezizifundisayo zisiza khona: ukuze ungaqalisi kabusha uchungechunge olude lokucubungula, ungaliqala kabusha ukusuka esinyathelweni osifunayo.
Ukusebenza kwePlatform
Isiteji sesivele sisebenza ngokukhiqiza, nsuku zonke senza ukudiliva nokuthunyelwa, sixhuma izikhungo ezintsha zokusabalalisa nezitolo. Njengengxenye yomshayeli wendiza, uhlelo lusebenza namaqembu emikhiqizo ethi "Ugwayi" kanye "Nezicathulo".
Ithimba lethu lonke libamba iqhaza ekuqhubeni abashayeli bezindiza, lihlaziye izinkinga ezivelayo futhi lenze iziphakamiso zokuthuthukisa umkhiqizo wethu, kusukela ekuthuthukiseni amalogi kuya ekushintsheni izinqubo.
Ukuze singaphindi amaphutha ethu, zonke izimo ezitholakala ngesikhathi somshayeli ziboniswa ezivivinyweni ezizenzakalelayo. Ukuba khona kwenani elikhulu lokuhlola okuzenzakalelayo nokuhlolwa kweyunithi kukuvumela ukuthi wenze ukuhlolwa kokuhlehla futhi ufake i-hotfix ngokoqobo emahoreni ambalwa.
Manje siyaqhubeka nokuthuthukisa nokuthuthukisa inkundla yethu, futhi sihlala sibhekana nezinselele ezintsha. Uma unesithakazelo, sizokhuluma ngezixazululo zethu ezihlokweni ezilandelayo.
Source: www.habr.com