Kuyambira 2019, Russia yakhala ndi lamulo lokhudza zovomerezeka. Lamulo silikugwira ntchito kumagulu onse azinthu, ndipo masiku oyambira kukakamiza kulembetsa magulu azogulitsa ndi osiyana. Fodya, nsapato, ndi mankhwala adzakhala oyamba kulembedwa movomerezeka; zinthu zina zidzawonjezedwa pambuyo pake, mwachitsanzo, mafuta onunkhira, nsalu, ndi mkaka. Kupanga kwamalamulo kumeneku kudapangitsa kuti pakhale njira zatsopano zothanirana ndi vuto la IT zomwe zipangitsa kuti zitheke kutsata moyo wonse wazinthu kuchokera pakupanga mpaka kugulidwa ndi ogula, kwa onse omwe akutenga nawo gawo pakukonzekera: boma lokha komanso mabungwe onse ogulitsa katundu kulemba kovomerezeka.
Mu X5, dongosolo lomwe lidzatsata katundu wolembedwa ndikusinthanitsa deta ndi boma ndi ogulitsa limatchedwa "Marcus". Tiwuzeni momwe zidapangidwira komanso ndani, momwe ukadaulo wake ulili, komanso chifukwa chake tili ndi chonyadira.
Real HighLoad
"Marcus" amathetsa mavuto ambiri, chachikulu ndi kuphatikizika kwa machitidwe a chidziwitso cha X5 ndi dongosolo lachidziwitso cha boma la zinthu zolembedwa (GIS MP) kuti azitsatira kayendedwe kazinthu zolembedwa. Pulatifomuyi imasunganso ma code onse omwe talandira ndi mbiri yonse ya kayendedwe ka ma code awa pa zinthu, ndikuthandizira kuthetsa kukonzanso kwazinthu zolembedwa. Pogwiritsira ntchito chitsanzo cha mankhwala a fodya, amene anaphatikizidwa m’magulu oyambirira a katundu wolembedwa, ndudu imodzi yokha yodzaza ndudu ili ndi mapaketi pafupifupi 600, iliyonse ili ndi zizindikiro zakezake. Ndipo ntchito ya dongosolo lathu ndikutsata ndikutsimikizira kulondola kwa mayendedwe a paketi iliyonse yotere pakati pa malo osungiramo zinthu ndi masitolo, ndikutsimikizira kuvomerezedwa kwa kugulitsa kwawo kwa wogula. Ndipo timajambulitsa ndalama zokwana 000 pa ola limodzi, ndipo tiyeneranso kulemba momwe paketi iliyonse yotereyi idalowa m'sitolo. Choncho, poganizira mayendedwe onse pakati pa zinthu, tikuyembekezera makumi mabiliyoni a zolemba pachaka.
Team M
Ngakhale Marcus amawonedwa ngati projekiti mkati mwa X5, ikugwiritsidwa ntchito pogwiritsa ntchito njira zopangira. Gululi limagwira ntchito molingana ndi Scrum. Ntchitoyi inayamba m'chilimwe chatha, koma zotsatira zoyamba zinabwera mu October - gulu lathu lomwe linasonkhanitsidwa mokwanira, zomangamanga zinapangidwa ndipo zida zinagulidwa. Tsopano gululi lili ndi anthu a 16, asanu ndi mmodzi omwe akugwira nawo ntchito za backend ndi frontend development, atatu omwe akukhudzidwa ndi kusanthula dongosolo. Anthu enanso asanu ndi mmodzi akutenga nawo gawo pa ntchito zamanja, zonyamula katundu, zoyeserera zokha, komanso kukonza zinthu. Kuphatikiza apo, tili ndi katswiri wa SRE.
Osati opanga okha omwe amalemba ma code mu gulu lathu; pafupifupi anyamata onse amadziwa kupanga ndi kulemba ma autotest, kulongedza zolemba ndi zolemba zokha. Timapereka chidwi chapadera kwa izi, chifukwa ngakhale chithandizo chamankhwala chimafunikira kukhazikika kwapamwamba. Nthawi zonse timayesetsa kulangiza ndi kuthandiza anzathu omwe sanakonzekerepo, ndikuwapatsa ntchito zing'onozing'ono zoti agwire.
Chifukwa cha mliri wa coronavirus, tidasamutsira gulu lonse kukagwira ntchito zakutali; kupezeka kwa zida zonse zowongolera chitukuko, kasamalidwe ka ntchito ku Jira ndi GitLab kunapangitsa kuti zidutse siteji iyi mosavuta. Miyezi yomwe idagwiritsidwa ntchito kutali idawonetsa kuti zokolola za gululo sizinavutike chifukwa cha izi; kwa ambiri, chitonthozo pantchito chidakula, chinthu chokhacho chomwe chidasowa chinali kulumikizana kwamoyo.
Msonkhano wamagulu akutali
Misonkhano pa ntchito yakutali
Technology okwana ya yankho
Chosungira chokhazikika ndi chida cha CI/CD cha X5 ndi GitLab. Timagwiritsa ntchito posungirako ma code, kuyesa kosalekeza, ndikutumiza kuyesa ndi kupanga ma seva. Timagwiritsanso ntchito mchitidwe wowunikira ma code, pamene osachepera 2 ogwira nawo ntchito amafunika kuvomereza kusintha kopangidwa ndi wopanga ku code. Ma static code analyzer SonarQube ndi JaCoCo amatithandiza kusunga khodi yathu yaukhondo ndikuwonetsetsa kuchuluka kwa mayeso a unit. Zosintha zonse pa code ziyenera kudutsa macheke awa. Zolemba zonse zoyeserera zomwe zimayendetsedwa pawokha zimangokhala zokha.
Kuti tikwaniritse bwino njira zamabizinesi ndi "Marcus", tidayenera kuthana ndi zovuta zingapo zaukadaulo, zamtundu uliwonse.
Ntchito 1. Kufunika kopingasa scalability dongosolo
Kuti tithetse vutoli, tidasankha njira ya microservice yomanga. Panthawi imodzimodziyo, kunali kofunika kwambiri kumvetsetsa mbali za udindo wa mautumiki. Tidayesetsa kuwagawa m'mabizinesi, poganizira zomwe zimachitika. Mwachitsanzo, kuvomereza ku nyumba yosungiramo katundu sikochitika kawirikawiri, koma ntchito yaikulu kwambiri, yomwe imayenera kupeza mwamsanga kuchokera kwa olamulira a boma zokhudzana ndi magawo a katundu omwe akuvomerezedwa, chiwerengero cha zomwe muzopereka umodzi zimafika 600000. , yang'anani kuvomerezedwa kwa kuvomera mankhwalawa m'nyumba yosungiramo katundu ndikubweza zidziwitso zonse zofunika pamakina osungira zinthu. Koma kutumiza kuchokera ku nyumba zosungiramo katundu kumakhala ndi mphamvu zambiri, koma nthawi yomweyo zimagwira ntchito ndi ma data ochepa.
Timagwiritsa ntchito mautumiki onse mopanda malire ndipo timayesa kugawa ntchito zamkati m'masitepe, pogwiritsa ntchito zomwe timazitcha kuti Kafka self-topics. Apa ndipamene microservice imadzitumizira yokha uthenga, yomwe imakupatsani mwayi woti muzitha kuyendetsa bwino ntchito zogwiritsa ntchito kwambiri komanso kumathandizira kukonza zinthu, koma zambiri pambuyo pake.
Tinaganiza zolekanitsa ma modules okhudzana ndi machitidwe akunja kukhala mautumiki osiyana. Izi zidapangitsa kuti zitheke kuthana ndi vuto lakusintha pafupipafupi ma API a machitidwe akunja, osakhudzidwa kwenikweni ndi mautumiki omwe ali ndi magwiridwe antchito.
Ma microservices onse amayikidwa mugulu la OpenShift, lomwe limathetsa vuto lakukulitsa microservice iliyonse ndikutilola kuti tisagwiritse ntchito zida zachitatu za Service Discovery.
Ntchito 2. Kufunika kosunga katundu wambiri komanso kusinthana kwakukulu kwa data pakati pa mautumiki apapulatifomu: Pa gawo lokhazikitsa polojekiti yokha, pafupifupi ntchito 600 pa sekondi iliyonse zimachitika. Tikuyembekeza kuti mtengowu uwonjezeke mpaka 5000 ops/sec ngati malo ogulitsa akulumikizana ndi nsanja yathu.
Vutoli lidathetsedwa potumiza gulu la Kafka ndikusiyiratu kuyanjana kolumikizana pakati pa ma microservices a nsanja. Izi zimafuna kusanthula mosamala kwambiri zofunikira za dongosolo, popeza sizinthu zonse zomwe zingakhale zosagwirizana. Nthawi yomweyo, sitimangofalitsa zochitika kudzera mwa broker, komanso timatumiza zidziwitso zonse zofunikira zamabizinesi muuthenga. Chifukwa chake, kukula kwa uthenga kumatha kufika ma kilobytes mazana angapo. Malire a kukula kwa uthenga ku Kafka amafuna kuti tilosere molondola kukula kwa uthenga, ndipo ngati kuli kofunikira, timawagawanitsa, koma kugawanika kuli komveka, kokhudzana ndi ntchito zamalonda.
Mwachitsanzo, timagawa katundu amene amafika m’galimoto m’mabokosi. Pakuchita ma synchronous, ma microservices osiyana amaperekedwa ndipo kuyezetsa kokwanira kumachitika. Kugwiritsa ntchito Kafka kunatibweretsera vuto lina - kuyesa magwiridwe antchito athu poganizira kuphatikiza kwa Kafka kumapangitsa kuti mayeso athu onse akhale osagwirizana. Tinathetsa vutoli polemba njira zathu zothandizira pogwiritsa ntchito Embedded Kafka Broker. Izi sizichotsa kufunika kolemba mayeso a mayunitsi panjira zapayekha, koma timakonda kuyesa milandu yovuta kugwiritsa ntchito Kafka.
Chisamaliro chochuluka chidaperekedwa pakutsata zipika kuti TraceId yawo isatayike pomwe zopatula zimachitika panthawi yantchito kapena pogwira ntchito ndi gulu la Kafka. Ndipo ngati panalibe zovuta zapadera ndi yoyamba, ndiye kuti chachiwiri timakakamizika kulemba ma TraceIds onse omwe gululo lidabwera nawo ndikusankha imodzi kuti tipitirize kutsatira. Kenako, posaka ndi TraceId yoyambirira, wogwiritsa ntchito apeza mosavuta zomwe kutsatira kudapitilira.
Ntchito 3. Kufunika kosunga deta yambiri: Zolemba zoposa 1 biliyoni pachaka za fodya zokha zimabwera ku X5. Amafuna mwayi wokhazikika komanso wachangu. Pazonse, dongosololi liyenera kukonza zolembedwa pafupifupi 10 biliyoni za mbiri yamayendedwe azinthu zolembedwazi.
Kuti athetse vuto lachitatu, database ya NoSQL ya MongoDB idasankhidwa. Tapanga shard ya node 5 ndipo node iliyonse ili ndi Replica Set ya ma seva atatu. Izi zimakupatsani mwayi wokulitsa dongosolo mozungulira, ndikuwonjezera ma seva atsopano pagulu, ndikuwonetsetsa kuti akulekerera zolakwika. Apa tidakumana ndi vuto lina - kuwonetsetsa kuti gulu la mongo likuyenda bwino, poganizira kugwiritsa ntchito ma microservices opingasa. Mwachitsanzo, imodzi mwa ntchito zamakina athu ndikuzindikira zoyesa kugulitsanso zinthu zomwe zili ndi zilembo zomwezo. Apa, zokutira zimawoneka ndi masikanidwe olakwika kapena ntchito zolakwika za osunga ndalama. Tidapeza kuti zobwereza ngati izi zitha kuchitika mkati mwa gulu limodzi la Kafka lomwe likukonzedwa, ndipo mkati mwa magulu awiri akukonzedwa mofanana. Chifukwa chake, kuyang'ana zobwerezedwa pofunsa nkhokwe sikunapereke kalikonse. Pa microservice iliyonse, tidathetsa vutoli padera kutengera malingaliro abizinesi amtunduwu. Mwachitsanzo, pa macheke, tidawonjezera cheke mkati mwa batch ndikupatula makonzedwe akuwoneka obwereza polowetsa.
Kuonetsetsa kuti ntchito ya owerenga ndi mbiri ya ntchito sizimakhudza mwanjira iliyonse chinthu chofunikira kwambiri - kugwira ntchito kwa bizinesi yathu, talekanitsa deta yonse ya mbiriyakale kukhala ntchito yosiyana ndi deta yosiyana, yomwe imalandiranso zambiri kudzera mu Kafka. . Mwanjira iyi, ogwiritsa ntchito amagwira ntchito ndi ntchito zawozawokha popanda kukhudza ntchito zomwe zimagwira ntchito nthawi zonse.
Ntchito 4: Kukonzanso ndi kuyang'anira pamzere:
M'machitidwe ogawidwa, mavuto ndi zolakwika zimakhalapo mosalephera pakupezeka kwa nkhokwe, mizere, ndi magwero a deta akunja. Pankhani ya Marcus, gwero la zolakwika zotere ndikuphatikizana ndi machitidwe akunja. Zinali zofunikira kupeza yankho lomwe lingalole zopempha mobwerezabwereza za mayankho olakwika ndi nthawi yodziwika, koma nthawi yomweyo osasiya kukonza zopempha zopambana pamzere waukulu. Pachifukwa ichi, lingaliro lotchedwa "topic based retry" lidasankhidwa. Pa mutu waukulu uliwonse, mutu umodzi kapena zingapo zoyeseranso zimapangidwira komwe mauthenga olakwika amatumizidwa ndipo nthawi yomweyo kuchedwa kwa mauthenga a mutu waukulu kumachotsedwa. Njira yolumikizirana -
Kuti tigwiritse ntchito chiwembu chotere, tidafunikira izi: kuphatikiza yankho ili ndi Spring ndikupewa kubwereza ma code. Tikuyang'ana pa intaneti, tidapeza yankho lomwelo kutengera Spring BeanPostProccessor, koma zidawoneka ngati zovuta kwa ife. Gulu lathu lapanga njira yosavuta yomwe imatilola kuphatikizira munyengo ya Spring popanga ogula ndikuwonjezera Yeseraninso Ogula. Tidapereka chitsanzo cha yankho lathu ku timu ya Spring, mutha kuziwona . Chiwerengero cha Ogwiritsanso Ntchito Yesaninso ndi kuchuluka kwa zoyeserera kwa wogula aliyense zimakonzedwa kudzera mu magawo, malingana ndi zosowa za bizinesi, komanso kuti chilichonse chigwire ntchito, chomwe chatsalira ndikuwonjezera ndemanga org.springframework.kafka.annotation.KafkaListener , zomwe ndizodziwika kwa onse opanga ma Spring.
Ngati uthengawo sunathe kusinthidwa pambuyo poyesanso, umapita ku DLT (mutu wa kalata yakufa) pogwiritsa ntchito Spring DeadLetterPublishingRecoverer. Mwa pempho la chithandizo, tidakulitsa ntchitoyi ndikupanga ntchito ina yomwe imakupatsani mwayi wowona mauthenga omwe akuphatikizidwa mu DLT, stackTrace, traceId ndi zina zambiri zothandiza za iwo. Kuphatikiza apo, kuyang'anira ndi zidziwitso zinawonjezeredwa pamitu yonse ya DLT, ndipo tsopano, kwenikweni, maonekedwe a uthenga mu mutu wa DLT ndi chifukwa chosanthula ndi kukonza cholakwika. Izi ndizothandiza kwambiri - ndi dzina la mutuwo, timamvetsetsa nthawi yomweyo kuti vutoli lidayamba liti, lomwe limafulumizitsa kufunafuna chifukwa chake.
Posachedwapa, takhazikitsa mawonekedwe omwe amatilola kuti titumizenso mauthenga pogwiritsa ntchito chithandizo chathu pambuyo pochotsa zomwe zimayambitsa (mwachitsanzo, kubwezeretsa ntchito za machitidwe akunja) ndipo, ndithudi, kukhazikitsa cholakwika chofanana kuti chiwunikidwe. Apa ndipamene mitu yathu yaumwini imabwera bwino: kuti musayambitsenso unyolo wautali wokonza, mutha kuyiyambitsanso kuchokera pagawo lomwe mukufuna.
Platform Operation
Pulatifomu ikugwira ntchito bwino, tsiku lililonse timanyamula katundu ndi kutumiza, kulumikiza malo atsopano ogulitsa ndi masitolo. Monga gawo la woyendetsa ndege, dongosololi limagwira ntchito ndi magulu a "Fodya" ndi "Nsapato".
Gulu lathu lonse limatenga nawo gawo poyendetsa oyendetsa ndege, kusanthula mavuto omwe akubwera ndikupanga malingaliro owongolera malonda athu, kuyambira kukonza zipika mpaka kusintha njira.
Kuti tisabwereze zolakwa zathu, milandu yonse yomwe imapezeka poyendetsa ndege imawonetsedwa pamayesero odzichitira okha. Kukhalapo kwa ma autotests ambiri ndi mayeso a unit kumakupatsani mwayi woyesa kuyambiranso ndikuyika hotfix kwenikweni mkati mwa maola ochepa.
Tsopano tikupitiriza kupanga ndi kukonza nsanja yathu, ndikukumana ndi zovuta zatsopano. Ngati mukufuna, tidzakambirana za mayankho athu m'nkhani zotsatirazi.
Source: www.habr.com