Kubva 2019, Russia yanga iine mutemo wekusungirwa kunyora. Mutemo haushande kumapoka ese ezvigadzirwa, uye mazuva ekupinda musimba rekumanikidza kunyoreswa kwemapoka ezvigadzirwa akasiyana. Fodya, bhutsu, uye mishonga ndiyo ichave yekutanga kunyorerwa; zvimwe zvigadzirwa zvichawedzerwa gare gare, semuenzaniso, pefiyumu, machira, nemukaka. Uku kuvandudzwa kwemutemo kwakakurudzira kugadzirwa kwezvitsva zveIT zvinozoita kuti zvikwanise kuteedzera hupenyu hwese hwechigadzirwa kubva pakugadzirwa kusvika kunotengwa nemutengi wekupedzisira, kune vese vatori vechikamu muchirongwa ichi: zvese nyika pachayo nemasangano ese anotengesa zvinhu mandatory labeling.
MuX5, iyo sisitimu inoteedzera zvinhu zvakanyorwa uye kuchinjanisa data nenyika uye vatengesi inonzi "Marcus". Ngatikuudzei kuti sei uye ndiani akaigadzira, ndeipi tekinoroji stack, uye nei tiine chimwe chinhu chekudada nacho.
Real HighLoad
"Marcus" inogadzirisa matambudziko mazhinji, chikuru ndechekubatana kwekudyidzana pakati peX5 ruzivo masisitimu uye yehurumende ruzivo system yezvigadzirwa zvakanyorwa (GIS MP) kuronda mafambiro ezvigadzirwa zvakanyorwa. Iyi puratifomu inochengetawo ese makodhi makodhi akagamuchirwa nesu uye nhoroondo yese yekufamba kwemakodhi aya pazvinhu zvese, uye inobatsira kubvisa kudzoreredzwa kwezvigadzirwa zvakanyorwa. Tichishandisa muenzaniso wezvigadzirwa zvefodya, izvo zvakabatanidzwa mumapoka ekutanga ezvigadzirwa zvakanyorwa mazita, rori rimwe chete refodya rine mapeji anenge 600 000, rimwe nerimwe rine bumbiro rayo rimene rakasiyana. Uye basa rehurongwa hwedu nderekutevera uye kuona huchokwadi hwekufamba kwepakiti imwe neimwe yakadaro pakati pematura nezvitoro, uye pakupedzisira kuona kubvumidzwa kwekutengesa kwavo kune yekupedzisira mutengi. Uye isu tinorekodha nezve 125 mari yekutengeserana paawa, uye isu tinoda zvakare kurekodha kuti imwe neimwe paki yakadai yakapinda sei muchitoro. Saka, tichifunga nezvese mafambiro pakati pezvinhu, tiri kutarisira makumi emabhiriyoni emarekodhi pagore.
Team M
Kunyangwe chokwadi chekuti Marcus inoonekwa sepurojekiti mukati meX5, iri kuitwa uchishandisa chigadzirwa maitiro. Chikwata chinoshanda zvinoenderana neScrum. Iyo purojekiti yakatanga zhizha rapfuura, asi mhedzisiro yekutanga yakauya chete muna Gumiguru - timu yedu pachedu yakaunganidzwa zvizere, hurongwa hwekuvaka hwakagadzirwa uye midziyo yakatengwa. Iye zvino boka racho rine vanhu gumi nevatanhatu, vatanhatu vavo vanobatanidzwa mukusimudzira uye mberi, vatatu vavo vanobatanidzwa mukuongorora kwehurongwa. Vamwe vanhu vatanhatu vari kuita zvemanyorero, mutoro, otomatiki kuyedza, uye kugadzirisa chigadzirwa. Mukuwedzera, isu tine SRE nyanzvi.
Kwete chete vanogadzira vanonyora kodhi muchikwata chedu; vangangoita vakomana vese vanoziva kuronga nekunyora otomatiki, kurodha zvinyorwa uye otomatiki zvinyorwa. Isu tinobhadhara zvakanyanya kune izvi, sezvo kunyangwe kutsigirwa kwechigadzirwa kunoda mwero wepamusoro we automation. Isu tinogara tichiedza kupa zano nekubatsira vatinoshanda navo vasina kuronga kare, uye kuvapa mamwe mabasa madiki ekushanda.
Nekuda kwedenda rekoronavirus, takaendesa timu yese kubasa riri kure; kuvapo kwezvishandiso zvese zvekusimudzira manejimendi, kufambiswa kwebasa muJira neGitLab kwakaita kuti zvikwanise kupfuura nhanho iyi zviri nyore. Mwedzi yakashandiswa kure yakaratidza kuti kugadzirwa kwechikwata hakuna kutambura semhedzisiro; kune vazhinji, nyaradzo pabasa yakawedzera, chinhu chega chaishaikwa kwaive kutaura kwepamoyo.
Remote timu musangano
Misangano panguva yebasa riri kure
Technology stack yemhinduro
Iyo yakajairwa repository uye CI/CD chishandiso cheX5 iGitLab. Isu tinoishandisa kuchengetedza kodhi, kuyedza kuenderera, uye kutumira kuyedza uye kugadzira maseva. Isu tinoshandisawo tsika yekuongorora kodhi, apo vangangoita 2 vatinoshanda navo vanofanirwa kubvumidza shanduko dzakaitwa nemugadziri kune kodhi. Static code analyzers SonarQube uye JaCoCo inotibatsira kuchengeta kodhi yedu yakachena uye ive nechokwadi chinodiwa chiyero cheyuniti bvunzo yekuvhara. Shanduko dzese kukodhi dzinofanira kuenda kuburikidza nekutarisa idzi. Ese magwaro ebvunzo anoitwa nemaoko anobva aitwa otomatiki.
Nekubudirira kwekuita kwebhizinesi maitiro ne "Marcus", taifanira kugadzirisa akati wandei matambudziko etekinoroji, nezve imwe neimwe yakarongeka.
Basa 1. Kudiwa kwehorizontal scalability yehurongwa
Kuti tigadzirise dambudziko iri, takasarudza nzira ye microservice yekuvaka. Panguva imwecheteyo, zvaive zvakakosha zvikuru kunzwisisa nzvimbo dzebasa remasevhisi. Takaedza kuvapatsanura kuita bhizinesi mashandiro, tichifunga nezve chaiwo maitiro. Semuenzaniso, kugamuchirwa paimba yekuchengetera zvinhu hakusi kwakajairika, asi kwakakura kwazvo-kushanda, panguva iyo inofanirwa kukurumidza kuwana kubva kumutongi wehurumende ruzivo pamusoro pezvikamu zvezvinhu zvinogamuchirwa, nhamba iyo mune imwe kutakura inosvika 600000. , tarisa kubvumidzwa kwekugamuchira chigadzirwa ichi mudura uye dzosera ruzivo rwese rwunodiwa rwewarehouse automation system. Asi kutumirwa kubva kumatura kune hukuru hukuru, asi panguva imwechete inoshanda nediki mavhoriyamu e data.
Isu tinoshandisa masevhisi ese pasina mamiriro uye tinoedza kupatsanura mashandiro emukati kuita matanho, tichishandisa yatinodaidza kuti Kafka self-topics. Apa ndipo apo microservice inotumira meseji kune yega, iyo inokutendera iwe kuti uenzanise mutoro pane yakawanda-yakanyanya-simba mashandiro uye inorerutsa kugadzirisa kwechigadzirwa, asi zvimwe pane izvo gare gare.
Isu takasarudza kupatsanura mamodule ekudyidzana nekunze masisitimu kuita masevhisi akasiyana. Izvi zvakaita kuti zvikwanisike kugadzirisa dambudziko rekugara kuchinje maAPIs ekunze masisitimu, pasina kana kukanganisa masevhisi ane bhizinesi mashandiro.
Ese mamicroservices anoiswa muOpenShift cluster, iyo inogadzirisa ese ari maviri dambudziko rekuyera yega microservice uye inotitendera kuti tisashandise wechitatu-bato Service Discovery maturusi.
Basa 2. Kudiwa kuchengetedza mutoro wakakwirira uye yakanyanya kusimba kuchinjanisa data pakati pepuratifomu masevhisi: Panguva yechikamu chekutanga chirongwa chete, anenge mazana matanhatu emabasa pasekondi anoitwa. Isu tinotarisira kuti kukosha uku kuwedzere kusvika pa600 ops/sec sezvitoro zvinobatana papuratifomu yedu.
Dambudziko iri rakagadziriswa nekuisa sumbu reKafka uye rakapotsa kusiya zvachose kupindirana pakati pemasevhisi epuratifomu. Izvi zvinoda kunyatso ongorora zvinodiwa nehurongwa, nekuti kwete ese ma operation anogona kuve asynchronous. Panguva imwecheteyo, isu hatisi kungoendesa zviitiko kuburikidza nemutengesi, asi zvakare kuendesa zvese zvinodiwa zvebhizinesi ruzivo mumeseji. Saka, saizi yemeseji inogona kusvika mazana emakirobhayiti. Iyo meseji saizi muganho muKafka inoda kuti isu tinyatso kufanotaura saizi yemeseji, uye kana zvichidikanwa, tinovapatsanura, asi kupatsanurwa kunonzwisisika, kune chekuita nekushanda kwebhizinesi.
Somuenzaniso, tinogovanisa zvinhu zvinosvika nemotokari mumabhokisi. Kune mashandisirwo ekushanda, ma microservices akaparadzaniswa akagoverwa uye kunyatsoyedzwa kwemutoro kunoitwa. Kushandisa Kafka kwakatipa rimwe dambudziko - kuyedza kushanda kwesevhisi yedu uchifunga nezvekubatanidzwa kweKafka kunoita kuti bvunzo dzedu dzese dzive dzakafanana. Isu takagadzirisa dambudziko iri nekunyora zvedu nzira dzekushandisa tichishandisa Embedded Kafka Broker. Izvi hazvibvisi kukosha kwekunyora mayunitsi bvunzo kune ega nzira, asi isu tinosarudza kuyedza kesi dzakaomarara tichishandisa Kafka.
Kutarisisa kwakawanda kwakabhadharwa pakuronda matanda kuitira kuti TraceId yavo isarasikike kana kusarudzika kuchiitika panguva yekushanda kwemasevhisi kana pakushanda neKafka batch. Uye kana pakanga pasina nyaya dzakakosha neyekutanga, saka mune yechipiri nyaya tinomanikidzwa kurodha ese maTraceIds ayo batch akauya nawo uye sarudza imwe yekuenderera mberi nekutevera. Zvino, kana uchitsvaga neiyo yekutanga TraceId, mushandisi anozoona zviri nyore kuti kutsvaga kwakaenderera mberi nei.
Basa rechitatu. Kudiwa kwekuchengeta huwandu hwakawanda hwe data: Mapepa efodya chete anodarika bhiriyoni pagore anouya kuX1. Vanoda nguva dzose uye nekukurumidza kuwana. Pakazara, iyo sisitimu inofanirwa kugadzirisa anosvika bhiriyoni gumi marekodhi enhoroondo yekufamba kwezvinhu izvi zvakanyorwa.
Kugadzirisa dambudziko rechitatu, iyo NoSQL dhatabhesi MongoDB yakasarudzwa. Isu takavaka shard ye5 node uye imwe neimwe node ine Replica Seti yevatatu maseva. Izvi zvinokutendera iwe kuyera sisitimu yakatwasuka, uchiwedzera maseva matsva kusumbu, uye ive nechokwadi chekutadza kwayo kushivirira. Pano takasangana nerimwe dambudziko - kuve nechokwadi chekutengeserana muboka remongo, tichifunga nezvekushandiswa kweiyo horizontally scalable microservices. Semuenzaniso, rimwe remabasa ehurongwa hwedu kuona kuyedza kutengesa zvakare zvigadzirwa zvine iwo akafanana macode ekunyora. Pano, zvakafukidzirwa zvinoonekwa nezvisikeni zvisizvo kana mashandiro asina kunaka nevanobvisa mari. Isu takaona kuti zvakapetwa zvakadaro zvinogona kuitika mukati mechikamu chimwe cheKafka chiri kugadziriswa, uye mukati memabheji maviri ari kugadziriswa anoenderana. Saka, kutarisa kune zvakapetwa nekubvunza dhatabhesi hakuna chakapa. Kune yega yega microservice, isu takagadzirisa dambudziko zvakasiyana zvichibva pane bhizinesi logic yesevhisi iyi. Semuenzaniso, kune cheki, isu takawedzera cheki mukati mebatch uye kupatsanura gadziriso yekutaridzika kweanodzokorora kana uchiisa.
Kuti ive nechokwadi chekuti basa revashandisi nenhoroondo yekushanda harikanganisi chinhu chinonyanya kukosha - kushanda kwebhizinesi redu maitiro, takaparadzanisa data rese renhoroondo kuita sevhisi yakaparadzana ine dhatabhesi rakasiyana, iro rinogamuchirawo ruzivo kuburikidza neKafka. . Nenzira iyi, vashandisi vanoshanda nesevhisi yakasarudzika vasingakanganise masevhisi anogadzirisa data rekuenderera mberi mashandiro.
Basa rechina: Mutsara wekudzokorora uye kutarisa:
Mune masisitimu akagoverwa, matambudziko uye zvikanganiso zvinomuka mukuwanikwa kwedatabase, mitsetse, uye ekunze data masosi. Panyaya yaMarcus, kunobva zvikanganiso zvakadaro kubatanidzwa nekunze masisitimu. Zvakanga zvakafanira kutsvaga mhinduro yaizobvumira zvikumbiro zvakadzokororwa zvemhinduro dzisina kunaka nedzimwe nguva yakatarwa, asi panguva imwe chete usarega kugadzirisa zvikumbiro zvakabudirira mumutsetse mukuru. Nechinangwa ichi, iyo inonzi "topic based retry" pfungwa yakasarudzwa. Pamusoro pemusoro wega wega mukuru, imwe kana anopfuura edzaizve misoro inogadzirwa kunotumirwa meseji isiriyo uye panguva imwe chete kunonoka kugadzirisa mameseji kubva pachinyorwa chikuru kunobviswa. Chirongwa chekudyidzana -
Kuti tiite chirongwa chakadaro, taida zvinotevera: kubatanidza iyi mhinduro neChirimo uye kudzivirira kudzokorora kodhi. Tichiri kusefa pawebhu, takawana mhinduro yakafanana yakavakirwa paChirimo BeanPostProccessor, asi zvaiita sezvisina basa kwatiri. Chikwata chedu chagadzira mhinduro iri nyore iyo inotitendera kuti tibatanidze muChirimo kutenderera kwekugadzira vatengi uye nekuwedzera Edzazve Vatengi. Isu takapa prototype yemhinduro yedu kuchikwata cheSpring, unogona kuzviona . Huwandu hweVatengi Edzazve uye nhamba yekuedza kwemutengi wega wega inogadziriswa kuburikidza nemiganhu, zvichienderana nezvinodiwa zvebhizinesi maitiro, uye kuti zvese zvishande, chasara kuwedzera annotation org.springframework.kafka.annotation.KafkaListener , iyo inozivikanwa kune vese vanogadzira Spring.
Kana meseji yatadza kugadziriswa mushure mekuedza kwese kuedza, inoenda kuDLT (yakafa tsamba musoro) uchishandisa Spring DeadLetterPublishingRecoverer. Nekukumbira kwerutsigiro, isu takawedzera mashandiro aya uye takagadzira sevhisi yakaparadzana iyo inokutendera kuti utarise mameseji anosanganisirwa muDLT, stackTrace, traceId uye rumwe ruzivo rwakakosha nezvavo. Mukuwedzera, kutarisa uye zviyeuchidzo zvakawedzerwa kune dzose DLT misoro, uye ikozvino, chaizvoizvo, kuonekwa kwemashoko mumusoro weDLT chikonzero chekuongorora nekugadzirisa kukanganisa. Izvi zvakanyanya nyore - nezita remusoro wenyaya, isu tinobva tanzwisisa padanho ripi rekuita dambudziko rakamuka, iro rinonyanya kukurumidza kutsvaga kwechikonzero chayo.
Nguva pfupi yadarika, takaita chinongedzo chinotibvumira kutumira mameseji tichishandisa rutsigiro rwedu mushure mekubvisa zvikonzero zvavo (semuenzaniso, kudzoreredza kushanda kweiyo yekunze system) uye, chokwadi, kumisikidza chikanganiso chinoenderana nekuongorora. Apa ndipo panouya nyaya dzedu pachedu: kuti usazotangazve cheni refu yekugadzirisa, unogona kuitanga patsva kubva padanho raunoda.
Platform Operation
Iyi puratifomu yatove mukushanda kunobudirira, zuva rega rega isu tinotakura uye kutumira, batanidza nzvimbo nyowani dzekugovera uye zvitoro. Sechikamu chemutyairi, hurongwa hunoshanda nemapoka ezvigadzirwa zve "Fodya" uye "Shoes".
Chikwata chedu chese chinotora chikamu mukuitisa vatyairi vendege, kuongorora matambudziko ari kubuda uye kuita mazano ekuvandudza chigadzirwa chedu, kubva pakuvandudza matanda kusvika pakuchinja maitiro.
Kuti tisadzokorore kukanganisa kwedu, zviitiko zvese zvinowanikwa panguva yekutyaira zvinoratidzwa mune otomatiki bvunzo. Kuvapo kwenhamba yakakura yeautotests uye bvunzo dzeyuniti inobvumidza iwe kuti uite regression bvunzo uye nekuisa hotfix chaiyo mukati maawa mashoma.
Iye zvino isu tinoenderera mberi nekuvandudza nekuvandudza chikuva chedu, uye tinogara tichitarisana nematambudziko matsva. Kana iwe uchifarira, tichataura nezvemhinduro dzedu munyaya dzinotevera.
Source: www.habr.com