Kwi-RIT 2019, umlingane wethu u-Alexander Korotkov wenza malunga ne-automation yophuhliso kwi-CIAN: ukwenza lula ubomi kunye nomsebenzi, sisebenzisa eyethu iqonga le-Integro. Ilandelela umjikelo wobomi bemisebenzi, ikhulula abaphuhlisi bemisebenzi yesiqhelo kwaye inciphisa kakhulu inani leempazamo kwimveliso. Kule posi, siya kuzalisekisa ingxelo ka-Alexander kwaye sikuxelele ukuba sihambe njani ukusuka kwimibhalo elula ukuya ekudibaniseni iimveliso zomthombo ovulekileyo ngeqonga lethu kunye nento eyenziwa liqela lethu elizisebenzelayo.
Inqanaba elingu-zero
"Ayikho into efana nenqanaba elingu-zero, andiyazi into enjalo"
Master Shifu kwifilimu "Kung Fu Panda"
Ukuzenzekela kwi-CIAN kwaqala iminyaka eyi-14 emva kokuba inkampani yasekwa. Ngelo xesha kwakukho abantu abangama-35 kwiqela lophuhliso. Kunzima ukukholelwa, akunjalo? Ewe, i-automation yayikhona ngandlel 'ithile, kodwa isalathiso esahlukileyo sokudityaniswa okuqhubekayo kunye nokuhanjiswa kwekhowudi kwaqala ukumila ngo-2015.
Ngelo xesha, sasine-monolith enkulu yePython, i-C # kunye ne-PHP, isetyenziswe kwiiseva ze-Linux / Windows. Ukuhambisa le monster, sasineseti yezikripthi esiziqhuba ngesandla. Kwakukho nendibano ye-monolith, eyazisa intlungu nokubandezeleka ngenxa yongquzulwano xa kudityaniswa amasebe, ukulungisa iziphene, nokwakhiwa kwakhona βngemisebenzi eyahlukeneyo yokwakha.β Inkqubo eyenziwe lula ijongeka ngolu hlobo:
Ayonwabanga yile nto, kwaye besifuna ukwakha inkqubo yokwakha ephindaphindwayo, ezenzekelayo nelawulekayo kunye nokusasazwa. Kule nto, sasidinga inkqubo ye-CI / CD, kwaye sakhetha phakathi kwenguqulo yamahhala ye-Teamcity kunye ne-free version ye-Jenkins, ekubeni sisebenze nabo kwaye zombini zisifanelekele ngokubhekiselele kwisethi yemisebenzi. Sikhethe iTeamcity njengemveliso yamva nje. Ngelo xesha, sasingekasebenzisi i-microservice architecture kwaye asizange silindele inani elikhulu lemisebenzi kunye neeprojekthi.
Siza kumbono wenkqubo yethu
Ukuphunyezwa kwe-Teamcity kususwe kuphela inxalenye yomsebenzi wezandla: into eseleyo kukudalwa kwezicelo zokutsalwa, ukukhuthazwa kwemiba ngesimo seJira, kunye nokukhethwa kwemiba yokukhululwa. Inkqubo yeTeamcity ayisakwazi ukumelana noku. Kwakuyimfuneko ukukhetha indlela yokuqhubela phambili okuzenzekelayo. Sithathele ingqalelo iinketho zokusebenza kunye nezikripthi kwiTeamcity okanye ukutshintshela kwiinkqubo ezizenzekelayo zomntu wesithathu. Kodwa ekugqibeleni siye sagqiba kwelokuba sifuna ukuguquguquka okuphezulu, nto leyo esisisombululo sethu kuphela esinokubonelela. Yile ndlela inguqulelo yokuqala ye-automation system yangaphakathi ebizwa ngokuba yi-Integro yavela.
I-Teamcity ijongene ne-automation kwinqanaba lokusungula iinkqubo zokwakha kunye nokuthunyelwa, ngelixa i-Integro igxininise kwi-automation ephezulu yeenkqubo zophuhliso. Kwakuyimfuneko ukudibanisa umsebenzi kunye nemiba eJira kunye nokuqhutyelwa kwekhowudi yomthombo ohambelanayo kwi-Bitbucket. Ngeli nqanaba, i-Integro yaqala ukuba nomsebenzi wayo wokusebenza kunye nemisebenzi yeentlobo ezahlukeneyo.
Ngenxa yokunyuka kwe-automation kwiinkqubo zoshishino, inani leeprojekthi kunye nokuqhuba kwi-Teamcity liye landa. Ke ingxaki entsha yeza: umzekelo omnye wasimahla weTeamcity wawungonelanga (iiarhente ezi-3 kunye neeprojekthi ezili-100), songeze omnye umzekelo (iiarhente ezi-3 kunye neeprojekthi ezili-100), emva koko omnye. Ngenxa yoko, saphela sinenkqubo yamaqela amaninzi, ekwakunzima ukuyilawula:
Xa umbuzo womzekelo we-4 wavela, saqonda ukuba asikwazi ukuqhubeka siphila ngolu hlobo, kuba iindleko ezipheleleyo zokuxhasa iimeko ze-4 zazingekho kuyo nayiphi na imida. Umbuzo waphakama malunga nokuthenga iTeamcity ehlawulwayo okanye ukukhetha iJenkins yamahhala. Senza izibalo kwiimeko kunye nezicwangciso ezizenzekelayo kwaye sagqiba ekubeni siza kuhlala eJenkins. Emva kweeveki ezimbalwa, satshintshela kwi-Jenkins kwaye sasusa enye yeentloko ezinxulumene nokugcina iimeko ezininzi zeTeamcity. Ke ngoko, sakwazi ukugxila ekuphuhliseni i-Integro kunye nokwenza ngokwezifiso iJenkins ngokwethu.
Ngokukhula kwe-automation eyisiseko (ngendlela yokudala ngokuzenzekelayo kwezicelo zokutsalwa, ukuqokelela kunye nokupapashwa kwe-Code coverage kunye nezinye iitshekhi), kukho umnqweno onamandla wokulahla ukukhutshwa kwencwadana kangangoko kunokwenzeka kwaye unike lo msebenzi kwiirobhothi. Ukongeza, inkampani yaqala ukufudukela kwi-microservices ngaphakathi kwenkampani, ebifuna ukukhutshwa rhoqo, kwaye ngokwahlukileyo omnye komnye. Le yindlela esiye safika ngayo ngokuthe ngcembe ekukhutshweni okuzenzekelayo kwee-microservices zethu (ngoku sikhulula i-monolith ngesandla ngenxa yobunzima benkqubo). Kodwa, njengesiqhelo, kwavela ukuntsonkotha okutsha.
Sizenza uvavanyo oluzenzekelayo
Ngenxa yokuzenzekela kokukhutshwa, iinkqubo zophuhliso ziye zakhawuleza, ngokuyinxenye ngenxa yokutsiba amanye amanqanaba ovavanyo. Kwaye oku kukhokelele ekulahlekeni okwethutyana komgangatho. Ivakala iyinto engenamsebenzi, kodwa kunye nokukhawuleziswa kokukhutshwa, bekuyimfuneko ukutshintsha indlela yophuhliso lwemveliso. Kwakuyimfuneko ukucinga malunga nokuzenzekelayo kovavanyo, ukufaka uxanduva lomntu (apha sithetha "ngokwamkela ingcamango entloko", kungekhona isohlwayo semali) yomphuhlisi wekhowudi ekhutshweyo kunye neebhugi kuyo, kunye nesigqibo so khulula/ungawukhululi umsebenzi ngobeko oluzenzekelayo.
Ukuphelisa iingxaki ezisemgangathweni, safikelela kwizigqibo ezibini ezibalulekileyo: saqala ukwenza uvavanyo lwe-canary kwaye sazisa ukubeka iliso ngokuzenzekelayo kwimvelaphi yephutha kunye nempendulo ezenzekelayo ekugqithiseni kwayo. Isisombululo sokuqala senze ukuba kube lula ukufumana iimpazamo ezicacileyo ngaphambi kokuba ikhowudi ikhutshwe ngokupheleleyo kwimveliso, okwesibini yanciphisa ixesha lokuphendula kwiingxaki kwimveliso. Iimpazamo, kunjalo, ziyenzeka, kodwa sichitha ixesha lethu elininzi kunye nemizamo yethu kungekhona ekulungiseni, kodwa ekunciphiseni.
Iqela elizenzekelayo
Okwangoku sinabasebenzi abaphuhlisi be-130, kwaye siyaqhubeka . Ukuhlanganiswa okuqhubekayo kunye neqela lokuhanjiswa kwekhowudi (emva koku kuthiwa yi-Deploy and Integration okanye iqela le-DI) liquka abantu be-7 kwaye lisebenza kwiindlela ezi-2: ukuphuhliswa kwe-platform ye-Automation ye-Integro kunye ne-DevOps.
I-DevOps inoxanduva lwemeko ye-Dev/Beta yendawo ye-CIAN, indawo ye-Integro, inceda abaphuhlisi basombulule iingxaki kwaye baphuhlise iindlela ezintsha zokulinganisa iindawo. Umkhombandlela wophuhliso lwe-Integro ujongene ne-Integro ngokwayo kunye neenkonzo ezinxulumeneyo, umzekelo, iiplagi ze-Jenkins, iJira, i-Confluence, kwaye ikwaphuhlisa izixhobo ezincedisayo kunye nezicelo zamaqela ophuhliso.
Iqela le-DI lisebenza ngokubambisana neqela lePlatform, eliphuhlisa ulwakhiwo, amathala eencwadi, kunye neendlela zophuhliso ngaphakathi. Kwangaxeshanye, nawuphi na umphuhlisi ongaphakathi kwe-CIAN unokufaka isandla kwi-automation, umzekelo, ukwenza i-micro-automation ukuze ihambelane neemfuno zeqela okanye ukwabelana ngombono opholileyo malunga nendlela yokwenza i-automation ibengcono.
Ikeyiki ye-automation e-CIAN
Zonke iisistim ezibandakanyekayo kwi-automation zinokwahlulwa ngokwezigaba ezininzi:
- Iinkqubo zangaphandle (Jira, Bitbucket, njl.). Amaqela ophuhliso asebenza nabo.
- Iqonga le-Integro. Amaxesha amaninzi, abaphuhlisi abasebenzi ngayo ngokuthe ngqo, kodwa yeyona nto igcina yonke i-automation isebenza.
- Ukuhanjiswa, i-orchestration kunye neenkonzo zokufumanisa (umzekelo, i-Jeknins, i-Consul, i-Nomad). Ngoncedo lwabo, sihambisa ikhowudi kwiiseva kwaye siqinisekisa ukuba iinkonzo zisebenza kunye.
- Umaleko womzimba (iiseva, i-OS, isoftware ehambelana nayo). Ikhowudi yethu isebenza kweli nqanaba. Oku kunokuba yiseva yomzimba okanye eyenyani (LXC, KVM, Docker).
Ngokusekwe kulo mbono, sahlulahlula imimandla yoxanduva phakathi kweqela le-DI. Amanqanaba amabini okuqala akwindawo yoxanduva lomkhombandlela wophuhliso lwe-Integro, kwaye amanqanaba amabini okugqibela sele ekwindawo yoxanduva lweDevOps. Olu lwahlulo luvumela ukuba sigxininise kwimisebenzi kwaye aluphazamisi ukusebenzisana, kuba sisondelelene kwaye sitshintshiselana rhoqo ngolwazi kunye namava.
Igcwele
Masigxile kwi-Integro kwaye siqale ngesitaki setekhnoloji:
- I-CentOS 7
- Docker + Nomad + Consul + Vault
- Java 11 (integro monolith endala iya kuhlala kwiJava 8)
- I-Spring Boot 2.X + i-Spring Cloud Config
- I-PostgreSql 11
- UmvundlaMQ
- Apache Ignite
- I-Camunda (ifakwe)
- Grafana + Graphite + Prometheus + Jaeger + ELK
- Web UI: React (CSR) + MobX
- SSO: Isiqhoboshi
Sithobela umgaqo wophuhliso lwe-microservice, nangona sinelifa ngendlela ye-monolith yenguqulelo yokuqala ye-Integro. I-microservice nganye iqhuba kwisitya sayo se-Docker, kwaye iinkonzo zinxibelelana ngezicelo ze-HTTP kunye nemiyalezo yeRabbitMQ. I-Microservices ifumanana nge-Consul kwaye yenza isicelo kuyo, idlulisa isigunyaziso nge-SSO (Keycloak, OAuth 2/OpenID Connect).
Njengomzekelo wobomi bokwenyani, cinga ukusebenzisana noJenkins, obandakanya la manyathelo alandelayo:
- I-microservice yolawulo lokuhamba komsebenzi (emva koku ebizwa ngokuba yi-Flow microservice) ifuna ukuqhuba isakhiwo e-Jenkins. Ukwenza oku, usebenzisa i-Consul ukufumana i-IP: I-PORT ye-microservice yokudibanisa kunye ne-Jenkins (emva koku kuthiwa yi-Jenkins microservice) kwaye ithumela isicelo esingavumelaniyo kuyo ukuqala ukwakha e-Jenkins.
- Emva kokufumana isicelo, i-Jenkins microservice ivelisa kwaye iphendule nge-ID ye-Job, engasetyenziselwa ukuchonga umphumo womsebenzi. Ngexesha elifanayo, ibangela ukwakhiwa kweJenkins nge-REST API call.
- UJenkins wenza ukwakhiwa kwaye, emva kokugqitywa, uthumela i-webhook eneziphumo zokwenziwa kwi-Jenkins microservice.
- I-Jenkins microservice, emva kokuba ifumene i-webhook, ivelisa umyalezo malunga nokugqitywa kokulungiswa kwesicelo kwaye ifake iziphumo zokwenziwa kuyo. Umyalezo owenziweyo uthunyelwa kumgca weRabbitMQ.
- Nge-RabbitMQ, umyalezo oshicilelweyo ufikelela kwi-Flow microservice, efunda malunga nesiphumo sokucubungula umsebenzi wayo ngokufanisa i-ID yomsebenzi kwisicelo kunye nomyalezo ofunyenweyo.
Ngoku sinee-microservices ezimalunga nama-30, ezinokuthi zohlulwe ngokwamaqela amaninzi:
- Ulawulo loqwalaselo.
- Ulwazi kunye nokusebenzisana nabasebenzisi (abathunywa, iposi).
- Ukusebenza ngekhowudi yomthombo.
- Ukudibanisa kunye nezixhobo zokuhambisa (ijenkins, nomad, consul, njl.).
- Ukubeka iliso (ukukhutshwa, iimpazamo, njl.).
- Izixhobo zewebhu (i-UI yokulawula iimeko zovavanyo, ukuqokelela izibalo, njl.).
- Ukudityaniswa nabalandeleli bemisebenzi kunye neenkqubo ezifanayo.
- Ulawulo lokuhamba komsebenzi kwimisebenzi eyahlukeneyo.
Imisebenzi yokuhamba komsebenzi
I-Integro yenza ngokuzenzekelayo imisebenzi enxulumene nomjikelo wobomi bomsebenzi. Ngamagama alula, umjikelo wobomi bomsebenzi uya kuqondwa njengokuhamba komsebenzi eJira. Iinkqubo zethu zophuhliso zineenguqu ezininzi zokuhamba komsebenzi ngokuxhomekeke kwiprojekthi, uhlobo lomsebenzi kunye neenketho ezikhethiweyo kumsebenzi othile.
Makhe sijonge indlela yokusebenza esiyisebenzisa rhoqo:
Kumzobo, i-gear ibonisa ukuba utshintsho lubizwa ngokuzenzekelayo yi-Integro, ngelixa umzobo womntu ubonisa ukuba utshintsho lubizwa ngokuba ngumntu. Makhe sijonge iindlela ezininzi ezinokuthathwa ngumsebenzi kulo msebenzi.
Uvavanyo olupheleleyo lwezandla kwi-DEV+BETA ngaphandle kovavanyo lwe-canary (ngokuqhelekileyo le yindlela esiyikhulula ngayo i-monolith):
Kusenokubakho ezinye iindibaniselwano zenguqu. Ngamanye amaxesha umendo oza kuthatha umba unokukhethwa ngeendlela kwiJira.
Intshukumo yomsebenzi
Makhe sijonge awona manyathelo aphambili enziwayo xa umsebenzi uhamba βkuVavanyo lwe-DEV + noVavanyo lweCanaryβ:
1. Umphuhlisi okanye i-PM yenza umsebenzi.
2. Umphuhlisi uthatha umsebenzi ukuba asebenze. Emva kokugqitywa, itshintshela kwimo ye-REVIEW.
3. UJira uthumela iWebhook kwi-Jira microservice (ejongene nokudibanisa noJira).
4. I-Jira microservice ithumela isicelo kwinkonzo ye-Flow (ejongene nokuhamba komsebenzi wangaphakathi apho umsebenzi wenziwa khona) ukuqala ukuhamba komsebenzi.
5. Ngaphakathi kwenkonzo yokuHamba:
- Abahloli babelwa umsebenzi (Abasebenzisi microservice owazi yonke into ngabasebenzisi + Jira microservice).
- Ngokusebenzisa uMthombo we-microservice (iyazi malunga neendawo zokugcina kunye namasebe, kodwa ayisebenzi kunye nekhowudi ngokwayo), uphando lwenziwa kwiindawo zokugcina eziqulethe isebe lomcimbi wethu (ukwenza lula ukukhangela, igama lesebe lihambelana nomba. inombolo eJira). Amaxesha amaninzi, umsebenzi uba nesebe elinye kuphela kwindawo yokugcina; oku kwenza lula ulawulo lwemigca yokusasazwa kunye nokunciphisa uqhagamshelwano phakathi kweendawo zokugcina.
- Kwisebe ngalinye elifunyenweyo, ulandelelwano lwezenzo lulandelayo lwenziwa:
i) Ukuhlaziya i-master branch (Git microservice yokusebenza ngekhowudi).
ii) Isebe livaliwe kutshintsho ngumphuhlisi (Bitbucket microservice).
iii) Isicelo sokutsalwa senzelwe eli sebe (Bitbucket microservice).
iv) Umyalezo malunga nesicelo esitsha sokuTsalwa uthunyelwa kwiincoko zomphuhlisi (Yazisa i-microservice ngokusebenza ngezaziso).
v) Yakha, uvavanye kwaye uhambise imisebenzi iqalwa kwi-DEV (Jenkins microservice yokusebenza noJenkins).
vi) Ukuba onke amanyathelo angaphambili agqitywe ngempumelelo, ngoko i-Integro ibeka i-Approve yayo kwisicelo sokutsalwa (Bitbucket microservice). - I-Integro ilindele ukuVunywa kwisicelo sokutsalwa kubavavanyi abakhethiweyo.
- Ngokukhawuleza ukuba zonke iimvume eziyimfuneko zifunyenwe (kubandakanywa neemvavanyo ezizenzekelayo ziphumelele ngokuqinisekileyo), i-Integro idlulisela umsebenzi kuVavanyo kwi-Dev (Jira microservice) isimo.
6. Abavavanyi bayawuvavanya umsebenzi. Ukuba akukho ngxaki, ngoko umsebenzi udluliselwa kwi-Ready For Build status.
7. I-Integro "ibona" ββukuba umsebenzi ulungele ukukhululwa kwaye uqala ukuthunyelwa kwayo kwimodi ye-canary (Jenkins microservice). Ukulungela ukukhululwa kugqitywa liqela lemigaqo. Umzekelo, umsebenzi ukwimo efunekayo, akukho zitshixo kweminye imisebenzi, okwangoku akukho zilayisho zisebenzayo zale microservice, njl.
8. Umsebenzi utshintshelwa kwiCanary status ( Jira microservice ).
9. I-Jenkins iqalisa umsebenzi wokuthunyelwa nge-Nomad kwimodi ye-canary (ngokuqhelekileyo i-1-3 iimeko) kwaye ikwazisa ngenkonzo yokubeka iliso yokukhupha (i-DeployWatch microservice) malunga nokuthunyelwa.
10. I-microservice ye-DeployWatch iqokelela imvelaphi yempazamo kwaye isabela kuyo, ukuba kuyimfuneko. Ukuba okungemva kwempazamo kugqithiwe (ummiselo ongasemva ubalwa ngokuzenzekelayo), abaphuhlisi bayaziswa nge-Notea microservice. Ukuba emva kwemizuzu emi-5 umphuhlisi akaphendulanga (cofa Buyela okanye Hlala), ngoko ukubuyisela umva okuzenzekelayo kweemeko ze-canary kuqaliswa. Ukuba imvelaphi ayigqithiswanga, ngoko umphuhlisi kufuneka aqalise ngesandla ukuthunyelwa komsebenzi kwiMveliso (ngokuchofoza iqhosha kwi-UI). Ukuba umphuhlisi akazange aqalise ukuthunyelwa kwiMveliso ngaphakathi kwemizuzu engama-60, ke iimeko ze-canary ziya kuphinda zibuyiselwe umva ngenxa yezizathu zokhuseleko.
11. Emva kokuqaliswa kokusasazwa kwiMveliso:
- Umsebenzi ugqithiselwa kwisimo seMveliso (Jira microservice).
- I-Jenkins microservice iqala inkqubo yokuthunyelwa kwaye yazisa i-DeployWatch microservice malunga nokuthunyelwa.
- I-DeployWatch microservice ijonga ukuba zonke izikhongozeli kwiMveliso zihlaziyiwe (kwakukho iimeko xa zingahlaziywanga zonke).
- Nge-Notify microservice, isaziso malunga neziphumo zokuthunyelwa sithunyelwa kwiMveliso.
12. Abaphuhlisi baya kuba nemizuzu engama-30 ukuqalisa ukubuyisela umva umsebenzi ovela kwiMveliso ukuba kufunyenwe ukuziphatha okungalunganga kwe-microservice. Emva kweli xesha, umsebenzi uya kudityaniswa ngokuzenzekelayo kwi-master (Git microservice).
13. Emva kokudibanisa ngempumelelo kwi-master, i-task status iya kutshintshwa ibe yi-Valiwe (i-Jira microservice).
Umzobo awuzenzi ukuba uneenkcukacha ezipheleleyo (enyanisweni kukho amanyathelo angaphezulu), kodwa ikuvumela ukuba uhlole iqondo lokudibanisa kwiinkqubo. Asiyicingi le nkqubo ngokufanelekileyo kwaye siphucula iinkqubo zokukhululwa ngokuzenzekelayo kunye nenkxaso yokuthunyelwa.
Yintoni elandelayo
Sinezicwangciso ezinkulu zokuphuhliswa kwe-automation, umzekelo, ukuphelisa imisebenzi yezandla ngexesha lokukhutshwa kwe-monolith, ukuphucula ukubeka iliso ngexesha lokuthunyelwa ngokuzenzekelayo, kunye nokuphucula ukusebenzisana nabaphuhlisi.
Kodwa masime apha okwangoku. Sigubungele izihloko ezininzi kuphononongo oluzenzekelayo phezulu, ezinye azizange zichukunyiswe kwaphela, ngoko siya kukuvuyela ukuphendula imibuzo. Silindele iingcebiso malunga nokuba yintoni esiza kuyigubungela ngokubanzi, bhala kwizimvo.
umthombo: www.habr.com