Kwa-RIT 2019, uzakwethu u-Alexander Korotkov wenza mayelana nokuzenzakalela kwentuthuko e-CIAN: ukwenza ukuphila nomsebenzi kube lula, sisebenzisa inkundla yethu ye-Integro. Ilandelela umjikelezo wempilo wemisebenzi, ikhulula onjiniyela emisebenzini yenjwayelo futhi yehlise ngokuphawulekayo inani leziphazamisi ekukhiqizweni. Kulokhu okuthunyelwe, sizogcwalisa umbiko ka-Alexander futhi sikutshele ukuthi sisuke kanjani emibhalweni elula saya ekuhlanganiseni imikhiqizo yomthombo ovulekile sisebenzisa inkundla yethu nokuthi ithimba lethu elihlukile le-automation lenzani.
Izinga elinguziro
"Ayikho into efana ne-zero level, angiyazi into enjalo"
I-Master Shifu evela kwifilimu ethi "Kung Fu Panda"
I-Automation e-CIAN yaqala eminyakeni engu-14 ngemuva kokusungulwa kwenkampani. Ngaleso sikhathi, ithimba lokuthuthukisa lalinabantu abangu-35. Kunzima ukukukholwa, akunjalo? Vele, i-automation yayikhona ngandlela thize, kodwa isiqondiso esihlukile sokuhlanganiswa okuqhubekayo nokulethwa kwekhodi kwaqala ukwakheka ngo-2015.
Ngaleso sikhathi, sasine-monolith enkulu yePython, C # kanye ne-PHP, esetshenziswe kumaseva e-Linux/Windows. Ukuze sikhiphe lesi silo, sibe nesethi yemibhalo esiyisebenzisa mathupha. Kwaphinde kwaba nokuhlanganiswa kwe-monolith, okwaletha ubuhlungu nokuhlupheka ngenxa yezingxabano lapho kuhlanganiswa amagatsha, kulungiswa amaphutha, futhi kwakha kabusha βngemisebenzi ehlukene esakhiweni.β Inqubo eyenziwe lula ibukeke kanje:
Asizange sijabule ngalokhu, futhi besifuna ukwakha inqubo yokwakha ephindaphindwayo, ezenzakalelayo nelawulekayo kanye nokusetshenziswa. Ngalokhu, besidinga uhlelo lwe-CI/CD, futhi sakhetha phakathi kwenguqulo yamahhala ye-Teamcity kanye nenguqulo yamahhala ye-Jenkins, njengoba sisebenze nabo futhi sobabili besifaneleka ngokwesethi yemisebenzi. Sikhethe i-Teamcity njengomkhiqizo wakamuva. Ngaleso sikhathi, sasingakasebenzisi i-microservice architecture futhi sasingalindele inani elikhulu lemisebenzi namaphrojekthi.
Sifika emcabangweni wesistimu yethu
Ukuqaliswa kwe-Teamcity kususe ingxenye kuphela yomsebenzi wezandla: okusele ukudalwa Kwezicelo Zokudonsa, ukukhushulwa kwezinkinga ngesimo ku-Jira, nokukhethwa kwezinkinga ezizokhishwa. Uhlelo lweTeamcity alusakwazi ukubhekana nalokhu. Kwakudingekile ukukhetha indlela yokuzenzakalela okwengeziwe. Sicabangele izinketho zokusebenza ngemibhalo ku-Teamcity noma ukushintshela kumasistimu wokuzenzakalela wenkampani yangaphandle. Kodwa ekugcineni sinqume ukuthi sidinga ukuguquguquka okukhulu, okungasinikeza isisombululo sethu kuphela. Lena yindlela inguqulo yokuqala yohlelo lwangaphakathi oluzenzakalelayo olubizwa ngokuthi i-Integro.
I-Teamcity ibhekana nokuzenzakalela ezingeni lokwethula izinqubo zokwakha nokuphakela, kuyilapho i-Integro igxile ekuzenzekeleni kwezinga eliphezulu lezinqubo zokuthuthukisa. Bekudingeka ukuhlanganisa umsebenzi nezinkinga ku-Jira nokucutshungulwa kwekhodi yomthombo ehlobene ku-Bitbucket. Kulesi sigaba, i-Integro yaqala ukuba nokugeleza komsebenzi wayo wokusebenza ngemisebenzi yezinhlobo ezahlukene.
Ngenxa yokwanda kokuzenzakalela ezinqubweni zebhizinisi, inani lamaphrojekthi kanye nokuqhutshwa kwe-Teamcity linyukile. Ngakho-ke kwafika inkinga entsha: isibonelo samahhala se-Teamcity sasinganele (ama-ejenti ama-3 namaphrojekthi ayi-100), sengeza esinye isibonelo (ama-ejenti angu-3 ngaphezulu namaphrojekthi ayi-100), kwase kuba esinye. Ngenxa yalokho, sagcina sinesistimu yamaqoqo amaningana, okwakunzima ukuwaphatha:
Lapho kuphakama umbuzo wesibonelo se-4, saqaphela ukuthi asikwazanga ukuqhubeka siphila kanje, ngoba izindleko eziphelele zokusekela izimo ezingu-4 zazingasekho ngaphakathi kwanoma yimiphi imingcele. Umbuzo uphakame mayelana nokuthenga iTeamcity ekhokhelwayo noma ukukhetha amaJenkins amahhala. Senza izibalo ngezimo nezinhlelo zokuzenzakalela futhi sanquma ukuthi sizohlala ku-Jenkins. Ngemva kwamasonto ambalwa, sashintshela ku-Jenkins futhi saqeda ezinye zekhanda elihlobene nokugcina izimo eziningi ze-Teamcity. Ngakho-ke, sikwazile ukugxila ekuthuthukiseni i-Integro nokwenza ngokwezifiso i-Jenkins ngokwethu.
Ngokukhula kwe-automation eyisisekelo (ngendlela yokudala okuzenzakalelayo kwezicelo zokudonsa, ukuqoqwa nokushicilelwa kokufakwa kweKhodi nokunye ukuhlola), kunesifiso esinamandla sokushiya ukukhishwa okwenziwa ngesandla ngangokunokwenzeka futhi unikeze lo msebenzi kumarobhothi. Ngaphezu kwalokho, inkampani yaqala ukuthuthela kuma-microservices ngaphakathi kwenkampani, eyayidinga ukukhishwa njalo, futhi ngokuhlukana komunye nomunye. Lena yindlela esifike ngayo kancane kancane ekukhishweni okuzenzakalelayo kwama-microservices ethu (okwamanje sikhulula i-monolith mathupha ngenxa yobunkimbinkimbi benqubo). Kodwa, njengoba ngokuvamile kwenzeka, kwavela inkimbinkimbi entsha.
Senza ukuhlola ngokuzenzakalelayo
Ngenxa yokukhishwa okuzenzakalelayo, izinqubo zokuthuthukisa ziye zashesha, ngokwengxenye ngenxa yokweqiwa kwezinye izigaba zokuhlola. Futhi lokhu kuholele ekulahlekeni kwekhwalithi okwesikhashana. Kuzwakala kuyinto encane, kodwa kanye nokusheshisa ukukhishwa, bekudingekile ukushintsha indlela yokuthuthukisa umkhiqizo. Kwakudingeka ukucabanga ngokuzenzakalelayo kokuhlola, ukufaka umthwalo wemfanelo womuntu siqu (lapha sikhuluma "ngokwamukela umbono ekhanda", hhayi izinhlawulo zemali) zonjiniyela wekhodi ekhishwe kanye nezimbungulu ezikuyo, kanye nesinqumo khulula/ungawukhiphi umsebenzi ngokufakwa okuzenzakalelayo.
Ukuqeda izinkinga zekhwalithi, sifike ezinqumweni ezimbili ezibalulekile: saqala ukwenza ukuhlolwa kwe-canary futhi sethula ukuqapha okuzenzakalelayo kwesizinda sephutha ngempendulo ezenzakalelayo ekudluleleni kwayo. Isixazululo sokuqala senze kwaba nokwenzeka ukuthola amaphutha asobala ngaphambi kokuba ikhodi ikhishwe ngokugcwele ekukhiqizeni, okwesibili kunciphise isikhathi sokuphendula ezinkingeni ekukhiqizeni. Yebo, amaphutha ayenzeka, kodwa isikhathi esiningi nemizamo yethu asichithi ekuwalungiseni, kodwa ekuwehliseni.
Ithimba le-Automation
Njengamanje sinezisebenzi zonjiniyela abangu-130, futhi sisaqhubeka . Ukuhlanganiswa okuqhubekayo kanye nethimba lokulethwa kwekhodi (ngemuva kwalokhu okubizwa ngokuthi i-Deploy and Integration noma ithimba le-DI) lihlanganisa abantu abangu-7 futhi lisebenza ezindleleni ezingu-2: ukuthuthukiswa kweplathifomu ye-Integro automation kanye ne-DevOps.
I-DevOps inesibopho semvelo ye-Dev/Beta yesayithi ye-CIAN, imvelo ye-Integro, isiza onjiniyela baxazulule izinkinga futhi ithuthukise izindlela ezintsha zokukala izindawo. Isiqondisindlela sokuthuthukiswa kwe-Integro sisebenzelana nakho kokubili kwe-Integro ngokwayo namasevisi ahlobene, isibonelo, ama-plugin we-Jenkins, Jira, Confluence, futhi ithuthukisa izinsiza ezisizayo nezicelo zamathimba okuthuthukisa.
Ithimba le-DI lisebenza ngokubambisana nethimba le-Platform, elithuthukisa izakhiwo, amalabhulali, nezindlela zokuthuthukisa ngaphakathi. Ngesikhathi esifanayo, noma yimuphi umthuthukisi ongaphakathi kwe-CIAN angakwazi ukufaka isandla ku-automation, isibonelo, enze i-micro-automation ukuze ivumelane nezidingo zeqembu noma babelane ngombono opholile wokuthi ungakwenza kanjani okuzenzakalelayo kube ngcono nakakhulu.
Ikhekhe lesendlalelo le-automation e-CIAN
Wonke amasistimu abandakanyeka ku-automation angahlukaniswa abe izingqimba eziningana:
- Amasistimu angaphandle (Jira, Bitbucket, njll.). Amaqembu entuthuko asebenza nabo.
- Isiteji se-Integro. Ezikhathini eziningi, abathuthukisi abasebenzi nayo ngokuqondile, kodwa iyona egcina konke okuzenzakalelayo kusebenza.
- Ukulethwa, i-orchestration kanye nezinsizakalo zokuthola (isibonelo, i-Jeknin, i-Consul, i-Nomad). Ngosizo lwabo, sisebenzisa amakhodi kumaseva futhi siqinisekisa ukuthi izinsiza zisebenzisana.
- Isendlalelo esibonakalayo (amaseva, i-OS, isofthiwe ehlobene). Ikhodi yethu isebenza kuleli zinga. Lokhu kungaba iseva ebonakalayo noma eyodwa (LXC, KVM, Docker).
Ngokusekelwe kulo mqondo, sihlukanisa izindawo zesibopho ngaphakathi kwethimba le-DI. Amaleveli amabili okuqala asendaweni yesibopho sokuqondiswa kokuthuthukiswa kwe-Integro, futhi amazinga amabili okugcina asevele esendaweni yesibopho se-DevOps. Lokhu kuhlukaniswa kusivumela ukuthi sigxile emisebenzini futhi akuphazamisi ukuxhumana, ngoba sisondelene futhi sishintshana njalo ulwazi nolwazi.
Iphelele
Ake sigxile ku-Integro futhi siqale ngesitaki sobuchwepheshe:
- I-CentOS 7
- I-Docker + Nomad + Consul + Vault
- I-Java 11 (i-Integro monolith endala izohlala ku-Java 8)
- I-Spring Boot 2.X + I-Spring Cloud Config
- I-PostgreSql 11
- RabbitMQ
- I-Apache Ignite
- I-Camunda (eshumekiwe)
- Grafana + Graphite + Prometheus + Jaeger + ELK
- I-Web UI: React (CSR) + MobX
- SSO: Isikhiye
Sithobela umgomo wokuthuthukiswa kwe-microservice, nakuba sinefa ngendlela ye-monolith yenguqulo yokuqala ye-Integro. I-microservice ngayinye isebenza esitsheni sayo se-Docker, futhi izinsiza zixhumana zodwa ngezicelo ze-HTTP nemiyalezo ye-RabbitMQ. Ama-Microservices atholana nge-Consul futhi enze isicelo kuwo, adlulise ukugunyazwa nge-SSO (Keycloak, OAuth 2/OpenID Connect).
Njengesibonelo sangempela, cabanga ukusebenzisana no-Jenkins, okuhlanganisa lezi zinyathelo ezilandelayo:
- I-microservice yokuphatha ukugeleza komsebenzi (okuzobizwa kamuva ngokuthi yi-Flow microservice) ifuna ukwenza isakhiwo e-Jenkins. Ukuze enze lokhu, usebenzisa i-Consul ukuze athole i-IP: PORT ye-microservice ukuze ihlanganiswe ne-Jenkins (ngemuva kwalokhu ebizwa ngokuthi i-Jenkins microservice) futhi ithumela isicelo esivumelanayo kuyo ukuze iqale ukwakha e-Jenkins.
- Ngemva kokuthola isicelo, i-Jenkins microservice ikhiqiza futhi iphendule nge-ID Yomsebenzi, engase isetshenziselwe ukukhomba umphumela womsebenzi. Ngasikhathi sinye, iqala ukwakhiwa e-Jenkins ngekholi ye-REST API.
- U-Jenkins wenza ukwakha futhi, ngemva kokuphothula, uthumela i-webhook enemiphumela yokusebenzisa ku-Jenkins microservice.
- I-Jenkins microservice, ngemva kokuthola i-webhook, ikhiqiza umlayezo mayelana nokuqedwa kokucubungula isicelo futhi inamathisele kuyo imiphumela yokwenziwa. Umlayezo okhiqiziwe uthunyelwa kulayini we-RabbitMQ.
- Nge-RabbitMQ, umlayezo oshicilelwe ufinyelela ku-Flow microservice, efunda ngomphumela wokucubungula umsebenzi wayo ngokufanisa i-ID Yomsebenzi esicelweni kanye nomlayezo owamukelwe.
Manje sinama-microservices angaba ngu-30, angahlukaniswa ngamaqembu amaningana:
- Ukuphathwa kokucushwa.
- Ulwazi nokuxhumana nabasebenzisi (izithunywa, i-imeyili).
- Ukusebenza ngekhodi yomthombo.
- Ukuhlanganiswa namathuluzi okuthumela (i-jenkins, nomad, i-consul, njll.).
- Ukuqapha (ukukhishwa, amaphutha, njll.).
- Izinsiza zewebhu (i-UI yokuphatha izindawo zokuhlola, ukuqoqa izibalo, njll.).
- Ukuhlanganiswa nama-trackers emisebenzi namasistimu afanayo.
- Ukuphathwa kokuhamba komsebenzi kwemisebenzi ehlukene.
Imisebenzi yohlelo lokusebenza
I-Integro yenza ngokuzenzakalelayo imisebenzi ehlobene nomjikelezo wokuphila komsebenzi. Ngamagama alula, umjikelezo wempilo yomsebenzi uzoqondwa njengokuhamba komsebenzi ku-Jira. Izinqubo zethu zokuthuthukisa zinokuhlukahluka kokuhamba komsebenzi okumbalwa kuye ngephrojekthi, uhlobo lomsebenzi kanye nezinketho ezikhethiwe emsebenzini othile.
Ake sibheke ukuhamba komsebenzi esisebenzisa kakhulu:
Emdwebeni, igiya libonisa ukuthi uguquko lubizwa ngokuzenzakalelayo yi-Integro, kuyilapho isibalo somuntu sibonisa ukuthi ukuguquka kubizwa ngumuntu ngesandla. Ake sibheke izindlela ezimbalwa umsebenzi ongazithatha kulokhu kuhamba komsebenzi.
Ukuhlola okwenziwa mathupha ngokuphelele ku-DEV+BETA ngaphandle kokuhlolwa kwe-canary (imvamisa yile ndlela esikhipha ngayo i-monolith):
Kungase kube nezinye izinhlanganisela zoshintsho. Kwesinye isikhathi indlela inkinga ezothathwa ngayo ingakhethwa ngezinketho ku-Jira.
Ukunyakaza komsebenzi
Ake sibheke izinyathelo eziyinhloko ezenziwa lapho umsebenzi uhamba ngohlelo lokusebenza lwe-βDEV Testing + Canary Testsβ:
1. Unjiniyela noma u-PM udala umsebenzi.
2. Umthuthukisi uthatha umsebenzi ukuze asebenze. Ngemva kokuqedwa, ishintshela esimweni esithi UKUBUYEKEZWA.
3. U-Jira uthumela i-Webhook ku-Jira microservice (enomthwalo wemfanelo wokuhlanganisa ne-Jira).
4. I-Jira microservice ithumela isicelo kusevisi ye-Flow (ebhekele ukugeleza komsebenzi kwangaphakathi lapho umsebenzi owenziwa khona) ukuze iqale ukugeleza komsebenzi.
5. Ngaphakathi kwesevisi ye-Flow:
- Ababuyekezi babelwa emsebenzini (Abasebenzisi microservice abazi konke ngabasebenzisi + Jira microservice).
- Nge-Source microservice (iyazi ngamanqolobane namagatsha, kodwa ayisebenzi nekhodi ngokwayo), kuseshwa amaqoqo aqukethe igatsha lenkinga yethu (ukwenza ukusesha kube lula, igama legatsha lihambisana nenkinga. inombolo eJira). Ezikhathini eziningi, umsebenzi uba negatsha elilodwa kuphela endaweni yokugcina izinto; lokhu kwenza ukuphathwa komugqa wokuthunyelwa kube lula futhi kunciphisa ukuxhumana phakathi kwamakhosombe.
- Egatsheni ngalinye elitholiwe, ukulandelana kwezenzo okulandelayo kwenziwa:
i) Ukubuyekeza igatsha eliyinhloko (Git microservice yokusebenza ngekhodi).
ii) Igatsha livinjiwe ezinguqulweni zonjiniyela (Bitbucket microservice).
iii) Isicelo Sokudonsa senzelwe leli gatsha (Bitbucket microservice).
iv) Umlayezo mayelana nesicelo esisha sokudonsa uthunyelwa ezingxoxweni zonjiniyela (Yazisa i-microservice ngokusebenza ngezaziso).
v) Yakha, hlola futhi sebenzisa imisebenzi iqalwa ku-DEV (Jenkins microservice yokusebenza noJenkins).
vi) Uma zonke izinyathelo zangaphambilini ziqedwa ngempumelelo, khona-ke i-Integro ibeka i-Approve yayo ku-Pull Application (Bitbucket microservice). - I-Integro ilindele Ukugunyaza Ekudonseni Isicelo kubabuyekezi abaqokiwe.
- Ngokushesha nje lapho zonke izimvume ezidingekayo sezitholiwe (okuhlanganisa nezivivinyo ezizenzakalelayo ziphumelele kahle), i-Integro idlulisela umsebenzi ku-Test on Dev (Jira microservice) isimo.
6. Abahloli bahlola umsebenzi. Uma zingekho izinkinga, umsebenzi udluliselwa kusimo se-Ready For Build.
7. I-Integro "ibona" ββukuthi umsebenzi usulungele ukukhululwa futhi uqala ukuthunyelwa kwawo kumodi ye-canary (Jenkins microservice). Ukulungela ukukhululwa kunqunywa isethi yemithetho. Isibonelo, umsebenzi usesimweni esidingekayo, akukho ukukhiya kweminye imisebenzi, okwamanje akukho okulayishiwe okusebenzayo kwale microservice, njll.
8. Umsebenzi udluliselwa esimweni se-Canary (Jira microservice).
9. I-Jenkins iqala umsebenzi wokuthunyelwa nge-Nomad kumodi ye-canary (imvamisa izikhathi ezingu-1-3) futhi yazisa isevisi yokuqapha ukukhishwa (i-DeployWatch microservice) mayelana nokusetshenziswa.
10. I-DeployWatch microservice iqoqa ingemuva lephutha bese isabela kulo, uma kudingeka. Uma ingemuva lephutha leqiwe (inkambiso yangemuva ibalwa ngokuzenzakalelayo), onjiniyela bayaziswa nge-Notify microservice. Uma ngemva kwemizuzu engu-5 unjiniyela engaphendulanga (chofoze Buyela noma Hlala), kuzobe sekwethulwa ukubuyisela emuva okuzenzakalelayo kwezenzakalo ze-canary. Uma ingemuva lingeqiwe, khona-ke unjiniyela kufanele aqalise mathupha ukuthunyelwa komsebenzi Ekukhiqizeni (ngokuchofoza inkinobho ku-UI). Uma unjiniyela engakaqalisi ukuthunyelwa Ekukhiqizeni phakathi nemizuzu engama-60, izimo ze-canary nazo zizohlehliswa ngenxa yezizathu zokuphepha.
11. Ngemva kokwethula ukuthunyelwa Kokukhiqiza:
- Umsebenzi udluliselwa esimweni Sokukhiqiza (i-Jira microservice).
- I-Jenkins microservice iqala inqubo yokusebenzisa futhi yazisa i-DeployWatch microservice mayelana nokusetshenziswa.
- I-DeployWatch microservice ihlola ukuthi zonke iziqukathi ezikuKhiqizo zibuyekeziwe (kube nezimo lapho zingabuyekeziwe zonke).
- Nge-Notify microservice, isaziso mayelana nemiphumela yokuthunyelwa sithunyelwa Emkhiqizweni.
12. Onjiniyela bazoba nemizuzu engama-30 ukuze baqale ukuhlehlisa umsebenzi ovela ku-Production uma kutholwa ukuziphatha okungalungile kwe-microservice. Ngemuva kwalesi sikhathi, umsebenzi uzohlanganiswa ngokuzenzakalelayo ku-master (Git microservice).
13. Ngemva kokuhlanganisa ngempumelelo kube okuyinhloko, isimo somsebenzi sizoshintshwa sithi Kuvaliwe (Jira microservice).
Umdwebo awuzenzi sengathi unemininingwane ngokuphelele (empeleni kunezinyathelo ezengeziwe), kodwa ikuvumela ukuthi uhlole izinga lokuhlanganiswa kuzinqubo. Asicabangi ukuthi lolu hlelo lufanelekile futhi sithuthukisa izinqubo zokukhishwa okuzenzakalelayo nokusekelwa kokuphakwa.
Yini elandelayo
Sinezinhlelo ezinkulu zokuthuthukisa i-automation, isibonelo, ukuqeda ukusebenza okwenziwa ngesandla ngesikhathi sokukhishwa kwe-monolith, ukuthuthukisa ukuqapha ngesikhathi sokuthunyelwa okuzenzakalelayo, nokuthuthukisa ukusebenzisana nonjiniyela.
Kodwa ake sime lapha okwamanje. Sihlanganise izihloko eziningi ekubuyekezweni okuzenzakalelayo, ezinye azithintwanga nhlobo, ngakho-ke sizokujabulela ukuphendula imibuzo. Silindele iziphakamiso ngalokho okumele sikuhlanganise ngokuningiliziwe, bhala kumazwana.
Source: www.habr.com