PaRIT 2019, mumwe wedu Alexander Korotkov akaita nezve otomatiki yekusimudzira paCIAN: kurerutsa hupenyu nebasa, isu tinoshandisa yedu Integro chikuva. Iyo inoteedzera kutenderera kwehupenyu hwemabasa, inorerutsira vagadziri venguva dzose mashandiro uye inoderedza zvakanyanya huwandu hwebugs mukugadzira. Mune ino positi, isu tinozadzisa chirevo chaAlexander uye tokuudza mafambiro atakaita kubva kune akareruka zvinyorwa kusvika kusanganisa yakavhurika sosi zvigadzirwa kuburikidza neyedu papuratifomu uye izvo zvedu zvakaparadzana otomatiki timu inoita.
Zero level
"Hapana chinhu chakadai sezero level, ini handizive chinhu chakadaro"
Master Shifu kubva mufirimu "Kung Fu Panda"
Automation paCIAN yakatanga makore gumi nemana mushure mekunge kambani yatangwa. Panguva iyoyo kwaive nevanhu makumi matatu nevashanu muboka rebudiriro. Zvakaoma kutenda, handiti? Ehe, otomatiki yaivepo mune imwe nzira, asi imwe nzira yakaparadzana yekuenderera mberi kwekubatanidza uye kuburitsa kodhi yakatanga kuumbwa muna 14.
Panguva iyoyo, takanga tine hombe monolith yePython, C# uye PHP, yakaiswa paLinux/Windows maseva. Kuti titumire chikara ichi, takanga tine seti yezvinyorwa zvataimhanya nemaoko. Kwakanga kunewo kuungana kwe monolith, uko kwakaunza marwadzo nokutambura nemhaka yerwisano pakubatanidza mapazi, kururamisa zvikanganiso, uye kuvakazve βnegadziriro yakasiyana yebasa mukuvaka.β Nzira yakareruka yakaita seizvi:
Hatina kufara neizvi, uye taida kuvaka inodzokororwa, otomatiki uye inogoneka kuvaka uye kuendesa maitiro. Nokuda kweizvi, taida CI / CD system, uye takasarudza pakati pehurukuro yakasununguka yeTeamcity uye yeJenkins yemahara, sezvo takashanda navo uye zvose zvakatikodzera maererano neseti yemabasa. Isu takasarudza Teamcity sechigadzirwa chazvino. Panguva iyoyo, takanga tisati tashandisa microservice architecture uye hatina kutarisira nhamba huru yemabasa nemapurojekiti.
Isu tinouya kune pfungwa yehurongwa hwedu
Kuitwa kweTeamcity kwakabvisa chikamu chebasa remanyorero: chasara kugadzirwa kweDhonza Zvikumbiro, kukwidziridzwa kwezvinhu nechimiro muJira, uye kusarudzwa kwenyaya dzekuburitswa. Iyo Teamcity system yakanga isisagone kurarama neizvi. Zvaive zvakakodzera kusarudza nzira yekuwedzera otomatiki. Isu takafunga sarudzo dzekushanda nemascript muTeamcity kana kushandura kune wechitatu-bato otomatiki masisitimu. Asi pakupedzisira takasarudza kuti tinoda kuchinjika zvakanyanya, izvo chete mhinduro yedu pachedu inogona kupa. Iyi ndiyo nzira yekutanga vhezheni yemukati otomatiki system inonzi Integro yakaonekwa.
Teamcity inobata otomatiki padanho rekutanga kuvaka uye kuendesa maitiro, uku Integro yakatarisana nepamusoro-chikamu otomatiki maitiro ebudiriro. Zvaive zvakakodzera kusanganisa basa nenyaya muJira nekugadziriswa kweiyo yakabatana kodhi kodhi muBitbucket. Panguva ino, Integro yakatanga kuve neyayo mafambiro ekushanda nemabasa emhando dzakasiyana.
Nekuda kwekuwedzera kwe otomatiki mumabhizinesi maitiro, nhamba yemapurojekiti uye inomhanya muTeamcity yakawedzera. Saka dambudziko idzva rakauya: imwe yemahara Teamcity muenzaniso yakanga isina kukwana (3 vamiririri uye zana mapurojekiti), isu takawedzera imwe muenzaniso (100 mamwe maajenti uye zana mapurojekiti), kozoti imwe. Nekuda kweizvozvo, takapedzisira tave nehurongwa hwemasumbu akati wandei, ayo aive akaoma kubata:
Apo mubvunzo wechiitiko che4 wakasimuka, takaona kuti hatingagoni kuramba tichirarama seizvi, nokuti mari yose yekutsigira zviitiko zve4 yakanga isisiri mukati memiganhu ipi zvayo. Mubvunzo wakamuka pamusoro pekutenga Teamcity yakabhadharwa kana kusarudza Jenkins yemahara. Isu takaverenga pane zviitiko uye otomatiki zvirongwa uye takasarudza kuti taizogara paJenkins. Mushure memavhiki mashoma, takachinjira kuna Jenkins uye takabvisa mamwe emusoro ane chekuita nekuchengetedza akawanda Teamcity zviitiko. Naizvozvo, takakwanisa kuisa pfungwa pakugadzira Integro uye kugadzirisa Jenkins isu pachedu.
Nekukura kweiyo otomatiki otomatiki (nenzira yekugadzira otomatiki yePull Zvikumbiro, kuunganidza uye kuburitswa kweCode coverage uye mamwe macheki), pane chishuwo chakasimba chekusiya zvinyorwa zvemanyorerwo zvakanyanya sezvinobvira uye kupa basa iri kumarobhoti. Mukuwedzera, kambani yakatanga kutamira kune microservices mukati mekambani, iyo yaida kubudiswa kakawanda, uye zvakasiyana kubva kune mumwe. Aya ndiwo maitiro atakaita zvishoma nezvishoma kuburitsa otomatiki kwemamicroservices edu (parizvino tiri kuburitsa monolith nemaoko nekuda kwekuoma kwemaitiro). Asi, sezvinowanzoitika, kuoma kutsva kwakamuka.
Isu tinozviitisa bvunzo
Nekuda kwe automation yekuburitsa, maitirwo ebudiriro akawedzera, zvishoma nekuda kwekusvetuka kwemamwe matanho ekuyedza. Uye izvi zvakakonzera kurasikirwa kwekanguva kwehutano. Zvinonzwika sezvisina basa, asi pamwe nekukasira kwekuburitswa, zvaive zvakafanira kushandura nzira yekuvandudza chigadzirwa. Zvakanga zvakafanira kufunga nezve otomatiki yekuyedzwa, kuisa mutoro wega (pano tiri kutaura nezve "kugamuchira iyo pfungwa mumusoro", kwete zvemari faindi) yemugadziri wekodhi yakaburitswa uye tsikidzi mairi, pamwe nesarudzo yekuita. kusunungura / kusasunungura basa kuburikidza nekuisa otomatiki.
Kubvisa matambudziko emhando, takasvika kune maviri akakosha sarudzo: isu takatanga kuita canary test uye takaunza otomatiki yekutarisa yekukanganisa kumashure nemhinduro otomatiki kune yakawandisa. Mhinduro yekutanga yakaita kuti zvikwanise kuwana zvikanganiso zvakajeka musati kodhi yakasunungurwa zvizere mukugadzirwa, yechipiri yakaderedza nguva yekupindura kumatambudziko mukugadzirwa. Zvikanganiso, hongu, zvinoitika, asi tinopedza yakawanda yenguva yedu nesimba kwete pakuzviruramisa, asi pakudzideredza.
Automation Team
Isu parizvino tine vashandi vevagadziri ve130, uye tinoenderera mberi . Chikwata chekuenderera mberi kwekubatanidza uye kuendesa kodhi (inozonzi Deploy uye Integration kana DI timu) ine vanhu vanomwe uye inoshanda mumativi maviri: kuvandudza kweIntegro otomatiki chikuva uye DevOps.
DevOps inotarisira Dev/Beta nharaunda yeCIAN saiti, iyo Integro nharaunda, inobatsira vanogadzira kugadzirisa matambudziko uye kugadzira nzira nyowani dzekuyera nharaunda. Iyo Integro budiriro inotungamira inobata nezvose zviri zviviri Integro pachayo uye ane hukama masevhisi, semuenzaniso, plugins yeJenkins, Jira, Confluence, uye zvakare inogadzira zvekubatsira uye maapplication ezvikwata zvebudiriro.
Chikwata cheDI chinoshanda pamwe chete nechikwata chePlatform, chinovandudza zvivakwa, maraibhurari, uye nzira dzekusimudzira mukati. Panguva imwecheteyo, chero mugadziri mukati meCIAN anogona kubatsira kune otomatiki, semuenzaniso, kuita micro-otomatiki kuti ienderane nezvinodiwa nechikwata kana kugovera zano rinotonhorera rekuita otomatiki zviri nani.
Layer keke ye otomatiki kuCIAN
Ese masisitimu anobatanidzwa mune otomatiki anogona kukamurwa kuita akati wandei zvidimbu:
- Zvirongwa zvekunze (Jira, Bitbucket, nezvimwewo). Zvikwata zvebudiriro zvinoshanda navo.
- Integro platform. Kazhinji, vanogadzira havashande nayo zvakananga, asi ndizvo zvinoita kuti zvese otomatiki zvishande.
- Delivery, orchestration uye kuwanikwa masevhisi (semuenzaniso, Jeknins, Consul, Nomad). Nerubatsiro rwavo, isu tinotumira kodhi pamaseva uye tinoona kuti masevhisi anoshanda pamwe chete.
- Physical layer (maseva, OS, ine hukama software). Kodhi yedu inoshanda pamwero uyu. Izvi zvinogona kunge zviri sevha yemuviri kana chaiyo (LXC, KVM, Docker).
Zvichienderana neiyi pfungwa, tinogovanisa nzvimbo dzemutoro mukati meDI timu. Iwo maviri ekutanga mazinga ari munzvimbo yebasa reIntegro budiriro yekutungamira, uye maviri ekupedzisira mazinga atove munzvimbo yebasa reDevOps. Kuparadzaniswa uku kunotibvumira kuti titarise pamabasa uye hakukanganise kudyidzana, sezvo isu tiri pedyo kune mumwe nemumwe uye tinogara tichitsinhana ruzivo uye ruzivo.
Intact
Ngatitarisei paIntegro uye titange neiyo tekinoroji stack:
- CentOS 7
- Docker + Nomad + Consul + Vault
- Java 11 (iyo yekare Integro monolith icharamba iri paJava 8)
- Spring Boot 2.X + Spring Cloud Config
- PostgreSql 11
- TsuroMQ
- Apache Ignite
- Camunda (embedded)
- Grafana + Graphite + Prometheus + Jaeger + ELK
- Webhu UI: React (CSR) + MobX
- SSO: Keycloak
Isu tinonamatira kumusimboti wekusimudzira microservice, kunyangwe isu tiine nhaka muchimiro chemonolith yepakutanga vhezheni yeIntegro. Imwe neimwe microservice inomhanya mumudziyo wayo weDocker, uye masevhisi anotaurirana kuburikidza neHTTP zvikumbiro uye mameseji eRabbitMQ. Microservices inowanana kuburikidza neConsul uye ita chikumbiro kwairi, ichipfuura mvumo kuburikidza neSSO (Keycloak, OAuth 2/OpenID Batanidza).
Semuenzaniso wehupenyu chaihwo, funga kutaurirana naJenkins, iyo ine matanho anotevera:
- Iyo workflow manejimendi microservice (inozonzi iyo Flow microservice) inoda kumhanyisa kuvaka muJenkins. Kuti aite izvi, anoshandisa Consul kuti awane IP: PORT ye microservice yekubatanidzwa neJenkins (pano inonzi Jenkins microservice) uye inotumira chikumbiro cheasynchronous kwairi kuti itange kuvaka muJenkins.
- Mushure mekugamuchira chikumbiro, Jenkins microservice inogadzira uye inopindura neJobho ID, iyo inogona kushandiswa kuona mhedzisiro yebasa. Panguva imwecheteyo, inokonzera kuvaka muJenkins kuburikidza neREST API call.
- Jenkins anoita kuvaka uye, mushure mekupedza, anotumira webhook nemhedzisiro yekuuraya kuJenkins microservice.
- Iyo Jenkins microservice, yagamuchira iyo webhook, inogadzira meseji nezve kupedzwa kwekukumbira kugadzirisa uye inonamatira mhedzisiro yekuurayiwa kwairi. Iyo meseji yakagadzirwa inotumirwa kuRabbitMQ mutsara.
- Kuburikidza neRabbitMQ, iyo meseji yakaburitswa inosvika kuFlow microservice, iyo inodzidza nezve mhedzisiro yekugadzirisa basa rayo nekufananidza ID yeJobho kubva pakukumbira uye meseji yakagamuchirwa.
Iye zvino tine anenge makumi matatu microservices, ayo anogona kugoverwa mumapoka akati wandei:
- Configuration management.
- Ruzivo uye kudyidzana nevashandisi (nhume, tsamba).
- Kushanda nesource code.
- Kubatanidzwa nemidziyo yekushandisa (jenkins, nomad, consul, nezvimwewo).
- Monitoring (kuburitswa, kukanganisa, nezvimwewo).
- Zvishandiso zveWebhu (UI yekugadzirisa nharaunda dzekuyedza, kuunganidza manhamba, nezvimwewo).
- Kubatanidzwa nevateki vebasa uye masisitimu akafanana.
- Workflow manejimendi kune akasiyana mabasa.
Workflow mabasa
Integro inogadzirisa zviitiko zvine chekuita nehupenyu hwebasa. Mumashoko akareruka, kutenderera kwehupenyu hwebasa kunozonzwisiswa sekufamba kwebasa muJira. Maitiro edu ekusimudzira ane akati wandei mafambiro akasiyana zvichienderana nepurojekiti, rudzi rwebasa uye sarudzo dzakasarudzwa mune rimwe basa.
Ngatitarisei mafambiro ebasa atinonyanya kushandisa:
Mumufananidzo, giya rinoratidza kuti shanduko inodaidzwa otomatiki neIntegro, nepo chimiro chemunhu chinoratidza kuti shanduko inodaidzwa nemaoko nemunhu. Ngatitarisei nzira dzinoverengeka dzinogona kutorwa nebasa mukufambiswa kwebasa iri.
Kunyatsoyedza manyore paDEV+BETA pasina canary bvunzo (kazhinji iyi ndiyo nzira yatinoburitsa monolith):
Panogona kunge paine mamwe masanganiswa ekuchinja. Dzimwe nguva nzira iyo nyaya ichatora inogona kusarudzwa kuburikidza nesarudzo muJira.
Basa rekufamba
Ngatitarisei matanho makuru anoitwa kana basa richifamba kuburikidza ne "DEV Testing + Canary Tests" mafambiro ebasa:
1. Mugadziri kana PM anogadzira basa.
2. Mugadziri anotora basa racho kushanda. Mushure mekupedzwa, inochinja kune IN REVIEW chimiro.
3. Jira anotumira Webhook kuJira microservice (ine basa rekubatanidza neJira).
4. Iyo Jira microservice inotumira chikumbiro kune Flow sevhisi (inotarisira mukati mekufamba kwemukati umo basa rinoitwa) kuvhura mafambiro ebasa.
5. Mukati meKuyerera sevhisi:
- Vaongorori vanopihwa basa racho (Vashandisi microservice inoziva zvese nezve vashandisi + Jira microservice).
- Kuburikidza neiyo Source microservice (inoziva nezve repositori uye matavi, asi isingashande nekodhi pachayo), kutsvaga kunoitwa matura ane bazi renyaya yedu (kurerutsa kutsvaga, zita rebazi rinoenderana nenyaya yacho. nhamba muJira). Kazhinji, basa rinongove nebazi rimwe chete mune imwe repository; izvi zvinorerutsa manejimendi emutsetse wekutumira uye kuderedza kubatana pakati pemarepositori.
- Kune rimwe nerimwe bazi rinowanikwa, zvinotevera kutevedzana kwezviito kunoitwa:
i) Kugadziridza tenzi bazi (Git microservice yekushanda nekodhi).
ii) Bazi rakavharwa kubva kune shanduko nemugadziri (Bitbucket microservice).
iii) Dhonza Chikumbiro chakagadzirirwa bazi iri (Bitbucket microservice).
iv) Meseji nezve nyowani Dhonza Chikumbiro inotumirwa kune yekuvandudza chats (Zivisa microservice yekushanda nezviziviso).
v) Vaka, edza uye kuendesa mabasa anotangwa paDEV (Jenkins microservice yekushanda naJenkins).
vi) Kana nhanho dzese dzakapfuura dzakapedzwa zvinobudirira, ipapo Integro inoisa yayo Bvumira muChikumbiro chekudhonza (Bitbucket microservice). - Integro inomirira Kubvumidza muKudhonza Chikumbiro kubva kune vakasarudzwa vanoongorora.
- Pakangotambirwa mvumo dzese dzinodiwa (kusanganisira otomatiki bvunzo dzapfuura zvakanaka), Integro inotamisa basa kuTest paDev (Jira microservice) chimiro.
6. Matesta anoedza basa racho. Kana pasina matambudziko, saka basa rinoendeswa kune Ready For Build status.
7. Integro "inoona" kuti basa racho rakagadzirira kusunungurwa uye rinotanga kushandiswa kwayo mu canary mode (Jenkins microservice). Kugadzirira kusunungurwa kunogadziriswa nemitemo yemitemo. Semuenzaniso, iro basa riri muchimiro chinodiwa, hapana makiyi pane mamwe mabasa, parizvino hapana anoshanda ekurodha eiyi microservice, nezvimwe.
8. Basa racho rinoendeswa kuCanary status (Jira microservice).
9. Jenkins anotanga basa rekutumira kuburikidza neNomad mu canary mode (kazhinji 1-3 zviitiko) uye anozivisa kusunungurwa kwekutarisa sevhisi (DeployWatch microservice) pamusoro pekushandiswa.
10. Iyo DeployWatch microservice inounganidza iyo yekukanganisa kumashure uye inogadzirisa kwairi, kana zvichidikanwa. Kana iyo yekukanganisa yakapfuura (iyo yekumashure yakajairwa inoverengerwa otomatiki), vagadziri vanoziviswa kuburikidza neZivisa microservice. Kana mushure memaminitsi mashanu mugadziri asina kupindura (kudzvanya Revert kana Gara), ipapo otomatiki rollback yezviitiko zvecanary inotangwa. Kana iyo yekumashure isina kupfuurwa, saka mugadziri anofanirwa kuvhura nemaoko basa rekuendesa kuKugadzirwa (nekudzvanya bhatani muUI). Kana mugadziri asina kuvhura kuendeswa kuKugadzirwa mukati memaminitsi makumi matanhatu, saka zviitiko zvecanary zvinozodzoserwa kumashure nekuda kwezvikonzero zvekuchengetedza.
11. Mushure mekutangisa kutumirwa kuKugadzirwa:
- Basa racho rinoendeswa kuKugadzira chimiro (Jira microservice).
- Iyo Jenkins microservice inotanga maitiro ekutumira uye inozivisa iyo DeployWatch microservice nezve kutumirwa.
- Iyo DeployWatch microservice inotarisa kuti midziyo yese paKugadzirwa yakagadziridzwa (paive nezviitiko apo zvisiri zvese zvakagadziridzwa).
- Kuburikidza neNotify microservice, chiziviso nezve mhedzisiro yekutumirwa chinotumirwa kuKugadzirwa.
. Mushure menguva ino, basa racho rinozobatanidzwa otomatiki kuita tenzi (Git microservice).
13. Mushure mekubudirira kusanganisa kuita tenzi, chinzvimbo chebasa chinoshandurwa kuita Yakavharwa (Jira microservice).
Iyo dhairekitori haiite kunge yakanyatso kutsanangurwa (muchokwadi kune mamwe matanho), asi inobvumidza iwe kuti uongorore dhigirii rekubatanidza mumaitiro. Isu hatitarise chirongwa ichi chakanakira uye tiri kuvandudza maitiro ekuburitsa otomatiki uye nerutsigiro rwekutumira.
Chii chinotevera
Isu tine hurongwa hukuru hwekuvandudza otomatiki, semuenzaniso, kubvisa mashandiro emaoko panguva yekuburitswa kwe monolith, kuvandudza kutarisa panguva yekuendesa otomatiki, nekuvandudza kudyidzana nevagadziri.
Asi ngatimirei pano izvozvi. Takafukidza misoro yakawanda mukuongorora otomatiki pamusoro, mamwe haana kubatwa zvachose, saka isu tichafara kupindura mibvunzo. Isu takamirira mazano pamusoro pezvinofanira kuvhara zvakadzama, nyora mumashoko.
Source: www.habr.com