Pa RIT 2019, mnzathu Alexander Korotkov adapanga za chitukuko chokhazikika ku CIAN: kufewetsa moyo ndi ntchito, timagwiritsa ntchito nsanja yathu ya Integro. Imatsata moyo wa ntchito, imathandizira opanga machitidwe achizolowezi ndikuchepetsa kwambiri kuchuluka kwa nsikidzi zomwe zikupanga. Mu positi iyi, tithandizira lipoti la Alexander ndikukuwuzani momwe tachokera ku zolemba zosavuta kupita kuphatikizira zinthu zotsegula kudzera papulatifomu yathu komanso zomwe gulu lathu losiyana limachita.
Zero mlingo
"Palibe chinthu ngati zero level, sindikudziwa zotere"
Master Shifu kuchokera ku filimu "Kung Fu Panda"
Zodzichitira ku CIAN zidayamba zaka 14 kampaniyo itakhazikitsidwa. Panthawiyo panali anthu 35 mu gulu lachitukuko. Zovuta kukhulupirira, sichoncho? Zachidziwikire, zodzipangira zokha zidalipo mwanjira ina, koma njira ina yophatikizira mosalekeza ndi kutumiza ma code idayamba kuchitika mu 2015.
Panthawiyo, tinali ndi monolith yayikulu ya Python, C # ndi PHP, yotumizidwa pa ma seva a Linux / Windows. Kuti titumize chilombochi, tinali ndi zolemba zomwe tinkayendetsa pamanja. Panalinso msonkhano wa monolith, womwe unabweretsa zowawa ndi zowawa chifukwa cha mikangano pogwirizanitsa nthambi, kukonza zolakwika, ndi kumanganso "ndi ntchito zosiyanasiyana pomanga." Njira yosavuta idawoneka motere:
Sitinasangalale ndi izi, ndipo tinkafuna kupanga njira yobwerezabwereza, yokhazikika komanso yotheka kuyendetsedwa ndi kutumiza. Pachifukwa ichi, tinkafunikira dongosolo la CI / CD, ndipo tinasankha pakati pa Teamcity yaulere ndi Jenkins yaulere, popeza tinagwira nawo ntchito ndipo onsewa ankatiyenerera malinga ndi ntchito. Tidasankha Teamcity ngati chinthu chaposachedwa kwambiri. Panthawiyo, tinali tisanagwiritsebe ntchito zomangamanga za microservice ndipo sitinayembekezere ntchito ndi ntchito zambiri.
Timafika ku lingaliro la dongosolo lathu
Kukhazikitsidwa kwa Teamcity kunachotsa gawo limodzi lokha la ntchito yamanja: chomwe chatsalira ndikupanga Pull Requests, kukwezedwa kwa nkhani ndi udindo ku Jira, ndi kusankha nkhani zomasulidwa. Teamcity sinathenso kuthana ndi izi. Zinali zofunikira kusankha njira yowonjezera makina. Taganizirani zosankha zogwirira ntchito ndi zolemba mu Teamcity kapena kusintha makina amtundu wachitatu. Koma pamapeto pake tinaganiza kuti tifunika kusinthasintha kwakukulu, komwe yankho lathu lokha lingapereke. Umu ndi momwe mtundu woyamba wa makina opangira mkati otchedwa Integro adawonekera.
Teamcity imagwira ntchito yodzipangira okha pamlingo woyambitsa njira zomangira ndi kutumiza, pomwe Integro imayang'ana kwambiri njira zapamwamba zachitukuko. Zinali zofunikira kuphatikiza ntchito ndi nkhani ku Jira ndikukonza ma code source ogwirizana mu Bitbucket. Pakadali pano, Integro idayamba kukhala ndi njira zake zogwirira ntchito ndi ntchito zamitundu yosiyanasiyana.
Chifukwa cha kuchuluka kwa ma automation pamabizinesi, kuchuluka kwa ma projekiti ndikuyendetsa mu Teamcity kwawonjezeka. Chifukwa chake vuto latsopano linabwera: chitsanzo chimodzi chaulere cha Teamcity sichinali chokwanira (othandizira atatu ndi mapulojekiti 3), tidawonjezeranso chochitika china (othandizira ena atatu ndi mapulojekiti 100), kenako china. Zotsatira zake, tidakhala ndi dongosolo lamagulu angapo, lomwe linali lovuta kuwongolera:
Pamene funso la chitsanzo cha 4 linabuka, tinazindikira kuti sitingathe kupitiriza kukhala ndi moyo wotere, chifukwa ndalama zonse zothandizira zochitika za 4 sizinalinso malire. Funso lidabuka pa kugula Teamcity yolipira kapena kusankha Jenkins waulere. Tidawerengera pazochitika ndi mapulani opangira makina ndipo tinaganiza zokhala pa Jenkins. Pambuyo pa milungu ingapo, tinasinthana ndi Jenkins ndikuchotsa mutu wina wokhudzana ndi kusunga maulendo angapo a Teamcity. Chifukwa chake, tinatha kuyang'ana kwambiri pakupanga Integro ndikusinthira mwamakonda a Jenkins tokha.
Ndi kukula kwa ma automation oyambira (munjira yodzipangira zokha Zopempha Zokoka, kusonkhanitsa ndi kufalitsa ma Code coverage ndi macheke ena), pali chikhumbo champhamvu chosiya kutulutsa kwamanja momwe tingathere ndikupereka ntchitoyi kwa maloboti. Kuphatikiza apo, kampaniyo idayamba kusamukira ku ma microservices mkati mwa kampani, zomwe zimafunikira kutulutsidwa pafupipafupi, komanso padera wina ndi mnzake. Umu ndi momwe tidafikira pang'onopang'ono kutulutsa ma microservices athu (pakali pano tikutulutsa monolith pamanja chifukwa chazovuta za njirayi). Koma, monga momwe zimakhalira, zovuta zatsopano zidayamba.
Timayesa ma automate
Chifukwa cha makina otulutsa, njira zachitukuko zapita patsogolo, pang'ono chifukwa cha kulumpha kwa magawo ena oyesera. Ndipo izi zinapangitsa kuti khalidwe liwonongeke kwakanthawi. Zikumveka ngati zazing'ono, koma pamodzi ndi kuthamangitsidwa kwa kutulutsidwa, kunali koyenera kusintha njira yopangira mankhwala. Zinali zofunikira kuganizira zoyeserera zokha, kuyika udindo wamunthu (pano tikukamba za "kuvomereza lingaliro pamutu", osati chindapusa chandalama) cha wopanga ma code omwe adatulutsidwa ndi nsikidzi momwemo, komanso chigamulo choti kumasula/osamasula ntchito poyiyika yokha.
Kuchotsa zovuta zamtundu, tidafika pazisankho ziwiri zofunika: tidayamba kuyesa ma canary ndikuyambitsa kuwunika kodziwikiratu kwa zolakwikazo ndikuyankha modzidzimutsa pakuwonjezera kwake. Yankho loyamba lidapangitsa kuti apeze zolakwika zoonekeratu code isanatulutsidwe kwathunthu mukupanga, yachiwiri idachepetsa nthawi yoyankha kumavuto opanga. Zolakwa, ndithudi, zimachitika, koma timathera nthawi yathu yambiri ndi kuyesetsa osati kukonza, koma kuchepetsa.
Automation Team
Pakali pano tili ndi antchito 130 opanga, ndipo tikupitiriza . Gulu lophatikizana mosalekeza ndi kutumiza ma code (omwe atchedwa Deploy and Integration kapena DI team) lili ndi anthu 7 ndipo limagwira ntchito mbali ziwiri: chitukuko cha Integro automation platform ndi DevOps.
DevOps imayang'anira chilengedwe cha Dev/Beta patsamba la CIAN, malo a Integro, amathandizira otukula kuthana ndi mavuto ndikupanga njira zatsopano zopangira makulitsidwe. Upangiri wa chitukuko cha Integro umachita ndi Integro yokha ndi ntchito zina zofananira, mwachitsanzo, mapulagini a Jenkins, Jira, Confluence, komanso amapanga zida zothandizira ndi mapulogalamu amagulu achitukuko.
Gulu la DI limagwira ntchito limodzi ndi gulu la Platform, lomwe limapanga zomanga, malaibulale, ndi njira zachitukuko mkati. Nthawi yomweyo, wopanga mapulogalamu aliwonse mkati mwa CIAN atha kuthandizira kupanga makina, mwachitsanzo, kupanga ma micro-automation kuti agwirizane ndi zosowa za gulu kapena kugawana malingaliro abwino amomwe angapangire makinawo kukhala abwinoko.
Layer keke ya automation ku CIAN
Machitidwe onse omwe amakhudzidwa ndi makina amatha kugawidwa m'magulu angapo:
- Machitidwe akunja (Jira, Bitbucket, etc.). Magulu achitukuko amagwira nawo ntchito.
- Integro nsanja. Nthawi zambiri, opanga sagwira nawo ntchito mwachindunji, koma ndizomwe zimapangitsa kuti makina onse aziyenda.
- Ntchito zotumizira, zoyimba ndi zotulukira (mwachitsanzo, Jeknin, Consul, Nomad). Ndi chithandizo chawo, timayika ma code pa seva ndikuwonetsetsa kuti mautumiki akugwira ntchito wina ndi mzake.
- Zosanjikiza thupi (maseva, OS, mapulogalamu okhudzana). Khodi yathu imagwira ntchito pamlingo uwu. Izi zitha kukhala seva yeniyeni kapena yeniyeni (LXC, KVM, Docker).
Kutengera lingaliro ili, timagawa magawo omwe ali ndi udindo mu gulu la DI. Magawo awiri oyambilira ali m'dera laudindo wa Integro Development direction, ndipo magawo awiri omaliza ali kale paudindo wa DevOps. Kupatukana kumeneku kumatithandiza kuyang'ana pa ntchito ndipo sikusokoneza kuyanjana, popeza timayandikana wina ndi mzake ndipo timasinthasintha nthawi zonse chidziwitso ndi zochitika.
Osasintha
Tiyeni tiyang'ane pa Integro ndikuyamba ndi stack yaukadaulo:
- CentOS 7
- Docker + Nomad + Consul + Vault
- Java 11 (Integro monolith yakale ikhalabe pa Java 8)
- Spring Boot 2.X + Spring Cloud Config
- Postgresql 11
- KaluluMQ
- Apache Ignite
- Camunda (embedded)
- Grafana + Graphite + Prometheus + Jaeger + ELK
- Web UI: React (CSR) + MobX
- SSO: Chophimba
Timatsatira mfundo ya chitukuko cha microservice, ngakhale tili ndi cholowa mu mawonekedwe a monolith wa mtundu woyambirira wa Integro. Microservice iliyonse imayenda mu chidebe chake cha Docker, ndipo mautumikiwa amalumikizana wina ndi mnzake kudzera pa zopempha za HTTP ndi mauthenga a RabbitMQ. Microservices amapezana wina ndi mnzake kudzera kwa Consul ndikupanga pempho, ndikudutsa chilolezo kudzera pa SSO (Keycloak, OAuth 2/OpenID Connect).
Monga chitsanzo chenicheni, taganizirani kucheza ndi Jenkins, zomwe zimakhala ndi izi:
- Microservice yoyendetsera kayendetsedwe ka ntchito (yomwe idatchedwa Flow microservice) ikufuna kuyendetsa ntchito yomanga ku Jenkins. Kuti achite izi, amagwiritsa ntchito Consul kuti apeze IP: PORT ya microservice kuti agwirizane ndi Jenkins (pano amatchedwa Jenkins microservice) ndikutumiza pempho losavomerezeka kwa izo kuti ayambe kumanga ku Jenkins.
- Atalandira pempho, Jenkins microservice imapanga ndikuyankha ndi ID ya Job, yomwe ingagwiritsidwe ntchito kuzindikira zotsatira za ntchitoyo. Nthawi yomweyo, imayambitsa kumanga ku Jenkins kudzera pa foni ya REST API.
- Jenkins amamanga ndipo, akamaliza, amatumiza webhook ndi zotsatira zake ku Jenkins microservice.
- Jenkins microservice, italandira webhook, imapanga uthenga wokhudza kutsirizidwa kwa pempho ndikugwirizanitsa zotsatira zake. Uthenga wopangidwa umatumizidwa ku mzere wa RabbitMQ.
- Kupyolera mu RabbitMQ, uthenga wofalitsidwa umafika ku Flow microservice, yomwe imaphunzira za zotsatira za kukonza ntchito yake pofananiza ID ya Job kuchokera pa pempho ndi uthenga wolandiridwa.
Tsopano tili ndi ma microservices pafupifupi 30, omwe amatha kugawidwa m'magulu angapo:
- Kasamalidwe ka kasinthidwe.
- Zambiri ndi kulumikizana ndi ogwiritsa ntchito (amithenga, makalata).
- Kugwira ntchito ndi source code.
- Kuphatikiza ndi zida zotumizira (jenkins, nomad, consul, etc.).
- Kuwunika (zotulutsa, zolakwika, ndi zina).
- Zothandizira pa intaneti (UI yoyang'anira malo oyeserera, kusonkhanitsa ziwerengero, ndi zina).
- Kuphatikizana ndi otsata ntchito ndi machitidwe ofanana.
- Kuwongolera magwiridwe antchito amitundu yosiyanasiyana.
Zochita zolimbitsa thupi
Integro imagwiritsa ntchito zochitika zokhudzana ndi moyo wantchito. M'mawu osavuta, kuzungulira kwa moyo wa ntchito kumamveka ngati mayendedwe a ntchito ku Jira. Njira zathu zachitukuko zimakhala ndi zosiyana zingapo za kayendetsedwe ka ntchito malinga ndi polojekiti, mtundu wa ntchito ndi zosankha zomwe zasankhidwa pa ntchito inayake.
Tiyeni tiwone njira zomwe timagwiritsa ntchito nthawi zambiri:
Pachithunzichi, zida zimasonyeza kuti kusintha kumatchedwa Integro, pamene chiwerengero cha anthu chimasonyeza kuti kusintha kumatchedwa pamanja ndi munthu. Tiyeni tiwone njira zingapo zomwe ntchito ingatengere mumayendedwe awa.
Kuyesa pamanja pa DEV+BETA popanda mayeso a canary (nthawi zambiri umu ndi momwe timatulutsira monolith):
Pakhoza kukhala zosakaniza zina zosinthira. Nthawi zina njira yomwe nkhani ingatenge imatha kusankhidwa kudzera muzosankha za Jira.
Kusuntha kwa ntchito
Tiyeni tiwone njira zazikulu zomwe zimachitika ntchito ikadutsa mumayendedwe a "DEV Testing + Canary Tests":
1. Wopanga mapulogalamu kapena PM amapanga ntchitoyi.
2. Wopanga mapulogalamu amatenga ntchitoyo kuti agwire ntchito. Ikamaliza, imasinthira kukhala IN REVIEW status.
3. Jira amatumiza Webhook ku Jira microservice (yoyang'anira kuphatikiza ndi Jira).
4. Jira microservice imatumiza pempho ku Flow service (yoyang'anira kayendedwe ka mkati momwe ntchito imagwiridwa) kuti ayambe ntchito.
5. M'kati mwa Flow service:
- Owunikira amapatsidwa ntchitoyo (Ogwiritsa ntchito microservice omwe amadziwa zonse za ogwiritsa ntchito + Jira microservice).
- Kupyolera mu Source microservice (ikudziwa za nkhokwe ndi nthambi, koma sichigwira ntchito ndi code yokha), kufufuza kumapangidwa kwa nkhokwe zomwe zili ndi nthambi ya nkhani yathu (kuchepetsa kusaka, dzina la nthambi likugwirizana ndi nkhaniyo. nambala mu Jira). Nthawi zambiri, ntchito imakhala ndi nthambi imodzi yokha m'malo amodzi; izi zimathandizira kasamalidwe ka mizere yotumizira ndikuchepetsa kulumikizana pakati pa nkhokwe.
- Pa nthambi iliyonse yomwe yapezeka, zotsatirazi zimachitika:
i) Kusintha nthambi ya master (Git microservice yogwira ntchito ndi code).
ii) Nthambi yatsekedwa kuti isasinthe ndi wopanga (Bitbucket microservice).
iii) Kupempha Kokoka kumapangidwira nthambi iyi (Bitbucket microservice).
iv) Uthenga wokhudza Kukokera Kwatsopano umatumizidwa kumacheza opanga mapulogalamu (Dziwani microservice yogwira ntchito ndi zidziwitso).
v) Kumanga, kuyesa ndi kutumiza ntchito kumayambika pa DEV (Jenkins microservice yogwira ntchito ndi Jenkins).
vi) Ngati masitepe onse am'mbuyomu amalizidwa bwino, ndiye Integro imayika Kuvomereza Kupempha (Bitbucket microservice). - Integro ikuyembekezera Kuvomereza mu Kukoka Pempho kuchokera kwa owunikira osankhidwa.
- Zivomerezo zonse zikangolandiridwa (kuphatikiza zoyeserera zokha zadutsa bwino), Integro imasamutsa ntchitoyi ku Test pa Dev (Jira microservice).
6. Oyesa amayesa ntchitoyo. Ngati palibe mavuto, ndiye kuti ntchitoyi imasamutsidwa ku Ready For Build status.
7. Integro "akuwona" kuti ntchitoyi ndi yokonzeka kumasulidwa ndipo imayamba kutumizidwa mu canary mode (Jenkins microservice). Kukonzekera kumasulidwa kumatsimikiziridwa ndi malamulo. Mwachitsanzo, ntchitoyi ili m'malo ofunikira, palibe maloko pa ntchito zina, pakadali pano palibe zokweza za microservice iyi, ndi zina.
8. Ntchitoyi imasamutsidwa ku Canary status (Jira microservice).
9. Jenkins amayambitsa ntchito yotumizira kudzera ku Nomad mu canary mode (nthawi zambiri zochitika za 1-3) ndikudziwitsa ntchito yowunikira kumasulidwa (DeployWatch microservice) za kutumizidwa.
10. DeployWatch microservice imasonkhanitsa zolakwika zakumbuyo ndikuchitapo kanthu, ngati kuli kofunikira. Ngati cholakwika chakumbuyo chapyola (chizoloΕ΅ezi chakumbuyo chimawerengedwa zokha), opanga amadziwitsidwa kudzera pa Notify microservice. Ngati pakatha mphindi 5 wopangayo sanayankhe (kudina Revert kapena Khalani), ndiye kuti kubweza mayendedwe a canary kumayambika. Ngati maziko sanapitirire, ndiye kuti wopangayo ayenera kuyambitsa ntchitoyo pamanja ku Production (podina batani mu UI). Ngati mkati mwa mphindi 60 wopangayo sanayambitse kutumiza ku Production, ndiye kuti zochitika za canary zidzabwezeredwanso chifukwa chachitetezo.
11. Pambuyo poyambitsa kutumiza ku Production:
- Ntchitoyi imasamutsidwa ku Production status (Jira microservice).
- Jenkins microservice imayamba ntchito yotumizira ndikudziwitsa a DeployWatch microservice za kutumizidwa.
- DeployWatch microservice imayang'ana kuti zotengera zonse pa Production zasinthidwa (panali zochitika pomwe sizinasinthidwe zonse).
- Kudzera mu Notify microservice, chidziwitso chokhudza zotsatira za kutumiza chimatumizidwa ku Production.
12. Madivelopa adzakhala ndi mphindi 30 kuti ayambe kubweza ntchito kuchokera ku Production ngati kachitidwe kolakwika ka microservice kazindikirika. Pambuyo pa nthawiyi, ntchitoyi idzaphatikizidwa yokha kukhala master (Git microservice).
13. Mukaphatikiza bwino kukhala master, ntchitoyo idzasinthidwa kukhala Yotsekedwa (Jira microservice).
Chithunzicho sichimadziwonetsera mwatsatanetsatane (kwenikweni pali masitepe ochulukirapo), koma chimakupatsani mwayi wowunika kuchuluka kwa kuphatikiza munjira. Sitikuwona chiwembuchi kukhala chabwino ndipo tikuwongolera njira zotulutsira zokha ndikuthandizira kutumiza.
Chotsatira
Tili ndi mapulani akuluakulu opangira makina opangira makina, mwachitsanzo, kuthetsa ntchito zamanja panthawi yotulutsidwa kwa monolith, kuwongolera kuwunika panthawi yotumiza, ndikuwongolera kulumikizana ndi opanga.
Koma tiyeni tiyime apa. Tidakambirana mitu yambiri pakuwunika kodzichitira zokha, zina sizinakhudzidwe nkomwe, ndiye tidzakhala okondwa kuyankha mafunso. Tikuyembekezera malingaliro pazomwe tingafotokoze mwatsatanetsatane, lembani mu ndemanga.
Source: www.habr.com