Igama lami ngingu-Viktor Yagofarov, futhi ngithuthukisa inkundla ye-Kubernetes e-DomClick njengomphathi wokuthuthukiswa kobuchwepheshe eqenjini le-Ops (ukusebenza). Ngingathanda ukukhuluma ngesakhiwo sezinqubo zethu ze-Dev <-> Ops, izici zokusebenza kweqoqo elikhulu lama-k8s e-Russia, kanye nezinqubo ze-DevOps/SRE ezisetshenziswa ithimba lethu.
Iqembu le-Ops
Ithimba le-Ops njengamanje linabantu abangu-15. Abathathu babo banesibopho sehhovisi, ababili basebenza endaweni yesikhathi ehlukile futhi bayatholakala, kuhlanganisa nasebusuku. Ngakho-ke, othile ovela kwa-Ops uhlale eqaphile futhi elungele ukuphendula isigameko sanoma ibuphi ubunzima. Asinawo amashifu ebusuku, okugcina i-psyche yethu futhi anikeze wonke umuntu ithuba lokulala ngokwanele futhi achithe isikhathi sokuphumula hhayi kukhompyutha kuphela.
Wonke umuntu unamakhono ahlukene: amanethiwekhi, ama-DBA, ochwepheshe besitaki be-ELK, abalawuli/abathuthukisi be-Kubernetes, ukuqapha, ukwenza izinto ezibonakalayo, ochwepheshe behadiwe, njll. Into eyodwa ehlanganisa wonke umuntu - wonke umuntu angangena esikhundleni sanoma yimuphi wethu ngokwezinga elithile: ngokwesibonelo, ngenisa ama-node amasha kuqoqo le-k8s, ubuyekeze i-PostgreSQL, ubhale ipayipi le-CI/CD + Ansible, wenze okuthile ku-Python/Bash/Go, xhuma i-hardware ku- Isikhungo sedatha. Amakhono aqinile kunoma iyiphi indawo awakuvimbeli ukuthi uguqule indlela osebenza ngayo futhi uqale ukuthuthuka kwenye indawo. Isibonelo, ngijoyine inkampani njengochwepheshe be-PostgreSQL, futhi manje indawo yami eyinhloko yomthwalo wemfanelo ngamaqoqo e-Kubernetes. Eqenjini, noma yikuphi ukuphakama kwamukelekile futhi umuzwa wokusebenzisa amandla uthuthukiswe kakhulu.
Phela siyazingela. Izimfuneko zamakhandidethi zisezingeni. Kimina uqobo, kubalulekile ukuthi umuntu ahlangane neqembu, akangqubuzani, kodwa futhi uyazi ukuthi angavikela kanjani umbono wakhe, ufuna ukuthuthukisa futhi akesabi ukwenza into entsha, anikeze imibono yakhe. Futhi, amakhono okuhlela ezilimini zokubhala, ulwazi lwezisekelo zeLinux nesiNgisi kuyadingeka. IsiNgisi sidingeka kalula ukuze umuntu uma kwenzeka i-fakap akwazi uku-google isixazululo senkinga ngemizuzwana eyi-10, hhayi ngemizuzu eyi-10. Manje sekunzima kakhulu ukuthola ochwepheshe abanolwazi olujulile lwe-Linux: kuyahlekisa, kodwa ababili kwabathathu abakwazi ukuphendula umbuzo othi “Siyini Isilinganiso Somthwalo? Yenziwe ngani?", Futhi umbuzo othi "Indlela yokuhlanganisa ukulahla okuyisisekelo kusuka ohlelweni lwe-C" kubhekwa njengento evela emhlabeni wama-supermen ... noma ama-dinosaurs. Kufanele sikubekezelele lokhu, ngoba ngokuvamile abantu banamakhono athuthuke kakhulu, kodwa sizofundisa i-Linux. Impendulo yombuzo othi "kungani unjiniyela we-DevOps edinga ukwazi konke lokhu emhlabeni wanamuhla wamafu" kuzodingeka ashiywe ngaphandle kwe-athikili, kodwa ngamagama amathathu: konke lokhu kuyadingeka.
Amathuluzi Ethimba
Ithimba Lamathuluzi lidlala indima ebalulekile ekuzisebenzeleni. Umsebenzi wabo oyinhloko ukudala amathuluzi afanele we-graphic kanye ne-CLI yabathuthukisi. Isibonelo, i-Confer yethu yokuthuthukiswa kwangaphakathi ikuvumela ukuthi ukhiphe ngokoqobo uhlelo lokusebenza ku-Kubernetes ngokuchofoza amagundane okumbalwa, ulungiselele izinsiza zayo, okhiye abavela ku-vault, njll. Ngaphambilini, kwakukhona i-Jenkins + Helm 2, kodwa kwadingeka ngithuthukise ithuluzi lami lokuqeda ukukopisha nokunamathisela nokuletha ukufana kumjikelezo wokuphila wesofthiwe.
Ithimba le-Ops aliwabhali amapayipi onjiniyela, kodwa lingaluleka nganoma yiziphi izinkinga ekubhaleni kwalo (abanye abantu basenayo i-Helm 3).
I-DevOps
Ngokuqondene ne-DevOps, siyibona kanje:
Amaqembu e-Dev abhala ikhodi, ayikhiphe nge-Confer to dev -> qa/stage -> prod. Isibopho sokuqinisekisa ukuthi ikhodi ayinensisi futhi ayiqukethe amaphutha iseqenjini le-Dev ne-Ops. Emini, umuntu osemsebenzini wethimba le-Ops kufanele aqale aphendule ngesigameko ngesicelo sakhe, futhi kusihlwa nasebusuku, umqondisi osemsebenzini (Ops) kufanele avuse unjiniyela osemsebenzini uma azi qiniseka ukuthi inkinga ayikho kwingqalasizinda. Wonke amamethrikhi nezixwayiso ekuqapheni zivela ngokuzenzakalelayo noma kancane ngokuzenzakalelayo.
Indawo yesibopho se-Ops iqala kusukela ngesikhathi isicelo senziwa ekukhiqizeni, kodwa umthwalo wemfanelo kaDev awugcini lapho - senza into efanayo futhi sisesikebheni esifanayo.
Onjiniyela baluleka abalawuli uma bedinga usizo lokubhala i-microservice yomqondisi (isibonelo, i-Go backend + HTML5), futhi abalawuli baluleka onjiniyela nganoma yiziphi izinkinga zengqalasizinda noma izinkinga ezihlobene nama-k8s.
Ngendlela, asinayo i-monolith nhlobo, ama-microservices kuphela. Inani labo kuze kube manje liyashintshashintsha phakathi kuka-900 no-1000 kuqoqo le-prod k8s, uma likalwa ngenombolo. ukuthunyelwa. Inani lama-pods liyashintshashintsha phakathi kuka-1700 no-2000. Okwamanje kukhona cishe ama-pods angu-2000 kuqoqo le-prod.
Angikwazi ukunikeza izinombolo eziqondile, njengoba siqapha ama-microservices angadingekile futhi sinqamule ngokuzenzakalelayo. Ama-K8 asisiza ukuthi silandelele amabhizinisi angadingekile , egcina izinsiza eziningi nemali.
Ukuphathwa kwezinsiza
Ukuqapha
Ukuqapha okuhleleke kahle futhi okufundisayo kuba yisisekelo ekusebenzeni kweqoqo elikhulu. Asikasitholi isixazululo esijwayelekile esingamboza u-100% wazo zonke izidingo zokuqapha, ngakho-ke ngezikhathi ezithile sidala izixazululo zangokwezifiso ezihlukile kule ndawo.
- Zabbix. Ukuqapha okuhle kwakudala, okuhloswe ngokuyinhloko ukulandelela isimo sonke sengqalasizinda. Lisitshela uma i-node ifa mayelana nokucubungula, inkumbulo, amadiski, inethiwekhi, njalonjalo. Akukho okungaphezu kwemvelo, kodwa futhi sine-DaemonSet ehlukile yama-ejenti, ngosizo, isibonelo, siqapha isimo se-DNS kuqoqo: sibheka ama-coredns pods ayisiphukuphuku, sihlola ukutholakala kwababungazi bangaphandle. Kungabonakala sengathi kungani uzikhathaza ngalokhu, kodwa ngomthamo omkhulu wethrafikhi le ngxenye iyiphuzu elibi lokwehluleka. Sengivele , ngizabalaze kanjani ngokusebenza kwe-DNS kuqoqo.
- I-Prometheus Operator. Iqoqo labathekelisi abahlukahlukene linikeza umbono omkhulu wazo zonke izingxenye zeqoqo. Okulandelayo, sikubona ngeso lengqondo konke lokhu kumadeshibhodi amakhulu e-Grafana, futhi sisebenzisa isiphathi se-alert ukuze uthole izexwayiso.
Elinye ithuluzi eliwusizo kithi kwaba . Siyibhale ngemuva kwezikhathi ezimbalwa sihlangabezane nesimo lapho iqembu elithile lidlulela kwezinye izindlela ze-Ingress, okuholele kumaphutha angu-50x. Manje ngaphambi kokuthumela ekukhiqizeni, abathuthukisi bahlola ukuthi akekho ozothinteka, futhi eqenjini lami leli ithuluzi elihle lokuxilongwa kokuqala kwezinkinga nge-Ingresses. Kuyahlekisa ukuthi ekuqaleni yayibhalelwe ama-admins futhi kubukeka sengathi “kubi”, kepha ngemuva kokuthi amathimba e-dev ethanda leli thuluzi, lashintsha kakhulu futhi laqala ukubukeka lingafani “nomphathi wenze ubuso bewebhu kubaphathi. ” Maduze sizoliyeka leli thuluzi futhi izimo ezinjalo zizoqinisekiswa nangaphambi kokuba ipayipi likhishwe.
Izinsiza zeqembu kuCube
Ngaphambi kokuthi singene ezibonelweni, kufanelekile ukuchaza ukuthi sizabela kanjani izinsiza microservices.
Ukuze uqonde ukuthi yimaphi amaqembu futhi ngamanani awasebenzisayo izinsiza (iprosesa, inkumbulo, i-SSD yendawo), sabela umyalo ngamunye owawo indawo yamagama ku-"Cube" futhi ukhawule amandla ayo aphezulu ngokuya ngeprosesa, inkumbulo kanye nediski, njengoba uxoxile ngaphambilini ngezidingo zamaqembu. Ngokufanelekile, umyalo owodwa, ngokuvamile, ngeke uvimbele lonke iqoqo ukuze lisetshenziswe, kwabelwa izinkulungwane zamacores nama-terabytes enkumbulo. Ukufinyelela endaweni yamagama kunikezwa nge-AD (sisebenzisa i-RBAC). Izikhala zamagama nemikhawulo yazo zengezwa ngesicelo sokudonsa endaweni yokugcina ye-GIT, bese yonke into ikhishwa ngokuzenzakalelayo ngepayipi le-Ansible.
Isibonelo sokwaba izinsiza eqenjini:
namespaces:
chat-team:
pods: 23
limits:
cpu: 11
memory: 20Gi
requests:
cpu: 11
memory: 20Gi
Izicelo nemikhawulo
Cubed" Isicelo inombolo yezinsiza eziqinisekisiwe ezibekelwe pod (isitsha sedokhu esisodwa noma ngaphezulu) kuqoqo. Umkhawulo uwumkhawulo ongaqinisekisiwe. Ngokuvamile ungabona kumagrafu ukuthi ithimba elithile lizibekele kanjani izicelo eziningi kakhulu zazo zonke izinhlelo zokusebenza futhi alikwazi ukuthumela uhlelo lokusebenza ku-“Cube”, njengoba zonke izicelo ngaphansi kwendawo yazo yamagama sezivele “zichithiwe”.
Indlela eyiyo yokuphuma kulesi simo iwukubheka ukusetshenziswa kwangempela kwezinsiza bese uqhathanisa nenani eliceliwe (Isicelo).
Kuzithombe-skrini ezingenhla ungabona ukuthi ama-CPU "Aceliwe" afaniswe nenombolo yangempela yochungechunge, futhi Imikhawulo ingadlula inombolo yangempela yezintambo ze-CPU =)
Manje ake sibheke enye indawo yamagama ngokuningiliziwe (ngikhethe i-namespace kube-system - indawo yamagama yesistimu yezingxenye ze-“Cube” ngokwayo) futhi sibone isilinganiso sesikhathi sokucubungula esisetshenziswe ngempela kanye nenkumbulo kuleyo eceliwe:
Kusobala ukuthi inkumbulo eningi kanye ne-CPU ibekelwe izinsiza zohlelo kunaleyo esetshenziswa ngempela. Endabeni ye-kube-system, lokhu kuyafaneleka: kwenzeka ukuthi isilawuli se-nginx ingress noma ama-nodelocaldns phezulu ashaye i-CPU futhi adle i-RAM eningi, ngakho-ke lapha ukugodla okunjalo kuyafaneleka. Ngaphezu kwalokho, asikwazi ukuthembela kumashadi emahoreni angu-3 okugcina: kuyathandeka ukubona amamethrikhi omlando phakathi nenkathi enkulu yesikhathi.
Kwasungulwa uhlelo “lwezincomo”. Isibonelo, lapha ungabona ukuthi iziphi izinsiza ezingaba ngcono uma uphakamisa "imikhawulo" (ibha ephezulu evunyelwe) ukuze "ukuncibilika" kungenzeki: isikhathi lapho insiza isivele isebenzise i-CPU noma inkumbulo esiqeshini sesikhathi esabiwe futhi ilindile ize "ingabe iqandisiwe":
Nawa ama-pods okufanele anciphise ukudla kwawo:
Mayelana throttling + ukuqapha kwezinsiza, ungabhala indatshana engaphezu kweyodwa, ngakho-ke buza imibuzo kumazwana. Ngamagama ambalwa, ngingasho ukuthi umsebenzi wokuzenzakalela amamethrikhi anjalo unzima kakhulu futhi udinga isikhathi esiningi nokulinganisa isenzo nemisebenzi "yewindi" kanye "CTE" Prometheus / VictoriaMetrics (la magama acashuniwe, njengoba kukhona cishe akukho okufana nalokhu ku-PromQL, futhi kufanele uhlukanise imibuzo esabekayo ibe izikrini ezimbalwa zombhalo futhi uzilungiselele).
Ngenxa yalokho, abathuthukisi banamathuluzi okuqapha izindawo zabo zamagama ku-Cube, futhi bayakwazi ukuzikhethela ukuthi yiziphi izinhlelo zokusebenza ezingase "zinqunywe," futhi yiziphi iziphakeli ezinganikezwa yonke i-CPU ubusuku bonke.
Izindlela
Enkampanini njengoba kwenzeka manje imfashini, sinamathela ku-DevOps- kanye SRE-umsebenzi Lapho inkampani inama-microservices ayi-1000, abathuthukisi abangaba ngu-350 kanye nama-admins ayi-15 kuyo yonke ingqalasizinda, kufanele "ube nemfashini": ngemuva kwawo wonke lawa "ma-baswords" kunesidingo esiphuthumayo sokwenza konke okuzenzakalelayo nawo wonke umuntu, futhi abaphathi akufanele babe ibhodlela. ezinqubweni.
Njenge-Ops, sinikeza amamethrikhi ahlukahlukene namadeshibhodi konjiniyela abahlobene namazinga okuphendula kwesevisi namaphutha.
Sisebenzisa izindlela ezifana nalezi: , и ngokuzihlanganisa ndawonye. Sizama ukunciphisa inani lamadeshibhodi ukuze kuthi shazi kucace ukuthi iyiphi isevisi eyehlisa isithunzi (isibonelo, amakhodi okuphendula ngomzuzwana, isikhathi sokuphendula ngamaphesenti angama-99), njalo njalo. Ngokushesha nje lapho amanye amamethrikhi amasha edingeka kumadeshibhodi avamile, siyawadweba ngokushesha futhi siyengeze.
Sekuphele inyanga ngingadwebi amagrafu. Lokhu mhlawumbe kuwuphawu oluhle: kusho ukuthi “okufunwayo” okuningi sekufeziwe. Kwenzeka ukuthi phakathi nesonto ngidwebe igrafu entsha okungenani kanye ngosuku.
Umphumela ubalulekile ngoba manje onjiniyela abavamile ukuya kubaphathi benemibuzo “lapho bangabheka khona uhlobo oluthile lwemethrikhi.”
Ukuqaliswa I-Service Mesh isiseduze futhi kufanele yenze ukuphila kube lula kuwo wonke umuntu, ozakwethu abavela ku-Tools sebevele sebesondele ekusebenziseni i-abstract "Istio yomuntu onempilo": umjikelezo wempilo wesicelo ngasinye se-HTTP uzobonakala ekuqashweni, futhi kuzokwazi njalo ukuqonda ukuthi “kusiphi isigaba lapho yonke into yaphuka khona” ngesikhathi sokusebenzisana (hhayi nje kuphela) nokusebenzisana. Bhalisela izindaba ezivela kuhabhu le-DomClick. =)
Ukusekelwa kwengqalasizinda ye-Kubernetes
Ngokomlando, sisebenzisa inguqulo enamagqabhagqabha Kubespray - Indima efanelekile yokukhipha, ukunweba kanye nokuvuselela i-Kubernetes. Ngesinye isikhathi, ukusekelwa kokufakwa okungeyona i-kubeadm kwanqanyulwa egatsheni elikhulu, futhi inqubo yokushintshela ku-kubeadm ayizange iphakanyiswe. Ngenxa yalokho, inkampani yaseSouthbridge yenze eyayo imfoloko (ngokusekelwa kwe-beadm kanye nokulungiswa okusheshayo kwezinkinga ezibucayi).
Inqubo yokubuyekeza wonke amaqoqo e-k8s ibukeka kanje:
- Thatha Kubespray kusuka eSouthbridge, hlola ngentambo yethu, Merjim.
- Sikhiphela isibuyekezo Ukucindezeleka- "Cube".
- Sikhipha i-node yokubuyekeza eyodwa ngesikhathi (ku-Ansible lokhu kungukuthi “serial: 1”) ngaphakathi I-Dev- "Cube".
- Siyabuyekeza Prod ngoMgqibelo kusihlwa inodi eyodwa ngesikhathi.
Kunezinhlelo zokuyibuyisela esikhathini esizayo Kubespray kokuthile ngokushesha bese uya kuyo kubeadm.
Sekukonke sinama-"Cubes" amathathu: I-Stress, i-Dev ne-Prod. Sihlela ukwethula enye (okubekwe eceleni okushisayo) I-Prod-“Cube” esikhungweni sedatha sesibili. Ukucindezeleka и I-Dev hlala “emishinini ebonakalayo” (oVirt for Stress kanye nefu le-VMWare le-Dev). Prod- "I-Cube" ihlala "ngensimbi engenalutho": lawa angama-node afanayo anezintambo ze-CPU ezingu-32, inkumbulo engu-64-128 GB kanye ne-300 GB SSD RAID 10 - kukhona angu-50 esewonke. Amanodi “amancane” amathathu anikezelwe “kumakhosi” Prod- "Cuba": 16 GB yememori, 12 CPU imicu.
Ukuze sithengise, sikhetha ukusebenzisa "insimbi engenalutho" futhi sigweme izendlalelo ezingadingekile ezifana I-OpenStack: asibadingi "omakhelwane abanomsindo" kanye ne-CPU ukweba isikhathi. Futhi inkimbinkimbi yokuphatha icishe iphindwe kabili esimweni se-OpenStack yangaphakathi.
Ku-CI/CD “Cubic” nezinye izingxenye zengqalasizinda sisebenzisa iseva ehlukile ye-GIT, i-Helm 3 (bekuwushintsho olubuhlungu ukusuka ku-Helm 2, kodwa sijabule kakhulu ngezinketho atomic), Jenkins, Ansible kanye neDocker. Siyawathanda amagatsha esici nokuthunyelwa ezindaweni ezahlukahlukene kusuka endaweni eyodwa yokugcina.
isiphetho
Lokhu, ngokwemibandela ejwayelekile, ukuthi inqubo ye-DevOps ibukeka kanjani kwa-DomClick ngokombono kanjiniyela wokusebenza. I-athikili ibonakale ingaphansi kobuchwepheshe kunalokho ebengikulindele: ngakho-ke, landela izindaba ze-DomClick ku-Habré: kuzoba nezindatshana eziningi "eziqinile" mayelana ne-Kubernetes nokuningi.
Source: www.habr.com