Igama lam nguViktor Yagofarov, kwaye ndiphuhlisa iqonga leKubernetes eDomClick njengomphathi wophuhliso lobugcisa kwiqela le-Ops (ukusebenza). Ndingathanda ukuthetha ngesakhiwo se-Dev yethu Iinkqubo ze-Ops, iimpawu zokusebenza enye yamaqela amakhulu e-k8s eRashiya, kunye nezenzo ze-DevOps / SRE ezisetyenziswa liqela lethu.
Iqela le-Ops
Iqela le-Ops okwangoku linabantu abali-15. Abathathu kubo banoxanduva lweofisi, ezimbini zisebenza kwindawo yexesha elahlukileyo kwaye ziyafumaneka, kubandakanya ebusuku. Ngaloo ndlela, umntu ovela kwi-Ops uhlala esweni kwaye ekulungele ukuphendula kwisiganeko saso nasiphi na ubunzima. Asinayo i-night shift, egcina i-psyche yethu kwaye inika wonke umntu ithuba lokulala ngokwaneleyo kwaye achithe ixesha lokuphumla kungekhona kwikhompyutheni kuphela.
Wonke umntu unezakhono ezahlukeneyo: iinethiwekhi, ii-DBA, iingcali ze-ELK stack, i-Kubernetes abalawuli/abaphuhlisi, ukubeka iliso, ukubonwa, iingcali zehardware, njl. Inye into edibanisa wonke umntu - wonke umntu unokuthatha indawo yethu kuye kwinqanaba elithile: umzekelo, ukwazisa iindawo ezintsha kwiqela le-k8s, hlaziya i-PostgreSQL, bhala i-CI/CD + umbhobho ongeyomfuneko, wenze into ethile kwiPython/Bash/Go, qhagamshela ihardware Iziko ledatha. Ubuchule obunamandla kuyo nayiphi na indawo akukuthinteli ekutshintsheni indlela osebenza ngayo kwaye uqalise ukuphucula kwenye indawo. Umzekelo, ndijoyine inkampani njengengcali yePostgreSQL, kwaye ngoku indawo yam ephambili yoxanduva ngamaqela eKubernetes. Kwiqela, nakuphi na ukuphakama kwamkelekile kwaye ingqiqo ye-leverage iphuhliswe kakhulu.
Phofu ke, siyazingela. Iimfuno zabaviwa zisemgangathweni. Kum ngokwam, kubalulekile ukuba umntu uhambelana neqela, akakho ingxabano, kodwa uyazi nendlela yokukhusela imbono yakhe, ufuna ukuphuhlisa kwaye akesabi ukwenza into entsha, unikezela ngeengcamango zakhe. Kwakhona, izakhono zokucwangcisa kwiilwimi zokubhala, ulwazi lweziseko zeLinux kunye nesiNgesi ziyafuneka. IsiNgesi sifuneka ngokulula ukuze umntu xa kwenzeka i-fakap akwazi uku-google isisombululo kwingxaki ngemizuzwana eyi-10, kwaye kungekhona kwimizuzu eyi-10. Ngoku kunzima kakhulu ukufumana iingcali ezinolwazi olunzulu lweLinux: iyahlekisa, kodwa abaviwa ababini kwabathathu abanakuphendula umbuzo othi “Yintoni uMyinge woMyinge? Yintoni eyenziwe ngayo? ", Kwaye umbuzo othi "Indlela yokudibanisa i-core dump kwiprogram ye-C" ithathwa njengento evela kwihlabathi le-supermen ... okanye iidinosaurs. Kuya kufuneka siyinyamezele le nto, kuba ngokwesiqhelo abantu baphuhlise kakhulu obunye ubuchule, kodwa siya kufundisa iLinux. Impendulo yombuzo othi "kutheni injineli ye-DevOps idinga ukwazi konke oku kwihlabathi lanamhlanje lamafu" kuya kufuneka ishiywe ngaphandle kwendawo yenqaku, kodwa ngamagama amathathu: konke oku kuyadingeka.
Izixhobo zeQela
Iqela leZixhobo lidlala indima ebalulekileyo kwi-automation. Umsebenzi wabo ophambili kukwenza izixhobo ezifanelekileyo zegraphic kunye neCLI zabaphuhlisi. Umzekelo, i-Confer yethu yophuhliso lwangaphakathi ikuvumela ukuba ukhuphele ngokwenyani isicelo kwi-Kubernetes ngonqakrazo nje lwemouse okumbalwa, qwalasela izixhobo zayo, izitshixo ezivela kwivault, njl. Ngaphambili, bekukho iJenkins + Helm 2, kodwa kuye kwafuneka ndiphuhlise esam isixhobo sokuphelisa ikopi-uncamathisele kwaye ndizise ukufana kubomi besoftware.
Iqela le-Ops alibhali imibhobho kubaphuhlisi, kodwa banokucebisa ngayo nayiphi na imiba ekubhaleni kwabo (abanye abantu basenayo iHelm 3).
DevOps
Ngokuphathelele i-DevOps, siyibona ngolu hlobo:
Amaqela e-Dev abhala ikhowudi, ayikhuphe nge-Confer to dev -> qa/stage -> prod. Uxanduva lokuqinisekisa ukuba ikhowudi ayicothi kwaye ayinazo iimpazamo kumaqela e-Dev kunye ne-Ops. Emini, umntu osemsebenzini ovela kwiqela le-Ops kufuneka kuqala aphendule kwisiganeko kunye nesicelo sakhe, kwaye ngokuhlwa nasebusuku, umlawuli osemsebenzini (Ops) kufuneka avuse umphuhlisi emsebenzini ukuba uyazi. uqinisekile ukuba ingxaki ayikho kwiziseko zoncedo. Zonke iimetrics kunye nezilumkiso ekubekweni esweni zivela ngokuzenzekelayo okanye isiqingatha ngokuzenzekelayo.
Indawo ye-Ops yoxanduva iqala ukusukela oko isicelo sikhutshwa kwimveliso, kodwa uxanduva lukaDev alupheli apho - senza into efanayo kwaye sikwisikhephe esifanayo.
Abaphuhlisi bacebisa abalawuli ukuba bafuna uncedo lokubhala i-microservice yolawulo (umzekelo, Go backend + HTML5), kwaye abalawuli bacebisa abaphuhlisi kuyo nayiphi na imiba yeziseko okanye imiba enxulumene ne-k8s.
Ngendlela, asinayo i-monolith kwaphela, kuphela i-microservices. Inani labo ukuza kuthi ga ngoku liguquguquka phakathi kwama-900 kunye ne-1000 kwiqela le-prod k8s, ukuba lilinganiswe ngenani. ukuthunyelwa. Inani lemidumba liyaguquguquka phakathi kwe-1700 kunye ne-2000. Ngoku kukho malunga ne-2000 pods kwiqela lemveliso.
Andikwazi ukunika amanani achanekileyo, kuba sijonga ii-microservices ezingafunekiyo kwaye sizinqumle ngokuzenzekelayo. Ii-K8s zisinceda sigcine umkhondo wamaziko angeyomfuneko , egcina izibonelelo ezininzi kunye nemali.
Ulawulo lwezibonelelo
Ukubeka iliso
Ukubekwa kweliso okucwangciseke kakuhle nokufundisa kuba sisiseko ekusebenzeni kweqela elikhulu. Asikafumani sisombululo sendalo yonke esiya kugubungela i-100% yazo zonke iimfuno zokubeka iliso, ngoko ke sidala izisombululo ezahlukeneyo zesiko kule meko.
- Zabbix. Ukubeka iliso kwakudala okuhle, okujoliswe ikakhulu ukulandelela imeko yonke yeziseko zophuhliso. Isixelela xa i-node isifa ngokwenkqubo, inkumbulo, iidiski, inethiwekhi, njalo njalo. Akukho nto engaphezu kwendalo, kodwa sikwanayo i-DaemonSet eyahlukileyo yee-agent, ngoncedo apho, ngokomzekelo, sibeka iliso kwi-DNS kwi-cluster: sijonge i-coredns pods yesiphukuphuku, sijonga ukufumaneka kwemikhosi yangaphandle. Kuya kubonakala ngathi kutheni ukuzikhathaza ngale nto, kodwa ngomthamo omkhulu wetrafikhi eli candelo liyinqaku elibi lokusilela. Mna kakade , indlela endisokole ngayo kunye nokusebenza kwe-DNS kwiqela.
- Umqhubi wePrometheus. Iseti yabathengisi bangaphandle abahlukeneyo inika isishwankathelo esikhulu sawo onke amacandelo eqela. Okulandelayo, sijonga konke oku kwiideshibhodi ezinkulu eGrafana, kwaye sisebenzise umlawuli wesilumkiso kwizilumkiso.
Esinye isixhobo esiluncedo kuthi yayi . Siyibhale emva kwamatyeli aliqela sadibana nemeko apho elinye iqela lagqitha kwiindlela ze-Ingress zelinye, okubangele iimpazamo ezingama-50x. Ngoku ngaphambi kokuthunyelwa kwimveliso, abaphuhlisi bajonga ukuba akukho mntu uya kuchaphazeleka, kwaye kwiqela lam esi sisixhobo esilungileyo sokuxilongwa kokuqala kweengxaki nge-Ingresses. Kuyinto ehlekisayo ukuba ekuqaleni yayibhalelwe ii-admins kwaye ibonakala ngathi "inzima", kodwa emva kokuba amaqela e-dev ethandane nesixhobo, yatshintsha kakhulu kwaye yaqala ukubonakala ngathi "umlawuli wenze ubuso bewebhu kubaphathi. ” Kungekudala siya kusishiya esi sixhobo kwaye iimeko ezinjalo ziya kuqinisekiswa nangaphambi kokuba kukhutshwe umbhobho.
Izixhobo zeqela kwiCube
Ngaphambi kokuba singene kwimizekelo, kufanelekile ukuba sichaze indlela esizabela ngayo izibonelelo microservices.
Ukuqonda ukuba ngawaphi amaqela kwaye ngawaphi amanani asebenzisa awo izixhobo (iprosesa, imemori, i-SSD yendawo), sabela umyalelo ngamnye indawo yegama kwi "Cube" kwaye unciphise amandla ayo aphezulu ngokweprosesa, imemori kunye nediski, sele ixoxile ngaphambili iimfuno zamaqela. Ngokufanelekileyo, umyalelo omnye, ngokubanzi, awuyi kuthintela yonke i-cluster yokuthunyelwa, ukwaba amawaka eecores kunye neeterabytes zememori. Ukufikelela kwindawo yamagama kunikwe nge-AD (sisebenzisa i-RBAC). Izithuba zamagama kunye nemida yazo zongezwa ngesicelo sokutsalwa kwindawo yokugcina ye-GIT, kwaye emva koko yonke into ikhutshwe ngokuzenzekelayo ngombhobho oNgameleyo.
Umzekelo wokwabiwa kwezibonelelo kwiqela:
namespaces:
chat-team:
pods: 23
limits:
cpu: 11
memory: 20Gi
requests:
cpu: 11
memory: 20Gi
Izicelo kunye nemida
Cubed" isicelo linani lezibonelelo eziqinisekisiweyo ezigciniweyo ze ngaphantsi (isikhongozeli esinye okanye ezingaphezulu) kwiqela. Umda ngowona uphezulu ongaqinisekiswanga. Ungabona rhoqo kwiigrafu ukuba iqela elithile lizibeke njani izicelo ezininzi kuzo zonke izicelo zalo kwaye alikwazi ukuhambisa isicelo kwi "Cube", ekubeni zonke izicelo phantsi kwegama labo sele "zichithiwe".
Indlela echanekileyo yokuphuma kule meko kukujonga eyona ndlela isetyenziswayo yobutyebi kwaye uyithelekise nesixa esiceliweyo (Isicelo).
Kwizikrini ezingentla ungabona ukuba ii-CPU "eziceliwe" zihambelana nenani langempela leentambo, kwaye iMida ingadlula inani lokwenyani lemicu ye-CPU =)
Ngoku makhe sijonge isithuba samagama ngokweenkcukacha (ndikhethe isithuba segama kube-inkqubo-isithuba segama senkqubo yamacandelo e “Cube” ngokwayo) kwaye ubone umlinganiselo wexesha elisetyenzisiweyo lomqhubekekisi kunye nenkumbulo kulowo uceliweyo:
Kucacile ukuba inkumbulo eninzi kunye ne-CPU igcinelwe iinkonzo zenkqubo kunokuba isetyenziswa ngokwenene. Kwimeko ye-kube-system, oku kuyathetheleleka: kwenzekile ukuba i-nginx ingress isilawuli okanye i-nodelocaldns kwincopho yazo ibethe i-CPU kwaye idle i-RAM eninzi, ke apha ugcino olunje lufanelekile. Ukongeza, asikwazi ukuxhomekeka kwiitshathi kwiiyure ezi-3 zokugqibela: kuyanqweneleka ukubona iimethrikhi zembali kwixesha elikhulu.
Inkqubo “yezindululo” yaphuhliswa. Umzekelo, apha unokubona ukuba zeziphi izixhobo eziya kuba ngcono ekuphakamiseni "imida" (ibha ephezulu evunyelweyo) ukuze "i-throttling" ingenzeki: umzuzu xa uvimba sele uchithe i-CPU okanye imemori kwixesha elabelwe kwaye ilinde ide ibe "ingakhenceki":
Kwaye nantsi imidumba ekufuneka ithintele ukutya kwabo:
phezu ukutswina + ukujonga izixhobo, ungabhala ngaphezulu kwenqaku elinye, ke buza imibuzo kwizimvo. Ngamagama ambalwa, ndingatsho ukuba umsebenzi wokuzenzekelayo iimetriki ezinjalo unzima kakhulu kwaye ufuna ixesha elininzi kunye nesenzo sokulinganisa kunye nemisebenzi "yefestile" kunye ne "CTE" Prometheus / VictoriaMetrics (la magama akwisicatshulwa, kuba kukho phantse akukho nto inje kwi-PromQL, kwaye kuya kufuneka wahlule imibuzo eyoyikisayo kwizikrini ezininzi zokubhaliweyo kwaye uzilungiselele).
Ngenxa yoko, abaphuhlisi banezixhobo zokubeka iliso kwiindawo zabo zamagama kwiCube, kwaye bayakwazi ukuzikhethela apho kwaye ngaliphi ixesha apho izicelo zinokuthi "zinqunyulwe," kwaye zeziphi iiseva ezinokunikwa yonke i-CPU ubusuku bonke.
Iindlela
Kwinkampani njengoko injalo ngoku isefashonini, sibambelela kwi-DevOps- kunye SRE-ingcali Xa inkampani inama-microservices angama-1000, malunga nabaphuhlisi be-350 kunye nee-admins ze-15 kuzo zonke iziseko zophuhliso, kufuneka "ube nemfashini": emva kwazo zonke ezi "baswords" kukho isidingo esiphuthumayo sokwenza yonke into kunye nayo yonke into, kwaye i-admins ayifanele ibe yingxaki. kwiinkqubo.
Njenge-Ops, sibonelela ngeemetrics ezahlukeneyo kunye needashbhodi kubaphuhlisi abanxulumene namazinga empendulo yenkonzo kunye neempazamo.
Sisebenzisa iindlela ezifana: , и ngokuzidibanisa kunye. Sizama ukunciphisa inani leedeshibhodi ukwenzela ukuba ngokujonga kucace ukuba yeyiphi inkonzo ethotywayo ngoku (umzekelo, iikhowudi zokuphendula ngomzuzwana, ixesha lokuphendula nge-99th percentile), njalo njalo. Nje ukuba ezinye iimetrikhi ezintsha zibe yimfuneko kwiideshibhodi ngokubanzi, sizoba ngoko nangoko kwaye songeze.
Khange ndizobe iigrafu kangangenyanga. Oku mhlawumbi luphawu oluhle: kuthetha ukuba uninzi "lokufunwayo" sele lufezekile. Kwenzeka ukuba phakathi evekini ndizobe igrafu entsha kanye ngemini.
Iziphumo zibalulekile kuba ngoku abaphuhlisi abafane baye kubaphathi ngemibuzo “apho bajonge khona uhlobo oluthile lwemetric.”
Ukuphunyezwa I-Service Mesh ijikeleze ikona kwaye kufuneka benze ubomi bube lula kumntu wonke, oogxa abavela kwiZixhobo sele besondele ekuphumezeni i-abstract "Istio yomntu onempilo": umjikelo wobomi besicelo ngasinye seHTTP (s) siya kubonakala ekubekeni iliso, kwaye kuya kuhlala kunokwenzeka ukuqonda "kweliphi inqanaba yonke into yaphuka" ngexesha le-inter-service (kwaye kungekhona kuphela) ukusebenzisana. Bhalisela iindaba ezivela kwi-DomClick hub. =)
Inkxaso yeziseko zophuhliso kubernetes
Ngokwembali, sisebenzisa inguqulelo ekhutshiweyo Kubespray -Indima efanelekileyo yokuhambisa, ukwandisa kunye nokuhlaziya iKubernetes. Ngexesha elithile, inkxaso yofakelo olungelulo-kubeadm yasikwa kwisebe eliphambili, kwaye inkqubo yokutshintshela kwi-kubeadm ayizange icetywe. Ngenxa yoko, inkampani yaseSouthbridge yenze eyayo ifolokhwe (ngenkxaso ye-beadm kunye nokulungiswa ngokukhawuleza kweengxaki ezinzima).
Inkqubo yokuhlaziya onke amaqela e-k8s ijongeka ngolu hlobo:
- Thatha Kubespray ukusuka eSouthbridge, jonga ngentambo yethu, Merjim.
- Sihambisa uhlaziyo ku istres- "Cube".
- Sikhupha uhlaziyo lwenodi enye ngexesha (kwi-Ansible le yi "serial: 1") ngaphakathi dev- "Cube".
- Siyahlaziya Prod ngorhatya lwangoMgqibelo indawo enye ngexesha.
Kukho izicwangciso zokuyibuyisela kwixa elizayo Kubespray into ekhawulezayo kwaye uye kuyo kubeadm.
Lilonke sine "Cubes" ezintathu: Uxinzelelo, iDev kunye neProd. Siceba ukusungula enye (imo yokulinda eshushu) Imveliso-"Cube" kwiziko ledatha yesibini. istres и dev hlala “koomatshini bokwenene” (i-oVirt yoxinzelelo kunye nelifu le-VMWare le-Dev). Prod- "Cube" ihlala "kwintsimbi engenanto": ezi ziindawo ezifanayo ezinemicu ye-CPU engama-32, i-64-128 GB yememori kunye ne-300 GB SSD RAID 10 - kukho i-50 yazo iyonke. Iindawo ezintathu "ezincinci" zinikezelwe kwi "masters" Prod- "Cuba": I-16 GB yememori, i-12 intambo ye-CPU.
Ukuthengisa, sikhetha ukusebenzisa "isinyithi esingenanto" kwaye sigweme iileya ezingabalulekanga ezifana OpenStack: asifuni "abamelwane abanomsindo" kunye ne-CPU beba ixesha. Kwaye ubunzima bolawulo malunga nokuphindaphinda kabini kwimeko ye-OpenStack yangaphakathi.
Kwi-CI/CD "Cubic" kunye nezinye izinto ezisisiseko sisebenzisa i-server ye-GIT eyahlukileyo, i-Helm 3 (yayiyinguqu ebuhlungu ukusuka kwi-Helm 2, kodwa sivuya kakhulu ngeenketho atomic), Jenkins, Ansible kunye neDocker. Siyawathanda amasebe amacandelo kunye nokusasazwa kweemeko ezahlukeneyo ukusuka kwindawo yokugcina.
isiphelo
Oku, ngokubanzi, ukuba inkqubo ye-DevOps ijongeka njani kwi-DomClick ngokwembono yenjineli yokusebenza. Eli nqaku liye labonakala lingaphantsi kobugcisa kunokuba bendilindele: ngoko ke, landela iindaba ze-DomClick kwi-Habré: kuya kubakho amanqaku amaninzi "anzima" malunga ne-Kubernetes kunye nokunye.
umthombo: www.habr.com