Ini ndinonzi Viktor Yagofarov, uye ndiri kugadzira Kubernetes chikuva paDomClick semaneja wekuvandudza tekinoroji muboka reOps (operation). Ndinoda kutaura nezve chimiro cheDev yedu <-> Ops maitiro, maitiro ekushanda rimwe remasumbu makuru ek8s muRussia, pamwe chete neDevOps / SRE maitiro anoshandiswa nechikwata chedu.
Ops Team
Chikwata cheOps parizvino chine vanhu gumi nevashanu. Vatatu vavo vane basa rehofisi, maviri anoshanda mune imwe nzvimbo yenguva uye anowanikwa, kusanganisira manheru. Nekudaro, mumwe munhu anobva kuOps anogara ari pamonitor uye akagadzirira kupindura kune chiitiko chechero chakaoma. Hatina kuchinja kweusiku, iyo inochengetedza psyche yedu uye inopa munhu wose mukana wekuwana zvakakwana kurara uye kupedza nguva yekuzorora kwete pakombiyuta chete.
Wese munhu ane hunyanzvi hwakasiyana: networker, DBAs, ELK stack nyanzvi, Kubernetes admins/vagadziri, kutarisa, virtualization, hardware nyanzvi, nezvimwe. Chinhu chimwe chinobatanidza munhu wese - munhu wese anogona kutsiva chero wedu kusvika padanho rakati: semuenzaniso, suma node nyowani muboka reK8s, gadziridza PostgreSQL, nyora CI/CD + Ansible pombi, otomatiki chimwe chinhu muPython/Bash/Go, batanidza Hardware ku. Data center. Hunyanzvi hwakasimba munzvimbo ipi neipi haukutadzise kubva pakuchinja nzira yako yekuita uye kutanga kuvandudza mune imwe nzvimbo. Semuenzaniso, ndakabatana nekambani sePostgreSQL nyanzvi, uye ikozvino nzvimbo yangu huru yebasa ndeyeKubernetes masumbu. Muchikwata, chero hurefu hunogamuchirwa uye pfungwa yekuwedzera inogadzirwa zvakanyanya.
Nenzira, tiri kuvhima. Zvinodikanwa zvevavhoti zvakaringana. Kwandiri pachedu, zvakakosha kuti munhu akwanise kupinda muboka, haasi kupesana, asiwo anoziva nzira yekudzivirira maonero ake, anoda kukura uye haatyi kuita chimwe chinhu chitsva, anopa pfungwa dzake. Zvakare, hunyanzvi hwekuronga mumitauro yekunyora, ruzivo rwezvakakosha zveLinux neChirungu zvinodiwa. Chirungu chinodiwa zviri nyore kuti kana munhu aita fakap anogona google mhinduro yedambudziko mumasekondi gumi, uye kwete mumaminitsi gumi. Izvozvi zvakaoma kuwana nyanzvi dzine ruzivo rwakadzama rweLinux: zvinosekesa, asi vaviri kubva pavatatu vavhoti havagone kupindura mubvunzo "Chii chinonzi Avhareji yeMutoro? Chii chakaitwa? ", Uye mubvunzo wekuti "Nzira yekuunganidza marara kubva purogiramu yeC" inofungidzirwa chimwe chinhu kubva kunyika ye supermen ... kana dinosaurs. Isu tinofanirwa kushivirira izvi, sezvo kazhinji vanhu vakanyanya kuvandudza humwe hunyanzvi, asi isu tichadzidzisa Linux. Mhinduro kumubvunzo wekuti "sei injiniya weDevOps achida kuziva zvese izvi munyika yemazuva ano yemakore" ichafanira kusiiwa kunze kwechikamu chechinyorwa, asi mumashoko matatu: zvese izvi zvinodiwa.
Team Tools
Chikwata cheZvishandiso chinoita basa rakakosha mune otomatiki. Basa ravo guru nderekugadzira zviri nyore graphical uye CLI zvishandiso zvevagadziri. Semuyenzaniso, yedu yemukati budiriro Confer inokutendera kuti unyatso kuburitsa application kuKubernetes nekungodzvanya mbeva shoma, gadzirisa zviwanikwa zvayo, makiyi kubva kune vault, nezvimwe. Pakutanga, pakanga paine Jenkins + Helm 2, asi ini ndaifanira kugadzira yangu chishandiso kubvisa kopi-paste uye kuunza kufanana kune software lifecycle.
Chikwata cheOps hachinyore mapaipi evagadziri, asi vanogona kuraira pane chero nyaya mukunyora kwavo (vamwe vanhu vachiri neHelm 3).
DevOps
Kana iri DevOps, tinozviona seizvi:
Zvikwata zveDev zvinonyora kodhi, iburitse kuburikidza neConfer to dev -> qa/stage -> prod. Basa rekuona kuti kodhi hainonoke uye haina zvikanganiso iri neDev uye Ops zvikwata. Masikati, munhu ari pabasa kubva kuboka reOps anofanira kutanga apindura kune chiitiko nechikumbiro chavo, uye manheru nehusiku, maneja ari pabasa (Ops) anofanira kumutsa mugadziri ari pabasa kana achiziva. chokwadi chekuti dambudziko harisi muzvigadzirwa. Ese metrics uye zviyeuchidzo mukutarisisa zvinoonekwa otomatiki kana semi-otomatiki.
Nzvimbo yeOps yebasa inotanga kubva panguva iyo application yaiswa mukugadzirwa, asi basa raDev harigumire ipapo - isu tinoita chinhu chimwe chete uye tiri muchikepe chimwe chete.
Vagadziri vanorayira admins kana vachida rubatsiro kunyora admin microservice (semuenzaniso, Go backend + HTML5), uye maadmins anorayira vanogadzira pane chero nyaya dzezvivakwa kana nyaya dzine chekuita nek8s.
Nenzira, isu hatina monolith zvachose, chete microservices. Huwandu hwavo kusvika parizvino hunochinja pakati pe900 ne1000 muprod k8s cluster, kana ikayerwa nenhamba. deployments. Huwandu hwemapods hunochinja pakati pe1700 ne2000. Parizvino pane mapodhi anosvika 2000 muprod cluster.
Ini handikwanise kupa nhamba chaidzo, sezvo isu tichitarisa zvisina basa mamicroservices uye tinoacheka semi-otomatiki. K8s inotibatsira kuchengeta zvinyorwa zvisingakoshi , iyo inochengetedza zvinhu zvakawanda uye mari.
Resource management
Kuongorora
Kunyatsorongeka uye ruzivo rwekuongorora runova nheyo yekushanda kweboka guru. Isu hatisati tawana mhinduro yepasirese iyo inogona kuvhara 100% yezvese zvinodiwa zvekutarisisa, saka isu nguva nenguva tinogadzira akasiyana echinyakare mhinduro munzvimbo ino.
- Zabbix. Kutarisisa kwakanaka kwekare, uko kunoitirwa kunyanya kuteedzera mamiriro ese ezvivakwa. Inotitaurira kana node inofa maererano nekugadzirisa, ndangariro, disks, network, zvichingodaro. Hapana mashura, asi isu zvakare tine yakaparadzana DaemonSet yevamiririri, nerubatsiro rwekuti, semuenzaniso, isu tinotarisa mamiriro eDNS musumbu: isu tinotarisa benzi coredns pods, isu tinotarisa kuwanikwa kwevaenzi vekunze. Zvingaita sekuti sei uchinetseka neizvi, asi nehombe dzemotokari chikamu ichi chinhu chakakomba chekutadza. Kare ini nechekare , ndakanetsekana sei nekuita kweDNS muchikwata.
- Prometheus Operator. Seti yevashambadziri vakasiyana inopa tarisiro huru yezvikamu zvese zvesumbu. Tevere, tinoona zvese izvi pamadhibhodhi makuru muGrafana, uye shandisa alertmanager yekuzivisa.
Chimwe chishandiso chinobatsira kwatiri chaiva . Isu takazvinyora mushure menguva dzakati wandei takasangana nemamiriro ezvinhu apo chimwe chikwata chakapfuura nzira dzeIngress dzechimwe chikwata, zvichikonzera zvikanganiso makumi mashanu. Iye zvino vasati vaendesa kukugadzira, vanogadzira vanotarisa kuti hapana anozokanganiswa, uye kuchikwata changu ichi chishandiso chakanaka chekutanga kuongororwa kwematambudziko neIngresses. Zvinosekesa kuti pakutanga zvakanyorerwa maadmins uye zvaiita kunge “zvisinganzwisisike”, asi mushure mekunge zvikwata zvema dev zvadanana nechishandiso, zvakachinja zvakanyanya ndokutanga kusaita senge “admin akagadzira face yewebhu kune admins. ” Munguva pfupi isu tichasiya chishandiso ichi uye mamiriro akadai anozosimbiswa kunyangwe pombi isati yaburitswa.
Zviwanikwa zvechikwata muCube
Tisati tapinda mumienzaniso, zvakakodzera kutsanangura kuti tinogovera sei zviwanikwa microservices.
Kuti unzwisise kuti ndezvipi zvikwata uye muhuwandu hupi vanoshandisa yavo zviwanikwa (processor, ndangariro, yemunharaunda SSD), isu tinogovera yega yega kuraira kwayo namespace mu "Cube" uye kudzikisira kugona kwayo kwakanyanya maererano ne processor, ndangariro uye dhisiki, yakambokurukura zvinodiwa nezvikwata. Saizvozvo, murairo mumwechete, kazhinji, hauzovharidzi sumbu rese rekutumira, kugovera zviuru zvemacores uye terabytes endangariro. Kuwana nzvimbo yezita kunopihwa kuburikidza neAD (tinoshandisa RBAC). Mazita enzvimbo nemiganhu yawo anowedzerwa kuburikidza nechikumbiro chekudhonza kune iyo GIT repository, uyezve zvese zvinobva zvatenderedzwa kunze kuburikidza neAnsible pombi.
Muenzaniso wekugovera zviwanikwa kuchikwata:
namespaces:
chat-team:
pods: 23
limits:
cpu: 11
memory: 20Gi
requests:
cpu: 11
memory: 20Gi
Zvikumbiro nemiganhu
Cubed" chikumbiro ndiyo nhamba yezvakavimbiswa zvakachengetwa zviwanikwa pod (imwe kana anopfuura docker midziyo) musumbu. Limit ndeye isinga vimbiswa huwandu. Iwe unogona kazhinji kuona pamagirafu kuti imwe timu yakazvimisira sei zvikumbiro zvakawandisa kune ese mashandisirwo ayo uye haigone kuendesa application ku "Cube", sezvo zvikumbiro zvese pasi pemazita avo "zvakatoshandiswa".
Nzira chaiyo yekubuda mumamiriro ezvinhu aya ndeyekutarisa iyo chaiyo yekushandisa zviwanikwa uye kuienzanisa nemari yakakumbirwa (Chikumbiro).
Mumascreenshots ari pamusoro unogona kuona kuti "Akakumbirwa" maCPU anofananidzwa nenhamba chaiyo yeshinda, uye Miganhu inogona kudarika iyo chaiyo nhamba yeCPU tambo =)
Zvino ngatitarisei imwe nzvimbo yezita zvakadzama (ndakasarudza namespace kube-system - iyo system namespace yezvikamu zve "Cube" pachayo) uye ona chiyero cheiyo yakashandiswa processor nguva uye ndangariro kune yakakumbirwa:
Zviripachena kuti yakawanda ndangariro uye CPU yakachengeterwa masevhisi ehurongwa kupfuura anoshandiswa chaizvo. Panyaya yekube-system, izvi zvinoruramiswa: zvakaitika kuti nginx ingress controller kana nodelocaldns panhongonya yavo yakarova CPU uye yakadya yakawanda RAM, saka pano kuchengetwa kwakadaro kwakakodzera. Pamusoro pezvo, isu hatigone kuvimba nemachati kwemaawa matatu ekupedzisira: zvinodikanwa kuona mametrics enhoroondo munguva yakakura.
Nzira ye "kukurudzira" yakagadzirwa. Semuenzaniso, pano iwe unogona kuona kuti ndezvipi zviwanikwa zvingave zviri nani pakusimudza "miganhu" (iyo yepamusoro inobvumidzwa bar) kuitira kuti "kutambisa" kurege kuitika: iyo nguva iyo sosi yakatoshandisa CPU kana ndangariro mune yakagoverwa nguva chidimbu uye. iri kumirira kusvika yave "isina chando":
Uye heano mapods anofanirwa kudzora havi yavo:
pamusoro throttling + yekutarisa zviwanikwa, unogona kunyora zvinopfuura chinyorwa chimwe, saka bvunza mibvunzo mune zvakataurwa. Mumashoko mashoma, ndinogona kutaura kuti basa rekuita otomatiki mametrics akadaro rakaoma kwazvo uye rinoda nguva yakawanda uye kuenzanisa kuita ne “hwindo” mabasa uye “CTE” Prometheus / VictoriaMetrics (aya mazwi ari mukotesheni, sezvo paine anenge ari hapana chakaita seizvi muPromQL, uye iwe unofanirwa kupatsanura mibvunzo inotyisa kuita akati wandei masikirini ezvinyorwa uye woagonesa).
Nekuda kweizvozvo, vanogadzira vane maturusi ekutarisa mazita avo muCube, uye vanokwanisa kuzvisarudzira kupi uye panguva ipi iyo maapplication anogona kuve nemidziyo yavo "kucheka," uye ndeapi maseva anogona kupihwa iyo CPU yese husiku hwese.
Nzira
Mukambani sezvazviri ikozvino fashionable, tinonamatira kuDevOps- uye SRE-mudzidzi Kana kambani iine 1000 microservices, vangangoita mazana matatu nemakumi mashanu evagadziri uye gumi nemashanu admins kune ese magadzirirwo, iwe unofanirwa "kuve fashoni": kuseri kwe "baswords" ese aya pane kudiwa kwekukurumidzira kuita zvese uye munhu wese, uye admins haifanire kunge iri bhodhoro. mumaitiro.
SeOps, tinopa akasiyana metrics uye dashboards kune vanogadzira zvine chekuita nereti yekupindura masevhisi uye kukanganisa.
Isu tinoshandisa nzira dzakadai se: , и nokuzvibatanidza pamwechete. Tinoedza kuderedza kuwanda kwedashboards kuitira kuti pakatarisa zviri pachena kuti ndeipi sevhisi iri kudzikisira parizvino (somuenzaniso, macode emhinduro pasekondi, nguva yekupindura ne99th percentile), zvichingodaro. Kana mamwe ma metrics matsva achinge ave madikanwa kune akajairwa madhibhodhi, isu tinobva tadhirowa nekuawedzera.
Ndave nemwedzi ndisina kudhirowa magirafu. Ichi chingangodaro chiratidzo chakanaka: zvinoreva kuti zvizhinji zve "zvinoda" zvakatoitika. Zvakaitika kuti mukati mevhiki ndaidhirowa rimwe girafu idzva kamwe chete pazuva.
Mhedzisiro yacho yakakosha nekuti ikozvino vagadziri havawanzo kuenda kune admins nemibvunzo "kupi kwekutarisa imwe mhando yemetric."
Kutevedzera Service Mesh yave pedyo nekona uye inofanirwa kuita kuti hupenyu huve nyore kune wese munhu, vatinoshanda navo kubva kuZvishandiso vatove pedyo nekuita iyo abstract "Istio yemunhu ane hutano": kutenderera kwehupenyu hweumwe neumwe HTTP (s) chikumbiro kuchaonekwa mukutarisa, uye zvichagara zvichigoneka kuti unzwisise "padanho ripi zvese zvakaputsika" panguva yekudyidzana (uye kwete chete) kudyidzana. Nyorera kune nhau kubva kuDomClick hub. =)
Kubernetes tsigiro yezvivakwa
Nhoroondo, isu tinoshandisa yakavharwa vhezheni Kubespray -Inogoneka basa rekutumira, kuwedzera uye kugadzirisa Kubernetes. Pane imwe nguva, rutsigiro rwekusina-kubeadm kumisikidzwa rwakatemwa kubva kubazi guru, uye maitiro ekuchinja kubeadm haana kutaurwa. Nekuda kweizvozvo, iyo Southbridge kambani yakagadzira yayo forogo (ine kubeadm rutsigiro uye nekukurumidza kugadzirisa kwematambudziko akakosha).
Maitiro ekugadzirisa ese k8s masumbu anotaridzika seizvi:
- Tora Kubespray kubva kuSouthbridge, tarisa neshinda yedu, Merjim.
- Isu tiri kuburitsa iyo update ku kushushikana- "Cube".
- Isu tinoburitsa iyo yekuvandudza node imwe panguva (muAnsible iyi "serial: 1") mukati Dev- "Cube".
- Isu tinovandudza Prod neMugovera manheru node imwe panguva.
Pane zvirongwa zvekuitsiva mune ramangwana Kubespray kune chimwe chinhu chinokurumidza uye enda kubeadm.
Pakazara isu tine matatu "Cubes": Stress, Dev uye Prod. Tiri kuronga kutanga imwe (inopisa standby) Prod-"Cube" mune yechipiri data center. kushushikana и Dev gara mu "virtual machines" (oVirt yeStress uye VMWare gore reDev). Prod- "Cube" inogara pa "simbi isina chinhu": aya akafanana node ane 32 CPU tambo, 64-128 GB yekuyeuka uye 300 GB SSD RAID 10 - pane makumi mashanu akazara. Node nhatu "tete" dzakatsaurirwa kune "masters" Prod- "Cuba": 16 GB yendangariro, gumi nemaviri CPU tambo.
Nekutengesa, isu tinosarudza kushandisa "simbi isina chinhu" uye kudzivirira zvisingakoshi maseru senge OpenStack: hatidi "vavakidzani vane ruzha" uye CPU kuba nguva. Uye kuomarara kwekutonga kunowedzera kaviri munyaya ye-mumba OpenStack.
Kune CI/CD "Cubic" uye zvimwe zvikamu zvezvivakwa isu tinoshandisa yakaparadzana GIT server, Helm 3 (yaive shanduko inorwadza kubva kuHelm 2, asi isu tinofara kwazvo nesarudzo. atomiki), Jenkins, Ansible uye Docker. Isu tinoda maficha matavi uye kuendesa kunzvimbo dzakasiyana kubva kune imwe repository.
mhedziso
Izvi ndizvo, mune zvakajairika, izvo maitiro eDevOps anotaridzika kuDomClick kubva pamaonero einjiniya anoshanda. Chinyorwa chakazove chisiri hunyanzvi pane zvandaitarisira: saka, tevera iyo DomClick nhau paHabré: pachave neakawanda "hardcore" zvinyorwa nezve Kubernetes nezvimwe.
Source: www.habr.com