Kudzoka muna 2016 isu kuBuffer , uye zvino angangoita 60 node (paAWS) uye 1500 midziyo iri kushanda pane yedu k8s cluster inotungamirwa ne . Nekudaro, takatamira kumamicroservices kuburikidza nekuyedza uye kukanganisa, uye kunyangwe mushure memakore akati wandei ekushanda nema k8s tichiri kutarisana nematambudziko matsva. Mune ino post tichataura nezvazvo processor zvisingakwanisi: sei taifunga kuti itsika dzakanaka uye nei dzakaguma dzisina kunaka.
processor kugumira uye throttling
Kufanana nevamwe vazhinji vashandisi veKubernetes, . Pasina kurongeka kwakadaro, midziyo mune node inogona kutora ese processor simba, izvo zvinozoita zvakakosha Kubernetes maitiro (semuenzaniso. kubelet) icharega kupindura zvikumbiro. Saka, kuseta CPU miganhu inzira yakanaka yekudzivirira node dzako.
processor miganho inoisa mudziyo kune yakanyanya CPU nguva yainogona kushandisa kwenguva yakati (default ndeye 100ms), uye mudziyo haufe wakapfuura uyu muganho. In Kubernetes for throttling mudziyo uye kudzivirira kubva pakupfuura muganhu, chishandiso chakakosha chinoshandiswa , asi aya ekugadzira eCPU miganho anopedzisira akuvadza kuita uye kuwedzera nguva yekupindura yemidziyo yako.
Chii chingaitika kana tikasaisa processor miganhu?
Zvinosuruvarisa, isu pachedu taifanira kutarisana nechinetso ichi. Imwe neimwe node ine maitiro ane chekuita nekugadzirisa midziyo kubelet, uye akarega kupindura zvikumbiro. Iyo node, kana izvi zvaitika, ichapinda mudunhu NotReady, uye midziyo kubva mairi ichaendeswa kune imwe nzvimbo uye kugadzira matambudziko akafanana pamanode matsva. Kwete yakanaka mamiriro, kutaura zvishoma.
Kuratidzira kwedambudziko rekupuruzira uye kupindura
Iyo yakakosha metric yekutevera mudziyo ndeye trottling, inoratidza kuti kangani mudziyo wako wakatsikwa. Isu takaona nekufarira kuvepo kwekupuruzira mune mamwe midziyo, zvisinei nekuti processor mutoro wakanyanya here kana kuti kwete. Semuenzaniso, ngatitarisei imwe yedu huru APIs:
Sezvauri kuona pazasi, isu takaisa muganhu 800m (0.8 kana 80% musimboti), uye nhanho dzepamusoro pakusvikika 200m (20% musimboti). Zvingaite sekunge tisati tapeta sevhisi tichine simba rakawanda re processor, zvisinei ...
Iwe unogona kunge waona kuti kunyangwe iyo processor mutoro uri pazasi peiyo yakatarwa miganhu - zvakanyanya pazasi - throttling ichiri kuitika.
Takatarisana neizvi, takakurumidza kuwana zviwanikwa zvakawanda (, , ) nezve kudonha kwekuita uye nguva yekupindura yemasevhisi nekuda kwe throttling.
Nei tichiona throttling at low CPU load? Iyo pfupi vhezheni ndeiyi: "kune bug muLinux kernel iyo inokonzeresa kudhuzesa kwemidziyo ine yakatarwa processor miganhu." Kana iwe uchifarira mamiriro edambudziko, unogona kuverenga mharidzo ( ΠΈ sarudzo) naDave Chiluk.
Kubvisa zvirambidzo zveCPU (nekungwarira kwakanyanya)
Mushure mehurukuro refu, takasarudza kubvisa zvirambidzo zve processor kubva kumasevhisi ese anokanganisa zvakananga kana zvisina kunanga mashandiro akakosha kune vashandisi vedu.
Sarudzo yanga isiri nyore nekuti tinokoshesa zvakanyanya kugadzikana kweboka redu. Munguva yakapfuura, takatoedza nekusagadzikana kweboka redu, uye ipapo masevhisi akadya zviwanikwa zvakawanda uye akadzikisa basa renode yavo yese. Zvino zvinhu zvese zvaive zvakati siyanei: takave nekunzwisisa kwakajeka kwezvataitarisira kubva kumasumbu edu, pamwe nehurongwa hwakanaka hwekuita shanduko dzakarongwa.
Kunyorerana kwebhizinesi pane iri kunetsa.
Nzira yekudzivirira node dzako kana zvirambidzo zvabviswa?
Kuzviparadzanisa nevamwe kwe "zvisina kurambidzwa" masevhisi:
Munguva yakapfuura, takatoona dzimwe node dzichipinda muhurumende notReady, zvakanyanya nekuda kwemasevhisi akashandisa zviwanikwa zvakawanda.
Isu takasarudza kuisa masevhisi akadaro munzvimbo dzakasiyana ("dzakanyorwa") kuitira kuti dzirege kukanganisa masevhisi "ane hukama". Nekuda kweizvozvo, nekumaka dzimwe node uye nekuwedzera kushivirira parameter kune "asina hukama" masevhisi, takawana hukuru hwekutonga pamusoro pesumbu, uye zvakava nyore kwatiri kuziva matambudziko nemanodhi. Kuti uite maitiro akafanana iwe pachako, unogona kujairana nawo .
Kugovera chaiyo processor uye chikumbiro chekuyeuka:
Kutya kwedu kukuru kwaive kwekuti nzira yacho yaizopedza zviwanikwa zvakawanda uye node yaizomira kupindura zvikumbiro. Kubva zvino (nekuda kweDatadog) isu taigona kunyatso tarisa masevhisi ese ari pasumbu redu, ndakaongorora mwedzi yakati wandei yekushanda kweaya atakaronga kudoma se "asina hukama". Ini ndinongoseta iyo yakanyanya CPU kushandiswa nemuganho we20%, uye nekudaro ndakagovera nzvimbo mune node kana k8s ichiedza kugovera mamwe masevhisi kune node.
Sezvauri kuona mugirafu, iyo yakanyanya kuremerwa pane processor yasvika 242m CPU cores (0.242 processor cores). Pakukumbira processor, zvakakwana kutora nhamba yakakura zvishoma pane iyi kukosha. Ndokumbira utarise kuti sezvo masevhisi ari mushandisi-centric, huwandu hwepamusoro hunoenderana netraffic.
Ita zvimwe chete nekushandiswa kwendangariro uye mibvunzo, uye voila - mese makagadzirira! Kuti uchengetedze zvakanyanya, unogona kuwedzera yakachinjika pod autoscaling. Nekudaro, pese kana resource reload rakwira, autoscaling ichagadzira mapodhi matsva, uye kubernetes inoagovera kune node nenzvimbo yemahara. Kana pasina nzvimbo yasara musumbu pachayo, unogona kuzviseta yambiro kana kugadzirisa kuwedzera kwenodhi nyowani kuburikidza neavo autoscaling.
Pamaminuses, zvakakosha kuziva kuti takarasikirwa mu "", i.e. nhamba yemidziyo inomhanya pane imwe node. Isu tinogonawo kuva ne "relaxations" yakawanda pakaderera traffic density, uye pane zvakare mukana wekuti iwe uchasvika kune yakakwira processor mutoro, asi autoscaling node dzinofanira kubatsira neyekupedzisira.
Mhinduro
Ndiri kufara kuburitsa iyi mibairo yakanaka kubva mukuedza kwemavhiki mashoma apfuura; takatoona kuvandudzwa kwakakosha mumhinduro kumasevhisi ese akagadziridzwa:
Isu takawana zvakanakisa mhedzisiro pane yedu peji peji (), ipapo sevhisi yakakurumidza kupinda makumi maviri nembiri!
Iyo Linux kernel bug yakagadziriswa here?
Hongu kugovera shanduro 4.19 uye pamusoro.
Zvisinei, pakuverenga kwechipiri chaSeptember 2020 isu tichiri kusangana nekutaurwa kwemamwe mapurojekiti eLinux ane bug yakafanana. Ini ndinotenda kuti kumwe kugoverwa kweLinux kuchine iyi bug uye vari kushanda pakuigadzirisa.
Kana yako yekugovera vhezheni yakaderera pane 4.19, ini ndingakurudzira kuvandudza kune ichangoburwa, asi chero zvakadaro iwe unofanirwa kuedza kubvisa zvinorambidzwa processor uye ona kana iyo throttling ichienderera. Pazasi iwe unogona kuona chikamu chikamu cheKubernetes manejimendi masevhisi uye Linux kugovera:
- Debian: gadzirisa yakabatanidzwa mune yazvino vhezheni yekugovera, , uye inotaridzika chaizvo () Dzimwe shanduro dzakapfuura dzinogonawo kugadziriswa.
- Ubuntu: gadzirisa yakabatanidzwa mune yazvino vhezheni
- EKS ine kugadzirisa parizvino . Kana vhezheni yako yakadzikira pane iyi, unofanirwa kugadzirisa iyo AMI.
- kops: Ρ
kops 1.18+Mufananidzo mukuru uchave Ubuntu 20.04. Kana vhezheni yako yekops yakura, ungafanira kumirira kugadzirisa. Isu pachedu takamirira zvino. - GKE (Google Cloud): Gadzirisa yakabatanidzwa , zvisinei kune matambudziko ne throttling .
Chii chaunofanira kuita kana kugadzirisa kwakagadzirisa dambudziko rekudonha?
Handina chokwadi chekuti dambudziko rakagadziriswa zvachose. Kana tasvika kune kernel vhezheni negadziriso, ini ndichayedza sumbu uye kugadzirisa iyo positi. Kana paine vakatogadziridza, ndingafarire kuverenga mhinduro dzako.
mhedziso
- Kana iwe ukashanda neDocker midziyo pasi peLinux (zvisinei Kubernetes, Mesos, Swarm kana zvimwe), midziyo yako inogona kurasikirwa nekuita nekuda kwekupuruzira;
- Edza kugadziridza kune yazvino vhezheni yekugovera kwako uine tariro yekuti bug yakatogadziriswa;
- Kubvisa processor miganho kunogadzirisa dambudziko, asi iyi inzira ine njodzi inofanirwa kushandiswa nekuchenjerera kwakanyanya (zviri nani kutanga ugadzirise kernel uye kuenzanisa mhedzisiro);
- Kana iwe wabvisa miganhu yeCPU, nyatso tarisa CPU yako uye ndangariro mashandisiro uye ita shuwa kuti CPU yako zviwanikwa zviri kupfuura zvaunoshandisa;
- Sarudzo yakachengeteka ingave yeautoscale mapodhi kugadzira mapodhi matsva kana yakakwira hardware mutoro, kuitira kuti kubernetes inovapa kumahara node.
Ndinovimba iyi positi inokubatsira kuvandudza mashandiro emidziyo yako masisitimu.
PS munyori anowirirana nevaverengi nevatsinhanisi (muChirungu).
Source: www.habr.com