Imikhawulo ye-CPU kanye nokudonsa kanzima ku-Kubernetes

Qaphela. transl.: Lo mlando ovula amehlo we-Omio—i-European travel aggregator—ususa abafundi kuthiyori eyisisekelo ubase ebunkingeni obumangalisayo obusebenzayo bokucushwa kwe-Kubernetes. Ukujwayelana nezimo ezinjalo akusizi nje ukwandisa ama-horizons akho, kodwa futhi kuvimbele izinkinga ezingezona ezincane.

Uke waba nesicelo sokubhajwa endaweni, uyeke ukuphendula ekuhlolweni kwezempilo, futhi ungakwazi ukuthola ukuthi kungani? Incazelo eyodwa engaba khona ihlobene nemikhawulo yesabelo sensiza ye-CPU. Yilokhu esizokhuluma ngakho kulesi sihloko.

TL; DR:
Sincoma ngokuqinile ukuthi ukhubaze imikhawulo ye-CPU ku-Kubernetes (noma ukhubaze ama-quota e-CFS ku-Kubelet) uma usebenzisa inguqulo ye-Linux kernel enesiphazamisi se-quota ye-CFS. Emongweni iyatholakala sina futhi owaziwayo isiphazamisi esiholela ekushayeni ngamandla nasekubambezelekeni.

E-Omio yonke ingqalasizinda iphethwe yiKubernetes. Yonke imithwalo yethu yokusebenza esezingeni eliphezulu nengenasisekelo isebenza ku-Kubernetes kuphela (sisebenzisa i-Google Kubernetes Engine). Ezinyangeni eziyisithupha ezedlule, siqale ukubona ukwehla okungahleliwe. Izinhlelo zokusebenza ziyamiswa noma ziyeke ukuphendula ekuhlolweni kwezempilo, zilahlekelwe ukuxhumeka kunethiwekhi, njll. Lokhu kuziphatha kwasixaka isikhathi eside, futhi ekugcineni sanquma ukuthatha le nkinga ngokungathi sína.

Isifinyezo sendatshana:

• Amagama ambalwa mayelana neziqukathi kanye ne-Kubernetes;
• Izicelo nemikhawulo ye-CPU zisetshenziswa kanjani;
• Indlela umkhawulo we-CPU osebenza ngayo ezindaweni ezinamanzi amaningi;
• Indlela yokulandelela i-CPU throttling;
• Isixazululo senkinga nama-nuances.

Amagama ambalwa mayelana neziqukathi kanye ne-Kubernetes

I-Kubernetes empeleni iyindinganiso yesimanje emhlabeni wengqalasizinda. Umsebenzi wayo oyinhloko ukucula kweziqukathi.

Izitsha

Esikhathini esedlule, bekufanele sakhe izinto zobuciko ezifana neJava JAR/WARs, Python Eggs, noma ezisebenzisekayo ukuze zisebenze kumaseva. Nokho, ukuze zisebenze, kwakudingeka kwenziwe umsebenzi owengeziwe: ukufaka imvelo yesikhathi sokusebenza (Java/Python), ukubeka amafayela adingekayo ezindaweni ezifanele, ukuqinisekisa ukuhambisana nenguqulo ethile yesistimu yokusebenza, njll. Ngamanye amazwi, ukunakwa ngokucophelela bekufanele kukhokhwe ekuphathweni kokucushwa (okwakuvame ukuba umthombo wengxabano phakathi konjiniyela nabaphathi bohlelo).

Iziqukathi zashintsha yonke into. Manje i-artifact iyisithombe sesitsha. Ingamelwa njengohlobo lwefayela elisebenzisekayo elinwetshiwe eliqukethe hhayi uhlelo kuphela, kodwa futhi nemvelo yokusebenza ephelele (i-Java/Python/...), kanye namafayela/amaphakheji adingekayo, afakwe ngaphambili futhi alungele ukuwenza. gijima. Iziqukathi zingasatshalaliswa futhi zisebenze kumaseva ahlukene ngaphandle kwanoma yiziphi izinyathelo ezengeziwe.

Ngaphezu kwalokho, iziqukathi zisebenza endaweni yazo ye-sandbox. Bane-adaptha yabo yenethiwekhi ebonakalayo, isistimu yabo yefayela enokufinyelela okulinganiselwe, isigaba sabo sezinqubo, ukulinganiselwa kwabo ku-CPU nenkumbulo, njll. Konke lokhu kwenziwa ngenxa yesistimu engaphansi ekhethekile ye-Linux kernel - izikhala zamagama.

Kubernetes

Njengoba kushiwo ngaphambili, uKubernetes uyi-orchestrator yeziqukathi. Isebenza kanje: uyinikeza inqwaba yemishini, bese uthi: "Hey, Kubernetes, ake sethule izimo eziyishumi zesitsha sami esinamaphrosesa angu-2 kanye nememori engu-3 GB ngayinye, futhi sizigcine zisebenza!" UKubernetes uzonakekela okunye. Izothola umthamo wamahhala, iqalise iziqukathi futhi iqale kabusha uma kunesidingo, ikhiphe izibuyekezo lapho ishintsha izinguqulo, njll. Empeleni, i-Kubernetes ikuvumela ukuthi ukhiphe ingxenye yezingxenyekazi zekhompuyutha futhi wenze inhlobonhlobo yezinhlelo ezilungele ukuthunyelwa nokusebenzisa izinhlelo zokusebenza.

Kubernetes ngokombono we-layman

Yiziphi izicelo nemikhawulo ku-Kubernetes

Kulungile, simboze iziqukathi kanye ne-Kubernetes. Siyazi futhi ukuthi iziqukathi eziningi zingahlala emshinini owodwa.

Isifaniso singadwetshwa ngefulethi elihlanganyelwe. Indawo ebanzi (imishini/amayunithi) iyathathwa futhi iqashelwe abaqashi abambalwa (izitsha). U-Kubernetes usebenza njengomthengisi. Umbuzo uphakama, kanjani ukugcina abaqashi ezingxabanweni nomunye? Kuthiwani uma omunye wabo, ethi, enquma ukuboleka indlu yokugezela isigamu sosuku?

Yilapho izicelo nemikhawulo ziqala khona ukusebenza. CPU Isicelo edingekayo ngezinhloso zokuhlela kuphela. Lena into efana "nohlu lwezifiso" lwesitsha, futhi isetshenziselwa ukukhetha indawo efanelekile kakhulu. Ngesikhathi esifanayo CPU Limit singaqhathaniswa nesivumelwano sokuqasha - ngokushesha nje lapho sikhetha iyunithi yesitsha, i ngeke dlulela ngale kwemikhawulo ebekiwe. Futhi yilapho inkinga iphakama khona...

Izicelo nemikhawulo zisetshenziswa kanjani ku-Kubernetes

I-Kubernetes isebenzisa indlela yokudonsa (imijikelezo yewashi) eyakhelwe ku-kernel ukuze isebenzise imikhawulo ye-CPU. Uma uhlelo lokusebenza ludlula umkhawulo, i-throttling inikwe amandla (okungukuthi, ithola imijikelezo ye-CPU embalwa). Izicelo nemikhawulo yenkumbulo ihlelwa ngendlela ehlukile, ngakho-ke kulula ukuyibona. Ukuze wenze lokhu, vele uhlole isimo sokuqalisa kabusha sokugcina se-pod: noma ngabe "I-OOMKilled". I-CPU throttling akulula kangako, njengoba i-K8s yenza amamethrikhi atholakale ngokusetshenziswa, hhayi ngamaqoqo.

Isicelo se-CPU

Isicelo se-CPU sisetshenziswa kanjani

Ukwenza kube lula, ake sibheke inqubo yokusebenzisa umshini one-CPU engu-4 njengesibonelo.

I-K8s isebenzisa indlela yeqembu lokulawula (amaqoqo) ukuze ilawule ukwabiwa kwezinsiza (inkumbulo nephrosesa). Imodeli ye-hierarchical iyatholakala kuyo: ingane izuza njengefa imikhawulo yeqembu labazali. Imininingwane yokusabalalisa igcinwa kusistimu yefayela ebonakalayo (/sys/fs/cgroup). Endabeni yokucubungula lokhu /sys/fs/cgroup/cpu,cpuacct/*.

I-K8s isebenzisa ifayela cpu.share ukwaba izinsiza zokuphrosesa. Esimweni sethu, i-root cgroup ithola amasheya angu-4096 wezinsiza ze-CPU - 100% wamandla okucubungula atholakalayo (1 core = 1024; leli inani elinqunyiwe). Iqembu lempande lisabalalisa izinsiza ngokulinganayo kuye ngamasheya enzalo abhaliswe kuwo cpu.share, futhi nabo benza okufanayo ngenzalo yabo, njll. Ku-node evamile ye-Kubernetes, iqoqo lempande linezingane ezintathu: system.slice, user.slice и kubepods. Amaqembu amancane okuqala amabili asetshenziselwa ukusabalalisa izinsiza phakathi kwemithwalo yesistimu ebalulekile nezinhlelo zabasebenzisi ngaphandle kwama-K8. Owokugcina - kubepods - idalwe nguKubernetes ukusabalalisa izinsiza phakathi kwama-pods.

Umdwebo ongenhla ubonisa ukuthi iqoqo elincane lokuqala nelesibili bathole ngalinye 1024 amasheya, neqembu elincane le-kuberpod elinikeziwe 4096 amasheya Kungenzeka kanjani lokhu: ngemuva kwakho konke, iqembu lempande linokufinyelela kuphela 4096 amasheya, futhi isamba samasheya enzalo yakhe siyidlula ngokuphawulekayo lesi sibalo (6144)? Iphuzu liwukuthi inani lenza umqondo onengqondo, ngakho umhleli we-Linux (CFS) uyisebenzisela ukwaba izinsiza ze-CPU ngokulinganayo. Esimweni sethu, amaqembu amabili okuqala athola 680 amasheya angempela (16,6% ka-4096), kanye ne-kubepod ithola okusele 2736 amasheya Esimeni sokungasebenzi, amaqembu amabili okuqala ngeke asebenzise izinsiza ezabiwe.

Ngenhlanhla, umhleli unendlela yokugwema ukumosha izinsiza ze-CPU ezingasetshenzisiwe. Idlulisela umthamo "ongasebenzi" echibini lomhlaba wonke, lapho isakazwa khona emaqenjini adinga amandla ephrosesa engeziwe (ukudlulisa kwenzeka ngamaqoqo ukugwema ukulahlekelwa okuzungezayo). Indlela efanayo isetshenziswa kuyo yonke inzalo yenzalo.

Le nqubo iqinisekisa ukusatshalaliswa okufanele kwamandla okucubungula futhi iqinisekisa ukuthi akekho ocubungulayo "ontshontsha" izinsiza kwabanye.

Umkhawulo we-CPU

Ngaphandle kweqiniso lokuthi ukumiswa kwemikhawulo nezicelo kuma-K8s kubukeka kufana, ukuqaliswa kwazo kuhluke kakhulu: lokhu edukisa kakhulu kanye nengxenye encane ebhalwe phansi.

I-K8s iyahlanganyela Indlela ye-quota ye-CFS ukusebenzisa imikhawulo. Izilungiselelo zabo zicaciswe kumafayela cfs_period_us и cfs_quota_us kumkhombandlela weqembu (ifayela nalo likhona cpu.share).

Ngokungafani cpu.share, isabelo sisekelwe ku isikhathi, futhi hhayi emandleni okucubungula atholakalayo. cfs_period_us icacisa ubude besikhathi (i-epoch) - ihlale ingu-100000 μs (100 ms). Kukhona inketho yokushintsha leli nani kuma-K8s, kodwa litholakala kuphela ku-alpha okwamanje. Umhleli usebenzisa i-epoch ukuze aqale kabusha ama-quota asetshenzisiwe. Ifayela lesibili cfs_quota_us, icacisa isikhathi esitholakalayo (i-quota) enkathini ngayinye. Qaphela ukuthi iphinde icaciswe kuma-microsecond. Isabelo singase seqe ubude benkathi; ngamanye amazwi, ingaba nkulu kuno-100 ms.

Ake sibheke izimo ezimbili emishinini engu-16-core (uhlobo lwekhompyutha oluvame kakhulu esinalo kwa-Omio):

Isimo 1: imicu emi-2 kanye nomkhawulo ongu-200 ms. Ayikho i-throttling

Isimo 2: Imicu engu-10 nomkhawulo ongu-200 ms. I-Throttling iqala ngemva kuka-20 ms, ukufinyelela kuzinsiza zokucubungula kuqaliswa kabusha ngemva kokunye okungu-80 ms.

Ake sithi usethe umkhawulo we-CPU kuwo 2 izinhlamvu; I-Kubernetes izohumusha leli nani libe ngu-200 ms. Lokhu kusho ukuthi isiqukathi singasebenzisa umkhawulo wesikhathi esingu-200ms we-CPU ngaphandle kokuphusha.

Futhi yilapho ubumnandi buqala khona. Njengoba kushiwo ngenhla, isilinganiso esitholakalayo singama-200 ms. Uma usebenza ngokuhambisana ishumi imicu emshinini we-12-core (bheka umfanekiso wesimo sesi-2), ngenkathi zonke ezinye izidumba zingenzi lutho, i-quota izophela ngo-20 ms nje (kusukela 10 * 20 ms = 200 ms), futhi yonke imicu yale pod izolenga. » (mpintsha) ku-80 ms elandelayo. Osekukhulunyiwe isihleli bug, ngenxa yokuthi kwenzeka ukuphihliza okweqile futhi isiqukathi asikwazi nokugcwalisa isabelo esikhona.

Ungayihlola kanjani i-throttling kuma-pods?

Vele ungene ngemvume ku-pod futhi wenze cat /sys/fs/cgroup/cpu/cpu.stat.

• nr_periods - inani eliphelele lezikhathi zomhleli;
• nr_throttled - inani lezikhathi ezimpintshayo ekwakhiweni nr_periods;
• throttled_time - Isikhathi esihlanganisiwe sokudonsa kuma-nanoseconds.

Kwenzekani ngempela?

Ngenxa yalokho, sithola ukudonsa okuphezulu kuzo zonke izinhlelo zokusebenza. Kwesinye isikhathi uyangena isikhathi esisodwa nengxenye kunamandla kunokubalwa!

Lokhu kuholela emaphutheni ahlukahlukene - ukwehluleka ukuhlolwa kokulungela, ukumiswa kweziqukathi, ukunqamuka kokuxhumeka kwenethiwekhi, ukuphela kwesikhathi phakathi kwezingcingo zesevisi. Lokhu ekugcineni kuphumela ekwenyukeni kokubambezeleka kanye namazinga aphezulu amaphutha.

Isinqumo nemiphumela

Konke kulula lapha. Silahle imikhawulo ye-CPU futhi saqala ukubuyekeza i-OS kernel ngamaqoqo ukuze ibe inguqulo yakamuva, lapho iphutha lalungiswa khona. Inani lamaphutha (HTTP 5xx) kumasevisi ethu lehle ngokushesha kakhulu:

Amaphutha e-HTTP 5xx

Amaphutha e-HTTP 5xx esevisi eyodwa ebalulekile

Isikhathi sokuphendula p95

Ukubambezeleka kwesicelo sesevisi ebalulekile, i-95th percentile

Izindleko zokusebenza

Inani lamahora esibonelo asetshenzisiwe

Yini ukubanjwa?

Njengoba kushiwo ekuqaleni kwesihloko:

Isifaniso singadwetshwa ngendlu yomphakathi... U-Kubernetes usebenza njengomthengisi. Kodwa kanjani ukugcina abaqashi kusukela izingxabano nomunye? Kuthiwani uma omunye wabo, ethi, enquma ukuboleka indlu yokugezela isigamu sosuku?

Nakhu okubanjwayo. Isiqukathi esisodwa esinganaki singadla zonke izinsiza ezitholakalayo ze-CPU emshinini. Uma une-smart application stack (isibonelo, i-JVM, i-Go, i-Node VM ilungiselelwe kahle), khona-ke lokhu akuyona inkinga: ungasebenza ezimweni ezinjalo isikhathi eside. Kodwa uma izinhlelo zokusebenza zingalungiselelwe kahle noma zingalungiselelwe nhlobo (FROM java:latest), isimo singase singalawuleki. E-Omio sinesisekelo esizenzakalelayo se-Dockerfiles anezilungiselelo ezanele zokuzenzakalelayo zesitaki solimi esikhulu, ngakho le nkinga ibingekho.

Sincoma ukuthi ugade amamethrikhi UKUSETSHENZISWA (ukusetshenziswa, ukugcwaliswa kwesikhala namaphutha), ukubambezeleka kwe-API namazinga amaphutha. Qinisekisa ukuthi imiphumela ihlangabezana nokulindelwe.

izithenjwa

Lena indaba yethu. Izinto ezilandelayo zisize kakhulu ukuqonda okwakwenzeka:

• kernel.org → Isihleli Se-CFS;
• kernel.org → Ukulawula Umkhawulokudonsa we-CFS;
• Ukuqonda Ukuhlela Isiqukathi se-Linux;
• Konke Odinga Ukukwazi Ngeziqukathi Ze-Linux, Ingxenye I: Amaqembu Okulawula I-Linux Nokuhlukaniswa Kwenqubo;
• I-Kubernetes Failure Stories - bheka "i-cpu throttling".

I-Kubernetes bug ibika:

• #51135: Gwema ukusetha imikhawulo ye-CPU yama-pod Aqinisekisiwe;
• #67577: Izilinganiso ze-CFS zingaholela ekushayeni ngokungadingekile;
• I-CFS enolaka ngokweqile.

Ingabe uke wahlangabezana nezinkinga ezifanayo ekusebenzeni kwakho noma unesipiliyoni esihlobene nokudonsa kanzima ezindaweni zokukhiqiza ezifakwe iziqukathi? Yabelana ngendaba yakho kumazwana!

I-PS evela kumhumushi

Funda futhi kubhulogi yethu:

• «Ukukala okuzenzakalelayo nokuphathwa kwezinsiza ku-Kubernetes (uhlolojikelele nombiko wevidiyo)";
• «Isebenza kanjani i-CPU Manager ku-Kubernetes";
• «Kwenzekani ku-Kubernetes uma usebenzisa i-kubectl run? Ingxenye 2".

Source: www.habr.com