Phawula. transl.: Le mbali ivula amehlo ka-Omio-i-aggregator yokuhamba yaseYurophu-ithatha abafundi ukusuka kwithiyori esisiseko ukuya kwizinto ezinomdla ezintsonkothileyo zoqwalaselo lwe-Kubernetes. Ukuqhelana neemeko ezinjalo akuncedi nje ukwandisa i-horizons yakho, kodwa kuthintele iingxaki ezingenamsebenzi.
Ngaba ukhe wafumana isicelo sibambeke endaweni, uyeke ukuphendula kwiitshekhi zezempilo, kwaye awukwazi ukufumanisa ukuba kutheni? Enye ingcaciso enokwenzeka inxulumene nemida yesabelo sobutyebi be-CPU. Yile nto siza kuthetha ngayo kweli nqaku.
TL; DR:
Sicebisa ngamandla ukukhubaza imida ye-CPU kwi-Kubernetes (okanye ukhubaze ulwabiwo lwe-CFS kwi-Kubelet) ukuba usebenzisa uguqulelo lwe-Linux kernel nge CFS quota bug. Embindini nzulu kwaye ibug ekhokelela ekubetheni okugqithisileyo kunye nokulibaziseka.
Kwi-Omio lonke iziseko zophuhliso lulawulwa yiKubernetes. Yonke imithwalo yethu yelizwe kunye nengenammiselo isebenza ngokukodwa kwi-Kubernetes (sisebenzisa i-Google Kubernetes Engine). Kwezi nyanga zintandathu zidlulileyo, siqale ukujonga ukucotha ngokungacwangciswanga. Izicelo zikhenkceza okanye ziyeke ukuphendula kwiisheke zezempilo, zilahlekelwe unxibelelwano kwinethiwekhi, njl. Le ndlela yokuziphatha yasixaka ixesha elide, yaye ekugqibeleni sagqiba kwelokuba siyithathele ingqalelo le ngxaki.
Isishwankathelo senqaku:
- Amagama ambalwa malunga nezikhongozeli kunye neKubernetes;
- Izicelo zeCPU kunye nemida ziphunyezwa njani;
- Usebenza njani umda we-CPU kwiindawo ezininzi ezingundoqo;
- Indlela yokulandelela i-CPU throttling;
- Isisombululo sengxaki kunye nama-nuances.
Amagama ambalwa malunga nezikhongozeli kunye neKubernetes
I-Kubernetes ngoyena mgangatho wangoku kwihlabathi leziseko ezingundoqo. Owona msebenzi wayo yi-orchestration yesikhongozeli.
Izikhongozeli
Kwixesha elidlulileyo, kwafuneka senze izinto zakudala ezifana neJava JARs/WARs, Python Eggs, okanye eziphunyeziweyo ukuze ziqhube kwiiseva. Nangona kunjalo, ukwenza ukuba basebenze, kufuneka kwenziwe umsebenzi owongezelelweyo: ukufaka indawo yokusebenza (i-Java / Python), ukubeka iifayile eziyimfuneko kwiindawo ezifanelekileyo, ukuqinisekisa ukuhambelana nenguqu ethile yenkqubo yokusebenza, njl. Ngamanye amazwi, kufuneka kuqwalaselwe ngononophelo kulawulo lolungelelwaniso (oludla ngokuba ngumthombo wengxabano phakathi kwabaphuhlisi kunye nabalawuli benkqubo).
Iibhokisi zatshintsha yonke into. Ngoku i-artifact ngumfanekiso wesikhongozeli. Inokumelwa njengohlobo lwefayile eyandisiweyo ephunyeziweyo equlathe kungekuphela nje inkqubo, kodwa nemeko-bume yophumezo epheleleyo (iJava/Python/...), kunye neefayile eziyimfuneko/iipakethi, ezifakelwe kwangaphambili kwaye zilungele uku. baleka. Izikhongozeli zinokusasazwa kwaye ziqhube kwiiseva ezahlukeneyo ngaphandle kwamanyathelo ongezelelweyo.
Ukongeza, izitya zisebenza kwindawo yazo yesanti. Bane-adaptha yabo yomnatha enenyani, inkqubo yabo yefayile kunye nofikelelo olulinganiselweyo, ulawulo lwabo lweenkqubo, imida yabo kwi-CPU kunye nenkumbulo, njl.
Kubernetes
Njengoko bekutshiwo ngaphambili, uKubernetes yiorchestrator yesikhongozeli. Isebenza ngolu hlobo: uyinika ichibi loomatshini, kwaye emva koko uthi: "Heyi, Kubernetes, makhe siqalise imizekelo elishumi yesitya sam kunye neeprosesa ezi-2 kunye ne-3 GB yememori nganye, kwaye zigcine ziqhuba!" U-Kubernetes uya kukhathalela abanye. Iya kufumana umthamo wasimahla, iqalise izikhongozeli kwaye uziqalise kwakhona ukuba kuyimfuneko, khupha uhlaziyo xa utshintsha iinguqulelo, njl. Ngokusisiseko, i-Kubernetes ikuvumela ukuba ukhuphe icandelo le-hardware kwaye wenze iintlobo ngeentlobo zeenkqubo ezilungele ukuthunyelwa kunye nokuqhuba usetyenziso.
Kubernetes ukusuka kwindawo yokujonga i-layman
Ziziphi izicelo kunye nemida eKubernetes
Kulungile, sizigqumathele izitya kunye neKubernetes. Siyazi kwakhona ukuba izikhongozeli ezininzi zinokuhlala kumatshini omnye.
Isifaniso sinokuzotywa kunye nendlu yoluntu. Indawo ephangaleleyo (oomatshini/iiyunithi) iyathathwa kwaye iqeshiswe kubaqeshi abaliqela (iikhonteyina). U-Kubernetes usebenza njengomthengisi. Umbuzo uvela, indlela yokugcina abaqeshi kwiingxabano kunye nomnye? Kuthekani ukuba omnye wabo, uthi, unquma ukuboleka indlu yokuhlambela isiqingatha sosuku?
Kulapho izicelo kunye nemida zingena khona. ICPU isicelo efunekayo kuphela ngeenjongo zokucwangcisa. Le yinto efana "noluhlu lweminqweno" yesikhongozeli, kwaye isetyenziselwa ukukhetha eyona node ifanelekileyo. Ngexesha elifanayo CPU umda inokuthelekiswa nesivumelwano sokurenta - nje ukuba sikhethe iyunithi yesikhongozeli, i andinako dlulela ngaphaya kwemida ebekiweyo. Kwaye kulapho ingxaki ivela khona...
Izicelo kunye nemida ziphunyezwa njani kwi-Kubernetes
I-Kubernetes isebenzisa i-throttling mechanism (ukutsiba imijikelo yewotshi) eyakhelwe kwi-kernel ukuphumeza imida ye-CPU. Ukuba isicelo sigqithise kumda, i-throttling yenziwe (i.e. ifumana imijikelo ye-CPU embalwa). Izicelo kunye nemida yememori ihlelwe ngokwahlukileyo, ngoko kulula ukuyibona. Ukwenza oku, khangela nje isimo sokugqibela sokuqalisa kwakhona kwepod: nokuba yi "OOMKilled". I-CPU throttling ayilula kangako, kuba ii-K8s zenza kuphela iimetrics zifumaneke ngokusetyenziswa, hayi ngamaqela.
Isicelo seCPU
Siphunyezwa njani isicelo seCPU
Ukuze kube lula, makhe sijonge inkqubo usebenzisa umatshini one-4-core CPU njengomzekelo.
I-K8s isebenzisa indlela yeqela lolawulo (amaqela) ukulawula ulwabiwo lwezibonelelo (inkumbulo kunye neprosesa). Imodeli ye-hierarchical iyafumaneka kuyo: umntwana uzuza njengelifa imida yeqela labazali. Iinkcukacha zonikezelo zigcinwe kwindlela yefayile enenyani (/sys/fs/cgroup). Kwimeko yeprosesa oku /sys/fs/cgroup/cpu,cpuacct/*.
I-K8s isebenzisa ifayile cpu.share ukwaba izibonelelo zeprosesa. Kwimeko yethu, i-root cgroup ifumana izabelo ze-4096 zezixhobo ze-CPU - i-100% yamandla akhoyo akhoyo (i-1 core = 1024; eli lixabiso elimiselweyo). Iqela leengcambu lisasaza izixhobo ngokulinganayo ngokuxhomekeke kwizabelo zenzala ebhaliswe kuyo cpu.share, kwaye nabo benza okufanayo kwinzala yabo, njl. Kwi-node ye-Kubernetes eqhelekileyo, iqela leengcambu linabantwana abathathu: system.slice, user.slice ΠΈ kubepods. Amaqela amancinci amabini okuqala asetyenziselwa ukusasaza izibonelelo phakathi komthwalo wenkqubo ebalulekileyo kunye neenkqubo zabasebenzisi ngaphandle kwee-K8s. Eyokugqibela - kubepods - idalwe nguKubernetes ukuhambisa izixhobo phakathi kweepod.
Lo mzobo ungasentla ubonisa ukuba iqela lokuqala nelesibini lafumana elinye 1024 izabelo, kunye neqela elisezantsi le-kuberpod elabiweyo 4096 izabelo Kwenzeka njani oku: emva kwayo yonke into, iqela leengcambu linokufikelela kuphela 4096 izabelo, kwaye nenani lezabelo zenzala yakhe lingaphezulu ngokuphawulekayo kweli nani (6144)? Ingongoma kukuba ixabiso lenza ingqiqo, ngoko ke umcwangcisi weLinux (CFS) uyisebenzisela ukwaba ngokulinganayo izixhobo ze-CPU. Kwimeko yethu, amaqela amabini okuqala afumana 680 izabelo real (16,6% of 4096), kunye kubepod ufumana eseleyo 2736 izabelo Kwimeko yokunciphisa ixesha, amaqela amabini okuqala akayi kusebenzisa izibonelelo ezinikezelweyo.
Ngethamsanqa, umcwangcisi unendlela yokuphepha ukuchitha izibonelelo ze-CPU ezingasetyenziswanga. Idlulisela umthamo "ongasebenziyo" kwichibi lehlabathi, apho isasazwa khona kumaqela adinga amandla ongezelelweyo okuqhuba (ukudluliselwa kwenzeka kwiibhetshi ukuphepha ilahleko ejikelezayo). Kusetyenziswa indlela efanayo kuyo yonke inzala yenzala.
Le ndlela iqinisekisa ukusasazwa okufanelekileyo kwamandla eprosesa kwaye iqinisekisa ukuba akukho mntu "uba" izixhobo kwabanye.
Umda weCPU
Ngaphandle kwento yokuba uqwalaselo lwemida kunye nezicelo kwii-K8s zijongeka ngokufanayo, ukuphunyezwa kwazo kwahluke kakhulu: oku eyona nto ilahlekisayo kunye neyona nxalenye ibhalwe kancinci.
I-K8s iyazibandakanya ukuphumeza imida. Izicwangciso zabo zichazwe kwiifayile cfs_period_us ΠΈ cfs_quota_us kulawulo lweqela (ifayile ikwakhona cpu.share).
Ngokungafaniyo cpu.share, umlinganiselo usekwe kwi ixesha lelixa, kwaye hayi kumandla weprosesa akhoyo. cfs_period_us ichaza ubude bexesha (i-epoch) - ihlala i-100000 ΞΌs (100 ms). Kukho ukhetho lokutshintsha eli xabiso kwii-K8s, kodwa lifumaneka kuphela kwi-alpha okwangoku. Umcwangcisi usebenzisa ixesha ukuqalisa kwakhona umlinganiselo osetyenzisiweyo. Ifayile yesibini cfs_quota_us, ichaza ixesha elikhoyo (umlinganiselo) kwixesha ngalinye. Qaphela ukuba ikwacacisiwe kwii-microseconds. Umlinganiselo ungagqitha ubude bexesha; ngamanye amazwi, inokuba nkulu kuno 100 ms.
Makhe sijonge iimeko ezimbini koomatshini be-16-core (olona hlobo luqhelekileyo lwekhompyuter esinalo e-Omio):
Imeko 1: 2 imisonto kunye nomda we-200 ms. Akukho ukutswina
Imeko 2: 10 imisonto kunye 200 ms umda. I-Throttling iqala emva kwe-20 ms, ukufikelela kwimithombo yeprosesa kuphinda kuqaliswe emva kwenye i-80 ms.
Masithi usete umda we-CPU kuyo 2 iinkozo; I-Kubernetes iya kuguqulela eli xabiso kwi-200 ms. Oku kuthetha ukuba i-container ingasebenzisa ubuninzi be-200ms yexesha le-CPU ngaphandle kokugubha.
Kwaye kulapho ulonwabo luqala khona. Njengoko kukhankanyiwe ngasentla, i-quota ekhoyo yi-200 ms. Ukuba usebenza ngokuhambelanayo lishumi imisonto kumatshini we-12-core (jonga umzekeliso we-scenario 2), ngelixa zonke ezinye ii-pods zingasebenzi, i-quota iya kugqitywa nge-20 ms nje (ukusukela 10 * 20 ms = 200 ms), kwaye yonke imisonto yale pod iya kuxhoma. Β» (umqala) kwi-80 ms elandelayo. Le sele ikhankanyiwe , ngenxa yokwenzeka kwe-throttling egqithisileyo kwaye umgqomo awunako nokuzalisekisa isabelo esikhoyo.
Indlela yokuvavanya i-throttling kwiipods?
Ngena nje kwi-pod kwaye wenze cat /sys/fs/cgroup/cpu/cpu.stat.
-
nr_periods- inani lilonke lamaxesha abacwangcisi; -
nr_throttled- inani lexesha le-throttled ekubunjweninr_periods; -
throttled_time-Ixesha elongezelelekileyo le-throttled kwii-nanoseconds.
Kwenzeka ntoni kanye kanye?
Ngenxa yoko, sifumana i-throttling ephezulu kuzo zonke izicelo. Ngamanye amaxesha uyangena ixesha elinye elinesiqingatha yomelele kunokuba ibalwe!
Oku kukhokelela kwiimpazamo ezahlukeneyo - ukusilela ukujonga ukulungela, ukukhenkceza komgqomo, ukuqhawuka koqhagamshelo lwenethiwekhi, ukuphelelwa lixesha ngaphakathi kweefowuni zenkonzo. Oku ekugqibeleni kubangela ukubambezeleka okwandisiweyo kunye namazinga aphezulu eempazamo.
Isigqibo kunye neziphumo
Yonke into ilula apha. Siyishiyile imida ye-CPU kwaye saqalisa ukuhlaziya i-OS kernel ngamaqela kuguqulelo lwamva nje, apho ibug yalungiswa. Inani leempazamo (HTTP 5xx) kwiinkonzo zethu zehle ngokukhawuleza kakhulu:
HTTP 5xx iimpazamo
Iimpazamo ze-HTTP 5xx kwinkonzo enye ebalulekileyo
Ixesha lokuphendula p95
Isicelo senkonzo ebalulekileyo ye-latency, i-95th percentile
Iindleko zokusebenza
Inani leeyure zomzekelo ezichithiweyo
Yintoni ukubamba?
Njengoko kuchaziwe ekuqaleni kwenqaku:
Isifaniso sinokuzotywa kunye nendlu yoluntu ... I-Kubernetes isebenza njenge-realtor. Kodwa njani ukugcina abaqeshi kwiingxabano kunye nomnye? Kuthekani ukuba omnye wabo, uthi, unquma ukuboleka indlu yokuhlambela isiqingatha sosuku?
Nantsi into yokubambisa. Esinye isikhongozeli esingakhathaliyo sinokutya zonke izixhobo ezikhoyo ze-CPU kumatshini. Ukuba une-smart application stack (umzekelo, i-JVM, Hamba, i-Node VM iqwalaselwe ngokufanelekileyo), ngoko oku akuyona ingxaki: unokusebenza kwiimeko ezinjalo ixesha elide. Kodwa ukuba izicelo azilungiswanga kakuhle okanye azilungiswanga kwaphela (FROM java:latest), imeko inokuphuma ekulawuleni. E-Omio sinesiseko esizenzekelayo se-Dockerfiles ezinemimiselo eyaneleyo engagqibekanga yesipakisho esikhulu solwimi, ngoko lomba wawungekho.
Sicebisa ukuba ujonge iimetriki (ukusetyenziswa, ukugcwalisa kunye neempazamo), ukulibaziseka kwe-API kunye namazinga eempazamo. Qinisekisa ukuba iziphumo ziyahlangabezana nokulindelweyo.
iimbekiselo
Eli libali lethu. Ezi zixhobo zilandelayo zincede kakhulu ukuqonda okwenzekayo:
- ;
- ;
- ;
- ;
- - khangela "i-cpu throttling".
Iingxelo ze-Kubernetes bug:
- ;
- ;
- .
Ngaba ukhe wadibana neengxaki ezifanayo kwindlela yakho yokusebenza okanye unamava anxulumene nokuthontelana kwindawo yemveliso enezikhongozeli? Yabelana ngebali lakho kwizimvo!
PS evela kumguquleli
Funda nakwibhlog yethu:
- Β«";
- Β«";
- «».
umthombo: www.habr.com