Emuva ngo-2016 siseBuffer , futhi manje cishe ama-node angu-60 (ku-AWS) neziqukathi ezingu-1500 zisebenza kuqoqo lethu le-k8s eliphethwe . Nokho, sithuthele kuma-microservices ngokuzama nangephutha, futhi ngisho nangemva kweminyaka eminingana yokusebenza nama-k8 sisabhekene nezinkinga ezintsha. Kule post sizokhuluma ngayo ukulinganiselwa kweprosesa: kungani sicabange ukuthi bazijwayeze kahle nokuthi kungani bagcina bengemuhle kangako.
Imikhawulo yeprosesa kanye ne-throttling
Njengabanye abasebenzisi abaningi be-Kubernetes, . Ngaphandle kokusetha okunjalo, iziqukathi ku-node zingathatha wonke amandla okucubungula, okubuye kubangele izinqubo ezibalulekile ze-Kubernetes (ngokwesibonelo kubelet) izoyeka ukuphendula izicelo. Ngakho, ukusetha imikhawulo ye-CPU kuyindlela enhle yokuvikela ama-node akho.
Imikhawulo yephrosesa isetha isiqukathi kusikhathi esiphezulu se-CPU esingasisebenzisa isikhathi esithile (okuzenzakalelayo ngu-100ms), futhi isiqukathi asisoze seqe lo mkhawulo. In Kubernetes for throttling isitsha futhi uyivimbele ukuba idlule umkhawulo, kusetshenziswa ithuluzi elikhethekile , kodwa le mikhawulo ye-CPU yokwenziwa igcina ilimaze ukusebenza futhi yandise isikhathi sokuphendula seziqukathi zakho.
Kungenzekani uma singabeki imikhawulo yephrosesa?
Ngeshwa, thina ngokwethu kwadingeka sibhekane nale nkinga. Indawo ngayinye inenqubo enesibopho sokuphatha iziqukathi kubelet, futhi wayeka ukuphendula izicelo. I-node, lapho lokhu kwenzeka, izongena kuhulumeni NotReady, neziqukathi ezivela kuyo zizoqondiswa kabusha kwenye indawo futhi zidale izinkinga ezifanayo kumanodi amasha. Akusona isimo esikahle, ukusho okungenani.
Ukubonakaliswa kwenkinga yokudonsa nokuphendula
Imethrikhi eyinhloko yokulandelela isiqukathi ithi trottling, ikhombisa ukuthi isiqukathi sakho siphunyuzwe kangaki. Siqaphele ngentshisekelo ubukhona bokunyakaziswa kwezinye iziqukathi, kungakhathaliseki ukuthi umthwalo weprosesa weqile noma cha. Njengesibonelo, ake sibheke enye yama-API ethu amakhulu:
Njengoba ubona ngezansi, sibeke umkhawulo 800m (0.8 noma 80% core), kanye namanani aphezulu lapho kufinyelelwa khona kangcono 200m (20% core). Kubukeka sengathi ngaphambi kokuphonsela insizakalo sisenamandla amaningi okucubungula, nokho...
Kungenzeka ukuthi uqaphele ukuthi noma ngabe umthwalo weprosesa ungaphansi kwemikhawulo ebekiwe - kakhulu ngezansi - i-throttling isenzeka.
Sibhekene nalokhu, ngokushesha sathola izinsiza ezimbalwa (, , ) mayelana nokwehla kokusebenza nesikhathi sokuphendula kwezinsizakalo ngenxa yokudonsa kanzima.
Kungani sibona i-throttling ngomthwalo ophansi we-CPU? Inguqulo emfushane ithi: "kunesiphazamisi ku-Linux kernel ebangela ukunyakaziswa okungadingekile kweziqukathi ezinemikhawulo ecacisiwe yephrosesa." Uma unentshisekelo ngohlobo lwenkinga, ungafunda isethulo ( и ongakhetha) nguDave Chiluk.
Ukususa imikhawulo ye-CPU (ngokuqapha okukhulu)
Ngemuva kwezingxoxo ezinde, sinqume ukususa imikhawulo yephrosesa kuzo zonke izinsiza ezithinte ngokuqondile noma ngokungaqondile ukusebenza okubalulekile kubasebenzisi bethu.
Isinqumo besingelula ngoba sikwazisa kakhulu ukuzinza kweqoqo lethu. Esikhathini esedlule, sesivele sazama ukungazinzi kweqoqo lethu, futhi izinsiza zasebenzisa izinsiza eziningi futhi zehlisa umsebenzi wawo wonke ama-node. Manje konke kwase kuhluke ngandlela thize: sase sikuqonda ngokucacile lokho esasikulindele kumaqoqo ethu, kanye nesu elihle lokuqalisa izinguquko ezihleliwe.
Izincwadi zebhizinisi ngodaba olucindezelayo.
Ungawavikela kanjani ama-node akho lapho imikhawulo isusiwe?
Ukuhlukaniswa kwezinsizakalo "ezingenamkhawulo":
Esikhathini esedlule, sesivele sabona amanye ama-node eba sesimeni notReady, ngokuyinhloko ngenxa yamasevisi asebenzise izinsiza eziningi kakhulu.
Sinqume ukubeka lezi zinsizakalo ezindaweni ezihlukene (“ezinelebula”) ukuze zingaphazamisi izinsiza “ezihlobene”. Ngenxa yalokho, ngokumaka amanye ama-node nokwengeza ipharamitha yokubekezelelana ezinsizeni "ezingahlobene", sizuze ukulawula okukhulu phezu kweqoqo, futhi kwaba lula ngathi ukuhlonza izinkinga ngama-node. Ukuze wenze izinqubo ezifanayo ngokwakho, ungakwazi ukuzijwayeza .
Ukwabela iphrosesa elungile nesicelo sememori:
Ukwesaba kwethu okukhulu kwakuwukuthi inqubo izodla izinsiza eziningi kakhulu futhi i-node izoyeka ukuphendula izicelo. Njengoba manje (sibonga i-Datadog) sikwazi ukuqapha ngokucacile zonke izinsiza kuqoqo lethu, ngihlaziye izinyanga ezimbalwa zokusebenza kwalezo ebesihlele ukuziqoka “njengazingahlobene”. Ngimane ngisethe ukusetshenziswa okuphezulu kwe-CPU ngomkhawulo we-20%, futhi ngaleyo ndlela ngibe isikhala endaweni lapho kwenzeka ama-k8s ezama ukwabela ezinye izinsizakalo endaweni.
Njengoba ubona kugrafu, umthwalo omkhulu kuphrosesa usufinyelele 242m Ama-CPU cores (ama-processor cores angu-0.242). Ngesicelo sokucubungula, kwanele ukuthatha inombolo enkulu kancane kunaleli nani. Sicela uqaphele ukuthi njengoba izinsiza zigxile kubasebenzisi, amanani aphezulu omthwalo ahambisana nethrafikhi.
Yenza okufanayo ngokusebenzisa inkumbulo nemibuzo, futhi voila - nonke senimi ngomumo! Ukuze uthole ukuphepha okukhulu, ungakwazi ukwengeza i-pod autoscaling evundlile. Ngakho-ke, ngaso sonke isikhathi lapho umthwalo wezinsiza uphezulu, i-autoscaling izodala ama-pods amasha, futhi i-kubernetes izowasabalalisa kumanodi anendawo yamahhala. Esimeni lapho singekho isikhala esisele kuqoqo ngokwalo, ungazisethela isexwayiso noma ulungiselele ukungezwa kwamanodi amasha ngokusebenzisa i-autoscaling yawo.
Kumaminithi, kubalulekile ukuqaphela ukuthi silahlekelwe "", i. inani leziqukathi ezisebenza endaweni eyodwa. Singase futhi sibe “nokuphumula” okuningi ekuminyana kwethrafikhi ephansi, futhi kukhona nethuba lokuthi uzofinyelela umthwalo weprosesa ophezulu, kodwa ama-autoscaling node kufanele asize kulokhu kwakamuva.
Imiphumela
Ngiyajabula ukushicilela le miphumela emihle kakhulu evela ekuhlolweni kula maviki ambalwa adlule; sesivele sibone ukuthuthuka okubalulekile ekuphenduleni kuwo wonke amasevisi alungisiwe:
Sithole imiphumela emihle kakhulu ekhasini lethu lasekhaya (), lapho inkonzo yashesha yangena izikhathi ezingamashumi amabili nambili!
Ingabe i-Linux kernel bug ilungisiwe?
Yebo ukusatshalaliswa kwenguqulo 4.19 nangaphezulu.
Nokho, lapho ufunda okwesibili kaSepthemba 2020 sisahlangabezana nokukhulunywa ngamanye amaphrojekthi e-Linux anesiphazamisi esifanayo. Ngikholwa ukuthi okunye ukusatshalaliswa kweLinux kusenaso lesi siphazamisi futhi sisasebenza nje ukusilungisa.
Uma inguqulo yakho yokusabalalisa ingaphansi kuka-4.19, ngingancoma ukuthi ubuyekezele kwakamuva, kodwa kunoma yikuphi kufanele uzame ukususa imikhawulo yephrosesa futhi ubone ukuthi i-throttling iyaqhubeka yini. Ngezansi ungabona uhlu oluyingxenye yezinsizakalo zokuphatha ze-Kubernetes kanye nokusatshalaliswa kweLinux:
- I-Debian: lungisa kuhlanganiswe enguqulweni yakamuva yokusabalalisa, , futhi kubukeka kusha (). Ezinye izinguqulo zangaphambili nazo zingalungiswa.
- Ubuntu: lungisa kuhlanganiswe enguqulweni yakamuva
- I-EKS isinokulungiswa okwamanje . Uma inguqulo yakho iphansi kunalokhu, kufanele ubuyekeze i-AMI.
- ama-kops: у
kops 1.18+Isithombe esikhulu sokusingatha kuzoba Ubuntu 20.04. Uma inguqulo yakho ye-kops isindala, kungase kudingeke ulinde ukulungiswa. Thina ngokwethu silindile manje. - I-GKE (Ifu Le-Google): Lungisa okuhlanganisiwe , nokho kunezinkinga nge-throttling .
Okufanele ukwenze uma ukulungiswa kulungise inkinga ye-throttling?
Angiqiniseki ukuthi inkinga isixazululiwe ngokuphelele. Uma sifika enguqulweni ye-kernel ngokulungiswa, ngizohlola iqoqo futhi ngibuyekeze okuthunyelwe. Uma kukhona osekuvele kubuyekeziwe, ngingaba nentshisekelo yokufunda imiphumela yakho.
isiphetho
- Uma usebenza neziqukathi ze-Docker ngaphansi kwe-Linux (kungakhathaliseki ukuthi i-Kubernetes, i-Mesos, i-Swarm noma ezinye), iziqukathi zakho zingase zilahlekelwe ukusebenza ngenxa yokudonsa;
- Zama ukuthuthukela enguqulweni yakamuva yokusabalalisa kwakho ngethemba lokuthi iphutha selilungisiwe;
- Ukususa imingcele yeprosesa kuzoxazulula inkinga, kodwa lokhu kuyindlela eyingozi okufanele isetshenziswe ngokuqapha okwedlulele (kungcono ukuqala ubuyekeze i-kernel bese uqhathanisa imiphumela);
- Uma uyisusile imikhawulo ye-CPU, qapha ngokucophelela i-CPU yakho kanye nokusetshenziswa kwememori futhi uqiniseke ukuthi izinsiza ze-CPU yakho zingaphezu kokusetshenziswa kwakho;
- Inketho ephephile kungaba ukukala ama-pods ukuze udale ama-pods amasha uma kwenzeka umthwalo wehadiwe ephezulu, ukuze i-kubernetes iwabele amanodi amahhala.
Ngethemba ukuthi lokhu okuthunyelwe kukusiza ukuthi uthuthukise ukusebenza kwezinhlelo zakho zeziqukathi.
PS umbhali uhambisana nabafundi nabahlaziyi (ngesiNgisi).
Source: www.habr.com