Emva ngo-2016 siseBuffer , kwaye ngoku malunga ne-60 nodes (kwi-AWS) kunye nezikhongozeli ezili-1500 ezisebenzayo kwiqela lethu le-k8s elilawulwa . Nangona kunjalo, siye safudukela kwii-microservices ngolingo nangempazamo, kwaye nasemva kweminyaka emininzi sisebenza nee-k8s sisajongene neengxaki ezintsha. Kule post siza kuthetha ngayo unyino lweprosesa: kwakutheni ukuze sicinge ukuba baziqhelisa kakuhle kwaye kutheni begqibele bengasalunganga kangako.
Unyino lweprosesa kunye ne-throttling
Njengabanye abasebenzisi abaninzi beKubernetes, . Ngaphandle kolungiselelo olunjalo, izikhongozeli kwi-node zinokuthatha onke amandla eprosesa, nto leyo ebangela iinkqubo ezibalulekileyo zeKubernetes (umzekelo. kubelet) uya kuyeka ukuphendula izicelo. Ke, ukuseta imida ye-CPU yindlela elungileyo yokukhusela iindawo zakho.
Imida yeProcessor isete isikhongozeli sexesha eliphezulu le-CPU enokulisebenzisa ixesha elithile (okuhlala kukho li-100ms), kwaye isikhongozeli asisokuze sigqithe lo mda. Kwi Kubernetes for ukutswina isikhongozeli kunye nokuthintela ukuba ungagqithisi umda, isixhobo esikhethekileyo sisetyenziswa , kodwa le mida ye-CPU eyenziweyo igcina ilimaza ukusebenza kunye nokwandisa ixesha lokuphendula kwizikhongozeli zakho.
Kwenzeka ntoni ukuba asiyibeki imida yeprosesa?
Ngelishwa, kwafuneka sijamelane nale ngxaki. Indawo nganye inenkqubo enoxanduva lokulawula izikhongozeli kubelet, yaye wayeka ukuphendula izicelo. I-node, xa oku kusenzeka, kuya kungena kurhulumente NotReady, kunye nezikhongozeli ezisuka kuyo ziyakuqondiswa kwenye indawo kwaye zenze iingxaki ezifanayo kwiindawo ezintsha. Ayiyomeko ifanelekileyo, ukutsho oko.
Ukubonakaliswa kwengxaki ye-throttling kunye nokuphendula
I-metric engundoqo yokulandelela umgqomo yi trottling, ibonisa ukuba mangaphi amaxesha apho isikhongozeli sakho sicinezelwe. Siqaphele ngomdla ubukho bokuxinana kwezinye izikhongozeli, nokuba umthwalo weprosesa ubugqithisile okanye hayi. Njengomzekelo, makhe sijonge enye yee-API zethu eziphambili:
Njengoko ubona ngezantsi, sibeke umda ukuba 800m (0.8 okanye 80% engundoqo), kunye nencopho yamaxabiso ekufikeleleni 200m (20% engundoqo). Kubonakala ngathi ngaphambi kokuba siqhube inkonzo sisenamandla amaninzi eprosesa, nangona kunjalo ...
Usenokuba uqaphele ukuba naxa umthwalo weprosesa ungaphantsi kwemida echaziweyo - kakhulu ngezantsi - ukuxinana kusenzeka.
Ukujongana noku, kungekudala safumana izixhobo ezininzi (, , ) malunga nokwehla kokusebenza kunye nexesha lokuphendula leenkonzo ngenxa yokuxinana.
Kutheni sibona i-throttling kumthwalo ophantsi we-CPU? Inguqulelo emfutshane yile: "kukho ibug kwi-Linux kernel ebangela ukushukuma ngokungeyomfuneko kwezikhongozeli ezinemida yeprosesa echaziweyo." Ukuba unomdla kubume bengxaki, ungafunda umboniso ( и iinketho) nguDave Chiluk.
Ukususa izithintelo ze-CPU (ngononophelo olukhulu)
Emva kweengxoxo ezinde, sigqibe kwelokuba sisuse izithintelo zeprosesa kuzo zonke iinkonzo ezichaphazele ngokuthe ngqo okanye ngokungathanga ngqo ukusebenza okubalulekileyo kubasebenzisi bethu.
Isigqibo besingalula kuba sikuxabisa kakhulu uzinzo lweqela lethu. Kwixesha elidlulileyo, sele sele sizama ukungazinzi kweqela lethu, kwaye ke iinkonzo zichithe izibonelelo ezininzi kwaye zanciphisa umsebenzi we-node yazo yonke. Ngoku yonke into yayahluke ngandlel’ ithile: sasinokuqonda ngokucacileyo oko besikulindele kumaqela ethu, kunye nesicwangciso esilungileyo sokuphumeza iinguqu ezicwangcisiweyo.
Imbalelwano yoshishino malunga nomba oxingayo.
Indlela yokukhusela iindawo zakho xa izithintelo zisusiwe?
Ukwahlulwa kweenkonzo “ezingenamiqathango”:
Kwixesha elidlulileyo, sele sibonile ezinye iindawo zokuhlala zingena kwisimo notReady, ngokuyintloko ngenxa yeenkonzo ezitye izixhobo ezininzi kakhulu.
Sagqiba ekubeni sibeke ezo nkonzo kwiindawo ezahlukeneyo (“ezibhalwe”) ukuze zingaphazamisi iinkonzo “ezinxulumeneyo”. Ngenxa yoko, ngokumakisha ezinye iinqununu kunye nokongeza ipharamitha yokunyamezela kwiinkonzo "ezingahambelaniyo", siphumelele ukulawula okukhulu kwi-cluster, kwaye kwaba lula ngathi ukuchonga iingxaki ngeenodi. Ukwenza iinkqubo ezifanayo ngokwakho, unokuziqhelanisa nazo .
Ukwabela iprosesa echanekileyo kunye nesicelo sememori:
Uloyiko lwethu olukhulu yayikukuba inkqubo iya kusebenzisa izixhobo ezininzi kwaye i-node iya kuyeka ukuphendula izicelo. Ukusukela ngoku (enkosi kwi-Datadog) sinokubeka iliso ngokucacileyo zonke iinkonzo kwiqela lethu, ndihlalutye iinyanga ezininzi zokusebenza kwezo besicwangcise ukuzichaza "njengenganxulumananga". Ndiseta nje ubuninzi bokusetyenziswa kwe-CPU kunye nomda we-20%, kwaye ngaloo ndlela ndabelwe indawo kwindawo apho i-k8s izama ukwabela ezinye iinkonzo kwindawo.
Njengoko ubona kwigrafu, umthwalo ophezulu kwiprosesa ufikelele 242m CPU cores (0.242 processor cores). Kwisicelo somqhubekekisi, kwanele ukuthatha inani elikhulu kancinci kuneli xabiso. Nceda uqaphele ukuba njengoko iinkonzo ziphakathi komsebenzisi, amaxabiso omthwalo ophakamileyo ahambelana netrafikhi.
Yenza okufanayo kunye nokusetyenziswa kwememori kunye nemibuzo, kwaye voila - nonke nisekiwe! Ngokhuseleko olukhulu, unokongeza i-horizontal pod autoscaling. Ngaloo ndlela, ngalo lonke ixesha umthwalo wemithombo uphezulu, i-autoscaling iya kudala iipod ezintsha, kwaye i-kubernetes iya kusasaza kwiindawo ngeendawo zamahhala. Kwimeko apho akukho sithuba sishiywe kwiqela ngokwalo, ungaziseta isivuseleli okanye uqwalasele ukongezwa kweenodi ezintsha ngokusebenzisa i-autoscaling yazo.
Kwimizuzu, kubalulekile ukuqaphela ukuba siphulukene "", i.e. inani lezikhongozeli ezihamba kwindawo enye. Sinokuba "nokuphumla" okuninzi kuxinzelelo lwetrafikhi ephantsi, kwaye kukho nethuba lokuba uya kufikelela kumthwalo weprosesa ephezulu, kodwa i-autoscaling nodes kufuneka incede ngale yokugqibela.
Iziphumo
Ndiyavuya ukupapasha ezi ziphumo zibalaseleyo kwimifuniselo kwezi veki zimbalwa zidlulileyo; sele sibonile ukuphucuka okubalulekileyo kwiimpendulo kuzo zonke iinkonzo ezilungisiweyo:
Sifumene iziphumo ezilungileyo kwiphepha lethu lasekhaya (), apho inkonzo yakhawuleza yangena amashumi amabini anesibini amaxesha!
Ngaba i-Linux kernel bug ilungisiwe?
Ewe ulungile unikezelo uguqulelo 4.19 nangaphezulu.
Nangona kunjalo, xa ufunda okwesibini kaSeptemba 2020 sisafumana ukukhankanywa kwezinye iiprojekthi zeLinux ezinebug efanayo. Ndiyakholelwa ukuba ezinye izinikezelo zeLinux zisenayo le bug kwaye zisebenza nje ukuyilungisa.
Ukuba uguqulelo lwakho lokusabalalisa lungaphantsi kwe-4.19, ndingancoma ukuhlaziywa kwangoku, kodwa kuyo nayiphi na imeko kufuneka uzame ukususa izithintelo zeprosesa kwaye ubone ukuba i-throttling iyaqhubeka. Ngezantsi ungabona uluhlu oluyinxenye lweenkonzo zolawulo lweKubernetes kunye nokuhanjiswa kweLinux:
- I-Debian: lungisa udityaniswe kuguqulelo lwamva nje losasazo, , kwaye ibonakala intsha (). Ezinye iinguqulelo zangaphambili nazo zinokulungiswa.
- Ubuntu: lungisa udityaniswe kwinguqulelo yamva nje
- I-EKS inokulungiswa okwangoku . Ukuba uguqulelo lwakho lusezantsi kunolu, kufuneka uhlaziye i-AMI.
- kops: у
kops 1.18+Umfanekiso ophambili womkhosi uya kuba ngu-Ubuntu 20.04. Ukuba inguqulelo yakho yekops indala, kungafuneka ulinde ukulungiswa. Nathi silindile ngoku. - I-GKE (iLifu likaGoogle): Lungisa oludityanisiweyo , nangona kunjalo kukho iingxaki nge-throttling .
Yintoni onokuyenza ukuba isilungiso silungise ingxaki ye-throttling?
Andiqinisekanga ukuba ingxaki isonjululwe ngokupheleleyo. Xa sifika kwi-kernel version kunye nokulungiswa, ndiya kuvavanya i-cluster kwaye ndihlaziye isithuba. Ukuba kukho umntu osele ehlaziyiwe, ndingathanda ukufunda iziphumo zakho.
isiphelo
- Ukuba usebenza kunye nezikhongozeli zeDocker phantsi kweLinux (akunamsebenzi iKubernetes, Mesos, Swarm okanye ezinye), izikhongozeli zakho zinokuphulukana nokusebenza ngenxa yokuxinana;
- Zama ukuhlaziya kuguqulelo lwamva nje lonikezelo lwakho ngethemba lokuba ibug sele ilungisiwe;
- Ukususa imida yeprosesa kuya kusombulula ingxaki, kodwa le yindlela eyingozi ekufuneka isetyenziswe ngokuqaphela okukhulu (kungcono ukuhlaziya kuqala i-kernel kwaye uthelekise iziphumo);
- Ukuba uyisusile imida ye-CPU, jonga ngononophelo i-CPU yakho kunye nokusetyenziswa kwememori kwaye uqiniseke ukuba izixhobo zakho ze-CPU zingaphezulu kwendlela oyisebenzisa ngayo;
- Ukhetho olukhuselekileyo luya kuba kukwenza iipod ezizenzekelayo ukwenza ii-pods ezintsha kwimeko yomthwalo ophezulu we-hardware, ukuze i-kubernetes ibanike kwiindawo ezikhululekileyo.
Ndiyathemba ukuba esi sithuba siyakunceda uphucule ukusebenza kweenkqubo zesikhongozeli sakho.
PS umbhali uhambelana nabafundi kunye nabahlalutyi (ngesiNgesi).
umthombo: www.habr.com