Kubwerera mu 2016 ife ku Buffer , ndipo tsopano pafupifupi ma node 60 (pa AWS) ndi makontena 1500 akugwira ntchito pagulu lathu la k8s lomwe limayendetsedwa ndi . Komabe, tinasamukira ku microservices kupyolera mu kuyesa ndi zolakwika, ndipo ngakhale patapita zaka zingapo tikugwira ntchito ndi k8s tikukumanabe ndi mavuto atsopano. Mu positi iyi tikambirana malire a purosesa: chifukwa chiyani tinkaganiza kuti anali ochita bwino komanso chifukwa chake adamaliza kukhala osachita bwino.
Kulephera kwa purosesa ndi kugunda
Monga ogwiritsa ntchito ena ambiri a Kubernetes, . Popanda kuyika kotere, zotengera mu node zimatha kutenga mphamvu zonse za purosesa, zomwe zimayambitsa njira zofunika za Kubernetes (mwachitsanzo. kubelet) adzasiya kuyankha zopempha. Chifukwa chake, kukhazikitsa malire a CPU ndi njira yabwino yotetezera node zanu.
Malire a purosesa amayika chidebe ku nthawi yochuluka ya CPU yomwe ingagwiritse ntchito panthawi inayake (chosakhazikika ndi 100ms), ndipo chidebe sichidzadutsa malire awa. Ku Kubernetes kwa kugwedeza ΠΊΠΎΠ½ΡΠ΅ΠΉΠ½Π΅ΡΠ° ΠΈ Π½Π΅Π΄ΠΎΠΏΡΡΠ΅Π½ΠΈΡ ΠΏΡΠ΅Π²ΡΡΠ΅Π½ΠΈΡ ΠΈΠΌ ΠΏΡΠ΅Π΄Π΅Π»Π° ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΠΎΡΠΎΠ±ΡΠΉ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½Ρ , koma malire opangira a CPU awa amatha kuwononga magwiridwe antchito ndikuwonjezera nthawi yoyankhira zotengera zanu.
Kodi chingachitike ndi chiyani ngati sitiyika malire a purosesa?
Tsoka ilo, ife tokha tinakumana ndi vuto limeneli. Node iliyonse ili ndi njira yoyendetsera zotengera kubelet, ndipo anasiya kuyankha zopempha. Node, izi zikachitika, zidzalowa m'boma NotReady, ndi zotengera zochokera pamenepo zidzatumizidwa kwina ndikukhazikitsanso zovuta zomwezo pama node atsopano. Osati zochitika zabwino, kunena pang'ono.
Kuwonekera kwa vuto la throttling ndi kuyankha
Ma metric ofunikira pakutsata kotengera ndi trottling, zikuwonetsa kuti kangati chidebe chanu chagwedezeka. Tidawona ndi chidwi kukhalapo kwa kugwedezeka m'mitsuko ina, mosasamala kanthu kuti katundu wa purosesa anali wochuluka kapena ayi. Mwachitsanzo, tiyeni tiwone imodzi mwama API athu akulu:
Monga mukuonera pansipa, takhazikitsa malire 800m (0.8 kapena 80% pachimake), ndi nsonga zapamwamba zomwe zifika bwino 200m (20% yapakati). Zikuwoneka kuti tisanagwiritse ntchito ntchitoyi tikadali ndi mphamvu zambiri za purosesa, komabe ...
Mwina mwazindikira kuti ngakhale purosesa ikadzaza malire omwe atchulidwa - kwambiri pansipa - kugwedeza kumachitikabe.
Titakumana ndi izi, posakhalitsa tidapeza zinthu zingapo (, , ) za kuchepa kwa magwiridwe antchito ndi nthawi yoyankha ntchito chifukwa cha kugunda.
Chifukwa chiyani tikuwona kutsika kwa CPU yotsika? Mtundu waufupi ndi wakuti: "pali cholakwika mu Linux kernel chomwe chimayambitsa kugwedezeka kosafunikira kwa zotengera zomwe zili ndi malire a purosesa." Ngati muli ndi chidwi ndi momwe vutoli lilili, mutha kuwerenga ulaliki ( ΠΈ options) by Dave Chiluk.
Kuchotsa zoletsa za CPU (mosamala kwambiri)
Titakambirana kwanthawi yayitali, tidaganiza zochotsa zoletsa zama processor kuzinthu zonse zomwe zidakhudza mwachindunji kapena mwanjira ina yofunika kwambiri kwa ogwiritsa ntchito.
Chisankhocho sichinali chophweka chifukwa timayamikira kwambiri kukhazikika kwa gulu lathu. M'mbuyomu, tayesa kale kusakhazikika kwa gulu lathu, ndiyeno mautumikiwo adadya zinthu zambiri ndikuchepetsa ntchito ya node yawo yonse. Tsopano zonse zinali zosiyana pang'ono: tinali kumvetsetsa bwino zomwe tinkayembekezera kuchokera kumagulu athu, komanso njira yabwino yogwiritsira ntchito kusintha komwe kunakonzedwa.
Kulemberana makalata ndi bizinesi pavuto lalikulu.
Momwe mungatetezere ma node anu pamene zoletsa zachotsedwa?
Kupatula ntchito "zopanda malire":
M'mbuyomu, tawona kale ma node ena akulowa m'malo notReady, makamaka chifukwa cha ntchito zomwe zidawononga zinthu zambiri.
Tinaganiza zoyika mautumikiwa m'malo osiyana ("olembedwa") kuti asasokoneze ntchito "zokhudzana". Chotsatira chake, polemba ma node ena ndikuwonjezera chizindikiro chololera ku mautumiki "osagwirizana", tinapeza kulamulira kwakukulu pamagulu, ndipo zinakhala zosavuta kuti tizindikire mavuto ndi node. Kuti muchite izi nokha, mutha kuzidziwa bwino .
Kupereka purosesa yoyenera ndi pempho la kukumbukira:
Kuopa kwathu kwakukulu kunali kuti ndondomekoyi idzawononga zinthu zambiri ndipo node idzasiya kuyankha zopempha. Popeza tsopano (zikomo kwa Datadog) titha kuyang'anira bwino ntchito zonse pagulu lathu, ndidasanthula miyezi ingapo yogwira ntchito ya zomwe tidakonza zoti "zosagwirizana". Ndimangoyika kugwiritsa ntchito kwakukulu kwa CPU ndi malire a 20%, ndikugawa malo mu node ngati ma k8s ayesa kupereka mautumiki ena ku mfundo.
Monga mukuwonera pa graph, kuchuluka kwakukulu kwa purosesa kwafika 242m CPU cores (0.242 processor cores). Pa pempho la purosesa, ndikwanira kutenga nambala yokulirapo pang'ono kuposa mtengo uwu. Chonde dziwani kuti popeza ntchitozo ndizoyambira ogwiritsa ntchito, nsonga zamtengo wapatali zimagwirizana ndi kuchuluka kwa magalimoto.
Chitani zomwezo ndikugwiritsa ntchito kukumbukira ndi mafunso, ndipo voila - mwakonzeka! Kuti mutetezeke kwambiri, mutha kuwonjezera mayendedwe opingasa a pod autoscaling. Chifukwa chake, nthawi iliyonse katundu wazinthu akachuluka, autoscaling imapanga ma pod atsopano, ndipo kubernetes adzawagawira ku node ndi malo aulere. Ngati palibe danga lomwe latsala pagulu lokha, mutha kudziyika nokha tcheru kapena sinthani kuwonjezera ma node atsopano kudzera mu autoscaling yawo.
Mwa minuses, ndikofunikira kudziwa kuti tidataya "", ndi. kuchuluka kwa zotengera zomwe zikuyenda pa mfundo imodzi. Titha kukhalanso ndi "zopumula" zambiri pazambiri zamagalimoto otsika, ndipo palinso mwayi woti mufikire purosesa yayikulu, koma ma autoscaling node ayenera kuthandizira pomaliza.
Zotsatira
Ndine wokondwa kufalitsa zotsatira zabwino kwambiri izi kuchokera pazoyeserera m'masabata angapo apitawa; tawona kale kusintha kwakukulu pazantchito zonse zosinthidwa:
Tinapeza zotsatira zabwino kwambiri patsamba lathu loyambira (), pamenepo ntchito idafulumizitsa kawiri kawiri!
Kodi cholakwika cha Linux kernel chakhazikika?
Inde, magawo 4.19 ndi apamwamba.
Komabe, powerenga kwachiwiri kwa Seputembara 2020 timakumanabe ndi zonena za mapulojekiti ena a Linux okhala ndi cholakwika chofanana. Ndikukhulupirira kuti magawo ena a Linux akadali ndi cholakwika ichi ndipo akungoyesetsa kukonza.
Ngati mtundu wanu wogawa uli wotsika kuposa 4.19, ndingalimbikitse kukweza kwaposachedwa, koma mulimonsemo muyenera kuyesa kuchotsa zoletsa za purosesa ndikuwona ngati kugwedezeka kukupitilira. Pansipa mutha kuwona mndandanda wapanthawi ya Kubernetes kasamalidwe ka ntchito ndi magawo a Linux:
- Debian: konzani zophatikizidwa mu mtundu waposachedwa wa kugawa, , ndipo zikuwoneka zatsopano (). Mabaibulo ena am'mbuyomu akhoza kukonzedwanso.
- Ubuntu: Konzani zophatikizidwa mu mtundu waposachedwa
- EKS ili ndi kukonza . Ngati mtundu wanu ndi wotsika kuposa uwu, muyenera kusintha AMI.
- kops: Ρ
kops 1.18+Chithunzi chachikulu cha alendo chidzakhala Ubuntu 20.04. Ngati mtundu wanu wa kops ndi wakale, mungafunike kudikirira kuti mukonze. Ife tokha tikuyembekezera tsopano. - GKE (Google Cloud): Konzani zophatikizidwa , komabe pali mavuto ndi throttling .
Zoyenera kuchita ngati kukonzako kwathetsa vuto la throttling?
Sindikutsimikiza kuti vutoli lathetsedwa. Tikafika ku mtundu wa kernel ndikukonza, ndiyesa gululo ndikusinthira positi. Ngati wina wasintha kale, ndingakonde kuwerenga zotsatira zanu.
Pomaliza
- Ngati mumagwira ntchito ndi zotengera za Docker pansi pa Linux (zilibe kanthu Kubernetes, Mesos, Swarm kapena ena), zotengera zanu zitha kulephera kugwira ntchito chifukwa chakugwedezeka;
- Yesani kusinthira ku mtundu waposachedwa wa kugawa kwanu ndikuyembekeza kuti cholakwikacho chakonzedwa kale;
- Kuchotsa malire a purosesa kudzathetsa vutoli, koma iyi ndi njira yowopsa yomwe iyenera kugwiritsidwa ntchito mosamala kwambiri (ndi bwino kuti muyambe kusintha kernel ndikuyerekeza zotsatira);
- Ngati mwachotsa malire a CPU, yang'anani mosamala CPU yanu ndikugwiritsa ntchito kukumbukira ndikuwonetsetsa kuti zida zanu za CPU zikupitilira zomwe mumadya;
- Njira yotetezeka ingakhale kuyendetsa ma pods kuti apange ma pod atsopano ngati ali ndi katundu wambiri wa hardware, kotero kuti kubernetes amawagawira kuti akhale ndi ma node aulere.
Ndikukhulupirira kuti positi iyi ikuthandizani kukonza magwiridwe antchito a makina anu.
PS wolemba amafanana ndi owerenga ndi opereka ndemanga (mu Chingerezi).
Source: www.habr.com