Malire a CPU ndi kugwedezeka kwaukali ku Kubernetes

Zindikirani. transl.: Mbiri yotsegula maso iyi ya Omio—wophatikiza maulendo a ku Ulaya—imatenga owerenga kuchokera ku chiphunzitso choyambirira kupita ku zovuta zochititsa chidwi za Kubernetes configuration. Kudziwana bwino ndi milandu yotere sikumangokulitsa malingaliro anu, komanso kupewa mavuto omwe si ang'onoang'ono.

Kodi munayamba mwakhalapo ndi ntchito yokhazikika, kusiya kuyankha zoyezetsa zaumoyo, ndikulephera kudziwa chifukwa chake? Kufotokozera kumodzi komwe kungathe kukhudzana ndi malire a CPU resource quota. Izi ndi zomwe tikambirana m'nkhaniyi.

TL; DR:
Timalimbikitsa mwamphamvu kuletsa malire a CPU mu Kubernetes (kapena kuletsa magawo a CFS ku Kubelet) ngati mukugwiritsa ntchito mtundu wa Linux kernel wokhala ndi cholakwika cha CFS. Pakatikati likupezeka serious ndi wodziwika bwino cholakwika chomwe chimabweretsa kugunda kwambiri komanso kuchedwa.

Mu Omio zomangamanga zonse zimayendetsedwa ndi Kubernetes. Ntchito zathu zonse zodziwika bwino komanso zopanda malire zimayendera Kubernetes (timagwiritsa ntchito Google Kubernetes Engine). M'miyezi isanu ndi umodzi yapitayo, tinayamba kuona kuchepa kwachisawawa. Mapulogalamu amayimitsa kapena kusiya kuyankha pakuwunika zaumoyo, kutaya kulumikizana ndi netiweki, ndi zina. Khalidweli linatidodometsa kwa nthawi yaitali, ndipo pomalizira pake tinaganiza zoti tiganizire kwambiri za vutoli.

Chidule cha nkhaniyi:

• Mawu ochepa okhudza zotengera ndi Kubernetes;
• Momwe zopempha za CPU ndi malire amatsatiridwa;
• Momwe malire a CPU amagwirira ntchito m'malo amitundu yambiri;
• Momwe mungayang'anire kuthamanga kwa CPU;
• Yankho lamavuto ndi ma nuances.

Mawu ochepa okhudza zotengera ndi Kubernetes

Kubernetes ndiye muyeso wamakono m'dziko la zomangamanga. Ntchito yake yayikulu ndikuyimba kwa zidebe.

Zotengera

M'mbuyomu, tidayenera kupanga zinthu zakale monga Java JARs/WARs, Python Eggs, kapena executables kuti ziyendetse pa maseva. Komabe, kuti izi zitheke, ntchito yowonjezera inayenera kuchitidwa: kukhazikitsa malo othamanga (Java / Python), kuyika mafayilo ofunikira m'malo oyenera, kuonetsetsa kuti zikugwirizana ndi mtundu wina wa opaleshoni, ndi zina zotero. Mwa kuyankhula kwina, kusamala kunayenera kuperekedwa ku kasamalidwe ka kasinthidwe (komwe nthawi zambiri kunkayambitsa mikangano pakati pa omanga ndi oyang'anira dongosolo).

Zotengera zinasintha chilichonse. Tsopano chojambulacho ndi chithunzi cha chidebe. Itha kuyimiridwa ngati mtundu wa fayilo yowonjezereka yomwe ili ndi pulogalamu yokhayo, komanso malo ophera anthu onse (Java/Python/...), komanso mafayilo/maphukusi ofunikira, oyikiratu komanso okonzeka thamanga. Zotengera zimatha kutumizidwa ndikuyendetsedwa pamaseva osiyanasiyana popanda njira zina zowonjezera.

Kuphatikiza apo, zotengera zimagwira ntchito m'malo awoawo a sandbox. Iwo ali awoawo pafupifupi maukonde adaputala, wapamwamba dongosolo ndi mwayi wochepa, ulamuliro wawo wa ndondomeko, zofooka zawo pa CPU ndi kukumbukira, etc. Zonsezi akuyendera chifukwa cha subsystem wapadera wa Linux kernel - namespaces.

Kubernetes

Monga tanena kale, Kubernetes ndi woyimba ziwiya. Zimagwira ntchito motere: mumapatsa makina ambiri, ndiyeno nkuti: "Hei, Kubernetes, tiyeni tiyambitse zitsanzo khumi za chidebe changa chokhala ndi mapurosesa a 2 ndi 3 GB ya kukumbukira iliyonse, ndikuwasunga!" Kubernetes adzasamalira ena onse. Ipeza kuchuluka kwaulere, kuyambitsa zotengera ndikuziyambitsanso ngati kuli kofunikira, tulutsani zosintha mukasintha mitundu, ndi zina. Kwenikweni, Kubernetes imakupatsani mwayi wochotsa gawo la hardware ndikupanga makina osiyanasiyana oyenera kuyika ndikuyendetsa mapulogalamu.

Kubernetes kuchokera pamalingaliro a munthu wamba

Zopempha ndi malire ku Kubernetes ndi chiyani

Chabwino, taphimba zotengera ndi Kubernetes. Tikudziwanso kuti zotengera zingapo zimatha kukhala pamakina amodzi.

Fanizo likhoza kujambulidwa ndi nyumba ya anthu onse. Malo akulu (makina/mayunitsi) amatengedwa ndikubwerekedwa kwa alendi angapo (zotengera). Kubernetes amagwira ntchito ngati wogulitsa. Funso likubuka, momwe mungatetezere obwereketsa ku mikangano wina ndi mnzake? Bwanji ngati mmodzi wa iwo, kunena, aganiza kubwereka bafa kwa theka la tsiku?

Apa ndipamene zopempha ndi malire zimalowa. CPU pempho zofunikira pazolinga zokonzekera. Ichi ndi china chake ngati "mndandanda wazofuna" wa chidebecho, ndipo chimagwiritsidwa ntchito posankha node yoyenera kwambiri. Pa nthawi yomweyo CPU malire titha kufananizidwa ndi mgwirizano wobwereketsa - tikangosankha gawo la chidebe, the sindingathe kupyola malire okhazikitsidwa. Ndipo apa ndipamene vuto limabuka...

Momwe zopempha ndi malire amatsatiridwa ku Kubernetes

Kubernetes amagwiritsa ntchito makina othamanga (kudumpha mawotchi) opangidwa mu kernel kuti agwiritse ntchito malire a CPU. Ngati pulogalamu yadutsa malire, kugwedeza kumayatsidwa (ie, imalandira ma CPU ochepa). Zopempha ndi malire a kukumbukira zimakonzedwa mosiyana, kotero zimakhala zosavuta kuzizindikira. Kuti muchite izi, ingoyang'anani momwe mungayambitsirenso pomaliza: kaya ndi "OOMKilled". CPU throttling sikophweka, popeza ma K8s amangopangitsa kuti ma metric apezeke pogwiritsa ntchito, osati ndi magulu.

Kufunsira kwa CPU

Momwe pempho la CPU limagwiritsidwira ntchito

Kuti zikhale zosavuta, tiyeni tiwone njira yogwiritsira ntchito makina okhala ndi 4-core CPU monga chitsanzo.

K8s amagwiritsa ntchito makina olamulira (magulu) kuti athetse kugawidwa kwazinthu (kukumbukira ndi purosesa). Chitsanzo chotsatira chikupezeka kwa icho: mwanayo adzalandira malire a gulu la makolo. Zambiri zogawa zimasungidwa mu fayilo yeniyeni (/sys/fs/cgroup). Pankhani ya purosesa izi ndi /sys/fs/cgroup/cpu,cpuacct/*.

K8s amagwiritsa ntchito fayilo cpu.share kugawa zothandizira purosesa. Kwa ife, gulu la mizu limapeza magawo 4096 azinthu za CPU - 100% ya mphamvu ya purosesa yomwe ilipo (1 core = 1024; ichi ndi mtengo wokhazikika). Gulu la mizu limagawa zinthu molingana kutengera magawo a mbadwa zolembetsedwa cpu.share, ndipo iwonso amachita chimodzimodzi ndi mbadwa zawo, ndi zina zotero. Pamalo wamba a Kubernetes, gulu la mizu lili ndi ana atatu: system.slice, user.slice и kubepods. Magulu awiri oyambirira amagwiritsidwa ntchito kugawa zinthu pakati pa katundu wovuta kwambiri ndi mapulogalamu a ogwiritsa ntchito kunja kwa K8s. Womaliza - kubepods - yopangidwa ndi Kubernetes kuti azigawa zinthu pakati pa ma pod.

Chithunzi pamwambapa chikuwonetsa kuti timagulu tating'ono tating'ono tomwe tidalandira 1024 magawo, ndi gulu laling'ono la kuberpod lomwe laperekedwa 4096 magawo Izi zingatheke bwanji: pambuyo pa zonse, gulu la mizu likhoza kupeza kokha 4096 magawo, ndipo kuchuluka kwa magawo a mbadwa zake kumaposa chiwerengerochi (6144)? Mfundo ndi yakuti mtengo wake umamveka bwino, kotero Linux scheduler (CFS) amaigwiritsa ntchito kuti igawanitse zinthu za CPU molingana. Kwa ife, magulu awiri oyambirira amalandira 680 magawo enieni (16,6% a 4096), ndipo kubepod amalandira zotsalira 2736 magawo Pakakhala nthawi yochepa, magulu awiri oyambirira sangagwiritse ntchito zinthu zomwe zaperekedwa.

Mwamwayi, wokonza mapulani ali ndi njira yopewera kuwononga zida za CPU zomwe sizinagwiritsidwe ntchito. Imasamutsa mphamvu "yopanda pake" ku dziwe lapadziko lonse lapansi, komwe imagawidwa kumagulu omwe amafunikira mphamvu yowonjezera ya purosesa (kutengerako kumachitika m'magulu kuti apewe kuwonongeka kozungulira). Njira yofananayi imagwiritsidwa ntchito kwa mbadwa zonse za mbadwa.

Njirayi imatsimikizira kugawidwa koyenera kwa mphamvu ya purosesa ndikuwonetsetsa kuti palibe njira "yobera" zothandizira ena.

CPU Limit

Ngakhale kuti masinthidwe a malire ndi zopempha mu K8s amawoneka ofanana, kukhazikitsidwa kwawo kuli kosiyana kwambiri: izi. osokeretsa kwambiri ndi gawo lochepera lolembedwa.

K8s amakumana CFS quota mechanism kukhazikitsa malire. Zokonda zawo zimafotokozedwa m'mafayilo cfs_period_us и cfs_quota_us mu cgroup directory (fayilo ilinso pamenepo cpu.share).

Mosiyana cpu.share, chiwerengerocho chimachokera pa nthawi, osati pa mphamvu ya purosesa yomwe ilipo. cfs_period_us imatchula nthawi ya nthawi (epoch) - nthawi zonse imakhala 100000 μs (100 ms). Pali njira yosinthira mtengowu mu K8s, koma umapezeka mu alpha pakadali pano. Wokonza amagwiritsira ntchito epoch kuti ayambitsenso ma quotas omwe agwiritsidwa ntchito. Fayilo yachiwiri cfs_quota_us, imatchula nthawi yomwe ilipo (chiwerengero) mu nthawi iliyonse. Dziwani kuti imatchulidwanso mu ma microseconds. Chiwerengerocho chikhoza kupitirira kutalika kwa nthawi; mwa kuyankhula kwina, ikhoza kukhala yayikulu kuposa 100 ms.

Tiyeni tiwone zochitika ziwiri pamakina apakati-16 (mtundu wamba wamakompyuta omwe tili nawo ku Omio):

Chitsanzo 1: 2 ulusi ndi malire a 200 ms. Palibe throttling

Nkhani 2: 10 ulusi ndi 200 ms malire. Kuthamanga kumayamba pambuyo pa 20 ms, mwayi wopeza purosesa umayambiranso pambuyo pa 80 ms.

Tinene kuti mwakhazikitsa malire a CPU 2 maso; Kubernetes adzamasulira mtengo uwu kukhala 200 ms. Izi zikutanthauza kuti chidebecho chitha kugwiritsa ntchito nthawi yayitali ya 200ms ya CPU popanda kugwedezeka.

Ndipo apa ndi pomwe zosangalatsa zimayambira. Monga tafotokozera pamwambapa, kuchuluka komwe kulipo ndi 200 ms. Ngati mukugwira ntchito limodzi khumi ulusi pa makina a 12-core (onani fanizo lachitsanzo 2), pomwe ma pod ena onse ali opanda ntchito, gawolo lidzathetsedwa mu 20 ms (kuyambira 10 * 20 ms = 200 ms), ndipo ulusi wonse wa pod iyi udzakhazikika. » (fulumizitsa) kwa 80 ms yotsatira. Zomwe zatchulidwa kale scheduler bug, chifukwa cha kugwedezeka kwambiri ndipo chidebe sichingathe kukwaniritsa zomwe zilipo.

Kodi mungayese bwanji throttling mu pods?

Ingolowetsani ku pod ndikuchita cat /sys/fs/cgroup/cpu/cpu.stat.

• nr_periods - chiwerengero chonse cha nthawi zokonzekera;
• nr_throttled - chiwerengero cha nthawi throttled mu zikuchokera nr_periods;
• throttled_time - nthawi yowonjezereka mu nanoseconds.

Kodi kwenikweni chikuchitika ndi chiyani?

Zotsatira zake, timakhala ndi kugunda kwakukulu pamapulogalamu onse. Nthawi zina amakhala nthawi imodzi ndi theka wamphamvu kuposa kuwerengetsera!

Izi zimabweretsa zolakwika zosiyanasiyana - kulephera kokonzekera, kuzizira kwa chidebe, kusweka kwa ma netiweki, kutha kwa nthawi pakuyimba foni. Izi pamapeto pake zimabweretsa kuchuluka kwa latency komanso kuchuluka kwa zolakwika.

Chisankho ndi zotsatira zake

Zonse ndi zophweka apa. Tinasiya malire a CPU ndikuyamba kukonzanso OS kernel m'magulu kupita ku mtundu waposachedwa, momwe cholakwikacho chidakhazikitsidwa. Chiwerengero cha zolakwika (HTTP 5xx) muntchito zathu chatsika kwambiri:

HTTP 5xx zolakwika

Zolakwika za HTTP 5xx pa ntchito imodzi yovuta

Nthawi yoyankhira p95

Pempho lofunikira lantchito yochedwa latency, 95th percentile

Ndalama zoyendetsera ntchito

Nambala ya maola achiwonetsero omwe agwiritsidwa ntchito

Kodi nsomba ndi ziti?

Monga tanenera kumayambiriro kwa nkhaniyi:

Fanizo likhoza kujambulidwa ndi nyumba ya anthu wamba ... Kubernetes amachita ngati wogulitsa. Koma kodi mungawateteze bwanji kuti asamakangane? Bwanji ngati mmodzi wa iwo, kunena, aganiza kubwereka bafa kwa theka la tsiku?

Nayi kugwira. Chidebe chimodzi chosasamala chimatha kudya zonse zomwe zilipo za CPU pamakina. Ngati muli ndi stack yogwiritsira ntchito mwanzeru (mwachitsanzo, JVM, Go, Node VM yakonzedwa bwino), ndiye kuti izi sizovuta: mutha kugwira ntchito motere kwa nthawi yayitali. Koma ngati mapulogalamu sakukometsedwa bwino kapena osakometsedwa konse (FROM java:latest), zinthu zitha kusokonekera. Ku Omio, tili ndi ma Dockerfiles okhazikika okhala ndi zoikika zokhazikika pagulu lalikulu la zilankhulo, kotero nkhaniyi kunalibe.

Timalimbikitsa kuyang'anira ma metric Gwiritsani ntchito (kugwiritsa ntchito, kuchuluka ndi zolakwika), kuchedwa kwa API ndi mitengo yolakwika. Onetsetsani kuti zotsatira zikukwaniritsa zomwe mukuyembekezera.

powatsimikizira

Iyi ndi nkhani yathu. Zinthu zotsatirazi zidathandizira kwambiri kumvetsetsa zomwe zikuchitika:

• kernel.org → CFS Scheduler;
• kernel.org → CFS Bandwidth Control;
• Kumvetsetsa Kukonzekera kwa Chikho cha Linux;
• Chilichonse Chimene Muyenera Kudziwa Zokhudza Linux Containers, Gawo I: Linux Control Groups and Process Isolation;
• Kubernetes Kulephera Nkhani - yang'anani "cpu throttling".

Kubernetes bug akuti:

• #51135: Pewani kuyika malire a CPU pa ma pod Otsimikizika;
• #67577: Magawo a CFS atha kubweretsa kugwedezeka kosafunikira;
• CFS yaukali kwambiri.

Kodi mudakumanapo ndi zovuta zofananira m'machitidwe anu kapena muli ndi chidziwitso chokhudzana ndi kugwedezeka m'malo opangira zida? Gawani nkhani yanu mu ndemanga!

PS kuchokera kwa womasulira

Werenganinso pa blog yathu:

• «Autoscaling and resources management in Kubernetes (review and video report)";
• «Momwe CPU Manager amagwirira ntchito ku Kubernetes";
• «Kodi chimachitika ndi chiyani ku Kubernetes mukathamanga kubectl run? Gawo 2".

Source: www.habr.com