April 27 kumsonkhanowu , monga gawo la gawo la "DevOps", lipoti la "Autoscaling and resource management in Kubernetes" linaperekedwa. Imakamba za momwe mungagwiritsire ntchito ma K8s kuti muwonetsetse kupezeka kwa mapulogalamu anu ndikuwonetsetsa kuti magwiridwe antchito apamwamba kwambiri.
Mwa mwambo, ndife okondwa kupereka (Mphindi 44, yophunzitsa kwambiri kuposa nkhaniyo) komanso chidule chachikulu m'mawu. Pitani!
Tiyeni tiwunike mutu wa lipoti liwu ndi liwu ndikuyamba kuchokera kumapeto.
Kubernetes
Tinene kuti tili ndi zotengera za Docker pa omwe akutilandira. Zachiyani? Kuonetsetsa kubwereza komanso kudzipatula, zomwe zimalola kutumizidwa kosavuta komanso kwabwino, CI/CD. Tili ndi magalimoto ambiri otere okhala ndi makontena.
Kodi Kubernetes amapereka chiyani pankhaniyi?
- Timasiya kuganizira za makinawa ndikuyamba kugwira ntchito ndi "mtambo" gulu la zotengera kapena makoko (magulu a zotengera).
- Komanso, sitimaganiziranso za pods payekha, koma kusamalira zambiriΠΎmagulu akuluakulu. Chotero zoyamba zapamwamba tiloleni kunena kuti pali template yoyendetsera ntchito inayake, ndipo apa pali chiwerengero chofunikira cha zochitika kuti muyendetse. Ngati tisintha template pambuyo pake, zochitika zonse zidzasintha.
- Ndi chithandizo cha declarative API M'malo motsatira malamulo enaake, timalongosola "mapangidwe a dziko" (mu YAML), omwe amapangidwa ndi Kubernetes. Ndipo kachiwiri: pamene kufotokozera kumasintha, chiwonetsero chake chenicheni chidzasinthanso.
Kasamalidwe kazinthu
CPU
Tiyeni tithamangitse nginx, php-fpm ndi mysql pa seva. Ntchito izi zidzakhala ndi njira zambiri zomwe zikuyenda, iliyonse yomwe imafunikira zida zamakompyuta:
(zinambala zomwe zili patsambali ndi βzinkhweβ, kufunikira kwa njira iliyonse yamagetsi apakompyuta)
Kuti zikhale zosavuta kugwira ntchito ndi izi, ndizomveka kuphatikiza njira m'magulu (mwachitsanzo, njira zonse za nginx mu gulu limodzi "nginx"). Njira yosavuta komanso yodziwikiratu yochitira izi ndikuyika gulu lililonse mu chidebe:
Kuti mupitilize, muyenera kukumbukira chomwe chidebe ndi (mu Linux). Maonekedwe awo adatheka chifukwa cha zinthu zitatu zazikulu mu kernel, zomwe zidakhazikitsidwa kalekale: , ΠΈ . Ndipo chitukuko china chinayendetsedwa ndi matekinoloje ena (kuphatikiza "zipolopolo" zosavuta monga Docker):
Pankhani ya lipotilo, timangokonda magulu, chifukwa magulu owongolera ndi gawo la magwiridwe antchito a zotengera (Docker, etc.) zomwe zimagwiritsa ntchito kasamalidwe kazinthu. Njira zophatikizidwa m'magulu, monga momwe timafunira, ndi magulu olamulira.
Tiyeni tibwerere ku zofunikira za CPU panjira izi, ndipo tsopano pamagulu anjira:
(Ndibwerezanso kuti manambala onse ndi chisonyezero cha kufunikira kwa zinthu)
Panthawi imodzimodziyo, CPU yokha ili ndi gwero linalake (mu chitsanzo ichi ndi 1000), zomwe aliyense angasowe (chiwerengero cha zosowa zamagulu onse ndi 150+850+460=1460). Nanga chidzachitike nβchiyani pamenepa?
Kernel imayamba kugawa zinthu ndikuchita "mwachilungamo", kupereka ndalama zomwezo ku gulu lirilonse. Koma koyamba, pali zambiri kuposa zomwe zimafunikira (333> 150), kotero kuti zochulukirapo (333-150 = 183) zimakhalabe zosungidwa, zomwe zimagawidwanso mofanana pakati pa zotengera zina ziwiri:
Chotsatira chake: chidebe choyamba chinali ndi zinthu zokwanira, chachiwiri - chinalibe zinthu zokwanira, chachitatu - chinalibe zinthu zokwanira. Izi ndi zotsatira za zochita "woona mtima" ku Linux - . Ntchito yake ikhoza kusinthidwa pogwiritsa ntchito ntchitoyo kulemera chilichonse mwa zotengerazo. Mwachitsanzo, monga chonchi:
Tiyeni tiwone vuto la kusowa kwazinthu mu chidebe chachiwiri (php-fpm). Zida zonse zotengera zotengera zimagawidwa mofanana pakati pa njira. Zotsatira zake, ndondomekoyi imagwira ntchito bwino, koma ogwira ntchito onse amachepetsa, akulandira zosakwana theka la zomwe amafunikira:
Umu ndi momwe CFS scheduler imagwirira ntchito. Tidzayitananso zolemera zomwe timagawira ku makontena zopempha. Chifukwa chiyani izi zili choncho - onaninso.
Tiyeni tiyang'ane zochitika zonse kumbali inayo. Monga mukudziwira, misewu yonse imapita ku Roma, ndipo pankhani ya kompyuta, kupita ku CPU. CPU imodzi, ntchito zambiri - mumafunika magetsi. Njira yosavuta yoyendetsera zinthu ndi "kuwala kwamagalimoto": adapatsa njira imodzi nthawi yofikira ku CPU, kenako yotsatira, ndi zina zambiri.
Njira imeneyi imatchedwa hard quotas (kuchepetsa kwambiri). Tizikumbukira mophweka ngati malire. Komabe, ngati mugawira malire kuzinthu zonse, vuto limakhalapo: mysql anali kuyendetsa pamsewu ndipo panthawi ina kufunikira kwake kwa CPU kutha, koma njira zina zonse zimakakamizika kudikirira mpaka CPU. opanda ntchito.
Tiyeni tibwerere ku Linux kernel ndikuyanjana kwake ndi CPU - chithunzi chonse chili motere:
cgroup ili ndi makonda awiri - makamaka awa ndi "zopotoza" ziwiri zosavuta zomwe zimakulolani kudziwa:
- kulemera kwa chidebe (zopempha) ndi magawo;
- kuchuluka kwa nthawi yonse ya CPU yogwira ntchito pazotengera (malire) ndi gawo.
Kodi kuyeza CPU?
Pali njira zosiyanasiyana:
- Kodi mapuloti, palibe amene akudziwa - muyenera kukambirana nthawi zonse.
- Chidwi momveka bwino, koma wachibale: 50% ya seva yokhala ndi ma cores 4 ndi ma cores 20 ndi zinthu zosiyana kwambiri.
- Mutha kugwiritsa ntchito zomwe zatchulidwa kale kulemera, omwe Linux amadziwa, koma amakhalanso achibale.
- Njira yoyenera kwambiri ndikuyesa zida zamakompyuta masekondi. Iwo. mumasekondi a purosesa nthawi yofananira ndi masekondi a nthawi yeniyeni: 1 sekondi ya nthawi ya purosesa inaperekedwa pa sekondi imodzi yeniyeni - ichi ndi chimodzi chonse cha CPU core.
Kuti azilankhula mosavuta, anayamba kuyeza molunjika maso, kutanthauza kuti nthawi yomweyo CPU yokhudzana ndi yeniyeniyo. Popeza Linux imamvetsetsa zolemera, koma osati nthawi yochuluka ya CPU / cores, makina ankafunika kumasulira kuchokera kumodzi kupita ku imzake.
Tiyeni tiganizire chitsanzo chosavuta ndi seva yokhala ndi 3 CPU cores, pomwe ma pod atatu adzapatsidwa zolemera (500, 1000 ndi 1500) zomwe zimatembenuzidwa mosavuta ku zigawo zofanana za ma cores omwe apatsidwa (0,5, 1 ndi 1,5).
Ngati mutenga seva yachiwiri, pomwe padzakhala ma cores owirikiza kawiri (6), ndikuyika ma pod omwewo pamenepo, kugawa kwa ma cores kumatha kuwerengedwa mosavuta ndikungochulukitsa ndi 2 (1, 2 ndi 3, motsatana). Koma mphindi yofunikira imachitika pamene pod yachinayi ikuwonekera pa seva iyi, yomwe kulemera kwake, kuti ikhale yosavuta, kudzakhala 3000. Zimachotsa gawo lazinthu za CPU (theka la cores), ndipo kwa ma pod otsalawo amawerengedwanso (theka):
Kubernetes ndi CPU zothandizira
Ku Kubernetes, zida za CPU nthawi zambiri zimayesedwa miliadrax,ndi. 0,001 cores amatengedwa ngati kulemera koyambira. (Chinthu chomwecho mu Linux/cgroups terminology imatchedwa gawo la CPU, ngakhale, ndendende, 1000 millicores = magawo 1024 CPU.) K8s imawonetsetsa kuti sikuyika ma pods ambiri pa seva kuposa momwe zilili ndi CPU zothandizira kulemera kwa ma pod onse.
Kodi izi zimachitika bwanji? Mukawonjezera seva ku gulu la Kubernetes, zimanenedwa kuti ndi ma cores angati a CPU omwe alipo. Ndipo popanga pod yatsopano, wokonza Kubernetes amadziwa kuchuluka kwa ma cores omwe angafune. Chifukwa chake, podyo idzaperekedwa ku seva komwe kuli ma cores okwanira.
Zidzatheka bwanji ngati osati pempho latchulidwa (ie pod ilibe nambala yodziwika ya ma cores yomwe ikufunika)? Tiyeni tiwone momwe Kubernetes nthawi zambiri amawerengera zothandizira.
Pa pod mungathe kutchula zopempha zonse (CFS scheduler) ndi malire (mukumbukira kuwala kwa magalimoto?):
- Ngati atchulidwa ofanana, ndiye kuti pod imapatsidwa kalasi ya QoS zatsimikiziridwa. Chiwerengero cha ma cores omwe amapezeka nthawi zonse ndi otsimikizika.
- Ngati pempho lili locheperapo - kalasi ya QoS chophulika. Iwo. Tikuyembekeza kuti pod, mwachitsanzo, igwiritse ntchito 1 pachimake, koma mtengo uwu si malire ake: nthawi zina pod angagwiritse ntchito zambiri (pamene seva ili ndi zothandizira zaulere pa izi).
- Palinso gulu la QoS khama kwambiri - imaphatikizapo ma pod omwe pempho silinatchulidwe. Zothandizira zimaperekedwa kwa iwo komaliza.
chikumbukiro
Ndi kukumbukira, zinthu ndi zofanana, koma zosiyana pang'ono - pambuyo pake, chikhalidwe cha zinthu izi ndi chosiyana. Mwambiri, fanizoli lili motere:
Tiyeni tiwone momwe zopempha zimagwiritsidwira ntchito kukumbukira. Lolani ma pod azikhala pa seva, akusintha kagwiritsidwe ka kukumbukira, mpaka imodzi mwazo ikhale yaikulu kwambiri moti imatha kukumbukira. Pankhaniyi, wakupha OOM akuwonekera ndikupha njira yayikulu kwambiri:
Izi sizitiyendera nthawi zonse, kotero ndizotheka kuwongolera njira zomwe zili zofunika kwa ife ndipo siziyenera kuphedwa. Kuti muchite izi, gwiritsani ntchito parameter ooom_score_adj.
Tiyeni tibwerere m'makalasi a QoS a CPU ndikujambulitsa fanizo la oom_score_adj lomwe limatsimikizira zomwe zimafunikira kukumbukira pama pod:
- Mtengo wotsika kwambiri wa oom_score_adj wa pod - -998 - umatanthauza kuti pod yotere iyenera kuphedwa komaliza, izi. zatsimikiziridwa.
- Wapamwamba kwambiri - 1000 - ndi khama kwambiri, makoko oterowo amaphedwa poyamba.
- Kuwerengera zotsalira zotsalira (chophulika) pali chilinganizo, chomwe chimachokera ku mfundo yakuti ndalama zambiri zomwe pod yapempha, zimakhala zochepa kuti ziphedwe.
Wachiwiri "kupotoza" - malire_mu_mabayiti - kwa malire. Ndi izo, chirichonse chiri chosavuta: timangopereka kuchuluka kwa kukumbukira komwe kunaperekedwa, ndipo apa (mosiyana ndi CPU) palibe funso la momwe tingayesere (kukumbukira).
Chiwerengero
pod iliyonse ku Kubernetes imaperekedwa requests ΠΈ limits - magawo onse a CPU ndi kukumbukira:
- kutengera zopempha, Kubernetes scheduler amagwira ntchito, yomwe imagawa ma pod pakati pa ma seva;
- kutengera magawo onse, gulu la QoS la pod limatsimikiziridwa;
- Kulemera kwachibale kumawerengedwa kutengera zopempha za CPU;
- CFS scheduler imakonzedwa kutengera zopempha za CPU;
- OOM killer imakonzedwa kutengera zopempha zokumbukira;
- "kuwala kwa magalimoto" kumakonzedwa kutengera malire a CPU;
- Kutengera malire a kukumbukira, malire amapangidwira gululo.
Nthawi zambiri, chithunzichi chimayankha mafunso onse okhudza momwe gawo lalikulu la kasamalidwe kazinthu limachitikira ku Kubernetes.
Autoscaling
K8s cluster-autoscaler
Tiyerekeze kuti gulu lonse lakhala kale ndipo poto yatsopano iyenera kupangidwa. Ngakhale poto silingawonekere, imakhazikika pamalo ake Akuyembekeza. Kuti ziwonekere, tikhoza kulumikiza seva yatsopano ku gulu kapena ... kukhazikitsa cluster-autoscaler, zomwe zidzatichitire ife: kuitanitsa makina enieni kuchokera kwa opereka mtambo (pogwiritsa ntchito pempho la API) ndikugwirizanitsa ndi gululo. , pambuyo pake pod idzawonjezedwa .
Uku ndikuwongolera gulu la Kubernetes, lomwe limagwira ntchito bwino (muzochitikira zathu). Komabe, monga kwina kulikonse, pali ma nuances ena apa ...
Malingana ngati titakulitsa kukula kwa masango, zonse zinali bwino, koma zomwe zimachitika pamene gululo anayamba kudzimasula yekha? Vuto ndiloti kusamuka kwa ma pod (kumasula makamu) ndikovuta mwaukadaulo komanso kokwera mtengo pankhani yazachuma. Kubernetes amagwiritsa ntchito njira yosiyana kwambiri.
Ganizirani gulu la ma seva atatu omwe ali ndi Deployment. Ili ndi ma pod 3: tsopano pali 6 pa seva iliyonse. Pazifukwa zina tinkafuna kuzimitsa imodzi mwama seva. Kuti tichite izi tidzagwiritsa ntchito lamulo kubectl drain, amene:
- idzaletsa kutumiza mapodo atsopano ku seva iyi;
- idzachotsa ma pod omwe alipo pa seva.
Popeza Kubernetes ali ndi udindo wosunga kuchuluka kwa ma pod (6), mophweka adzalenganso iwo m'malo ena, koma osati pa yomwe ili yolemala, chifukwa idalembedwa kale kuti siyikupezeka kuti isungitse ma pod atsopano. Awa ndiye makina oyambira a Kubernetes.
Komabe, palinso nuance pano. Momwemonso, kwa StatefulSet (m'malo mwa Deployment), zochitazo zidzakhala zosiyana. Tsopano tili ndi pulogalamu yodziwika bwino - mwachitsanzo, ma pod atatu omwe ali ndi MongoDB, imodzi yomwe ili ndi vuto linalake (deta yawonongeka kapena cholakwika china chomwe chimalepheretsa pod kuyamba bwino). Ndipo timasankhanso kuletsa seva imodzi. Kodi chidzachitike n'chiyani?
MongoDB akhoza kufa chifukwa ikufunika quorum: pagulu la makhazikitsidwe atatu, osachepera awiri ayenera kugwira ntchito. Komabe, izi sizikuchitika -zikomo kwa PodDisruptionBudget. Izi chizindikiro chimatsimikizira chiwerengero chochepa chofunikira cha ma pods ogwira ntchito. Kudziwa kuti imodzi mwa ma pod a MongoDB sikugwiranso ntchito, ndikuwona kuti PodDisruptionBudget yakhazikitsidwa ku MongoDB. minAvailable: 2, Kubernetes sikukulolani kuti mufufuze pod.
Mfundo yofunika kwambiri: kuti kayendetsedwe kake (komanso, kukonzanso) kwa pods kugwire ntchito bwino pamene masango atulutsidwa, m'pofunika kukonza PodDisruptionBudget.
Kukula kopingasa
Tiyeni tikambirane nkhani ina. Pali pulogalamu yomwe ikuyenda ngati Kutumiza ku Kubernetes. Kuchuluka kwa ogwiritsa ntchito kumabwera ku mapoto ake (mwachitsanzo, pali atatu aiwo), ndipo timayesa chizindikiro china mwa iwo (titi, kuchuluka kwa CPU). Pamene katundu akuwonjezeka, timalemba izi pa ndondomeko ndikuwonjezera chiwerengero cha ma pod kuti tigawire zopempha.
Masiku ano ku Kubernetes izi siziyenera kuchitidwa pamanja: kuchulukira / kutsika kwachulukidwe kwa ma pod kumakonzedwa kutengera mayendedwe azizindikiro zoyezera.
Mafunso akulu ndi awa: kuti muyeze bwanji ΠΈ momwe mungamasulire adapeza zofunikira (popanga chisankho pakusintha kuchuluka kwa ma pod). Mutha kuyesa zambiri:
Momwe mungachitire izi mwaukadaulo - sonkhanitsani ma metric, ndi zina. - Ndinayankhula mwatsatanetsatane mu lipoti za . Ndipo upangiri waukulu pakusankha magawo abwino kwambiri ndi kuyesa!
pali (Kugwiritsa Ntchito Machulukidwe ndi Zolakwika), tanthauzo lake lili motere. Pazifukwa ziti ndizomveka kukulitsa, mwachitsanzo, php-fpm? Kutengera kuti ogwira ntchito akutha, izi ndi magwiritsidwe. Ndipo ngati ogwira ntchito atha ndipo kulumikizana kwatsopano sikuvomerezedwa, izi zili kale kukhuta. Magawo awiriwa ayenera kuyeza, ndipo kutengera milingo, makulitsidwe ayenera kuchitidwa.
M'malo mapeto
Lipotili lili ndi kupitiriza: za makulitsidwe ofukula ndi momwe mungasankhire zothandizira zoyenera. Ndilankhula za izi m'mavidiyo amtsogolo - lembetsani kuti musaphonye!
Makanema ndi zithunzi
Kanema wakuchita (mphindi 44):
Kafotokozedwe ka lipoti:
PS
Malipoti ena okhudza Kubernetes pabulogu yathu:
- «» (Andrey Polovov; Epulo 8, 2019 pa Saint HighLoad++);
- «» (Dmitry Stolyarov; Novembala 8, 2018 pa HighLoad++);
- «» (Dmitry Stolyarov; May 28, 2018 ku RootConf);
- «» (Dmitry Stolyarov; Novembala 7, 2017 pa HighLoad++);
- «» (Dmitry Stolyarov; June 6, 2017 ku RootConf).
Source: www.habr.com