Rov qab rau xyoo 2016 peb ntawm Buffer , thiab tam sim no txog 60 nodes (ntawm AWS) thiab 1500 ntim tau ua haujlwm ntawm peb k8s pawg tswj hwm los ntawm . Txawm li cas los xij, peb tau tsiv mus rau microservices los ntawm kev sim thiab ua yuam kev, thiab txawm tias tom qab ob peb xyoos ntawm kev ua haujlwm nrog k8s peb tseem ntsib teeb meem tshiab. Hauv tsab ntawv no peb yuav tham txog processor txwv: vim li cas peb thiaj xav tias lawv xyaum ua zoo thiab vim li cas lawv thiaj li tsis zoo.
Processor txwv thiab throttling
Zoo li ntau lwm tus neeg siv Kubernetes, . Yog tsis muaj qhov teeb tsa zoo li no, cov thawv ntim hauv ib lub node tuaj yeem nqa tag nrho lub zog processor, uas ua rau cov txheej txheem Kubernetes tseem ceeb (piv txwv li. kubelet) yuav tsum tsis txhob teb cov lus thov. Yog li, teeb tsa CPU txwv yog txoj hauv kev zoo los tiv thaiv koj cov nodes.
Cov txheej txheem txwv tau teeb tsa lub thawv rau lub sijhawm CPU siab tshaj plaws nws tuaj yeem siv rau lub sijhawm tshwj xeeb (default yog 100ms), thiab lub thawv yuav tsis pub dhau qhov kev txwv no. Hauv Kubernetes rau throttling ntim thiab tiv thaiv kom tsis txhob dhau qhov kev txwv, ib qho cuab yeej tshwj xeeb yog siv , tab sis cov kev txwv CPU dag no yuav ua rau muaj kev ua haujlwm hnyav thiab ua rau lub sijhawm teb ntawm koj cov ntim khoom.
Yuav ua li cas tshwm sim yog tias peb tsis teem processor txwv?
Hmoov tsis zoo, peb tus kheej yuav tsum tau ntsib qhov teeb meem no. Txhua lub node muaj cov txheej txheem los tswj cov ntim khoom kubelet, thiab nws tsis teb cov lus thov. Cov node, thaum qhov no tshwm sim, yuav mus rau hauv lub xeev NotReady, thiab ntim los ntawm nws yuav raug xa mus rau lwm qhov thiab tsim cov teeb meem tib yam ntawm cov nodes tshiab. Tsis yog qhov zoo tagnrho scenario, hais qhov tsawg kawg nkaus.
Kev tshwm sim ntawm qhov teeb meem ntawm throttling thiab teb
Lub metric tseem ceeb rau lub thawv taug qab yog trottling, nws qhia tau hais tias pes tsawg zaus koj lub thawv tau throttled. Peb tau pom nrog kev txaus siab ntawm qhov muaj qhov throttling hauv qee lub thawv, tsis hais seb lub processor load yog qhov hnyav lossis tsis. Ua piv txwv, cia peb saib ib qho ntawm peb lub ntsiab APIs:
Raws li koj tuaj yeem pom hauv qab no, peb tau teeb tsa qhov kev txwv rau 800m (0.8 los yog 80% core), thiab ncov qhov tseem ceeb ntawm qhov zoo tshaj plaws ncav cuag 200m (20% core). Nws yuav zoo li tias ua ntej throttling qhov kev pabcuam peb tseem muaj ntau lub zog processor, txawm li cas los xij ...
Tej zaum koj yuav tau pom tias txawm tias thaum lub processor thauj khoom qis dua qhov txwv tsis pub dhau - qhov tseem ceeb hauv qab - throttling tseem tshwm sim.
Tau ntsib nrog qhov no, tsis ntev peb nrhiav tau ntau yam kev pab (, , ) hais txog kev poob qis hauv kev ua haujlwm thiab lub sijhawm teb ntawm cov kev pabcuam vim muaj kev cuam tshuam.
Vim li cas peb pom throttling ntawm qis CPU load? Qhov luv luv version yog: "muaj kab laum nyob rau hauv Linux ntsiav uas ua rau tsis tsim nyog throttling ntawm cov ntim nrog cov txheej txheem kev txwv." Yog tias koj txaus siab rau qhov xwm txheej ntawm qhov teeb meem, koj tuaj yeem nyeem qhov kev nthuav qhia ( ΠΈ xaiv) los ntawm Dave Chiluk.
Tshem tawm CPU txwv (nrog ceev faj heev)
Tom qab kev sib tham ntev, peb tau txiav txim siab tshem tawm cov kev txwv processor los ntawm txhua qhov kev pabcuam uas cuam tshuam ncaj qha lossis tsis ncaj qha rau kev ua haujlwm tseem ceeb rau peb cov neeg siv.
Qhov kev txiav txim siab tsis yooj yim vim tias peb muaj txiaj ntsig zoo rau kev ruaj ntseg ntawm peb pawg. Yav dhau los, peb twb tau sim nrog qhov tsis ruaj khov ntawm peb pawg, thiab tom qab ntawd cov kev pabcuam tau siv ntau yam kev pabcuam thiab ua rau kev ua haujlwm ntawm lawv tag nrho cov node ua haujlwm qeeb. Tam sim no txhua yam txawv me ntsis: peb tau nkag siab meej txog qhov peb xav tau los ntawm peb pawg, nrog rau lub tswv yim zoo rau kev siv cov kev hloov pauv uas tau npaj tseg.
Kev lag luam ntawv xov xwm ntawm qhov teeb meem nyuaj.
Yuav tiv thaiv koj cov nodes li cas thaum kev txwv raug tshem tawm?
Kev cais tawm ntawm cov kev pabcuam "tsis txwv":
Yav dhau los, peb twb pom ib co nodes nkag mus rau hauv ib lub xeev notReady, feem ntau yog vim cov kev pabcuam uas siv ntau yam kev pabcuam.
Peb tau txiav txim siab muab cov kev pabcuam no cais ("labeled") nodes kom lawv tsis cuam tshuam nrog cov kev pabcuam "muaj feem cuam tshuam". Yog li ntawd, los ntawm kev khij qee cov nodes thiab ntxiv qhov kev zam rau cov kev pabcuam "tsis cuam tshuam", peb ua tiav kev tswj hwm ntau dua ntawm pawg, thiab nws tau yooj yim rau peb txheeb xyuas cov teeb meem nrog cov nodes. Txhawm rau ua cov txheej txheem zoo sib xws koj tus kheej, koj tuaj yeem paub koj tus kheej nrog .
Muab qhov tseeb processor thiab nco thov:
Peb qhov kev ntshai loj tshaj plaws yog tias cov txheej txheem yuav siv ntau yam kev pab thiab cov node yuav tsum tsis txhob teb cov lus thov. Txij li tam sim no (ua tsaug rau Datadog) peb tuaj yeem saib xyuas tag nrho cov kev pabcuam hauv peb pawg, kuv tau tshuaj xyuas ob peb lub hlis ntawm kev ua haujlwm ntawm cov uas peb npaj los xaiv "tsis cuam tshuam". Kuv tsuas yog teeb tsa qhov siab tshaj plaws CPU siv nrog cov npoo ntawm 20%, thiab yog li faib qhov chaw nyob rau hauv cov node nyob rau hauv rooj plaub k8s sim muab lwm cov kev pab cuam rau ntawm node.
Raws li koj tuaj yeem pom hauv daim duab, qhov siab tshaj plaws ntawm lub processor tau mus txog 242m CPU cores (0.242 processor cores). Rau qhov kev thov processor, nws yog txaus kom coj tus lej me ntsis loj dua li tus nqi no. Thov nco ntsoov tias txij li cov kev pab cuam yog cov neeg siv-centric, ncov load qhov tseem ceeb coincide nrog tsheb.
Ua tib yam nrog kev siv lub cim xeeb thiab cov lus nug, thiab voila - koj tau teeb tsa tag nrho! Rau kev ruaj ntseg ntau dua, koj tuaj yeem ntxiv kab rov tav pod autoscaling. Yog li, txhua zaus cov peev txheej thauj khoom siab, autoscaling yuav tsim cov pods tshiab, thiab kubernetes yuav faib rau cov nodes nrog qhov chaw dawb. Yog tias tsis muaj chaw seem hauv pawg nws tus kheej, koj tuaj yeem teeb tsa koj tus kheej ceeb toom lossis teeb tsa qhov sib ntxiv ntawm cov nodes tshiab los ntawm lawv qhov autoscaling.
Ntawm cov minuses, nws tsim nyog sau cia tias peb poob hauv "", i.e. tus naj npawb ntawm cov ntim tau khiav ntawm ib qho. Peb kuj tseem yuav muaj "kev so" ntau ntawm cov tsheb khiav ceev, thiab tseem muaj txoj hauv kev uas koj yuav ncav cuag lub siab processor load, tab sis autoscaling nodes yuav tsum pab nrog tom kawg.
Π Π΅Π·ΡΠ»ΡΡΠ°ΡΡ
Kuv zoo siab tshaj tawm cov txiaj ntsig zoo tshaj plaws los ntawm kev sim ob peb lub lis piam dhau los; peb twb pom kev txhim kho tseem ceeb hauv cov lus teb thoob plaws txhua qhov kev pabcuam hloov kho:
Peb ua tiav cov txiaj ntsig zoo tshaj plaws ntawm peb lub vev xaib (), muaj kev pabcuam nrawm hauv nees nkaum ob zaug!
Puas yog Linux kernel kab laum kho?
Yog, kev faib tawm version 4.19 thiab siab dua.
Txawm li cas los xij, thaum nyeem ntawv rau lub Cuaj Hlis 2020 thib ob peb tseem tuaj hla hais txog qee qhov haujlwm Linux nrog cov kab laum zoo sib xws. Kuv ntseeg tias qee qhov kev faib tawm Linux tseem muaj kab mob no thiab tsuas yog ua haujlwm kho nws.
Yog tias koj qhov kev faib tawm qis dua 4.19, kuv xav kom hloov kho mus rau qhov tseeb, tab sis nyob rau hauv txhua rooj plaub koj yuav tsum sim tshem tawm cov kev txwv processor thiab saib seb qhov throttling txuas ntxiv mus. Hauv qab no koj tuaj yeem pom cov npe ib nrab ntawm Kubernetes cov kev pabcuam tswj hwm thiab Linux faib:
- Debian: txhim kho kev koom ua ke rau hauv qhov tseeb version ntawm kev faib tawm, , thiab zoo li tshiab (). Qee cov versions dhau los kuj yuav raug kho.
- Ubuntu: txhim kho kev koom ua ke rau hauv qhov tseeb version
- EKS tseem tau kho dua . Yog tias koj qhov version qis dua qhov no, koj yuav tsum hloov kho AMI.
- kops: Ρ
kops 1.18+Lub ntsiab host duab yuav yog Ubuntu 20.04. Yog tias koj version ntawm kops laus dua, koj yuav tau tos kom kho. Peb tus kheej tab tom tos tam sim no. - GKE (Google Cloud): Txhim kho kev sib koom ua ke , txawm li cas los xij muaj teeb meem nrog throttling .
Yuav ua li cas yog qhov kho tau qhov teeb meem throttling?
Kuv tsis paub tseeb tias qhov teeb meem daws tau tag nrho. Thaum peb tau txais mus rau cov ntsiav version nrog qhov kho, Kuv yuav sim cov pawg thiab hloov kho cov ncej. Yog tias leej twg tau hloov kho, kuv xav nyeem koj cov txiaj ntsig.
xaus
- Yog tias koj ua haujlwm nrog Docker ntim hauv Linux (tsis hais Kubernetes, Mesos, Swarm lossis lwm tus), koj lub thawv yuav poob kev ua haujlwm vim muaj kev cuam tshuam;
- Sim hloov kho mus rau qhov tseeb version ntawm koj qhov kev faib tawm hauv kev cia siab tias kab laum twb tau kho lawm;
- Tshem tawm cov kev txwv processor yuav daws tau qhov teeb meem, tab sis qhov no yog cov txheej txheem txaus ntshai uas yuav tsum tau siv nrog ceev faj heev (nws yog qhov zoo dua los ua ntej hloov cov ntsiav thiab sib piv cov txiaj ntsig);
- Yog tias koj tau tshem CPU txwv, ua tib zoo saib xyuas koj lub CPU thiab kev siv lub cim xeeb thiab xyuas kom tseeb tias koj cov peev txheej CPU muaj ntau tshaj qhov koj noj;
- Ib qho kev xaiv muaj kev nyab xeeb yuav yog autoscale pods los tsim cov pods tshiab nyob rau hauv rooj plaub ntawm kev kho vajtse siab, kom kubernetes muab lawv rau cov nodes dawb.
Kuv vam tias cov ntawv tshaj tawm no yuav pab koj txhim kho kev ua haujlwm ntawm koj lub thawv ntim khoom.
PS tus sau sib raug nrog cov neeg nyeem thiab cov neeg tawm tswv yim (ua lus Askiv).
Tau qhov twg los: www.hab.com