Monga lamulo, nthawi zonse pamakhala kufunikira kopereka dziwe lodzipatulira lazinthu ku pempho la ntchito yake yolondola komanso yokhazikika. Koma bwanji ngati mapulogalamu angapo akugwiritsa ntchito mphamvu yomweyo? Kodi angapereke bwanji aliyense wa iwo ndi osachepera zofunika chuma? Kodi mungachepetse bwanji kugwiritsa ntchito zinthu? Momwe mungagawire katunduyo moyenera pakati pa node? Kodi mungawonetse bwanji kuti njira yolumikizira yopingasa ikugwira ntchito ngati kuchuluka kwa ntchito kukuwonjezeka?
Muyenera kuyamba ndi mitundu iti yazinthu zomwe zilipo mu dongosolo - izi, ndithudi, ndi nthawi ya purosesa ndi RAM. Mu k8s ikuwonetsa mitundu yazinthu izi imayesedwa m'magawo awa:
CPU - mu cores
RAM - mu mabayiti
Komanso, pazithandizo zilizonse ndizotheka kukhazikitsa mitundu iwiri ya zofunika - zopempha ΠΈ Malire. Zopempha - zimafotokoza zofunikira zochepa zazinthu zaulere za node yoyendetsa chidebe (ndi pod yonse), pamene malire amaika malire okhwima pazinthu zomwe zilipo ku chidebecho.
Iwo. Pankhaniyi, kuti muthamangitse chidebe chokhala ndi nginx, mufunika osachepera 1G ya RAM yaulere ndi 0.2 CPU pa node, pomwe chidebecho chimatha kudya 0.2 CPU ndi RAM yonse yomwe ilipo pa mfundo.
Iwo. kuchuluka kwa zopempha zonse zomwe zili mu ns zosasinthika sizingadutse 300m kwa CPU ndi 1G kwa OP, ndipo kuchuluka kwa malire onse ndi 700m kwa CPU ndi 2G kwa OP.
Iwo. m'malo osasinthika a mayina a zotengera zonse, pempho lidzakhazikitsidwa ku 100m kwa CPU ndi 1G ya OP, malire - 1 CPU ndi 2G. Nthawi yomweyo, malire amayikidwanso pazomwe zingatheke popempha / malire a CPU (50m <x <2) ndi RAM (500M <x <4G).
Apa tikuwona ma pod onse akuyenda pa node inayake, komanso zinthu zomwe pod iliyonse imapempha. Ndipo izi ndi zomwe zipika za scheduler zimawoneka ngati cronjob-cron-events-1573793820-xt6q9 pod ikhazikitsidwa (chidziwitsochi chidzawonekera mu chipika cholembera mukakhazikitsa mulingo wa 10 wodula mitengo poyambira -v=10):
chipika
I1115 07:57:21.637791 1 scheduling_queue.go:908] About to try and schedule pod nxs-stage/cronjob-cron-events-1573793820-xt6q9
I1115 07:57:21.637804 1 scheduler.go:453] Attempting to schedule pod: nxs-stage/cronjob-cron-events-1573793820-xt6q9
I1115 07:57:21.638285 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s5 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638300 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s6 is allowed, Node is running only 20 out of 110 Pods.
I1115 07:57:21.638322 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s3 is allowed, Node is running only 20 out of 110 Pods.
I1115 07:57:21.638322 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s4 is allowed, Node is running only 17 out of 110 Pods.
I1115 07:57:21.638334 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s10 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638365 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s12 is allowed, Node is running only 9 out of 110 Pods.
I1115 07:57:21.638334 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s11 is allowed, Node is running only 11 out of 110 Pods.
I1115 07:57:21.638385 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s1 is allowed, Node is running only 19 out of 110 Pods.
I1115 07:57:21.638402 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s2 is allowed, Node is running only 21 out of 110 Pods.
I1115 07:57:21.638383 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s9 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638335 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s8 is allowed, Node is running only 18 out of 110 Pods.
I1115 07:57:21.638408 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s13 is allowed, Node is running only 8 out of 110 Pods.
I1115 07:57:21.638478 1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s10 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638505 1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s8 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638577 1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s9 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638583 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s7 is allowed, Node is running only 25 out of 110 Pods.
I1115 07:57:21.638932 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: BalancedResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 2343 millicores 9640186880 memory bytes, score 9
I1115 07:57:21.638946 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: LeastResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 2343 millicores 9640186880 memory bytes, score 8
I1115 07:57:21.638961 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: BalancedResourceAllocation, capacity 39900 millicores 66620170240 memory bytes, total request 4107 millicores 11307422720 memory bytes, score 9
I1115 07:57:21.638971 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: BalancedResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 5847 millicores 24333637120 memory bytes, score 7
I1115 07:57:21.638975 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: LeastResourceAllocation, capacity 39900 millicores 66620170240 memory bytes, total request 4107 millicores 11307422720 memory bytes, score 8
I1115 07:57:21.638990 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: LeastResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 5847 millicores 24333637120 memory bytes, score 7
I1115 07:57:21.639022 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639030 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639034 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639041 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639053 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639059 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639061 1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639063 1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639073 1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639077 1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639085 1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639088 1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639103 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639109 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639114 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639127 1 generic_scheduler.go:781] Host nxs-k8s-s10 => Score 100037
I1115 07:57:21.639150 1 generic_scheduler.go:781] Host nxs-k8s-s8 => Score 100034
I1115 07:57:21.639154 1 generic_scheduler.go:781] Host nxs-k8s-s9 => Score 100037
I1115 07:57:21.639267 1 scheduler_binder.go:269] AssumePodVolumes for pod "nxs-stage/cronjob-cron-events-1573793820-xt6q9", node "nxs-k8s-s10"
I1115 07:57:21.639286 1 scheduler_binder.go:279] AssumePodVolumes for pod "nxs-stage/cronjob-cron-events-1573793820-xt6q9", node "nxs-k8s-s10": all PVCs bound and nothing to do
I1115 07:57:21.639333 1 factory.go:733] Attempting to bind cronjob-cron-events-1573793820-xt6q9 to nxs-k8s-s10
Apa tikuwona kuti poyambira zosefera za scheduler ndikupanga mndandanda wa node 3 zomwe zitha kukhazikitsidwa (nxs-k8s-s8, nxs-k8s-s9, nxs-k8s-s10). Kenako imawerengera zigoli potengera magawo angapo (kuphatikiza BalancedResourceAllocation, LeastResourceAllocation) pa iliyonse ya nodeyi kuti mudziwe malo oyenera kwambiri. Pamapeto pake, pod imakonzedwa pa mfundo ndi chiwerengero chapamwamba kwambiri (pano mfundo ziwiri nthawi imodzi zimakhala ndi chiwerengero chofanana cha 100037, kotero kuti chisawawa chimasankhidwa - nxs-k8s-s10).
Pomaliza: ngati node ikuyendetsa ma pods omwe palibe zoletsa zomwe zimayikidwa, ndiye kuti k8s (kuchokera kumalo ogwiritsira ntchito zothandizira) izi zidzakhala zofanana ndi ngati panalibe ma pods oterowo pa node iyi nkomwe. Chifukwa chake, ngati inu, mokhazikika, muli ndi poto yosusuka (mwachitsanzo, wowza) ndipo palibe zoletsa zomwe zakhazikitsidwa, ndiye kuti zinthu zitha kuchitika pamene pod iyi idadyadi zida zonse za node, koma kwa k8s mfundo iyi. imaonedwa ngati yotsitsidwa ndipo idzapatsidwa chiwerengero chofanana cha mfundo pamene kusanja (momwemo mu mfundo zowunika zinthu zomwe zilipo) monga mfundo yomwe ilibe ma pods ogwira ntchito, zomwe pamapeto pake zingayambitse kugawa kosagwirizana kwa katundu pakati pa mfundo.
Kuthamangitsidwa kwa Pod
Monga mukudziwa, pod iliyonse imapatsidwa imodzi mwamagulu atatu a QoS:
wotsimikizika - amapatsidwa pamene chidebe chilichonse mu pod pempho ndi malire amatchulidwa kukumbukira ndi cpu, ndipo izi ziyenera kufanana
chophulika - osachepera chidebe chimodzi mu pod ali ndi pempho ndi malire, ndi pempho <malire
khama kwambiri - pomwe palibe chidebe chimodzi chomwe chili m'bokosi chomwe chili ndi malire
Nthawi yomweyo, node ikakumana ndi kusowa kwazinthu (disk, memory), kubelet imayamba kuyika ndikutulutsa ma pods molingana ndi algorithm inayake yomwe imaganizira zoyambira za pod ndi gulu lake la QoS. Mwachitsanzo, ngati tikulankhula za RAM, ndiye kutengera gulu la QoS, mfundo zimaperekedwa molingana ndi mfundo iyi:
Iwo. ndikuyika patsogolo komweko, kubelet iyamba kutulutsa ma pod ndi khama labwino kwambiri la QoS pamfundo.
Pomaliza: ngati mukufuna kuchepetsa mwayi wa pod wofunidwa kuthamangitsidwa kuchokera ku node ngati kusowa kwazinthu pa izo, ndiye pamodzi ndi chofunika kwambiri, muyeneranso kusamalira kukhazikitsa pempho / malire ake.
Njira yopangira ma horizontal autoscaling of application pods (HPA)
Ntchito ikangowonjezera ndikuchepetsa kuchuluka kwa ma pod kutengera kugwiritsa ntchito zinthu (dongosolo - CPU/RAM kapena wosuta - rps), gulu la k8s monga HPA (Horizontal Pod Autoscaler). Algorithm yake ndi iyi:
Iwo. tikuwona kuti poda yogwiritsira ntchito imayambitsidwa muzochitika ziwiri, iliyonse yomwe ili ndi zotengera ziwiri za nginx ndi nginx-exporter, zomwe zimatchulidwa. zopempha za CPU.
Iwo. Tidapanga hpa yomwe imayang'anira kuyesa kwa pulogalamu ya Deployment ndikusintha kuchuluka kwa ma pod ndi pulogalamuyo kutengera chizindikiro cha cpu (tikuyembekeza kuti pod iyenera kudya 30% ya CPU yomwe ikufuna), ndi kuchuluka kwa zofananira zomwe zilimo. kutalika kwa 2-10.
Kuwerengera kuchuluka kwa ma CPU omwe amadya poyerekeza ndi pempho la pempho, i.e. 53m / 90m * 100% = 59%
Tsopano tili ndi zonse zomwe tingafune kuti tiwone ngati tikufunika kusintha kuchuluka kwa zolemba; kuti tichite izi, timawerengera coefficient:
ratio = 59% / 30% = 1.96
Iwo. chiwerengero cha zofananira chiyenera kuchulukitsidwa ndi ~ 2 nthawi ndi kuchuluka kwa [2 * 1.96] = 4.
Kutsiliza: Monga mukuonera, kuti makinawa agwire ntchito, chofunikira ndi kupezeka kwa zopempha zonse zomwe zili mu pod.
Njira yolumikizira ma node opingasa (Cluster Autoscaler)
Kuti muchepetse zovuta zoyipa pamakina panthawi yonyamula katundu, kukhala ndi hpa yokonzedwa sikokwanira. Mwachitsanzo, malinga ndi makonda omwe ali mu hpa controller manager, amasankha kuti chiwerengero cha replicas chiyenera kuchulukitsidwa ndi 2 nthawi, koma ma node alibe zipangizo zaulere zogwiritsira ntchito ma pod angapo (ie, node sangathe kupereka adapempha zothandizira pazopempha) ndi ma pods awa amasinthira ku Pending state.
Pankhaniyi, ngati woperekayo ali ndi IaaS/PaaS yofananira (mwachitsanzo, GKE/GCE, AKS, EKS, etc.), chida ngati Node Autoscaler. Zimakulolani kuti muyike chiwerengero chokwanira komanso chocheperako cha node mumagulu ndikusintha mosavuta chiwerengero cha node (poyitana API wothandizira mtambo kuti ayitanitsa / kuchotsa node) pamene pali kusowa kwazinthu mumagulu ndi ma pod. sizingakonzedwe (zili mu Pending state).
Kutsiliza: Kuti muthe kuwongolera ma node, ndikofunikira kukhazikitsa zopempha muzotengera zapodo kuti ma k8 athe kuwunika moyenera katundu pa node ndipo potero anene kuti palibe zothandizira pagulu kuti muyambitse pod yotsatira.