🥇Malangizo asanu ndi anayi opititsa patsogolo ntchito ya Kubernetes

Moni nonse! Dzina langa ndi Oleg Sidorenkov, ndimagwira ntchito ku DomClick monga mtsogoleri wa gulu la zomangamanga. Takhala tikugwiritsa ntchito Kubik pakupanga kwazaka zopitilira zitatu, ndipo panthawiyi takumana ndi nthawi zosangalatsa zosiyanasiyana nazo. Lero ndikuuzani momwe, ndi njira yoyenera, mutha kufinya magwiridwe antchito ambiri kuchokera ku vanila Kubernetes pagulu lanu. Okonzeka kupitiriza!

Inu nonse mukudziwa bwino kuti Kubernetes ndi scalable open source system for the container orchestration; chabwino, kapena ma binaries 5 omwe amagwira ntchito zamatsenga pakuwongolera moyo wa ma microservices anu pamalo a seva. Kuphatikiza apo, ndi chida chosinthika chomwe chitha kusonkhanitsidwa ngati Lego kuti musinthe makonda osiyanasiyana pantchito zosiyanasiyana.

Ndipo zonse zikuwoneka kuti zili bwino: ponyani ma seva mutsango ngati nkhuni mubokosi lamoto, ndipo simudzadziwa chisoni chilichonse. Koma ngati muli wokonda chilengedwe, mungaganize kuti: “Kodi ndingatani kuti moto uziyaka ndi kupulumutsa nkhalangoyi?” Mwa kuyankhula kwina, momwe mungapezere njira zowonjezera zowonongeka ndi kuchepetsa ndalama.

1. Yang'anirani gulu ndi zothandizira zothandizira

Njira imodzi yodziwika bwino, koma yothandiza ndikuyambitsa zopempha / malire. Gawani mapulogalamu ndi malo a mayina, ndi malo a mayina ndi magulu a chitukuko. Musanatumizidwe, khazikitsani zogwiritsira ntchito kuti mugwiritse ntchito nthawi ya purosesa, kukumbukira, ndi kusungirako kwa ephemeral.

resources:
   requests:
     memory: 2Gi
     cpu: 250m
   limits:
     memory: 4Gi
     cpu: 500m

Kupyolera mu zomwe takumana nazo, tinafika pa mfundo yakuti: musawonjezere zopempha kuchokera ku malire ndi kupitirira kawiri. Voliyumu ya gululo imawerengedwa kutengera zopempha, ndipo ngati mupereka mapulogalamu osiyanasiyana pazothandizira, mwachitsanzo, nthawi 5-10, ndiye lingalirani zomwe zidzachitike ku node yanu ikadzadza ndi ma pod ndikulandira katundu mwadzidzidzi. Palibe chabwino. Pang'ono ndi pang'ono, kugwedeza, komanso pamtunda, mudzatsanzikana ndi wogwira ntchitoyo ndikupeza katundu wozungulira pamanode otsala pambuyo poyambira kusuntha.

Komanso, ndi chithandizo limitranges Pachiyambi, mutha kuyika zofunikira pa chidebecho - chocheperako, chokwera komanso chosasinthika:

➜  ~ kubectl describe limitranges --namespace ops
Name:       limit-range
Namespace:  ops
Type        Resource           Min   Max   Default Request  Default Limit  Max Limit/Request Ratio
----        --------           ---   ---   ---------------  -------------  -----------------------
Container   cpu                50m   10    100m             100m           2
Container   ephemeral-storage  12Mi  8Gi   128Mi            4Gi            -
Container   memory             64Mi  40Gi  128Mi            128Mi          2

Musaiwale kuchepetsa zopezera mayina kuti gulu limodzi lisatengere zonse zomwe zili mgululi:

➜  ~ kubectl describe resourcequotas --namespace ops
Name:                   resource-quota
Namespace:              ops
Resource                Used          Hard
--------                ----          ----
limits.cpu              77250m        80
limits.memory           124814367488  150Gi
pods                    31            45
requests.cpu            53850m        80
requests.memory         75613234944   150Gi
services                26            50
services.loadbalancers  0             0
services.nodeports      0             0

Monga momwe tingawonere kuchokera kukufotokozera resourcequotas, ngati gulu la ops likufuna kuyika ma pods omwe angadye 10 cpu ina, wokonza mapulani sangalole izi ndipo ataya cholakwika:

Error creating: pods "nginx-proxy-9967d8d78-nh4fs" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=5,requests.cpu=5, used: limits.cpu=77250m,requests.cpu=53850m, limited: limits.cpu=10,requests.cpu=10

Kuti muthetse vutoli, mukhoza kulemba chida, mwachitsanzo, ngati izi, wokhoza kusunga ndi kuchita zinthu zoyendetsera malamulo.

2. Sankhani mulingo woyenera kwambiri wapamwamba yosungirako

Apa ndikufuna kukhudza pamutu wa ma voliyumu osalekeza ndi gawo la disk la Kubernetes node za ogwira ntchito. Ndikuyembekeza kuti palibe amene amagwiritsa ntchito "Cube" pa HDD popanga, koma nthawi zina SSD yokhazikika sikhala yokwanira. Tidakumana ndi vuto pomwe zipika zimapha diski chifukwa cha ntchito za I/O, ndipo palibe mayankho ambiri:

Gwiritsani ntchito ma SSD apamwamba kwambiri kapena sinthani ku NVMe (ngati mukuwongolera zida zanu).
Chepetsani kudula mitengo.
Pangani "mwanzeru" kusanja ma pod omwe amagwiririra diski (podAntiAffinity).

Chophimba pamwambapa chikuwonetsa zomwe zimachitika pansi pa nginx-ingress-controller ku diski pamene access_logs loggging imayatsidwa (~ 12 thousand logs/sec). Izi, ndithudi, zingayambitse kuwonongeka kwa mapulogalamu onse pa node iyi.

Ponena za PV, tsoka, sindinayesepo chilichonse mitundu Mawu Okhazikika. Gwiritsani ntchito njira yabwino yomwe ingakukwanireni. Zakale, zakhala zikuchitika m'dziko lathu kuti gawo laling'ono la mautumiki limafuna mavoti a RWX, ndipo kalekale anayamba kugwiritsa ntchito yosungirako NFS pa ntchitoyi. Zotsika mtengo komanso ... zokwanira. Zachidziwikire, ine ndi iye tidadya zoyipa - adalitseni, koma tidaphunzira kuyimba, ndipo mutu wanga sunawawanso. Ndipo ngati n'kotheka, sunthirani ku yosungirako zinthu za S3.

3. Sungani zithunzi zokongoletsedwa bwino

Ndibwino kugwiritsa ntchito zithunzi zokongoletsedwa ndi chidebe kuti Kubernetes azitenga mwachangu ndikuzipanga bwino.

Kukometsedwa kumatanthauza kuti zithunzi:

khalani ndi pulogalamu imodzi yokha kapena gwiritsani ntchito imodzi yokha;
kukula kochepa, chifukwa zithunzi zazikulu zimafalitsidwa moipitsitsa pa intaneti;
kukhala ndi mapeto athanzi ndi okonzeka omwe amalola Kubernetes kuchitapo kanthu pakagwa nthawi yopuma;
gwiritsani ntchito makina ogwiritsira ntchito ziwiya (monga Alpine kapena CoreOS), omwe sagonjetsedwa ndi zolakwika za kasinthidwe;
gwiritsani ntchito masitepe ambiri kuti muthe kugwiritsa ntchito mapulogalamu ophatikizidwa osati magwero omwe akutsagana nawo.

Pali zida ndi ntchito zambiri zomwe zimakupatsani mwayi wowona ndikuwongolera zithunzi pakuwuluka. Ndikofunika kuti nthawi zonse muzisunga nthawi zonse ndikuyesa chitetezo. Chifukwa chake mupeza:

Kuchepetsa kuchuluka kwa netiweki pagulu lonse.
Kuchepetsa nthawi yoyambira nkhokwe.
Kukula kwakung'ono kwa registry yanu yonse ya Docker.

4. Gwiritsani ntchito cache ya DNS

Ngati tilankhula za katundu wambiri, ndiye kuti moyo umakhala wovuta kwambiri popanda kukonza dongosolo la DNS la cluster. Kalekale, opanga Kubernetes adathandizira yankho lawo la kube-dns. Zinagwiritsidwanso ntchito pano, koma pulogalamuyi sinakonzedwe makamaka ndipo sinatulutse ntchito yofunikira, ngakhale kuti inkawoneka ngati ntchito yosavuta. Kenako ma coredns adawonekera, omwe tidasinthira ndipo tinalibe chisoni; pambuyo pake idakhala ntchito yokhazikika ya DNS mu K8s. Panthawi ina, tinakula kufika ku 40 zikwi rps ku dongosolo la DNS, ndipo yankholi linakhalanso losakwanira. Koma, mwamwayi, Nodelocaldns adatuluka, aka node local cache, aka NodeLocal DNSCache.

Chifukwa chiyani timagwiritsa ntchito izi? Pali cholakwika mu Linux kernel kuti, mafoni ambiri akamalumikizana ndi NAT pa UDP, amatsogolera ku mpikisano wolowera pamatebulo olumikizirana, ndipo gawo lina la magalimoto kudzera mu NAT limatayika (ulendo uliwonse kudzera mu Service ndi NAT). Ma Nodelocaldns amathetsa vutoli pochotsa NAT ndikukweza kulumikizana ndi TCP kumtunda kwa DNS, komanso kusungitsa mafunso akumtunda kwa DNS (kuphatikiza kache kakang'ono ka 5-sekondi imodzi).

5. Onjezani makoko mopingasa komanso molunjika basi

Kodi munganene ndi chidaliro kuti ma microservices anu onse ali okonzeka kuchulukitsa kawiri kapena katatu? Momwe mungagawire bwino zothandizira pazofunsira zanu? Kusunga ma pod angapo akupitilira kuchuluka kwa ntchito kumatha kukhala kocheperako, koma kuwasunga kumbuyo kumabweretsa chiwopsezo cha nthawi yocheperako kuchokera pakuwonjezeka kwadzidzidzi kwa magalimoto kupita kuntchito. Services monga Horizontal Pod Autoscaler и Vertical Pod Autoscaler.

VPA amakulolani kuti mukweze zopempha / malire a zotengera zanu mu pod kutengera momwe mungagwiritsire ntchito. Zingakhale zothandiza bwanji? Ngati muli ndi ma pods omwe sangathe kuwongoleredwa mopingasa pazifukwa zina (zomwe sizodalirika kwenikweni), mutha kuyesa kuyika zosintha kuzinthu zake ku VPA. Mawonekedwe ake ndi njira yolimbikitsira yotengera mbiri yakale komanso zamakono kuchokera pa seva ya metric, kotero ngati simukufuna kusintha zopempha / malire, mutha kungoyang'anira zomwe mwapereka pazotengera zanu ndikuwongolera zosintha kuti musunge CPU ndi kukumbukira mu gulu.

Chithunzi chojambulidwa kuchokera ku https://levelup.gitconnected.com/kubernetes-autoscaling-101-cluster-autoscaler-horizontal-pod-autoscaler-and-vertical-pod-2a441d9ad231

Wokonza ku Kubernetes nthawi zonse amatengera zopempha. Chilichonse chomwe mungaike pamenepo, wokonza mapulani amafufuza node yoyenera kutengerapo. Malire amafunikira kuti cubelet imvetsetse nthawi yopumira kapena kupha pod. Ndipo popeza gawo lokhalo lofunikira ndilofunika zopempha, VPA idzagwira ntchito nayo. Nthawi zonse mukakulitsa pulogalamu molunjika, mumatanthauzira zomwe zopemphazo ziyenera kukhala. Nanga malirewo adzatani? Parameter iyi idzakulitsidwanso molingana.

Mwachitsanzo, nayi zokonda za pod mwachizolowezi:

resources:
   requests:
     memory: 250Mi
     cpu: 200m
   limits:
     memory: 500Mi
     cpu: 350m

Injini yotsimikizira imatsimikizira kuti ntchito yanu imafuna 300m CPU ndi 500Mi kuti iyende bwino. Mupeza makonda awa:

resources:
   requests:
     memory: 500Mi
     cpu: 300m
   limits:
     memory: 1000Mi
     cpu: 525m

Monga tafotokozera pamwambapa, uku ndikukweza molingana kutengera kuchuluka kwa zopempha/malire mu chiwonetserochi:

CPU: 200m → 300m: chiŵerengero cha 1:1.75;
Memory: 250Mi → 500Mi: chiŵerengero 1:2.

chokhudza HPA, ndiye kuti njira yogwirira ntchito imakhala yowonekera. Ma metrics monga CPU ndi kukumbukira ndizocheperapo, ndipo ngati avareji ya zofananira zonse zipitilira malire, ntchitoyo imakulitsidwa ndi +1 sub mpaka mtengo utsike pachimake kapena mpaka kuchuluka kwa zofananira kufikire.

Chithunzi chojambulidwa kuchokera ku https://levelup.gitconnected.com/kubernetes-autoscaling-101-cluster-autoscaler-horizontal-pod-autoscaler-and-vertical-pod-2a441d9ad231

Kuphatikiza pa ma metric wamba ngati CPU ndi kukumbukira, mutha kuyika malire pamiyezo yanu yochokera ku Prometheus ndikugwira nawo ntchito ngati mukuganiza kuti ndiye chizindikiro cholondola kwambiri cha nthawi yoti muwonjezere ntchito yanu. Ntchito ikakhazikika pansi pamlingo womwe watchulidwa, HPA iyamba kutsitsa mpaka kuchuluka kwa zofananira kapena mpaka katunduyo akwaniritse malire omwe atchulidwa.

6. Musaiwale za Node Affinity ndi Pod Affinity

Sikuti ma node onse amayendera pa hardware yomweyo, ndipo si ma pod onse omwe amafunikira kugwiritsa ntchito makompyuta. Kubernetes imakupatsani mwayi wokhazikitsa ma node ndi ma pod pogwiritsa ntchito Node Affinity и Pod Affinity.

Ngati muli ndi ma node omwe ali oyenera kugwira ntchito mozama kwambiri, ndiye kuti kuti mugwire bwino ntchito ndi bwino kumangiriza mapulogalamu ku ma node ofanana. Kuchita izi ntchito nodeSelector ndi chizindikiro cha node.

Tiyerekeze kuti muli ndi mfundo ziwiri: imodzi ndi CPUType=HIGHFREQ ndi ambiri othamanga mitima, wina ndi MemoryType=HIGHMEMORY kukumbukira zambiri ndikuchita mwachangu. Njira yosavuta ndikugawira kutumizidwa ku node HIGHFREQpowonjezera ku gawolo spec selector monga chonchi:

…
nodeSelector:
	CPUType: HIGHFREQ

Njira yotsika mtengo komanso yeniyeni yochitira izi ndikugwiritsa ntchito nodeAffinity m'munda affinity gawo spec. Pali njira ziwiri:

requiredDuringSchedulingIgnoredDuringExecution: kukhazikitsa kolimba (wokonza mapulani adzatumiza ma pods okha pa mfundo zina (ndi kwina kulikonse));
preferredDuringSchedulingIgnoredDuringExecution: kukhazikitsa kofewa (wokonza ndondomekoyo ayesa kuyika ku mfundo zinazake, ndipo ngati izo zitalephera, zidzayesa kutumiza ku node yotsatira yomwe ilipo).

Mutha kufotokozera mawu omveka bwino pakuwongolera zolemba za node, monga In, NotIn, Exists, DoesNotExist, Gt kapena Lt. Komabe, kumbukirani kuti njira zovuta m'mindandanda yayitali ya zilembo zimachepetsa kupanga zisankho pakagwa zovuta. M'mawu ena, khalani osavuta.

Monga tafotokozera pamwambapa, Kubernetes amakulolani kuti muyike kuyanjana kwa ma pod apano. Ndiko kuti, mutha kuwonetsetsa kuti ma pod ena amagwirira ntchito limodzi ndi ma pod ena omwe ali pamalo omwe amapezeka (oyenera mitambo) kapena node.

В podAffinity minda affinity gawo spec Minda yemweyo zilipo monga mu nkhani ya nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution и preferredDuringSchedulingIgnoredDuringExecution. Kusiyana kokha ndiko matchExpressions adzamanga makoko ku mfundo yomwe ikuyendetsa kale poto yokhala ndi chizindikirocho.

Kubernetes imaperekanso gawo podAntiAffinity, zomwe, mosiyana, sizimangirira pod ku node ndi ma pods enieni.

Za mawu nodeAffinity Malangizo omwewo angaperekedwe: yesetsani kusunga malamulo osavuta komanso omveka, musayese kudzaza ndondomeko ya pod ndi malamulo ovuta. Ndizosavuta kupanga lamulo lomwe silingafanane ndi zikhalidwe za gululo, kupanga katundu wosafunika pa scheduler ndikuchepetsa magwiridwe antchito onse.

7. Zowonongeka & Kulekerera

Palinso njira ina yoyendetsera ndondomeko. Ngati muli ndi gulu lalikulu lokhala ndi mazana a node ndi masauzande a microservices, ndiye kuti n'zovuta kwambiri kuti musalole kuti ma pods ena azikhala nawo pamagulu ena.

Njira zowononga - kuletsa malamulo - zimathandizira izi. Mwachitsanzo, muzochitika zina mutha kuletsa ma node ena kuyendetsa ma pod. Kuti mugwiritse ntchito dothi ku node inayake muyenera kugwiritsa ntchito njirayo taint mu kubectl. Tchulani fungulo ndi mtengo ndiyeno sinthani ngati NoSchedule kapena NoExecute:

$ kubectl taint nodes node10 node-role.kubernetes.io/ingress=true:NoSchedule

Ndikoyeneranso kudziwa kuti makina otayira amathandizira zotsatira zazikulu zitatu: NoSchedule, NoExecute и PreferNoSchedule.

NoSchedule zikutanthauza kuti pakadali pano sipadzakhalanso cholowa chofananira pamatchulidwe a pod tolerations, sichitha kuyikidwa pa node (muchitsanzo ichi node10).
PreferNoSchedule - mtundu wosavuta NoSchedule. Pankhaniyi, wokonza mapulani ayesa kuti asagawire ma pod omwe alibe cholowa chofananira tolerations pa node, koma izi sizovuta. Ngati palibe zothandizira mgululi, ndiye kuti ma pod ayamba kuyika pa node iyi.
NoExecute - izi zimayambitsa kuthamangitsidwa kwaposachedwa kwa ma pods omwe alibe cholowa chofananira tolerations.

Chosangalatsa ndichakuti, izi zitha kuthetsedwa pogwiritsa ntchito njira yololera. Izi ndizosavuta ngati pali node "yoletsedwa" ndipo mumangofunika kuyikapo ntchito zachitukuko. Kodi kuchita izo? Lolani ma pod okhawo omwe ali ndi kulolerana koyenera.

Izi ndi zomwe ma pod angawonekere:

spec:
   tolerations:
     - key: "node-role.kubernetes.io/ingress"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"

Izi sizikutanthauza kuti kukonzanso kotsatira kudzagwera pa mfundo iyi, iyi si njira ya Node Affinity ndi nodeSelector. Koma kuphatikiza zinthu zingapo, mutha kukwaniritsa makonda osinthika kwambiri.

8. Khazikitsani Podi Kutumiza Patsogolo

Chifukwa chakuti muli ndi ma pods omwe amaperekedwa ku node sizikutanthauza kuti ma pod onse ayenera kuchitidwa mofanana. Mwachitsanzo, mungafunike kuyika ma pods ena asanakhale ena.

Kubernetes imapereka njira zosiyanasiyana zosinthira Pod Priority and Preemption. Kukonzekera kumakhala ndi magawo angapo: chinthu PriorityClass ndi mafotokozedwe a munda priorityClassName mu mawonekedwe a pod. Tiyeni tione chitsanzo:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 99999
globalDefault: false
description: "This priority class should be used for very important pods only"

Timalenga PriorityClass, perekani dzina, kufotokozera ndi mtengo. Kukwera value, m'pamenenso amaika patsogolo kwambiri. Mtengo ukhoza kukhala 32-bit integer kuchepera kapena wofanana ndi 1. Makhalidwe apamwamba amasungidwa pamitu yofunikira kwambiri yomwe nthawi zambiri sangayesedwe. Kusamuka kudzachitika kokha ngati pod yapamwamba kwambiri ilibe malo ozungulira, ndiye kuti zina za pods zochokera kumalo ena zidzachotsedwa. Ngati makinawa ndi okhwima kwambiri kwa inu, mutha kuwonjezera njirayo preemptionPolicy: Never, ndiyeno sipadzakhala kukhululukidwa, pod idzayima poyamba pamzere ndikudikirira wokonza kuti apeze zothandizira zaulere.

Kenaka, timapanga pod momwe timasonyezera dzina priorityClassName:

apiVersion: v1
kind: Pod
metadata:
  name: static-web
  labels:
    role: myrole
 spec:
  containers:
    - name: web
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP
  priorityClassName: high-priority

Mutha kupanga makalasi ofunikira monga momwe mukufunira, ngakhale tikulimbikitsidwa kuti musatengeke ndi izi (nenani, dzichepetseni kukhala otsika, apakati komanso apamwamba).

Chifukwa chake, ngati kuli kofunikira, mutha kukulitsa luso la kutumiza ntchito zofunikira monga nginx-ingress-controller, coredns, etc.

9. Konzani gulu la ETCD

ETCD imatha kutchedwa ubongo wa gulu lonse. Ndikofunikira kwambiri kusungitsa ntchito ya database iyi pamlingo wapamwamba, popeza kuthamanga kwa ntchito mu Cube kumadalira. Muyezo wachilungamo, ndipo nthawi yomweyo, yankho labwino lingakhale kusunga gulu la ETCD pazida zazikulu kuti muchepetse kuchedwa kwa kube-apiserver. Ngati simungathe kuchita izi, ikani ETCD pafupi ndi momwe mungathere, ndi bandwidth yabwino pakati pa otenga nawo mbali. Komanso samalani kuti ndi ma node angati ochokera ku ETCD omwe angagwe popanda kuvulaza gululo

Kumbukirani kuti kuchulukirachulukira kwa mamembala omwe ali mgululi kumatha kukulitsa kulolerana kwa zolakwika ndikuwononga magwiridwe antchito, zonse ziyenera kukhala zolimbitsa thupi.

Ngati tikulankhula za kukhazikitsa ntchito, pali malingaliro ochepa:

Khalani ndi zida zabwino, kutengera kukula kwa tsango (mutha kuwerenga apa).
Sinthani magawo angapo ngati mwafalitsa gulu pakati pa ma DC awiri kapena netiweki yanu ndi ma disks kusiya zambiri zomwe mukufuna (mutha kuwerenga apa).

Pomaliza

Nkhaniyi ikufotokoza mfundo zimene gulu lathu limayesetsa kutsatira. Uku sikungofotokozera pang'onopang'ono zochita, koma zosankha zomwe zitha kukhala zothandiza pakuwongolera masango. Zikuwonekeratu kuti gulu lirilonse liri lapadera mwa njira yakeyake, ndipo mayankho amasinthidwe amatha kusiyana kwambiri, kotero zingakhale zosangalatsa kupeza malingaliro anu momwe mumawonera gulu lanu la Kubernetes ndi momwe mumasinthira magwiridwe ake. Gawani zomwe mwakumana nazo mu ndemanga, zidzakhala zosangalatsa kudziwa.

Source: www.habr.com

Malangizo asanu ndi anayi a Kubernetes Performance