🥇10 zvikanganiso zvakajairika paunenge uchishandisa Kubernetes

Cherechedza. transl.: Vanyori vechinyorwa ichi mainjiniya kubva kudiki kambani yeCzech, pipetail. Vakakwanisa kuisa pamwe chete runyoro runoshamisa rwe [dzimwe nguva banal, asi zvakadaro] zvakanyanya kudzvanya matambudziko uye zvisizvo zvine chekuita nekushanda kweKubernetes masumbu.

Kwemakore ekushandisa Kubernetes, takashanda nenhamba huru yemasumbu (ese akatarisirwa uye asingatarisirwe - paGCP, AWS neAzure). Nokufamba kwenguva, takatanga kuona kuti dzimwe mhosho dzaigara dzichidzokororwa. Nekudaro, hapana kunyara mune izvi: isu takaita mazhinji acho isu pachedu!

Chinyorwa chacho chine zvikanganiso zvinowanzoitika uye zvakare chinodudza nzira yekuzvigadzirisa.

1. Zvishandiso: zvikumbiro nemiganhu

Ichi chinhu chinokodzera kunyatsotariswa uye nzvimbo yekutanga pane iyo rondedzero.

CPU chikumbiro kazhinji zvimwe zvisina kutaurwa zvachose kana kuti ine kukosha kwakaderera (kuisa mapodhi akawanda pane imwe neimwe node sezvinobvira). Nokudaro, node dzinova dzakawandisa. Munguva dzemutoro wakakwira, simba rekugadzirisa node rinoshandiswa zvizere uye rimwe basa rinogashira chete izvo "zvayakakumbira" CPU throttling. Izvi zvinotungamira kune yakawedzera application latency, nguva yekubuda, uye zvimwe zvisingafadzi mhedzisiro. (Verenga zvakawanda pamusoro peizvi mune imwe shanduro yedu yazvino: “CPU miganho uye hutsinye kurira muKubernetes"- approx. transl.)

BestEffort (zvakanyanya kwete zvinokurudzirwa):

resources: {}

Yakanyanya kuderera CPU chikumbiro (zvakanyanya kwete zvinokurudzirwa):

   resources:
      Requests:
        cpu: "1m"

Kune rimwe divi, kuvapo kweCPU muganho kunogona kutungamira kune zvisina musoro kusvetuka kwewachi kutenderera nemapods, kunyangwe iyo node processor isina kuzara zvizere. Zvakare, izvi zvinogona kukonzera kunonoka kunowedzera. Makakatanwa anoenderera mberi akatenderedza parameter CPU CFS chikamu muLinux kernel uye CPU throttling zvichienderana nemiganhu yakatarwa, pamwe nekudzivisa chikamu cheCFS ... Maiwe, miganhu yeCPU inogona kukonzera matambudziko akawanda kupfuura anogona kugadzirisa. Rumwe ruzivo nezve izvi runogona kuwanikwa pane iyi link iripazasi.

Kunyanya kusarudzwa (kupfuura) zvinetso zvendangariro zvinogona kutungamirira kuzvinetso zvikuru. Kusvika pamuganho weCPU kunosanganisira kusvetuka mawachi, ukuwo kusvika pamuganho wekurangarira kunosanganisira kuuraya pod. Wakambocherechedza here OOMkill? Hongu, ndizvo chaizvo zvatiri kutaura nezvazvo.

Iwe unoda kuderedza mukana wekuti izvi zviitike here? Usawedzere-kugovera ndangariro uye shandisa Guaranteed QoS (Hunhu hweSevhisi) nekuisa chikumbiro chekurangarira kusvika pamuganho (semumuenzaniso uri pazasi). Verenga zvakawanda pamusoro peizvi mu Henning Jacobs mharidzo (Lead Engineer kuZalando).

Burstable (mukana mukuru wekuwana OOMkilled):

   resources:
      requests:
        memory: "128Mi"
        cpu: "500m"
      limits:
        memory: "256Mi"
        cpu: 2

Wakavimbiswa:

   resources:
      requests:
        memory: "128Mi"
        cpu: 2
      limits:
        memory: "128Mi"
        cpu: 2

Chii chingangobatsira pakugadzira zviwanikwa?

Nekubatsirwa kwe metrics-server iwe unogona kuona yazvino CPU sosi yekushandisa uye ndangariro kushandiswa nemapods (uye midziyo mukati mawo). Zvingangodaro, uri kutoishandisa. Ingomhanya mirairo inotevera:

kubectl top pods
kubectl top pods --containers
kubectl top nodes

Zvisinei, vanongoratidza kushandiswa kwemazuva ano. Inogona kukupa iwe pfungwa yakaoma yehurongwa hwehukuru, asi pakupedzisira iwe unozoda nhoroondo yekuchinja kwemetrics nekufamba kwenguva (kupindura mibvunzo yakaita senge: "Chii chaive chepamusoro CPU mutoro?", "Chii chaive mutoro nezuro mangwanani?", nezvimwewo). Nokuda kweizvi unogona kushandisa Prometheus, DataDog nezvimwe zvishandiso. Ivo vanongotora metrics kubva metrics-server voichengeta, uye mushandisi anogona kuvabvunza uye kuronga ivo zvinoenderana.

VerticalPodAutoscaler Kunoitawo chinja nzira iyi. Inoteedzera CPU uye nhoroondo yekushandisa ndangariro uye inomisikidza zvikumbiro zvitsva uye miganhu zvichienderana neruzivo urwu.

Kushandisa simba rekombuta zvakanaka harisi basa riri nyore. Zvakafanana nekutamba Tetris nguva dzese. Kana iwe uchibhadhara yakawandisa kune compute power ine yakaderera avhareji yekushandisa (taura ~ 10%), tinokurudzira kutarisa zvigadzirwa zvinoenderana neAWS Fargate kana Virtual Kubelet. Iwo akavakirwa pane serverless/pay-per-useji yekubhadharisa modhi, iyo inogona kuve yakachipa mumamiriro akadaro.

2. Upenyu uye kugadzirira kuongorora

Nekumisikidza, kurarama uye kugadzirira cheki hakugoneswe muKubernetes. Uye dzimwe nguva vanokanganwa kuvabatidza...

Asi ndeipi imwe nzira yaungatanga sevhisi kutangazve kana paine chikanganiso chinouraya? Uye muyeri wemutoro anoziva sei kuti pod yakagadzirira kugamuchira traffic? Kana kuti inogona kubata traffic yakawanda?

Iyi miedzo inowanzo vhiringika kune imwe neimwe:

Hupenyu - "kupona" cheki, iyo inotangazve pod kana ikakundikana;
Kugadzirira - Kugadzirira kutarisa, kana ikatadza, inobvisa iyo pod kubva kuKubernetes sevhisi (izvi zvinogona kutariswa uchishandisa kubectl get endpoints) uye traffic haisviki kwairi kusvika cheki inotevera yapera zvinobudirira.

Macheki ese aya ZVINOITWA PANGUVA YOSE HUPENYU HONDO YEPOD. Chinokosha zvikuru.

Imwe pfungwa isiriyo ndeyekuti kugadzirira probes inongoitwa pakutanga kuitira kuti muenzanisi azive kuti pod yagadzirira (Ready) uye inogona kutanga kugadzirisa traffic. Zvisinei, iyi ndiyo imwe chete yezvisarudzo zvekushandisa kwavo.

Imwe mukana wekuona kuti traffic pane pod yakawandisa uye inoremedza (kana iyo podhi inoita masvomhu-akanyanya kuverenga). Muchiitiko ichi, kutarisa kwekugadzirira kunobatsira kuderedza mutoro pane podhi uye "inotonhorera" iyo. Kubudirira kupedzisa cheki yekugadzirira mune ramangwana kunobvumira wedzera mutoro pane podhi zvakare. Muchiitiko ichi (kana bvunzo yekugadzirira ikatadza), kukundikana kweiyo liveness bvunzo kungave kunopesana. Sei uchitangazve pod ine hutano uye uchishanda nesimba?

Naizvozvo, mune zvimwe zviitiko, hapana cheki zvachose zviri nani pane kuvagonesa nemaparameter asina kurongeka. Sezvakataurwa pamusoro apa, kana liveness cheki makopi kugadzirira cheki, ipapo muri mudambudziko guru. Possible sarudzo ndeyekugadzirisa kugadzirira test cheteuye hupenyu hune ngozi siya parutivi.

Marudzi ese ari maviri echeki haafanire kukundikana kana zvakajairika zvinotsamira zvikatadza, zvikasadaro izvi zvinozotungamira kune cascading (avalanche-like) kutadza kwemapodhi ese. Nemamwe mashoko, usazvikuvadza.

3. LoadBalancer kune yega yega sevhisi yeHTTP

Zvingangodaro, iwe une masevhisi eHTTP musumbu rako raungade kuendesa kune yekunze.

Kana iwe ukavhura sevhisi se type: LoadBalancer, mutongi wayo (zvichienderana nemupi webasa) achapa uye kutaurirana LoadBalancer yekunze (isingarevi kuti inoshanda paL7, asi kunyange paL4), uye izvi zvinogona kukanganisa mutengo (kunze static IPv4 kero, simba rekombiyuta, kubhadhara kwesekondi imwe neimwe. ) nekuda kwekudiwa kwekugadzira nhamba huru yezvinhu zvakadaro.

Mune ino kesi, zvine musoro zvakanyanya kushandisa imwe yekunze mitoro yekuenzanisa, kuvhura masevhisi se type: NodePort. Kana zviri nani zvakadaro, wedzera chimwe chinhu chakadai nginx-ingress-controller (kana traefik), achava mumwe chete NodePort endpoint yakabatana neyekunze mutoro wekuenzanisa uye ichafambisa traffic musumbu uchishandisa ingress-Kubernetes zviwanikwa.

Mamwe masevhisi e-intra-cluster (micro) anodyidzana anogona "kutaurirana" vachishandisa masevhisi senge. ClusterIP uye yakavakirwa-mukati sevhisi yekutsvaga nzira kuburikidza neDNS. Ingo usashandise yavo yeruzhinji DNS/IP, sezvo izvi zvichigona kukanganisa latency uye kuwedzera mutengo wemafu masevhisi.

4. Autoscaling a cluster pasina kufunga nezvayo maitiro

Kana uchiwedzera node kune uye nekuabvisa kubva musumbu, haufanirwe kuvimba nemamwe ma metrics ekutanga sekushandiswa kweCPU pane idzo node. Podhi kuronga kunofanirwa kufunga nezvakawanda zvirambidzo, senge pod/node affinity, tsvina uye kushivirira, zvikumbiro zvekushandisa, QoS, nezvimwe. Kushandisa yekunze autoscaler iyo isingatore aya nuances muaccount inogona kuunza matambudziko.

Fungidzira kuti imwe pod inofanira kurongwa, asi ese aripo CPU simba anokumbirwa / kupatsanurwa uye pod. anonamira muchimiro Pending. Yekunze autoscaler inoona avhareji yazvino CPU mutoro (kwete iyo yakakumbirwa) uye haisi kutanga kuwedzera. (chiyero-kunze) - haiwedzere imwe node. Nekuda kweizvozvo, iyi podhi haizorongwa.

Muchiitiko ichi, reverse scaling (chikero-mukati) - kubvisa node kubva musumbu kunogara kwakaoma kuita. Fungidzira kuti une podhi ine hunyanzvi (ine inoenderera chengetedzo yakabatana). Mavhoriyamu anopfuurira kazhinji ndezva chaiyo inowanikwa nzvimbo uye hazvidzokororwe munharaunda. Saka, kana autoscaler yekunze ikadzima node neiyi podhi, mugadziri haakwanisi kuronga iyi pod pane imwe node, sezvo izvi zvichigona kuitwa chete munzvimbo inowanikwa iyo inoenderera kuchengetedza iripo. Pod ichanamira muhurumende Pending.

Yakanyanya kufarirwa munharaunda yeKubernetes cluster-autoscaler. Iyo inomhanya pane sumbu, inotsigira APIs kubva kune makuru gore vanopa, inofunga nezvese zvirambidzo uye inogona kukwira mumatambudziko ari pamusoro. Iyo zvakare inokwanisa kukwira-mukati uchichengetedza ese akatemerwa miganho, nekudaro ichichengetedza mari (iyo yaizoshandiswa pane isina kushandiswa).

5. Kuregeredza IAM / RBAC kugona

Chenjerera kushandisa vashandisi veIAM vane zvakavanzika zvakavanzika zve michina uye maapplication. Ronga kupinda kwenguva pfupi uchishandisa mabasa uye maakaundi ebasa (akaundi yebasa).

Isu tinowanzo sangana nenyaya yekuti makiyi ekuwana (uye zvakavanzika) akaomeswa mukugadziriswa kwekushandisa, pamwe nekuregeredza kutenderera kwezvakavanzika kunyangwe kuwana Cloud IAM. Shandisa mabasa eIAM nemaakaundi esevhisi pachinzvimbo chevashandisi pazvinenge zvakakodzera.

Kanganwa nezve kube2iam uye enda wakananga kuIAM mabasa emaakaundi ebasa (sezvinotsanangurwa mu chinyorwa chezita rimwe chete Štěpán Vraný):

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role
  name: my-serviceaccount
  namespace: default

Imwe tsananguro. Kwete izvozvo zvakaoma, handiti?

Zvakare, usape masevhisi maakaundi uye semuenzaniso profiles ropafadzo admin и cluster-adminkana vasingachida. Izvi zvakati omei kuita, kunyanya muRBAC K8s, asi zvechokwadi zvakakodzera kuedza.

6. Usavimbe ne otomatiki anti-affinity yemapods

Fungidzira kuti une matatu replicas eimwe deployment pane node. Iyo node inodonha, uye pamwe chete nayo yese replicas. Mamiriro asingafadzi, handiti? Asi nei mifananidzo yose yakanga iri panzvimbo imwe chete? Ko Kubernetes haifanirwe kupa kuwanikwa kwepamusoro (HA)?!

Nehurombo, iyo Kubernetes scheduler, pachayo yega, haienderane nemitemo yekuvapo kwakasiyana (anti-affinity) zvemapods. Vanofanira kutaurwa zvakajeka:

// опущено для краткости
      labels:
        app: zk
// опущено для краткости
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"

Ndizvo zvose. Iye zvino mapodhi acharongerwa pane dzakasiyana node (ichi mamiriro anotariswa chete panguva yekuronga, asi kwete panguva yekushanda kwavo - saka requiredDuringSchedulingIgnoredDuringExecution).

Pano tiri kutaura nezvazvo podAntiAffinity pamanode akasiyana: topologyKey: "kubernetes.io/hostname", - uye kwete nezve nzvimbo dzakasiyana dzekuwanikwa. Kuti uite yakazara-yakazara HA, unozofanirwa kuchera zvakadzama muchinyorwa ichi.

7. Kuregeredza PodDisruptionBudgets

Fungidzira kuti une mutoro wekugadzira pane Kubernetes cluster. Nguva nenguva, node uye cluster pachayo inofanirwa kuvandudzwa (kana kubviswa). PodDisruptionBudget (PDB) chimwe chinhu chakaita sechibvumirano chevimbiso yebasa pakati pevatariri vemasumbu nevashandisi.

PDB inokutendera kuti udzivise kukanganisa kwesevhisi kunokonzerwa nekushaikwa kwemanodhi:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: zookeeper

Mumuenzaniso uyu, iwe, semushandisi wechikwata ichi, taura kune maadmins: "Hei, ndine sevhisi yezvekuchengetedza mhuka, uye zvisinei nezvaunoita, ndinoda kuve neanokwana maviri ezvesevhisi aya anogara aripo."

Unogona kuverenga zvakawanda pamusoro peizvi pano.

8. Vashandisi vakawanda kana nharaunda muboka rakafanana

Kubernetes namespaces (mazita) usape simba rekudzivirira.

Pfungwa isiriyo yakajairika ndeyekuti kana iwe ukatumira isiri-prod mutoro mune imwe namespace uye prod mutoro mune imwe, ipapo ivo. hazvizofurirane neimwe nzira... Zvisineyi, imwe nhanho yekuzviparadzanisa nevamwe inogona kuwanikwa uchishandisa zvikumbiro zvezviwanikwa/kuganhurirwa, kuseta quotas, uye kuseta zvakakoshaMakirasi. Kuzviparadzanisa kwe "muviri" mundege yedata kunopihwa nehukama, kushivirira, tsvina (kana nodeselectors), asi kupatsanurwa kwakadaro kwakanyanya. zvakaoma shandisa.

Avo vanoda kubatanidza marudzi maviri emabasa akawanda muboka rimwe chete vachafanira kutarisana nekuoma. Kana pasina kudiwa kwakadaro, uye iwe unogona kuwana imwe rimwe sumbu rimwe (taura, mugore reruzhinji), saka zviri nani kuzviita. Izvi zvichawana yakanyanya nhanho yekudzivirira.

9. externalTrafficPolicy: Cluster

Kazhinji tinoona kuti traffic yese mukati meboka inouya kuburikidza nesevhisi senge NodePort, iyo iyo yakasarudzika mutemo inoisirwa. externalTrafficPolicy: Cluster... Zvinoreva kuti NodePort yakavhurika pane yega node musumbu, uye unogona kushandisa chero yadzo kupindirana nebasa raunoda (seti yemapods).

Panguva imwecheteyo, mapodhi chaiwo akabatana neataurwa pamusoro apa NodePort sevhisi anowanzo kuwanikwa pane imwe chete subset yemanode aya. Mune mamwe mazwi, kana ndikabatanidza kune node isina iyo pod inodiwa, inoendesa traffic kune imwe node, kuwedzera hop uye kuwedzera latency (kana nodes dziri munzvimbo dzakasiyana-siyana dzinowanikwa / data centers, iyo latency inogona kunge yakakwirira kwazvo; uyezve, egress traffic traffic ichawedzera).

Kune rimwe divi, kana imwe Kubernetes sevhisi ine mutemo wakaiswa externalTrafficPolicy: Local, ipapo NodePort inovhura chete pane idzo node uko mapodhi anodiwa ari kunyatso mhanya. Paunenge uchishandisa kunze kwemutoro balancer iyo inotarisa nyika (healthchecking) endpoints (zvinoita sei AWS ELB), Iye inotumira traffic chete kune inodiwa node, iyo ichava nemigumisiro inobatsira pakunonoka, computing zvido, egress mabhiri (uye pfungwa dzinoziva dzinorayira zvakafanana).

Pane mukana wakakura wekuti uri kutoshandisa chinhu chakadai traefik kana nginx-ingress-controller seNodePort endpoint (kana LoadBalancer, inoshandisawo NodePort) kufambisa HTTP ingress traffic, uye kuseta iyi sarudzo kunogona kuderedza zvakanyanya latency yezvikumbiro zvakadaro.

В chinyorwa ichi Iwe unogona kudzidza zvakawanda nezve ekunzeTrafficPolicy, zvakanakira uye zvazvakaipira.

10. Usasungirirwa kumasumbu uye usashandise zvisina kunaka ndege inodzora

Pakutanga, yaive tsika yekudaidza maseva nemazita chaiwo: Anton, HAL9000 uye Colossus... Nhasi vakatsiviwa nezviziviso zvakangogadzirwa. Zvisinei, tsika yacho yakaramba iripo, uye zvino mazita akakodzera anoenda kumasumbu.

Nyaya yakajairika (yakavakirwa pazviitiko chaizvo): zvese zvakatanga nehumbowo hwepfungwa, saka sumbu racho raive nezita rekudada. yokuedzwa… Makore apfuura uye ACHIRI kushandiswa mukugadzira, uye munhu wese ari kutya kuibata.

Hapana chinonakidza pakushanduka kwemapoka kuita mhuka dzinovaraidza, saka tinokurudzira kudzibvisa nguva nenguva uchidzidzira kupora kwenjodzi (izvi zvichabatsira Chaos engineering - approx. transl.). Mukuwedzera, hazvingakuvadzi kushanda pane iyo control layer (ndege yekudzora). Kutya kumubata hachisi chiratidzo chakanaka. Etc. akafa? Varume, muri munyatwa zvamazvirokwazvo!

Kune rimwe divi, haufanirwe kutorwa moyo nekuishandisa. Nenguva iyo control layer inogona kunonoka. Zvingangodaro, izvi zvinokonzerwa nenhamba huru yezvinhu kugadzirwa pasina kutenderera kwavo (yakajairika mamiriro kana uchishandisa Helm ine default marongero, ndosaka mamiriro ayo mune configmaps/zvakavanzika asina kuvandudzwa - semhedzisiro, zviuru zvezvinhu zvinoungana mukati. iyo control layer) kana nekugara kugadziridzwa kwezvinhu kube-api (yeotomatiki kuyera, yeCI/CD, yekutarisa, matanda echiitiko, controllers, nezvimwewo).

Uye zvakare, isu tinokurudzira kutarisa zvibvumirano zveSLA/SLO neanogadziriswa Kubernetes mupi uye kuterera kune garandi. Mutengesi anogona kuvimbisa control layer kuwanikwa (kana zvikamu zvayo), asi kwete kunonoka kwep99 kwezvikumbiro zvaunotumira kwairi. Mune mamwe mazwi, unogona kupinda kubectl get nodes, uye ugamuchire mhinduro chete mushure memaminitsi gumi, uye izvi hazvizove kutyora kwemitemo yechibvumirano chebasa.

11. Bonus: kushandisa ichangoburwa tag

Asi ichi chatova classic. Nguva pfupi yadarika takasangana nehunyanzvi uhu kazhinji, sezvo vazhinji, vadzidza kubva kune zvinorwadza, vakamira kushandisa tag. :latest ndokutanga kupinza mavhezheni. Hooray!

ECR inochengetedza kusachinjika kwema tag emifananidzo; Tinokurudzira kuti uzvijaire nechinhu ichi chinoshamisa.

Summary

Usatarisira kuti zvese zvichashanda husiku: Kubernetes haisi panacea. Yakashata app icharamba yakadaro kunyange muKubernetes (uye zvingangowedzera kuipa). Kusava nehanya kunotungamira kune yakawandisa kuomarara, inononoka uye inoshungurudza basa rekudzora layer. Pamusoro pezvo, iwe uri panjodzi yekusiiwa usina zano rekudzorera njodzi. Usatarisire Kubernetes kupa yekuzviparadzanisa nevamwe uye kuwanikwa kwakanyanya kunze kwebhokisi. Pedza imwe nguva uchiita kuti application yako iite gore chairo.

Iwe unogona kujairana nezvisina kubudirira zviitiko zvezvikwata zvakasiyana mukati muunganidzwa wenyaya uyu naHenning Jacobs.

Vanoda kuwedzera kune rondedzero yezvikanganiso zvakapihwa muchinyorwa chino vanogona kutibata pa Twitter (@MarekBartik, @MstrsObserver).

PS kubva kumushanduri

Verenga zvakare pablog yedu:

«Kugadzira Kubernetes masumbu: mangani anofanira kunge aripo?";
«Iyo ABCs yeKubernetes Chengetedzo: Kusimbisa, Mvumo, Kuongorora";
«Autoscaling uye zviwanikwa manejimendi muKubernetes"(wongororo uye mushumo wevhidhiyo);
«ConfigMaps muKubernetes: nuances yakakodzera kuziva nezvayo".

Source: www.habr.com

10 Zvakajairika Kukanganisa Paunenge Uchishandisa Kubernetes