Zolakwa 10 Zomwe Zimachitika Pogwiritsa Ntchito Kubernetes

Zindikirani. transl.: Olemba nkhaniyi ndi mainjiniya ochokera ku kampani yaying'ono yaku Czech, pipetail. Iwo adatha kuphatikiza mndandanda wabwino wa [nthawi zina oletsedwa, komabe] zovuta zovuta komanso malingaliro olakwika okhudzana ndi magwiridwe antchito a magulu a Kubernetes.

Kwa zaka zambiri zogwiritsa ntchito Kubernetes, tagwira ntchito ndi magulu ambiri (onse oyendetsedwa ndi osayendetsedwa - pa GCP, AWS ndi Azure). M’kupita kwa nthaŵi, tinayamba kuona kuti zolakwa zina zimabwerezedwa nthaŵi zonse. Komabe, palibe manyazi mu izi: tachita ambiri a iwo tokha!

Nkhaniyi ili ndi zolakwika zofala kwambiri ndipo imatchulanso momwe mungakonzere.

1. Zothandizira: zopempha ndi malire

Chinthuchi chikuyenera kuyang'aniridwa kwambiri ndi malo oyamba pamndandanda.

CPU pempho kawirikawiri mwina sizinatchulidwe nkomwe kapena zili ndi mtengo wotsika kwambiri (kuyika makoko ambiri pa mfundo iliyonse momwe mungathere). Choncho, mfundozo zimakhala zodzaza. Panthawi ya katundu wambiri, mphamvu yogwiritsira ntchito node imagwiritsidwa ntchito mokwanira ndipo ntchito inayake imalandira zomwe "inapempha" ndi. CPU throttling. Izi zimabweretsa kuchuluka kwa latency, kutha kwa nthawi, ndi zotsatira zina zosasangalatsa. (Werengani zambiri za izi m'matembenuzidwe athu aposachedwa: "Malire a CPU ndi kugwedezeka kwaukali ku Kubernetes"- pafupifupi. transl.)

BestEffort (kwambiri osati analimbikitsa):

resources: {}

Pempho lotsika kwambiri la CPU (kwambiri osati analimbikitsa):

   resources:
      Requests:
        cpu: "1m"

Kumbali inayi, kukhalapo kwa malire a CPU kungayambitse kudumpha kosayenera kwa mawotchi ndi ma pods, ngakhale purosesa ya node siidadzaza. Apanso, izi zingayambitse kuchedwa kowonjezereka. Kukangana kumapitirirabe kuzungulira pa parameter Mtengo wa CPU CFS mu Linux kernel ndi CPU throttling malinga ndi malire oikidwa, komanso kulepheretsa chiwerengero cha CFS ... Kalanga, malire a CPU angayambitse mavuto ambiri kuposa momwe angathetsere. Zambiri za izi zitha kupezeka pa ulalo womwe uli pansipa.

Kusankha mopambanitsa (kupitilira) mavuto a kukumbukira angayambitse mavuto aakulu. Kufikira malire a CPU kumaphatikizapo kudumpha mawotchi, pomwe kufika malire a kukumbukira kumaphatikizapo kupha pod. Kodi munaonapo OOMkill? Inde, ndizo ndendende zomwe tikukamba.

Kodi mukufuna kuchepetsa mwayi woti izi zichitike? Osagawa kwambiri kukumbukira ndikugwiritsa ntchito Guaranteed QoS (Quality of Service) pokhazikitsa pempho lokumbukira mpaka malire (monga momwe zilili pansipa). Werengani zambiri za izi mu Zowonetsera za Henning Jacobs (Lead Engineer at Zalando).

Kuphulika (mwayi waukulu wopeza OOMkilled):

   resources:
      requests:
        memory: "128Mi"
        cpu: "500m"
      limits:
        memory: "256Mi"
        cpu: 2

Kutsimikiziridwa:

   resources:
      requests:
        memory: "128Mi"
        cpu: 2
      limits:
        memory: "128Mi"
        cpu: 2

Kodi nchiyani chomwe chingathandize pakukhazikitsa zinthu?

Ndi chithandizo cha metrics-server mutha kuwona momwe CPU ikugwiritsidwira ntchito komanso kugwiritsa ntchito kukumbukira ndi ma pod (ndi zotengera mkati mwake). Mwinamwake, mukugwiritsa ntchito kale. Ingoyendetsani malamulo awa:

kubectl top pods
kubectl top pods --containers
kubectl top nodes

Komabe, amangowonetsa kugwiritsidwa ntchito kwamakono. Itha kukupatsirani lingaliro lovuta la dongosolo la kukula, koma pamapeto pake mudzafunika mbiri ya kusintha kwa ma metrics pakapita nthawi (kuyankha mafunso monga: "Kodi kuchuluka kwa CPU kunali kotani?", "Kodi katundu wake anali wotani dzulo m'mawa?", ndi zina zotero). Kwa ichi mungagwiritse ntchito Prometheus, DataDog ndi zida zina. Amangotenga ma metrics kuchokera ku metrics-server ndikuwasunga, ndipo wogwiritsa ntchito amatha kuwafunsa ndikuwakonzera moyenera.

VerticalPodAutoscaler timatha zokha ndondomeko iyi. Imatsata CPU ndi mbiri yogwiritsa ntchito kukumbukira ndikukhazikitsa zopempha zatsopano ndi malire kutengera chidziwitsochi.

Kugwiritsa ntchito mphamvu zamakompyuta moyenera sikophweka. Zili ngati kusewera Tetris nthawi zonse. Ngati mukulipira mochulukira kuti mugwiritse ntchito mphamvu zamagetsi ndikugwiritsa ntchito pang'ono (kunena ~ 10%), timalimbikitsa kuyang'ana zinthu zochokera ku AWS Fargate kapena Virtual Kubelet. Amamangidwa pamtundu wolipira wopanda seva / wolipira pakugwiritsa ntchito, zomwe zitha kukhala zotsika mtengo mumikhalidwe yotere.

2. Kukhala ndi moyo komanso kukonzekera kumafufuza

Mwachikhazikitso, macheke amoyo komanso okonzeka saloledwa ku Kubernetes. Ndipo nthawi zina amaiwala kuwayatsa...

Koma mungataninso kuti muyambitsenso kuyambiranso ntchito pakachitika vuto lalikulu? Ndipo wolinganiza katundu akudziwa bwanji kuti poda yakonzeka kuvomereza magalimoto? Kapena kuti imatha kuthana ndi magalimoto ambiri?

Mayesowa nthawi zambiri amasokonezedwa wina ndi mnzake:

• Moyo - "kupulumuka" cheke, yomwe imayambiranso pod ngati ilephera;
• Kukonzekera - cheke chokonzekera, ngati sichikanika, chimachotsa pod ku Kubernetes service (izi zitha kuwonedwa pogwiritsa ntchito kubectl get endpoints) ndipo magalimoto safika kwa icho mpaka cheke chotsatira chitsirizidwa bwino.

Macheke onsewa ZOCHITIKA PA MOYO WONSE WA POD. Ndikofunikira kwambiri.

Lingaliro lolakwika lodziwika bwino ndiloti zokonzekera zokonzekera zimangoyambika poyambira kuti wolinganiza adziwe kuti pod yakonzeka (Ready) ndipo akhoza kuyamba kukonza magalimoto. Komabe, ichi ndi chimodzi mwazosankha zogwiritsa ntchito.

Chinanso ndikuthekera kopeza kuti kuchuluka kwa magalimoto pamapod ndikokwanira komanso imadzaza (kapena pod imachita kuwerengera mozama kwambiri). Pankhaniyi, kufufuza kokonzekera kumathandiza kuchepetsa katundu pa pod ndi "kuziziritsa" izo. Kukwaniritsa bwino cheke chokonzekera mtsogolo kudzalola onjezani katundu pa pod kachiwiri. Pamenepa (ngati mayeso okonzeka alephera), kulephera kwa kuyesa kwa moyo kungakhale kopanda phindu. Chifukwa chiyani muyambitsenso pod yomwe ili yathanzi komanso yogwira ntchito molimbika?

Chifukwa chake, nthawi zina, palibe cheke chilichonse chomwe chili bwino kuposa kuwapangitsa kukhala ndi magawo osinthidwa molakwika. Monga tafotokozera pamwambapa, ngati liveness fufuzani makope kukonzekera cheke, ndiye kuti muli m’mavuto aakulu. zotheka njira ndi configure mayeso okonzeka okhandi moyo wowopsa siyani pambali.

Mitundu yonse iwiri ya macheke sayenera kulephera pamene kudalira wamba kulephera, apo ayi izi zipangitsa kulephera kwa ma pod onse. Mwanjira ina, musadzipweteka nokha.

3. LoadBalancer pa ntchito iliyonse ya HTTP

Mwinamwake, muli ndi mautumiki a HTTP mumagulu anu omwe mungafune kutumizira kunja.

Ngati mutsegula utumiki ngati type: LoadBalancer, wolamulira wake (malingana ndi wopereka chithandizo) adzapereka ndi kukambirana LoadBalancer yakunja (osati kwenikweni ikuyenda pa L7, koma ngakhale pa L4), ndipo izi zingakhudze mtengo (adiresi yakunja ya IPv4, mphamvu yamakompyuta, kulipira kwa sekondi iliyonse. ) chifukwa chofuna kupanga zinthu zambiri zoterezi.

Pankhaniyi, ndizomveka kugwiritsa ntchito chowongolera chimodzi chakunja, kutsegula ntchito ngati type: NodePort. Kapena bwino, onjezerani china chake nginx-ingress-controller (kapena alireza), amene adzakhala yekhayo NodePort endpoint yolumikizidwa ndi chojambulira chakunja chonyamula katundu ndipo idzayendetsa magalimoto mumagulu akugwiritsa ntchito ingress-Kubernetes zothandizira.

Ntchito zina za intra-cluster (micro) zomwe zimalumikizana zimatha "kulumikizana" pogwiritsa ntchito ntchito ngati ClusterIP ndi njira yopezera ntchito yomangidwira kudzera pa DNS. Osagwiritsa ntchito DNS/IP yawo yapagulu, chifukwa izi zitha kukhudza kuchedwa ndikuwonjezera mtengo wantchito zamtambo.

4. Autoscaling gulu popanda kuganizira mbali zake

Mukawonjezera ma node ndi kuwachotsa pagulu, musadalire ma metrics ena ofunikira monga kugwiritsa ntchito CPU pamfundozo. Kukonzekera kwa Pod kuyenera kuganizira zambiri zoletsa, monga kuyanjana kwa pod / node, kutayira ndi kulolerana, zopempha zothandizira, QoS, ndi zina zotero. Kugwiritsa ntchito makina ojambulira akunja omwe samatengera ma nuances awa kungayambitse mavuto.

Ingoganizirani kuti pod ina iyenera kukonzedwa, koma mphamvu zonse za CPU zomwe zilipo zikupemphedwa / kuphatikizidwa ndi pod. amakakamira mu chikhalidwe Pending. Autoscaler yakunja imawona kuchuluka kwa CPU komwe kulipo (osati kofunsidwa) ndipo sikuyambitsa kukulitsa (kuchepetsa) - sichiwonjezera node ina. Chifukwa chake, chida ichi sichingaganizidwe.

Pankhaniyi, reverse makulitsidwe (mulingo) - kuchotsa mfundo pagulu nthawi zonse kumakhala kovuta kwambiri kukhazikitsa. Tangoganizani kuti muli ndi poto yowoneka bwino (yokhala ndi chosungira chosalekeza cholumikizidwa). Ma voliyumu osalekeza nthawi zambiri amakhala a malo enieni opezeka ndipo sizimatsatiridwa m'derali. Chifukwa chake, ngati autoscaler yakunja ichotsa node ndi pod iyi, wokonza mapulani sangathe kukonza pod iyi pamfundo ina, chifukwa izi zitha kuchitika pamalo opezeka komwe kusungirako kosalekeza kuli. Pod idzakhazikika mu state Pending.

Zodziwika kwambiri m'dera la Kubernetes cluster-autoscaler. Imayendera gulu, imathandizira ma API kuchokera kwa opereka mtambo akuluakulu, imaganizira zoletsa zonse ndipo imatha kukula pazomwe zili pamwambapa. Imathanso kukulitsa ndikusunga malire onse, potero imapulumutsa ndalama (zomwe zikanagwiritsidwa ntchito pazosagwiritsidwa ntchito).

5. Kunyalanyaza luso la IAM/RBAC

Chenjerani kugwiritsa ntchito ogwiritsa ntchito a IAM okhala ndi zinsinsi zosalekeza za makina ndi ntchito. Konzani mwayi wofikira kwakanthawi pogwiritsa ntchito maudindo ndi maakaunti a ntchito (akaunti yautumiki).

Nthawi zambiri timakumana ndi mfundo yoti makiyi olowera (ndi zinsinsi) amasungidwa molimba pakusintha kwa pulogalamuyo, komanso kunyalanyaza kusinthasintha kwa zinsinsi ngakhale ali ndi mwayi wopeza Cloud IAM. Gwiritsani ntchito maudindo a IAM ndi maakaunti a ntchito m'malo mwa ogwiritsa ntchito ngati kuli koyenera.

Iwalani za kube2iam ndikupita molunjika ku maudindo a IAM pamaakaunti a ntchito (monga tafotokozera mu chizindikiro cha dzina lomwelo Štěpán Vraný):

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role
  name: my-serviceaccount
  namespace: default

Ndemanga imodzi. Osati zovuta, chabwino?

Komanso, musapereke maakaunti amautumiki ndi mwayi wambiri admin и cluster-adminngati sachifuna. Izi ndizovuta kwambiri kukhazikitsa, makamaka mu RBAC K8s, koma ndizoyenera kuyesetsa.

6. Osadalira odana ndi kugwirizana basi poto

Tangoganizani kuti muli ndi zifaniziro zitatu za kutumizidwa pa node. Node imagwa, ndipo pamodzi ndi zofananira zonse. Zosasangalatsa, chabwino? Koma nchifukwa ninji zofananira zonse zinali pa mfundo imodzi? Kodi Kubernetes sakuyenera kupereka kupezeka kwakukulu (HA)?!

Tsoka ilo, Kubernetes scheduler, mwakufuna kwake, samatsatira malamulo akukhalako kosiyana (anti-affinity) za matumba. Ayenera kunenedwa momveka bwino:

// опущено для краткости
      labels:
        app: zk
// опущено для краткости
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"

Ndizomwezo. Tsopano ma pod adzakonzedwa pama node osiyanasiyana (zimenezi zimawunikidwa panthawi yokonzekera, koma osati panthawi ya ntchito - chifukwa chake requiredDuringSchedulingIgnoredDuringExecution).

Apa tikukamba za podAntiAffinity pamitundu yosiyanasiyana: topologyKey: "kubernetes.io/hostname", - osati za magawo osiyanasiyana opezeka. Kuti mugwiritse ntchito HA kwathunthu, muyenera kukumba mozama pamutuwu.

7. Kunyalanyaza PodDisruptionBudgets

Tangoganizani kuti muli ndi katundu wopanga gulu la Kubernetes. Nthawi ndi nthawi, ma node ndi masango omwewo ayenera kusinthidwa (kapena kuchotsedwa). PodDisruptionBudget (PDB) ndi china chake ngati mgwirizano wotsimikizira ntchito pakati pa oyang'anira magulu ndi ogwiritsa ntchito.

PDB imakulolani kuti mupewe kusokonezeka kwa ntchito chifukwa cha kusowa kwa ma node:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: zookeeper

Muchitsanzo ichi, inu, monga wogwiritsa ntchito gululi, munene kwa ma admins kuti: "Hei, ndili ndi ntchito yoyang'anira malo osungira nyama, ndipo ziribe kanthu zomwe mungachite, ndikufuna kukhala ndi zosachepera 2 zofananira za ntchitoyi nthawi zonse."

Mutha kuwerenga zambiri za izi apa.

8. Ogwiritsa ntchito angapo kapena malo omwe ali mugulu limodzi

Kubernetes namespaces (malo a mayina) osapereka zotsekemera zolimba.

Lingaliro lolakwika lodziwika bwino ndilakuti ngati mutumiza katundu wosagwirizana ndi dzina limodzi ndi katundu wowonjezera kupita kwina, ndiye sizidzakhudza wina ndi mzake mwanjira iliyonse... Komabe, mulingo wina wodzipatula ukhoza kupezedwa pogwiritsa ntchito zopempha / zolepheretsa, kukhazikitsa ma quotas, ndi kukhazikitsa Zofunika Kwambiri. Kudzipatula kwina kwa "thupi" mu ndege ya data kumaperekedwa ndi ma affinities, kulolerana, zonyansa (kapena nodeselectors), koma kulekanitsa koteroko kulidi. zovuta kwaniritsa.

Amene akufunika kuphatikiza mitundu yonse iwiri ya ntchito mumagulu amodzi adzayenera kuthana ndi zovuta. Ngati palibe chosowa chotero, ndipo mungakwanitse kukhala nacho tsango linanso (kunena, mumtambo wapagulu), ndiye kuti ndibwino kutero. Izi zidzakwaniritsa mulingo wapamwamba kwambiri wa insulation.

9. ForeignTrafficPolicy: Cluster

Nthawi zambiri timawona kuti magalimoto onse mkati mwa gululo amabwera kudzera muutumiki ngati NodePort, yomwe ndondomeko yosasinthika imayikidwa. externalTrafficPolicy: Cluster... Zikutanthauza kuti NodePort imatsegulidwa pa mfundo iliyonse mgululi, ndipo mutha kugwiritsa ntchito iliyonse kuti mulumikizane ndi ntchito yomwe mukufuna (mapoko).

Panthawi imodzimodziyo, ma pods enieni okhudzana ndi utumiki wa NodePort womwe watchulidwa pamwambapa nthawi zambiri umapezeka pamtundu wina gawo la node izi. Mwanjira ina, ngati ndilumikizana ndi node yomwe ilibe pod yofunikira, imatumiza magalimoto kumalo ena, kuwonjezera hop ndi kuchuluka kwa latency (ngati ma node ali m'malo osiyanasiyana opezeka / malo opangira data, latency ikhoza kukhala yokwera kwambiri; kuphatikizanso, egress mitengo yamagalimoto idzawonjezeka).

Kumbali ina, ngati ntchito ina ya Kubernetes ili ndi ndondomeko yokhazikitsidwa externalTrafficPolicy: Local, ndiye NodePort imatsegula pazigawo zomwe ma pods ofunikira akuyenda. Pamene ntchito kunja katundu balancer kuti amafufuza boma (zaumoyo) endpoints (zimachita bwanji AWS ELB), Iye adzatumiza magalimoto okha ku mfundo zofunika, zomwe zidzakhala ndi zotsatira zopindulitsa pa kuchedwa, zosowa zamakompyuta, mabilu a egress (ndi kulingalira bwino kumapereka zomwezo).

Pali mwayi waukulu kuti mukugwiritsa ntchito kale zinthu ngati alireza kapena nginx-ingress-controller monga NodePort endpoint (kapena LoadBalancer, yomwe imagwiritsanso ntchito NodePort) kuti iyendetse HTTP ingress traffic, ndipo kukhazikitsa njirayi kungachepetse kwambiri latency ya zopempha zoterezi.

В chofalitsidwa ichi Mutha kuphunzira zambiri za externalTrafficPolicy, ubwino wake ndi kuipa kwake.

10. Osamangidwa kumagulu ndipo musagwiritse ntchito molakwika ndege yowongolera

M'mbuyomu, zinali chizolowezi kuitana ma seva ndi mayina oyenera: Anton, HAL9000 ndi Colossus... Masiku ano asinthidwa ndi zozindikiritsa zopangidwa mwachisawawa. Komabe, chizoloŵezicho chinakhalabe, ndipo tsopano mayina oyenerera amapita kumagulu.

Nkhani yeniyeni (yotengera zochitika zenizeni): zonse zinayamba ndi umboni wa lingaliro, kotero gululo linali ndi dzina lonyada. kuyezetsa… Zaka zapita ndipo ikugwiritsidwabe ntchito popanga, ndipo aliyense akuwopa kuigwira.

Palibe chosangalatsa ngati magulu akusintha kukhala ziweto, kotero timalimbikitsa kuwachotsa nthawi ndi nthawi poyeserera kuchira tsoka (izi zidzathandiza chisokonezo engineering - pafupifupi. transl.). Kuphatikiza apo, sizingakhale zopweteka kugwira ntchito pagawo lowongolera (ndege yoyang'anira). Kuopa kumugwira si chizindikiro chabwino. Ndi zina zotero wakufa? Anyamata, mulidi m'mavuto!

Kumbali ina, musatengeke ndi kuwongolera. Ndi nthawi gawo lowongolera litha kukhala lochedwa. Mwachidziwikire, izi zimachitika chifukwa cha kuchuluka kwa zinthu zomwe zimalengedwa popanda kuzungulira kwawo (nthawi zambiri mukamagwiritsa ntchito Helm yokhala ndi zosintha zosasintha, ndiye chifukwa chake mawonekedwe ake mu configmaps/zinsinsi samasinthidwa - chifukwa chake, zinthu masauzande zimadziunjikira mkati. wosanjikiza wowongolera) kapena ndikusintha kosalekeza kwa zinthu za kube-api (zowonjezera zokha, za CI/CD, zowunikira, zipika za zochitika, owongolera, ndi zina).

Kuphatikiza apo, timalimbikitsa kuyang'ana mapangano a SLA/SLO ndi wothandizira Kubernetes woyendetsedwa ndi kulabadira zitsimikizo. Wogulitsa akhoza kutsimikizira kuwongolera kupezeka kwa gawo (kapena zigawo zake), koma osati kuchedwa kwa p99 kwa zopempha zomwe mumatumiza kwa izo. Mwanjira ina, mutha kulowa kubectl get nodes, ndikulandira yankho pokhapokha patatha mphindi 10, ndipo izi sizidzakhala kuphwanya malamulo a mgwirizano wautumiki.

11. Bonasi: kugwiritsa ntchito tag yaposachedwa

Koma izi ndi zakale kale. Posachedwapa takumana ndi njira imeneyi kawirikawiri, popeza ambiri, ataphunzira kuchokera ku zowawa, asiya kugwiritsa ntchito tag. :latest ndikuyamba kusindikiza ma versions. Uwu!

ECR imasunga kusasinthika kwa ma tag azithunzi; Tikukulimbikitsani kuti muzolowere mbali yodabwitsayi.

Chidule

Musayembekezere kuti chilichonse chigwire ntchito usiku umodzi: Kubernetes si mankhwala. Pulogalamu yoyipa ikhalabe chonchi ngakhale ku Kubernetes (ndipo mwina zitha kuipiraipira). Kusasamala kudzatsogolera ku zovuta kwambiri, pang'onopang'ono komanso kupsinjika kwa gawo lowongolera. Kuphatikiza apo, mutha kukhala pachiwopsezo chosiyidwa popanda njira yopulumutsira masoka. Musayembekezere Kubernetes kuti apereke kudzipatula komanso kupezeka kwakukulu m'bokosi. Tengani nthawi kuti pulogalamu yanu ikhale yamtambo.

Mutha kuzolowerana ndi zochitika zomwe sizinachite bwino zamagulu osiyanasiyana nkhani imeneyi ndi Henning Jacobs.

Iwo omwe akufuna kuwonjezera pamndandanda wa zolakwika zomwe zaperekedwa m'nkhaniyi atha kulumikizana nafe pa Twitter (@MarekBartik, @MstrsObserver).

PS kuchokera kwa womasulira

Werenganinso pa blog yathu:

• «Kupanga magulu a Kubernetes: payenera kukhala angati?";
• «ABC ya Chitetezo ku Kubernetes: Kutsimikizika, Kuvomerezeka, Kuwerengera";
• «Autoscaling ndi kasamalidwe kazinthu ku Kubernetes"(ndemanga ndi lipoti lamavidiyo);
• «ConfigMaps ku Kubernetes: ma nuances oyenera kudziwa".

Source: www.habr.com