🥇Faasaoina i luga ole tau o ao Kubernetes ile AWS

O le faaliliuga o le tusiga na saunia i le afiafi o le amataga o le vasega "Tulafono fa'avae e fa'avae ile Kubernetes".

E faʻafefea ona faʻasaoina i luga ole tau ole ao pe a galulue ma Kubernetes? E leai se fofo sa'o e tasi, ae o lenei tusiga o loʻo faʻamatalaina ai le tele o mea faigaluega e mafai ona fesoasoani ia te oe e pulea lelei au punaoa ma faʻaitiitia ai tau tau o le ao.

Na ou tusia lenei tusiga ma Kubernetes mo AWS i le mafaufau, ae o le a faʻaaogaina (toetoe lava) tutusa le auala i isi kamupani ao. Ou te manatu o lau fuifui (s) ua uma ona faʻapipiʻiina le autoscaling (fuifui-autoscaler). O le aveeseina o punaoa ma le faʻaititia i lalo o lau faʻaogaina o le a naʻo le faʻaolaina o tupe pe a faʻaitiitia ai foi lau vaʻa o nodes tagata faigaluega (EC2 taimi).

O lenei tusiga o le a aofia ai:

fa'amama puna'oa e le'i fa'aaogaina (kube-janitor)
Fa'aiti'itia fa'alelei i itula e le faigaluega (kube-downscaler)
fa'aogaina le fa'ata'atiaga fa'ata'atia (HPA),
fa'aitiitiga o le fa'asaoina o puna'oa tele (kube-resource-report, VPA)
fa'aogaina fa'ata'ita'iga Spot

Faʻamamaina o punaoa e leʻi faʻaaogaina

O le galue i se siosiomaga fa'anatinati e lelei tele. Matou te mananaʻo i faʻalapotopotoga tekonolosi faatelevave. O le fa'avaveina o polokalame fa'akomepiuta o lona uiga o le tele o fa'agaioiga PR, va'aiga va'aiga, fa'ata'ita'iga, ma au'ili'ili fofo. O loʻo faʻapipiʻiina mea uma ile Kubernetes. O ai e iai le taimi e fa'amamā ai ma le lima ia fa'ata'ita'iga su'ega? E faigofie lava ona galo le tapeina o se fa'ata'ita'iga vaiaso. O le a fa'ai'u i luga le pili o le ao ona o se mea ua galo ona matou tapunia:

(Henning Jacobs:
Zhiza:
(upusii) Corey Quinn:
Talatuu: O lau tala AWS o se galuega o le numera o tagata faʻaoga o loʻo ia te oe.
Mea moni: O lau sikoa AWS ose galuega ole numera o inisinia o iai.

Ivan Kurnosov (i le tali):
Mea moni: O lau sikoa AWS o se galuega o le numera o mea na galo ia te oe e tape / tape.)

Kubernetes Janitor (kube-janitor) fesoasoani e fa'amamā lau fuifui. E fetuutuuna'i le fa'atulagaina o le fa'atonu mo le fa'aoga i le lalolagi atoa ma le fa'alotoifale:

O tulafono lautele e mafai ona faʻamalamalamaina le maualuga o le taimi e ola ai (TTL) mo PR / suʻega deployments.
E mafai ona fa'ailoa mai punaoa ta'ito'atasi i le janitor/ttl, mo se fa'ata'ita'iga e otometi ona aveese le spike/prototype pe a uma le 7 aso.

O tulafono lautele o loʻo faʻamalamalamaina i le faila YAML. O lona ala e ui atu i le parakalafa --rules-file i le kube-janitor. O se fa'ata'ita'iga tulafono e aveese uma ai igoa avanoa ma -pr- i le igoa ina ua mavae aso e lua:

- id: cleanup-resources-from-pull-requests
  resources:
    - namespaces
  jmespath: "contains(metadata.name, '-pr-')"
  ttl: 2d

O le faʻataʻitaʻiga o loʻo mulimuli mai e faʻatonutonu ai le faʻaogaina o le igoa o le talosaga i luga o le Deployment ma StatefulSet pods mo Deployments/StatefulSets fou uma i le 2020, ae i le taimi lava e tasi e faʻatagaina le faʻatinoina o suʻega e aunoa ma lenei igoa mo le vaiaso:

- id: require-application-label
  # удалить deployments и statefulsets без метки "application"
  resources:
    - deployments
    - statefulsets
  # см. http://jmespath.org/specification.html
  jmespath: "!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'"
  ttl: 7d

Fa'atino se fa'ata'ita'iga fa'atapula'aina mo le 30 minute i luga o se fuifui o lo'o ta'avale kube-janitor:

kubectl run nginx-demo --image=nginx
kubectl annotate deploy nginx-demo janitor/ttl=30m

O le isi faʻapogai o le faʻateleina o tau o le faʻaauau pea (AWS EBS). O le tapeina o le Kubernetes StatefulSet e le tapeina ai ana voluma faifai pea (PVC - PersistentVolumeClaim). O voluma EBS e le'i fa'aaogaina e faigofie ona maua ai tau o le fiaselau tala ile masina. Kubernetes Janitor ei ai se vaega e fa'amamā ai PVC e le'i fa'aaogaina. Mo se faʻataʻitaʻiga, o lenei tulafono o le a aveesea uma PVC e le o faʻapipiʻiina e se module ma e le o faʻasinoina e se StatefulSet poʻo CronJob:

# удалить все PVC, которые не смонтированы и на которые не ссылаются StatefulSets
- id: remove-unused-pvcs
  resources:
  - persistentvolumeclaims
  jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
  ttl: 24h

E mafai e le Kubernetes Janitor ona fesoasoani ia te oe e fa'amama lau fuifui ma puipuia tau mai le fa'aputu malie. Mo le fa'atulagaina ma le fa'atonuga o fa'atonuga, mulimuli README kube-janitor.

Fa'aiti'itia le fa'asala i itula e le faigaluega

O faiga fa'ata'ita'iga ma fa'asologa e masani ona mana'omia mo na'o itula pisinisi. O nisi o talosaga mo le gaosiga, e pei o le back office/admin tools, e mana'omia fo'i na'o le fa'atapula'aina o avanoa ma e ono fa'aletonu i le po.

Kubernetes Downscaler (kube-downscaler) e mafai ai e tagata faʻaoga ma faʻalapotopotoga ona faʻaititia le faiga i taimi e le faigaluega ai. Deployments ma StatefulSets e mafai ona fua i le leai o ni kopi. CronJobs e ono taofia lē tumau. Kubernetes Downscaler ua fa'atulagaina mo se fuifui atoa, tasi pe sili atu igoa avanoa, po'o punaoa ta'itasi. E mafai ona e setiina le "taimi le aoga" poʻo le isi itu, "taimi faigaluega". Mo se faʻataʻitaʻiga, ia faʻaitiitia le faʻaogaina i le tele e mafai ai i po ma faaiuga o vaiaso:

image: hjacobs/kube-downscaler:20.4.3
args:
  - --interval=30
  # не отключать компоненты инфраструктуры
  - --exclude-namespaces=kube-system,infra
  # не отключать kube-downscaler, а также оставить Postgres Operator, чтобы исключенными БД можно было управлять
  - --exclude-deployments=kube-downscaler,postgres-operator
  - --default-uptime=Mon-Fri 08:00-20:00 Europe/Berlin
  - --include-resources=deployments,statefulsets,stacks,cronjobs
  - --deployment-time-annotation=deployment-time

Ole kalafi lea mo le fa'atupuina o nodes o tagata faigaluega i fa'ai'uga o vaiaso:

O le fa'aitiitia mai le ~13 i le 4 nodes tagata faigaluega e mautinoa lava e faia ai se eseesega iloga i lau pili AWS.

Ae fa'afefea pe a ou mana'omia le galue i le taimi ole fuifui "downtime"? O nisi mea fa'apipi'i e mafai ona fa'ate'a tumau mai le fa'asili e ala i le fa'aopoopoina o le fa'alalo/tuuese: fa'amatalaga moni. E mafai ona fa'ate'aina le fa'agaoioiga mo se taimi le tumau e fa'aaoga ai le fa'alalo/fa'aesea-seia o'o i fa'amatalaga fa'atasi ai ma se fa'ailoga taimi i le faatulagaga YYYY-MM-DD HH:MM (UTC). Afai e mana'omia, e mafai ona fa'aitiitia le fuifui atoa e ala i le fa'apipi'iina o se pusa fa'atasi ma le fa'amatalaga downscaler/force-uptime, mo se faʻataʻitaʻiga, e ala i le tatalaina o le nginx avanoa:

kubectl run scale-up --image=nginx
kubectl annotate deploy scale-up janitor/ttl=1h # удалить развертывание через час
kubectl annotate pod $(kubectl get pod -l run=scale-up -o jsonpath="{.items[0].metadata.name}") downscaler/force-uptime=true

Vaai README kube-downscaler, pe afai e te fiafia i faʻatonuga faʻapipiʻi ma isi filifiliga.

Fa'aoga le fa'ata'atiaga fa'ata'atia

Le tele o talosaga/au'aunaga e feagai ma se faiga fa'amomoli fa'amomoli: o nisi taimi o latou modules e le aoga, ma o nisi taimi latou te galulue i le malosi atoatoa. O le fa'agaioia o se va'a tumau o pods e fa'afetauia le uta maualuga maualuga e le fa'atauaina. E lagolagoina e Kubernetes le fa'ata'atiaga fa'ata'atia i luga o se puna'oa HorizontalPodAutoscaler (HPA). O le faʻaogaina o le CPU e masani lava o se faʻailoga lelei mo le faʻavasegaina:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 100
        type: Utilization

Ua faia e Zalando se vaega e faigofie ona faʻafesoʻotaʻi fua faʻatatau mo le faʻavasegaina: Su'ega Su'ega Kube (kube-metrics-adapter) o se mea fa'apipi'i fua fa'atatau mo Kubernetes e mafai ona fa'aputuina ma tu'uina atu fua fa'ale-aganu'u ma fafo mo le fa'ata'atiaga fa'alava o pods. E lagolagoina le faʻavasegaina e faʻavae i luga ole Prometheus metrics, SQS queues, ma isi tulaga. Mo se faʻataʻitaʻiga, ia fua lau faʻaogaina i se fua faʻaleaganuʻu o loʻo faʻatusalia e le talosaga lava ia o le JSON i / metrics faʻaoga:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.pods.requests-per-second.json-path/json-key: "$.http_server.rps"
    metric-config.pods.requests-per-second.json-path/path: /metrics
    metric-config.pods.requests-per-second.json-path/port: "9090"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: requests-per-second
      target:
        averageValue: 1k
        type: AverageValue

O le fa'atulagaina o le fa'ata'ita'i fa'ata'atia ma le HPA e tatau ona avea ma se tasi o gaioiga fa'aletonu e fa'aleleia atili ai le lelei mo auaunaga e leai ni setete. Spotify ei ai se faʻamatalaga ma o latou poto masani ma fautuaga mo HPA: fua lau fa'agaioiga, ae le o lau atotupe.

Fa'aitiitia le fa'atauva'a o puna'oa

O galuega mamafa a Kubernetes e fuafua ai o latou mana'oga CPU/manatua e ala i "talosaga punaoa." O puna'oa o le CPU e fuaina i totonu o mata'itusi po'o sili atu i le "millicores", mo se fa'ata'ita'iga 500m o lona uiga 50% vCPU. O mea e manatua e fuaina ile paita, ma e mafai ona fa'aoga suffix masani, pei ole 500Mi, o lona uiga 500 megabytes. Talosaga puna'oa "loka" gafatia i nodes tagata faigaluega, o lona uiga o se pod ma se talosaga 1000m CPU i luga o se node ma 4 vCPUs o le a tuua na o le 3 vCPUs avanoa i isi pods. [1]

Slack (sili atu fa'aagaga) o le eseesega i le va o punaoa talosaga ma le faʻaaogaina moni. Mo se faʻataʻitaʻiga, o se pusa e talosagaina le 2 GiB o le manatua ae naʻo le faʻaaogaina o le 200 MiB o loʻo i ai ~ 1,8 GiB o le "sili atu" manatua. O le tele e alu ai tupe. E mafai e se tasi ona faʻatusatusaina o le 1 GiB o le faʻaaogaina o manatuaga tau ~ $10 i le masina. [2]

Kubernetes Punaoa Lipoti (kube-resource-report) faʻaalia le tele o tupe faʻaagaga ma e mafai ona fesoasoani ia te oe e faʻamautinoa le gafatia o tupe teu:

Kubernetes Punaoa Lipoti o lo'o fa'aalia ai le fa'asili fa'atasi e ala i le fa'aoga ma le fa'atonuga. Ole mea lea e mafai ai ona e su'e nofoaga e mafai ona fa'aitiitia ai mana'oga o punaoa. O le lipoti o le HTML na faia na'o se ata o le fa'aogaina o punaoa. E tatau ona e va'ava'ai ile fa'aogaina ole CPU/manatua ile taimi e fa'amautu ai mana'oga talafeagai. O le siata lea o le Grafana mo se 'au'aunaga mamafa a le PPU: o pusa uma o lo'o fa'aogaina le itiiti ifo nai lo le 3 o lo'o talosagaina ai le CPU:

O le fa'aitiitia o le talosaga a le PPU mai le 3000m i le ~400m e fa'asa'oloto ai punaoa mo isi galuega mamafa ma fa'ataga ai le fuifui e la'ititi.

"O le faʻaogaina o le CPU o faʻataʻitaʻiga EC2 e masani ona faʻafefe i le numera tasi pasene pasene," tusia e Corey Quinn. A o le EC2 o le fuaina o le tele sa'o atonu o se filifiliga leagaSuia nisi fesili a Kubernetes punaoa i se faila YAML e faigofie ma e mafai ona aumaia se tupe tele.

Ae tatou te manaʻo moni e suia e tagata tulaga taua i faila YAML? Leai, e mafai e masini ona sili atu ona lelei! Kubernetes Vertical Pod Autoscaler (VPA) e na'o le mea lena: fetuutuuna'i talosaga punaoa ma fa'agata e tusa ai ma le mamafa o galuega. O se faʻataʻitaʻiga kalafi o talosaga Prometheus CPU (laina lanumoana manifinifi) faʻafetaui e le VPA i le taimi:

Fa'aaoga e Zalando le VPA i ona fuifui uma mo vaega tetele. E mafai fo'i ona fa'aoga le VPA e fa'aoga e le fa'atauaina.

Goldilock mai Fairwind o se meafaigaluega e fatuina ai se VPA mo faʻapipiʻi taʻitasi i se igoa avanoa ona faʻaalia ai lea o se fautuaga VPA i luga o lana dashboard. E mafai ona fesoasoani i tagata atiaʻe e seti le saʻo CPU / manatua talosaga mo a latou talosaga:

Sa ou tusia se tamai blogpost e uiga i le VPA i le 2019, ma talu ai nei i CNCF End User Community na talanoaina le mataupu VPA.

Fa'aaogā EC2 Spot Instances

Mulimuli ae le itiiti, AWS EC2 tau e mafai ona faʻaititia e ala i le faʻaaogaina o faʻataʻitaʻiga Spot e pei o le Kubernetes tagata faigaluega nodes [3]. E maua avanoa fa'apitoa ile fa'aitiitiga e o'o ile 90% pe a fa'atusatusa ile tau i luga ole mana'oga. O le ta'avaleina o Kubernetes i luga o le EC2 Spot o se tu'ufa'atasiga lelei: e mana'omia ona e fa'ama'oti ituaiga eseese o fa'ata'ita'iga mo avanoa maualuga atu, o lona uiga e mafai ona e maua se node lapo'a mo le tau tutusa pe maualalo, ma o le fa'atuputeleina o le gafatia e mafai ona fa'aogaina e uta o galuega Kubernetes.

E faʻafefea ona faʻatautaia Kubernetes ile EC2 Spot? E tele filifiliga: fa'aoga se isi vaega e pei ole SpotInst (lea ua ta'ua nei o le "Spot", 'aua le fesili mai pe aisea), pe fa'aopoopo se Spot AutoScalingGroup (ASG) i lau fuifui. Mo se faʻataʻitaʻiga, o le CloudFormation snippet mo se "mafai-optimized" Spot ASG ma le tele o faʻataʻitaʻiga ituaiga:

MySpotAutoScalingGroup:
 Properties:
   HealthCheckGracePeriod: 300
   HealthCheckType: EC2
   MixedInstancesPolicy:
     InstancesDistribution:
       OnDemandPercentageAboveBaseCapacity: 0
       SpotAllocationStrategy: capacity-optimized
     LaunchTemplate:
       LaunchTemplateSpecification:
         LaunchTemplateId: !Ref LaunchTemplate
         Version: !GetAtt LaunchTemplate.LatestVersionNumber
       Overrides:
         - InstanceType: "m4.2xlarge"
         - InstanceType: "m4.4xlarge"
         - InstanceType: "m5.2xlarge"
         - InstanceType: "m5.4xlarge"
         - InstanceType: "r4.2xlarge"
         - InstanceType: "r4.4xlarge"
   LaunchTemplate:
     LaunchTemplateId: !Ref LaunchTemplate
     Version: !GetAtt LaunchTemplate.LatestVersionNumber
   MinSize: 0
   MaxSize: 100
   Tags:
   - Key: k8s.io/cluster-autoscaler/node-template/label/aws.amazon.com/spot
     PropagateAtLaunch: true
     Value: "true"

O nisi faʻamatalaga i le faʻaaogaina o Spot ma Kubernetes:

E te mana'omia le fa'aogaina o le fa'amutaina o Spot, mo se fa'ata'ita'iga e ala i le tu'ufa'atasia o le node pe a taofi le fa'ata'ita'iga
Fa'aaoga e Zalando tui fa'avasegaina ta'avale fa'apitoa fa'atasi ma mea e fa'amuamua i node pool
Nodes togitogi e mafai ona faamalosia talia "resitala" o galuega mamafa e tamoe i Spot

Aotelega

Ou te faʻamoemoe e te mauaina nisi o meafaigaluega o loʻo tuʻuina atu e aoga i le faʻaitiitia o lau pili ao. E mafai ona e mauaina le tele o mea o loʻo i totonu o le tusiga foi i laʻu lauga i le DevOps Gathering 2019 i luga o le YouTube ma ata.

O a au faiga sili mo le fa'asaoina o tau ile ao ile Kubernetes? Faamolemole ta'u mai i Twitter (@try_except_).

[1] O le mea moni, e itiiti ifo i le 3 vCPUs o le a fa'aaoga pea ona o le fa'aitiitiga o le node e fa'aitiitia e ala i puna'oa fa'aagaga. Kubernetes e fa'aeseesega i le va o le malosi o le node faaletino ma punaoa "tuuina atu" (Node Fa'asoa).

[2] Fa'ata'ita'iga fa'atatau: tasi m5.large fa'ata'ita'iga ma le 8 GiB o le manatua e ~$84 i le masina (eu-central-1, On-Demand), i.e. poloka 1/8 node e tusa ma le ~$10/masina.

[3] E tele isi auala e fa'aitiitia ai lau pili EC2, e pei o Fa'amaumauga Fa'asao, Fuafuaga Fa'asao, ma isi.

Aoao atili e uiga i le kosi.

puna: www.habr.com

Fa'asao ile Kubernetes ao tau ile AWS