Sevha paKubernetes gore rinodhura paAWS

Kuturikirwa kwechinyorwa kwakagadzirirwa manheru ekutanga kwekosi "Infrastructure platform yakavakirwa paKubernetes".

Nzira yekuchengetedza sei pamutengo wegore paunenge uchishanda neKubernetes? Iko hakuna mhinduro imwechete chaiyo, asi chinyorwa ichi chinotsanangura akati wandei maturusi anogona kukubatsira kubata zviwanikwa zvako zvakanyanya uye kuderedza mutengo wako wekombuta.

Ndakanyora chinyorwa ichi naKubernetes yeAWS mupfungwa, asi ichashanda (inenge) nenzira imwechete kune vamwe vanopa makore. Ndiri kufunga kuti masumbu ako (s) atove neautoscaling akagadziridzwa (cluster-autoscaler) Kubvisa zviwanikwa uye kudzika pasi kutumira kwako kunongokuchengetera mari kana zvichideredzawo boka rako revashandi node (EC2 zviitiko).

Ichi chinyorwa chichabata:

• kuchenesa zviwanikwa zvisina kushandiswa (kube-janitor)
• Deredza kuyera panguva isiri yekushanda maawa (kube-downscaler)
• uchishandisa horizontal autoscaling (HPA),
• kuderedzwa kwekunyanya kuchengetedza zviwanikwa (kube-resource-report, VPA)
• kushandisa Spot zviitiko

Kuchenesa zviwanikwa zvisina kushandiswa

Kushanda munzvimbo inomhanya-mhanya kwakanaka. Tinoda tech masangano accelerated. Kukurumidza kutumirwa kwesoftware kunorevawo kutumirwa kwakawanda kwePR, preview nharaunda, prototypes, uye analytics mhinduro. Zvese zvinoiswa paKubernetes. Ndiani ane nguva yekuchenesa nemaoko ma test deployments? Zviri nyore kukanganwa nezvekudzima kuedza kwevhiki. Bhiri regore rinozoguma rakwira nekuda kwechimwe chinhu chatakakanganwa kuvhara:

(Henning Jacobs:
Zhiza:
(zvinyorwa) Corey Quinn:
Nhema: Yako AWS account ibasa rehuwandu hwevashandisi vaunavo.
Chokwadi: Yako AWS mamakisi ibasa rehuwandu hwemainjiniya auinawo.

Ivan Kurnosov (mumhinduro):
Chokwadi chaicho: Yako AWS mamakisi ibasa rehuwandu hwezvinhu zvawakanganwa kudzima / kudzima.)

Kubernetes Janitor (kube-janitor) inobatsira kuchenesa sumbu rako. Iyo janitor gadziriso inochinjika kune ese ari maviri epasi uye emuno kushandiswa:

• Mitemo yeCluster-wide inogona kutsanangura iyo yakanyanya nguva-ye-kurarama (TTL) yePR/test deployments.
• Zviwanikwa zvega zvinogona kutsanangurwa nejanitor/ttl, semuenzaniso kubvisa otomatiki spike/prototype mushure memazuva manomwe.

Mitemo yakajairika inotsanangurwa muYAML faira. Nzira yaro inopfuudzwa neparameter --rules-file mu kube-janitor. Heino muenzaniso mutemo wekubvisa ese mazita enzvimbo ne -pr- muzita mushure memazuva maviri:

- id: cleanup-resources-from-pull-requests
  resources:
    - namespaces
  jmespath: "contains(metadata.name, '-pr-')"
  ttl: 2d

Uyu unotevera muenzaniso unodzora kushandiswa kwechinyorwa chekunyorera paDeployment uye StatefulSet mapodhi kune ese matsva eDeployments/StatefulSets muna 2020, asi panguva imwechete inobvumira kuitwa kwebvunzo pasina iyi label kwevhiki:

- id: require-application-label
  # удалить deployments и statefulsets без метки "application"
  resources:
    - deployments
    - statefulsets
  # см. http://jmespath.org/specification.html
  jmespath: "!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'"
  ttl: 7d

Mhanyai demo-inogumira maminetsi makumi matatu pane cluster inomhanya kube-janitor:

kubectl run nginx-demo --image=nginx
kubectl annotate deploy nginx-demo janitor/ttl=30m

Imwezve sosi yekuwedzera mitengo inoramba iripo mavhoriyamu (AWS EBS). Kudzima Kubernetes StatefulSet hakubvisi mavhoriyamu ayo anoramba aripo (PVC - PersistentVolumeClaim). Mavhoriyamu eEBS asina kushandiswa anogona kukonzeresa mutengo wemazana emadhora pamwedzi. Kubernetes Janitor ine chimiro chekuchenesa maPVC asina kushandiswa. Semuenzaniso, mutemo uyu uchabvisa ese maPVC asina kuiswa nemodule uye asina kutaurwa neStatefulSet kana CronJob:

# удалить все PVC, которые не смонтированы и на которые не ссылаются StatefulSets
- id: remove-unused-pvcs
  resources:
  - persistentvolumeclaims
  jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
  ttl: 24h

Kubernetes Janitor inogona kukubatsira kuti uchengete cluster yako yakachena uye kudzivirira mutengo wekombuta wemakore kubva pakuwedzera zvishoma nezvishoma. Nekutumirwa uye mirairo yekumisikidza, tevera VERENGA kube-janitor.

Deredza kuyera panguva isiri yekushanda maawa

Yedzo uye masisitimu masisitimu anowanzo kudiwa kuti ashande chete panguva dzebhizinesi. Mamwe maapplication ekugadzira, senge kuseri hofisi/admin maturusi, anodawo kuwanikwa kushoma uye anogona kuremara husiku.

Kubernetes Downscaler (kube-downscaler) inobvumira vashandisi uye vashandisi kudzika pasi sisitimu panguva isiri-yekushanda maawa. Deployments uye StatefulSets inogona kukwira kusvika zero replicas. CronJobs inogona kumiswa. Kubernetes Downscaler inogadzirirwa sumbu rose, imwe kana akawanda mazita enzvimbo, kana zviwanikwa zvemunhu. Iwe unogona kuseta "nguva isina basa" kana, zvakasiyana, "nguva yebasa". Semuenzaniso, kuderedza kuyera zvakanyanya sezvinobvira mukati mehusiku nekupera kwevhiki:

image: hjacobs/kube-downscaler:20.4.3
args:
  - --interval=30
  # не отключать компоненты инфраструктуры
  - --exclude-namespaces=kube-system,infra
  # не отключать kube-downscaler, а также оставить Postgres Operator, чтобы исключенными БД можно было управлять
  - --exclude-deployments=kube-downscaler,postgres-operator
  - --default-uptime=Mon-Fri 08:00-20:00 Europe/Berlin
  - --include-resources=deployments,statefulsets,stacks,cronjobs
  - --deployment-time-annotation=deployment-time

Heino girafu rekuyera cluster worker node pakupera kwevhiki:

Kudzika kubva ~ 13 kusvika ku4 vashandi node zvechokwadi kunoita mutsauko unooneka mubhiri rako reAWS.

Asi zvakadini kana ndichida kushanda panguva yeboka re "downtime"? Kumwe kutumirwa kunogona kubviswa zvachose kubva pakuyera nekuwedzera iyo downscaler / kusabvisa: chokwadi chirevo. Deployments inogona kuvharirwa kwenguva pfupi uchishandisa downscaler/kusabatanidza-kusvika chirevo chine chidhindo chenguva chose mufomati YYYY-MM-DD HH:MM (UTC). Kana zvichidikanwa, sumbu rose rinogona kuderedzwa nekuisa podhi ine chirevo downscaler/force-uptime, semuenzaniso, nekutanga nginx isina chinhu:

kubectl run scale-up --image=nginx
kubectl annotate deploy scale-up janitor/ttl=1h # удалить развертывание через час
kubectl annotate pod $(kubectl get pod -l run=scale-up -o jsonpath="{.items[0].metadata.name}") downscaler/force-uptime=true

Maona VERENGA kube-downscaler, kana iwe uchifarira mirairo yekuendesa uye mamwe maitiro.

Shandisa horizontal autoscaling

Mazhinji maapplication/masevhisi anobata neane simba yekurodha pateni: dzimwe nguva mamodule avo haana chaanoita, uye dzimwe nguva anoshanda akazara. Kushandisa chigarire chikepe chepods kuti utsungirire nepamusoro peak load haisi mari. Kubernetes inotsigira yakatwasuka otomatiki kuyera pane imwe sosi HorizontalPodAutoscaler (HPA). Kushandiswa kweCPU kazhinji chiratidzo chakanaka chekuyera:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 100
        type: Utilization

Zalando yakagadzira chikamu kuti ubatanidze nyore metrics etsika yekuyera: Kube Metrics Adapter (kube-metrics-adapter) ndeye generic metrics adapta yeKubernetes iyo inogona kuunganidza uye kushandira tsika uye ekunze metrics kune yakachinjika autoscaling yemapods. Inotsigira kuyera kwakavakirwa paPrometheus metrics, SQS mitsara, uye mamwe marongero. Semuenzaniso, kuyera kutumirwa kwako kune yakasarudzika metric inomiririrwa neiyo application pachayo seJSON mu / metrics shandisa:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.pods.requests-per-second.json-path/json-key: "$.http_server.rps"
    metric-config.pods.requests-per-second.json-path/path: /metrics
    metric-config.pods.requests-per-second.json-path/port: "9090"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: requests-per-second
      target:
        averageValue: 1k
        type: AverageValue

Kugadzirisa yakatwasuka otomatiki neHPA inofanira kunge iri imwe yezviito zvekuvandudza mashandiro emabasa asina chirevo. Spotify ine mharidzo ine ruzivo rwavo uye kurudziro yeHPA: kuyera deployments yako, kwete chikwama chako.

Deredza zviwanikwa zvekushandisa zvakanyanya

Kubernetes mitoro yebasa inotarisisa yavo CPU / ndangariro zvinodiwa kuburikidza ne "zvikumbiro zvekushandisa." CPU zviwanikwa zvinoyerwa mu virtual cores kana zvakanyanya mu "millicores", semuenzaniso 500m inoreva 50% vCPU. Memory zviwanikwa zvinoyerwa nemabhayiti, uye zvivakwa zvakajairika zvinogona kushandiswa, senge 500Mi, zvinoreva 500 megabytes. Resource inokumbira "kiya" kugona pane yevashandi node, zvichireva kuti pod ine 1000m CPU chikumbiro pane node ine 4 vCPU inosiya matatu chete vCPU inowanikwa kune mamwe mapodhi. [1]

Slack (yakawandisa reserve) ndiwo musiyano pakati pezviwanikwa zvakakumbirwa uye kushandiswa chaiko. Semuenzaniso, pod inokumbira 2 GiB yendangariro asi inongoshandisa 200 MiB ine ~ 1,8 GiB ye "yakawandisa" ndangariro. Kuwedzera kunoda mari. Mumwe anogona kufungidzira kuti 1 GiB yekuyeuka inodhura ~ $ 10 pamwedzi. [2]

Kubernetes Resource Report (kube-resource-report) inoratidza zvakawandisa zvakachengetwa uye inogona kukubatsira kuona savings inogona:

Kubernetes Resource Report inoratidza kuwanda kwakaunganidzwa nekushandisa uye kuraira. Izvi zvinokutendera kuti uwane nzvimbo uko zviwanikwa zvinodikanwa zvinogona kudzikiswa. Iyo yakagadzirwa HTML mushumo inopa chete snapshot yekushandisa zviwanikwa. Iwe unofanirwa kutarisa kuCPU / ndangariro kushandiswa nekufamba kwenguva kuti uone zvakaringana zvikumbiro zvekushandisa. Heino Grafana chati ye "yakajairika" CPU-inorema sevhisi: ese mapodhi ari kushandisa zvakanyanya kushoma pane matatu akakumbirwa CPU cores:

Kudzikisa chikumbiro cheCPU kubva pa3000m kusvika ~ 400m kunosunungura zviwanikwa kune mamwe mabasa uye kunobvumira sumbu kuti rive diki.

"Avhareji yekushandiswa kweCPU yeEC2 mamiriro anowanzo kutenderera mune imwechete-digit muzana renji," anonyora Corey Quinn. Panguva yeEC2 kufungidzira saizi yakakodzera inogona kunge iri sarudzo yakaipaKuchinja mamwe maKubernetes zviwanikwa zvemibvunzo muYAML faira kuri nyore uye kunogona kuunza mari yakawanda.

Asi isu tinoda chaizvo kuti vanhu vachinje hunhu mumafaira eYAML? Aiwa, michina inogona kuzviita zvirinani! Kubernetes Vertical Pod Autoscaler (VPA) inoita izvo chaizvo: inogadzirisa zvikumbiro zvezvishandiso uye zvipingaidzo zvinoenderana nebasa rebasa. Heino muenzaniso girafu yePrometheus CPU zvikumbiro (mutsetse mutete webhuruu) wakagadziridzwa neVPA nekufamba kwenguva:

Zalando inoshandisa VPA mumasumbu ayo ese nokuda kwezvikamu zvezvivako. Asina-akakosha maapplication anogona zvakare kushandisa VPA.

Ndarama kubva kuFairwind chishandiso chinogadzira VPA chega chega chinotumirwa munzvimbo yezita uye chobva charatidza kurudziro yeVPA pane dashboard yayo. Inogona kubatsira vanogadzira kuseta iyo chaiyo CPU / ndangariro zvikumbiro zvekushandisa kwavo:

Ndakanyora diki blogpost nezveVPA muna 2019, uye munguva pfupi yapfuura CNCF End User Community yakakurukura nyaya yeVPA.

Kushandisa EC2 Spot Instances

Chekupedzisira asi chisiri chidiki, mitengo yeAWS EC2 inogona kudzikiswa nekushandisa Spot zviitiko seKubernetes mushandi node. [3]. Spot zviitiko zvinowanikwa kusvika pa90% kuderedzwa zvichienzaniswa nemitengo yeOn-Demand. Kumhanya Kubernetes paEC2 Spot musanganiswa wakanaka: iwe unofanirwa kutsanangura akati wandei akasiyana emhando mhando dzekuwanikwa kwepamusoro, zvichireva kuti unogona kuwana node hombe yemutengo mumwe chete kana wakaderera, uye huwandu hwakawedzerwa hunogona kushandiswa neyakaiswa Kubernetes mitoro yebasa.

Maitiro ekumhanyisa Kubernetes paEC2 Spot? Pane akati wandei sarudzo: shandisa yechitatu bato sevhisi seSpotInst (yava kunzi "Spot", usandibvunza kuti sei), kana kungo wedzera Spot AutoScalingGroup (ASG) kune yako cluster. Semuenzaniso, heino CloudFormation snippet ye "capacity-optimized" Spot ASG ine akawanda mienzaniso mhando:

MySpotAutoScalingGroup:
 Properties:
   HealthCheckGracePeriod: 300
   HealthCheckType: EC2
   MixedInstancesPolicy:
     InstancesDistribution:
       OnDemandPercentageAboveBaseCapacity: 0
       SpotAllocationStrategy: capacity-optimized
     LaunchTemplate:
       LaunchTemplateSpecification:
         LaunchTemplateId: !Ref LaunchTemplate
         Version: !GetAtt LaunchTemplate.LatestVersionNumber
       Overrides:
         - InstanceType: "m4.2xlarge"
         - InstanceType: "m4.4xlarge"
         - InstanceType: "m5.2xlarge"
         - InstanceType: "m5.4xlarge"
         - InstanceType: "r4.2xlarge"
         - InstanceType: "r4.4xlarge"
   LaunchTemplate:
     LaunchTemplateId: !Ref LaunchTemplate
     Version: !GetAtt LaunchTemplate.LatestVersionNumber
   MinSize: 0
   MaxSize: 100
   Tags:
   - Key: k8s.io/cluster-autoscaler/node-template/label/aws.amazon.com/spot
     PropagateAtLaunch: true
     Value: "true"

Mamwe manotsi ekushandisa Spot neKubernetes:

• Iwe unofanirwa kubata Spot kumiswa, semuenzaniso nekubatanidza node kana muenzaniso wamiswa
• Zalando inoshandisa fork official cluster autoscaling ine node dziva zvekutanga
• Spot nodes anogona kumanikidzwa gamuchira "kunyoresa" kwehuwandu hwebasa kuti umhanye muSpot

Summary

Ndinovimba iwe unowana zvimwe zvezvishandiso zvakaratidzwa zvinobatsira mukudzikisa yako gore bhiri. Unogona kuwana zvakawanda zvemukati zvechinyorwa zvakare pa hurukuro yangu paDevOps Kuunganidza 2019 paYouTube uye mumasiraidhi.

Ndeapi maitiro ako akanakisa ekuchengetedza mutengo wegore paKubernetes? Ndapota ndizivisei pa Twitter (@try_except_).

[1] Muchokwadi, isingasviki 3 vCPUs icharamba ichishandiswa sezvo iyo node's throughput inoderedzwa neyakachengetwa system zviwanikwa. Kubernetes anosiyanisa pakati pemuviri node simba uye "provisioned" zviwanikwa (Node Inogoverwa).

[2] Kuverengera muenzaniso: imwe m5.large muenzaniso ine 8 GiB yekuyeuka ~$84 ​​​​pamwedzi (eu-central-1, On-Demand), i.e. kuvhara 1/8 node ingangoita ~$10/mwedzi.

[3] Pane dzimwe nzira dzakawanda dzekuderedza bhiri rako reEC2, sekuti Reserved Instances, Savings Plan, etc. - Handichabata nyaya idzodzo pano, asi unofanira kunyatsodzitarisa!

Dzidza zvakawanda nezvekosi.

Source: www.habr.com