Ku keydi kharashaadka daruuraha ee Kubernetes AWS

Turjumaada maqaalka waxaa la diyaariyay ka hor bilowga koorsada "Meesha kaabayaasha kaabayaasha ee ku salaysan Kubernetes".

Sida loo badbaadiyo kharashyada daruuraha markaad la shaqaynayso Kubernetes? Ma jiro hal xal oo sax ah, laakiin maqaalkani wuxuu qeexayaa dhowr qalab oo kaa caawin kara inaad si waxtar leh u maamusho kheyraadkaaga oo aad hoos u dhigto kharashyada xisaabinta daruuraha.

Waxaan maqaalkan ku qoray Kubernetes ee AWS maskaxda, laakiin waxay ku dabaqi doontaa (ku dhawaad) si la mid ah bixiyeyaasha kale ee daruuraha. Waxaan u malaynayaa in kooxdaadu ay horeba u haysteen iskoorka iskoorkacluster-autoscaler). Ka saarida agabka iyo hoos u dhigista hawlgelintaada waxay kaliya ku badbaadin doontaa lacag haddii ay sidoo kale yarayso qandhadaada shaqaalaha (tusaale EC2).

Maqaalkani waxa uu dabooli doonaa:

• nadiifinta kheyraadka aan la isticmaalin (kube-janitor)
• Iska yaree miisaanka inta lagu jiro saacadaha aan shaqada ahayn (kube-downscaler)
• iyadoo la isticmaalayo toosan autoscaling (HPA),
• hoos u dhigista kaydinta kheyraadka ee xad dhaafka ah (kube-khayr-warbixin, VPA)
• iyadoo la isticmaalayo tusaaleyaal Spot

Nadiifinta agabka aan la isticmaalin

Ka shaqaynta jawi degdeg ah waa mid aad u fiican. Waxaan rabnaa ururada tignoolajiyada la dedejiyey. Soo gudbinta kombuyuutarrada degdega ah waxay sidoo kale ka dhigan tahay hawlgelin badan oo PR ah, bey'ad horudhac ah, tusaalayaal, iyo xalalka falanqaynta. Wax walba waxaa la geeyaa Kubernetes. Yaa wakhti u haysta inuu gacanta ku nadiifiyo meelaynta imtixaanada? Way fududahay in la iloobo tirtirida tijaabo toddobaad jirsatay. Biilasha daruurtu waxay ku dambayn doontaa kor u kaca sababtoo ah wax aan iloobin inaan xidhno:

(Henning Jacobs:
Zhiza:
(xigasho) Corey Quinn:
Khuraafaad: Koontadaada AWS waa shaqo ka mid ah tirada isticmaalayaasha aad haysato.
Xaqiiqda: Dhibcahaaga AWS waa shaqo ka mid ah tirada injineerada aad haysato.

Ivan Kurnosov (oo ka jawaabaya):
Xaqiiqda dhabta ah: Dhibcahaaga AWS waa shaqo ka mid ah tirada waxyaabaha aad illowday inaad joojiso/tirto.)

Kubernetes Janitor (kube-janitor) waxay kaa caawinaysaa nadiifinta kooxdaada. Qaabaynta waardiyaha ayaa u dabacsan isticmaalka caalamiga ah iyo kuwa maxaliga ah labadaba:

• Xeerarka guud ee kooxdu waxay qeexi karaan wakhtiga ugu badan ee lagu noolaan karo (TTL) ee geynta PR/imtixaanka.
• Ilaha shakhsi ahaaneed waxaa lagu sharxi karaa waardiye/ttl, tusaale ahaan in si toos ah meesha looga saaro spike/prototype ka dib 7 maalmood.

Xeerarka guud ayaa lagu qeexay faylka YAML Waddada ay marayso waxa la dhex maraa halbeegga --rules-file in kube-janitor. Halkan waxaa ah xeer tusaale ah oo meesha looga saarayo dhammaan meelaha magacyada -pr- magaca laba maalmood ka dib:

- id: cleanup-resources-from-pull-requests
  resources:
    - namespaces
  jmespath: "contains(metadata.name, '-pr-')"
  ttl: 2d

Tusaalaha soo socdaa waxa uu nidaaminayaa isticmaalka summada codsiga ee ku yaala Gelida iyo StatefulSet bosyada ee dhammaan Soo-dejinta/StatefulSet-ka cusub ee 2020, laakiin isla markaa waxa ay ogolaataa fulinta imtixaanada iyada oo aan summadan la hayn muddo toddobaad ah:

- id: require-application-label
  # удалить deployments и statefulsets без метки "application"
  resources:
    - deployments
    - statefulsets
  # см. http://jmespath.org/specification.html
  jmespath: "!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'"
  ttl: 7d

Ku socodsii demo-wakhti xadidan 30 daqiiqo duleel kutbada ordaya kube-janitor:

kubectl run nginx-demo --image=nginx
kubectl annotate deploy nginx-demo janitor/ttl=30m

Ilaha kale ee kor u kaca kharashku waa qiyaasaha joogtada ah (AWS EBS). Tirtiridda Kubernetes StatefulSet ma tirtirto tirooyinkeeda joogtada ah (PVC - PersistentVolumeClaim). Qiyaasta EBS ee aan la isticmaalin waxay si fudud u keeni kartaa kharashyo boqolaal doolar ah bishii. Kubernetes Janitor waxa uu leeyahay sifo lagu nadiifiyo PVC-yada aan la isticmaalin. Tusaale ahaan, sharcigani wuxuu meesha ka saarayaa dhammaan PVC-yada aan lagu rakibin cutubka oo aan tixraacin StatefulSet ama CronJob:

# удалить все PVC, которые не смонтированы и на которые не ссылаются StatefulSets
- id: remove-unused-pvcs
  resources:
  - persistentvolumeclaims
  jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
  ttl: 24h

Kubernetes Janitor wuxuu kaa caawin karaa inaad nadiifiso kooxdaada oo ka ilaaliso kharashyada xisaabinta daruuraha inay si tartiib tartiib ah u kordho. Si aad u dirto iyo tilmaamaha qaabeynta, raac README kube-janitor.

Iska yaree miisaanka inta lagu jiro saacadaha aan shaqada ahayn

Nidaamyada tijaabada iyo diyaarinta ayaa caadi ahaan looga baahan yahay inay shaqeeyaan kaliya saacadaha shaqada. Qaar ka mid ah codsiyada wax soo saarka, sida xafiiska dambe/qalabka maamulka, waxay sidoo kale u baahan yihiin kaliya helitaan xaddidan waxaana laga yaabaa in la naafo.

Kubernetes Downscaler (kube-downscaler) waxay u ogolaataa isticmaalayaasha iyo hawl-wadeenada inay hoos u dhigaan nidaamka saacadaha aan shaqaynayn. Hawlgelinta iyo StatefulSets waxay qiyaasi karaan nuqullo eber ah. CronJobs ayaa laga yaabaa in la hakiyo. Kubernetes Downscaler waxaa loo habeeyey koox dhan, hal ama in ka badan oo magacyo ah, ama ilo gaar ah. Waxaad dejin kartaa mid ka mid ah "waqtiga shaqo la'aanta" ama, caksigeed, "waqtiga shaqada". Tusaale ahaan, si loo yareeyo miisaanka inta ugu badan ee suurtogalka ah inta lagu jiro habeenkii iyo maalmaha fasaxa:

image: hjacobs/kube-downscaler:20.4.3
args:
  - --interval=30
  # не отключать компоненты инфраструктуры
  - --exclude-namespaces=kube-system,infra
  # не отключать kube-downscaler, а также оставить Postgres Operator, чтобы исключенными БД можно было управлять
  - --exclude-deployments=kube-downscaler,postgres-operator
  - --default-uptime=Mon-Fri 08:00-20:00 Europe/Berlin
  - --include-resources=deployments,statefulsets,stacks,cronjobs
  - --deployment-time-annotation=deployment-time

Halkan waxaa ah garaaf loogu talagalay xisaabinta qanjidhada shaqaalaha kooxda maalmaha fasaxa:

Hoos u dhigista laga bilaabo ~13 ilaa 4 qanjidhada shaqaalaha ayaa hubaal ka dhigaya farqi muuqda ee biilkaaga AWS.

Laakiin maxaa dhacaya haddii aan u baahdo inaan shaqeeyo inta lagu jiro kooxdu "waqti-dhimis"? Hawlgelinta qaarkood ayaa si joogto ah looga saari karaa miisaamidda iyadoo lagu darayo hoos-u-dhigga/ka-reebista: sharraxaad run ah. Gelitaanka si ku meel gaadh ah ayaa looga saari karaa iyadoo la isticmaalayo hoos-u-samaynta/ka-reebista-ilaa sharraxaad leh shaambad wakhti buuxa ah oo qaabka YYYY-MM-DD HH:MM (UTC). Haddii loo baahdo, kooxda oo dhan dib ayaa loo miisaami karaa iyadoo la geynayo boodh ay ku qoran tahay sharraxaad downscaler/force-uptime, tusaale ahaan, adoo bilaabaya nginx maran:

kubectl run scale-up --image=nginx
kubectl annotate deploy scale-up janitor/ttl=1h # удалить развертывание через час
kubectl annotate pod $(kubectl get pod -l run=scale-up -o jsonpath="{.items[0].metadata.name}") downscaler/force-uptime=true

Eeg README kube-downscaler, haddii aad xiisaynayso tilmaamaha geynta iyo doorashooyin dheeraad ah.

Isticmaal toosan autoscaling

Codsiyo/adeegyo badan ayaa wax ka qabta qaab rarid firfircoon: mararka qaarkood qaybahoodu waa shaqo-la'aan, mararka qaarkoodna waxay ku shaqeeyaan si buuxda. Ka shaqaynta raxanno goosad ah oo joogto ah si ay ula qabsadaan culayska ugu sarreeya maaha mid dhaqaale. Kubernetes waxa ay taageertaa isku-dheellitirka tooska ah ee agabka HorizontalPodAutoscaler (HPA). Isticmaalka CPU inta badan waa tilmaame wanaagsan ee cabbirka:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 100
        type: Utilization

Zalando waxa uu abuuray qayb si fudud ugu xidha qiyaasaha caadiga ah ee cabbirka: Kube Metrics Adapter (kube-metrics-adapter) waa adabtarada mitirka guud ee Kubernetes kaas oo ururin kara oo u adeegi kara caadadii iyo cabbirada dibadda ee toosan autoscaling of pods. Waxay taageertaa miisaan-qaadista ku salaysan cabbirka Prometheus, safafka SQS, iyo dejinta kale. Tusaale ahaan, si loo cabbiro geyntaada mitir caadadii uu matalo arjiga laftiisa sida JSON ee / cabbirada u isticmaasho:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.pods.requests-per-second.json-path/json-key: "$.http_server.rps"
    metric-config.pods.requests-per-second.json-path/path: /metrics
    metric-config.pods.requests-per-second.json-path/port: "9090"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: requests-per-second
      target:
        averageValue: 1k
        type: AverageValue

Ku-habaynta autoscaling-ka toosan ee HPA waa in ay ahaato mid ka mid ah ficillada aan caadiga ahayn ee lagu wanaajinayo waxtarka adeegyada aan dal lahayn. Spotify waxay leedahay soo bandhigid khibradooda iyo talooyinka HPA: cabbir meelaha aad geynayso, ee ma aha boorsadaada.

Yaree kaydka xad dhaafka ah

Kubernetes culayska shaqada ayaa go'aamiya baahidooda CPU/xusuusta iyada oo loo marayo "codsiyada kheyraadka." Ilaha CPU waxa lagu cabbiraa koofiyadaha farsamada ama in ka badan “millicores”, tusaale ahaan 500m waxa ay ka dhigan tahay 50% vCPU. Ilaha xusuusta waxaa lagu qiyaasaa bytes, waxaana la isticmaali karaa daba-galayaasha caadiga ah, sida 500Mi, oo macnaheedu yahay 500 megabytes. Codsiyada kheyraadka "quful" ee qanjidhada shaqaalaha, taasoo la macno ah boodh leh codsi 1000m CPU ah oo ku yaal noodhka leh 4 vCPUs ayaa ka tagi doona kaliya 3 vCPUs oo diyaar u ah gabo kale. [1]

Slack ( kayd xad dhaaf ah) waa farqiga u dhexeeya agabka la codsaday iyo isticmaalka dhabta ah. Tusaale ahaan, boodhka codsanaya 2 GiB ee xusuusta ah laakiin kaliya ee adeegsata 200 MiB ayaa leh ~ 1,8 GiB oo ah xusuusta "xad dhaaf". Xad-dhaafku waa lacag. Mid ayaa qiyaas ahaan qiyaasi kara in 1 GiB ee xusuusta aan yarayn uu ku kacayo ~ $10 bishii. [2]

Warbixinta Kheyraadka Kubernetes (kube-resource-report) waxay soo bandhigaysaa kayd xad dhaaf ah waxayna kaa caawin kartaa inaad go'aamiso kartida kaydka:

Warbixinta Kheyraadka Kubernetes waxay muujinaysaa xad-dhaafka la isku daray codsi iyo amar. Tani waxay kuu ogolaaneysaa inaad hesho meelo lagu yarayn karo dalabaadka kheyraadka. Warbixinta HTML ee la soo saaray ayaa bixisa oo kaliya sawirka isticmaalka kheyraadka. Waa inaad eegtaa isticmaalka CPU/xusuusta waqti ka dib si aad u go'aamiso codsiyada kheyraadka ee kugu filan. Halkan waxaa ah jaantuska Grafana ee adeegga "caadiga ah" ee CPU-culus: dhammaan boodhadhka ayaa si aad ah u isticmaalaya wax ka yar 3-da xargaha CPU ee la codsaday:

Hoos u dhigista codsiga CPU laga bilaabo 3000m ilaa ~400m waxay u fasaxaysaa agabka culaysyada kale ee shaqada waxayna u ogolaanaysaa kooxdu inay yaraato.

"Celcelis ahaan isticmaalka CPU ee xaaladaha EC2 inta badan waxay ku dul wareegaysaa tirada boqolleyda hal-god," ayuu qoray Corey Quinn. Marka loo eego EC2 qiyaasida cabbirka saxda ah waxay noqon kartaa go'aan xunBeddelka qaar ka mid ah weydiimaha kheyraadka Kubernetes ee faylka YAML waa sahlan tahay waxayna keeni kartaa kayd badan.

Laakiin dhab ahaantii ma waxaan rabnaa in dadku ku beddelaan qiyamka faylalka YAML? Maya, mashiinada ayaa si ka sii wanaagsan u qaban kara! Kubernetes Autoscaler Pod taagan (VPA) sidaas ayey u sameysaa: waxay la qabsataa codsiyada kheyraadka iyo caqabadaha iyadoo loo eegayo culeyska shaqada. Halkan waxaa ah garaaf tusaale ah oo ku saabsan codsiyada Prometheus CPU (khadka buluuga ah ee khafiifka ah) oo ay habaysay VPA waqti ka dib:

Zalando waxay isticmaashaa VPA dhammaan kooxaheeda qaybaha kaabayaasha. Codsiyada aan muhiimka ahayn waxay sidoo kale isticmaali karaan VPA.

dahabka Laga soo bilaabo Fairwind waa qalab u abuura VPA meel-ka-dhigid kasta oo magaceed ka dibna ku soo bandhigta talada VPA ee dashboardkeeda. Waxay ka caawin kartaa horumariyayaashu inay dejiyaan codsiyada xusuusta CPU ee saxda ah ee codsiyadooda:

Waxaan qoray wax yar blogpost ku saabsan VPA 2019, iyo dhawaanahanba CNCF Community User End ayaa ka wada hadlay arrinta VPA.

Isticmaalka Tusaalooyinka Spot EC2

Ugu dambayn, laakiin ugu yaraan, kharashyada AWS EC2 waa la dhimi karaa iyada oo la adeegsanayo tusaaleyaal Spot sida qanjidhada shaqaalaha Kubernetes [3]. Tusaalooyinka goobta ayaa lagu heli karaa qiimo dhimis ilaa 90% ah marka loo eego qiimayaasha Dalbashada. Kubernetes ku socodsiinta EC2 Spot waa isku darka wanaagsan: waxaad u baahan tahay inaad qeexdo noocyo kala duwan oo tusaaleyaal ah si aad u hesho helitaan sare, taasoo la micno ah inaad heli karto nood weyn oo isku mid ah ama qiimo jaban, iyo awooda korodhka waxaa isticmaali kara Kubernetes culeyska shaqada.

Sida loo maamulo Kubernetes EC2 Spot? Waxaa jira dhawr ikhtiyaar: isticmaal adeeg qolo saddexaad sida SpotInst (hadda loo yaqaan "Spot", ha i waydiin sababta), ama si fudud ugu dar Spot AutoScalingGroup (ASG) kooxdaada. Tusaale ahaan, halkan waxaa ah qayb yar oo CloudFormation ah oo loogu talagalay "awood-ku-habboon" Spot ASG oo leh noocyo badan oo tusaale ah:

MySpotAutoScalingGroup:
 Properties:
   HealthCheckGracePeriod: 300
   HealthCheckType: EC2
   MixedInstancesPolicy:
     InstancesDistribution:
       OnDemandPercentageAboveBaseCapacity: 0
       SpotAllocationStrategy: capacity-optimized
     LaunchTemplate:
       LaunchTemplateSpecification:
         LaunchTemplateId: !Ref LaunchTemplate
         Version: !GetAtt LaunchTemplate.LatestVersionNumber
       Overrides:
         - InstanceType: "m4.2xlarge"
         - InstanceType: "m4.4xlarge"
         - InstanceType: "m5.2xlarge"
         - InstanceType: "m5.4xlarge"
         - InstanceType: "r4.2xlarge"
         - InstanceType: "r4.4xlarge"
   LaunchTemplate:
     LaunchTemplateId: !Ref LaunchTemplate
     Version: !GetAtt LaunchTemplate.LatestVersionNumber
   MinSize: 0
   MaxSize: 100
   Tags:
   - Key: k8s.io/cluster-autoscaler/node-template/label/aws.amazon.com/spot
     PropagateAtLaunch: true
     Value: "true"

Qoraallada qaar ee ku saabsan adeegsiga Spot with Kubernetes:

• Waxaad u baahan tahay inaad xakamayso joojinta Spot, tusaale ahaan adoo isku daraya noodhka marka tusaale ahaan la joojiyo
• Zalando wuxuu isticmaalaa fargeeto autoscaling kooxda rasmiga ah oo leh mudnaanta barkada node
• Duubka barta lagu qasbi karaa aqbal "diiwaangelinta" culeyska shaqada si loogu shaqeeyo Spot

Soo koobid

Waxaan rajeynayaa inaad hesho qaar ka mid ah qalabka la soo bandhigay oo waxtar u leh dhimista biilka daruurahaaga. Waxa kale oo aad ka heli kartaa inta badan waxa ku jira maqaalka Hadalkayga DevOps Gathering 2019 YouTube iyo slides.

Waa maxay hababkaaga ugu fiican ee lagu badbaadinayo kharashyada daruuraha ee Kubernetes? Fadlan ii soo sheeg Twitter (@try_except_).

[1] Dhab ahaantii, in ka yar 3 vCPUs ayaa ahaan doona kuwo la isticmaali karo maadaama wax-soo-saarka noodhka la dhimay agabka nidaamka kaydsan. Kubernetes waxay kala saartaa inta u dhaxaysa awoodda noode jirka iyo agabka "la siiyey"Node loo qoondayn karo).

[2] Tusaalaha xisaabinta: hal m5.weyn oo leh 8 GiB xusuusta ah waa ~$84 ​​bishii (eu-central-1, On-Demand), i.e. xannibidda 1/8 noode waa qiyaastii ~ $10 bishii.

[3] Waxaa jira habab badan oo kale oo aad ku dhimi karto biilka EC2, sida Xaaladaha kaydsan, Qorshaha Kaydka, iwm. - Kuma hadli doono mawduucyadaas halkan, laakiin hubaal waa inaad fiirisaa iyaga!

Wax badan ka baro koorsada.

Source: www.habr.com