Gcina kwiindleko zamafu eKubernetes kwi-AWS

Ukuguqulelwa kwenqaku kwalungiselelwa kwangaphambi kokuqalisa kwekhosi "Iqonga leziseko zophuhliso esekwe kwiKubernetes".

Ukugcina njani iindleko zelifu xa usebenza neKubernetes? Akukho sisombululo sinye esilungileyo, kodwa eli nqaku lichaza izixhobo ezininzi ezinokukunceda ukulawula izixhobo zakho ngokufanelekileyo kwaye unciphise iindleko zakho ze-computing yefu.

Ndibhale eli nqaku nge-Kubernetes ye-AWS engqondweni, kodwa iya kusebenza (phantse) ngendlela efanayo nakwabanye ababoneleli belifu. Ndicinga ukuba (amaqela) enu sele ane-autoscaling emiselweyo (iqela-autoscaler). Ukususa izibonelelo kunye nokunciphisa ukuthunyelwa kwakho kuya kongela imali kuphela ukuba oko kunciphisa izithuthi zakho zabasebenzi (izihlandlo ze-EC2).

Eli nqaku liza kubandakanya:

• ukucoca izixhobo ezingasetyenziswanga (kube-janitor)
• Nciphisa ukukala ngeeyure ezingezizo ezokusebenza (kube-downscaler)
• usebenzisa i-autoscaling ethe tye (HPA),
• ukuncitshiswa kogcino lwezibonelelo olugqithisileyo (kube-resource-ngxelo, VPA)
• usebenzisa iSpot events

Ukucoca izixhobo ezingasetyenziswanga

Ukusebenza kwindawo ekhawulezayo kukuhle. Sifuna imibutho yezobuchwepheshe ikhawulezisiwe. Ukuhanjiswa kwesoftware okukhawulezayo kukwathetha ukusasazwa okungaphezulu kwePR, indawo yokujonga kwangaphambili, iiprototypes, kunye nezisombululo zohlalutyo. Yonke into ifakwe kwi-Kubernetes. Ngubani onexesha lokucoca ukusasazwa kovavanyo ngesandla? Kulula ukulibala malunga nokucima umfuniselo weveki. Ityala lelifu liya kuphelela phezulu ngenxa yento esilibele ukuyivala:

(UHenning Jacobs:
Zhiza:
(ucaphula) uCorey Quinn:
Intsomi: Iakhawunti yakho ye-AWS ngumsebenzi wenani labasebenzisi onawo.
Inyani: Amanqaku akho e-AWS ngumsebenzi wenani leenjineli onazo.

UIvan Kurnosov (impendulo):
Inyani yokwenyani: Amanqaku akho e-AWS ngumsebenzi wenani lezinto ozilibeleyo ukucima/ukucima.)

Kubernetes Janitor (kube-janitor) inceda ukucoca iqela lakho. Ubume be-janitor bubhetyebhetye kusetyenziso lwehlabathi kunye nolwendawo:

• Imithetho ye-Cluster-wide-wide inokuchaza elona xesha liphezulu lokuhlala (TTL) kwiPR/test deployments.
• Izibonelelo zomntu ngamnye zinokuchazwa nge-janitor/ttl, umzekelo ukususa ngokuzenzekelayo i-spike/prototype emva kweentsuku ezi-7.

Imigaqo ngokubanzi ichazwa kwifayile yeYAML. Umendo wayo ugqithiswa kwiparameter --rules-file kwi kube-janitor. Nanku umzekelo womthetho ukususa zonke izithuba zamagama nge -pr- egameni emva kweentsuku ezimbini:

- id: cleanup-resources-from-pull-requests
  resources:
    - namespaces
  jmespath: "contains(metadata.name, '-pr-')"
  ttl: 2d

Lo mzekelo ulandelayo ulawula ukusetyenziswa kwelebhile yesicelo kwi-Deployment kunye ne-StatefulSet pods kuzo zonke ii-Deployments ezintsha/StatefulSets ngo-2020, kodwa kwangaxeshanye ivumela ukwenziwa kovavanyo ngaphandle kwale lebhile kangangeveki:

- id: require-application-label
  # удалить deployments и statefulsets без метки "application"
  resources:
    - deployments
    - statefulsets
  # см. http://jmespath.org/specification.html
  jmespath: "!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'"
  ttl: 7d

Qhuba idemo emiselwe ixesha lemizuzu engama-30 kwiqela eliqhuba kube-janitor:

kubectl run nginx-demo --image=nginx
kubectl annotate deploy nginx-demo janitor/ttl=30m

Omnye umthombo wokunyuka kweendleko yimithamo eqhubekayo (AWS EBS). Ukucima i-Kubernetes StatefulSet ayicimi imiqulu yayo eqhubekayo (PVC - PersistentVolumeClaim). Imiqulu ye-EBS engasetyenziswanga ngokulula inokubangela iindleko ezingamakhulu eedola ngenyanga. I-Kubernetes Janitor inomsebenzi wokucoca iiPVC ezingasetyenziswanga. Umzekelo, lo mgaqo uzakususa zonke iiPVC ezinganyuswanga yimodyuli kwaye ezingabhekiselwanga yi StatefulSet okanye CronJob:

# удалить все PVC, которые не смонтированы и на которые не ссылаются StatefulSets
- id: remove-unused-pvcs
  resources:
  - persistentvolumeclaims
  jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
  ttl: 24h

I-Kubernetes Janitor inokukunceda ugcine iqela lakho licocekile kwaye uthintele iindleko zecomputing yamafu ekubeni zifumbe kancinane. Ngemiyalelo yokusasaza kunye nolungelelwaniso, landela FUNDA kube-janitor.

Nciphisa ukukala ngeeyure ezingezizo ezokusebenza

Iisistim zovavanyo kunye nenqanaba zifuneka ukuba zisebenze kuphela ngeeyure zomsebenzi. Ezinye izicelo zemveliso, ezinje nge-ofisi/izixhobo zolawulo, nazo zifuna ukufumaneka okulinganiselweyo kwaye zinokucinywa ngobusuku.

Kubernetes Downscaler (kube-downscaler) ivumela abasebenzisi kunye nabaqhubi ukuba banciphise inkqubo ngexesha leeyure ezingasebenziyo. Ukusasazwa kunye ne-StatefulSets zinokukhula ziye kwi-zero replicas. I-CronJobs inokunqunyanyiswa. I-Kubernetes Downscaler iqwalaselelwe iqela elipheleleyo, isithuba segama elinye okanye ngaphezulu, okanye izixhobo zomntu ngamnye. Unokuseta nokuba "ixesha elingasebenziyo" okanye, ngokuchaseneyo, "ixesha lomsebenzi". Umzekelo, ukunciphisa isikali kangangoko kunokwenzeka ngobusuku nangeempelaveki:

image: hjacobs/kube-downscaler:20.4.3
args:
  - --interval=30
  # не отключать компоненты инфраструктуры
  - --exclude-namespaces=kube-system,infra
  # не отключать kube-downscaler, а также оставить Postgres Operator, чтобы исключенными БД можно было управлять
  - --exclude-deployments=kube-downscaler,postgres-operator
  - --default-uptime=Mon-Fri 08:00-20:00 Europe/Berlin
  - --include-resources=deployments,statefulsets,stacks,cronjobs
  - --deployment-time-annotation=deployment-time

Nantsi igrafu yokwandisa iindawo zabasebenzi beqela ngeempelaveki:

Ukuhla ukusuka ~ 13 ukuya kwi-4 iindawo zabasebenzi ngokuqinisekileyo kwenza umahluko obonakalayo kwi-bill yakho ye-AWS.

Kodwa kuthekani ukuba kufuneka ndisebenze ngexesha leqela "lokuphumla"? Ukusasazwa okuthile kunokukhutshwa ngokusisigxina ekunyuseni ngokongeza isikali esisezantsi/kungabandakanyi: inkcazo eyinyani. Ukusasazwa kungakhutshelwa ngaphandle okwethutyana kusetyenziswa isikali esisezantsi/kungaquki-de kube yinkcazo enesitampu sexesha esipheleleyo kwifomathi YYYY-MM-DD HH:MM (UTC). Ukuba kukho imfuneko, iqela lilonke lingacuthwa umva ngokubeka ipod ngesihlomelo downscaler/force-uptime, umzekelo, ngokundulula nginx blank:

kubectl run scale-up --image=nginx
kubectl annotate deploy scale-up janitor/ttl=1h # удалить развертывание через час
kubectl annotate pod $(kubectl get pod -l run=scale-up -o jsonpath="{.items[0].metadata.name}") downscaler/force-uptime=true

Yabona FUNDA kube-downscaler, ukuba unomdla kwimiyalelo yokuthunyelwa kunye neenketho ezongezelelweyo.

Sebenzisa i-autoscaling ethe tye

Uninzi lwezicelo/iinkonzo zijongana nepateni yokulayisha eguqukayo: ngamanye amaxesha iimodyuli zazo azisebenzi, kwaye ngamanye amaxesha zisebenza ngokuzeleyo. Ukusebenzisa iqela elisisigxina lee-pods ukumelana nomthwalo ophezulu wencopho akuyonga. I-Kubernetes ixhasa ukulinganisa okuzenzekelayo okuthe tye kuwo wonke uvimba HorizontalPodAutoscaler (HPA). Ukusetyenziswa kwe-CPU ihlala isalathiso esilungileyo sokukala:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 100
        type: Utilization

I-Zalando idale icandelo ukuqhagamshela ngokulula iimetrikhi zesiko lokulinganisa: I-Kube Metrics Adapter (kube-metrics-adapter) sisixhobo sokulinganisa i-generic ye-Kubernetes enokuthi iqokelele kwaye isebenze isiko kunye neemetrics zangaphandle ze-autoscaling ethe tye yeepods. Ixhasa ukulinganisa okusekwe kwiimetrics zePrometheus, imigca ye-SQS, kunye nezinye iisetingi. Umzekelo, ukukala usebenziso lwakho kwimetric yesiko emelwe sisicelo ngokwaso njenge-JSON kwi/usetyenziso lweemetrics:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.pods.requests-per-second.json-path/json-key: "$.http_server.rps"
    metric-config.pods.requests-per-second.json-path/path: /metrics
    metric-config.pods.requests-per-second.json-path/port: "9090"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: requests-per-second
      target:
        averageValue: 1k
        type: AverageValue

Ukuqwalasela i-autoscaling ethe tyaba kunye ne-HPA kufuneka ibe yenye yeentshukumo ezingagqibekanga zokuphucula ukusebenza kakuhle kweenkonzo ezingenammiselo. USpotify unenkcazo ngamava kunye neengcebiso ze-HPA: linganisela ukuthunyelwa kwakho, hayi isipaji sakho.

Nciphisa ubhukisho olugqithisileyo lwezibonelelo

Umthwalo we-Kubernetes umisela iimfuno zabo ze-CPU / imemori ngokusebenzisa "izicelo zezixhobo." Izibonelelo ze-CPU zilinganiswa kwii-cores ezibonakalayo okanye ngokuqhelekileyo "kwi-millicores", umzekelo i-500m ithetha i-50% vCPU. Izixhobo zememori zilinganiswa ngee-bytes, kwaye izimamva eziqhelekileyo zinokusetyenziswa, njenge-500Mi, oku kuthetha ukuba i-500 megabytes. Izicelo zezibonelelo "zokutshixa" umthamo kwiindawo zabasebenzi, oku kuthetha ukuba i-pod ene-1000m yesicelo se-CPU kwi-node ene-4 vCPU iya kushiya kuphela ii-vCPU ezi-3 ezifumaneka kwezinye iipod. [1]

Ukuthoba (ugcino olugqithileyo) ngumahluko phakathi kwezibonelelo eziceliweyo kunye nokusetyenziswa okuyinyani. Umzekelo, i-pod ecela i-2 GiB yememori kodwa isebenzisa kuphela i-200 MiB ine ~ 1,8 GiB yememori "egqithileyo". Ukugqithisa kubiza imali. Umntu unokuqikelela ukuba i-1 GiB yeendleko zememori ezingafunekiyo ~$10 ngenyanga. [2]

Ingxelo yeSibonelelo seKubernetes (kube-resource-report) ibonisa oovimba abagqithileyo kwaye inokukunceda uqikelele ukonga okunokwenzeka:

Ingxelo yeSibonelelo seKubernetes ibonisa ugqithiso oludityaniswe sisicelo kunye nomyalelo. Oku kukuvumela ukuba ufumane iindawo apho iimfuno zobutyebi zinokuncitshiswa. Ingxelo yeHTML eyenziweyo inika umfanekiso okhawulezayo wosetyenziso lwemithombo. Kuya kufuneka ujonge ukusetyenziswa kwe-CPU / imemori ekuhambeni kwexesha ukumisela izicelo ezifanelekileyo zezibonelelo. Nantsi itshathi yaseGrafana yenkonzo "eqhelekileyo" ye-CPU-enzima: zonke ii-pods zisebenzisa kakhulu ngaphantsi kwe-3 eceliweyo ye-CPU cores:

Ukunciphisa isicelo se-CPU ukusuka kwi-3000m ukuya kwi-400m ikhulula izixhobo zeminye imithwalo yomsebenzi kwaye ivumela ukuba iqela libe lincinci.

"Umndilili wokusetyenziswa kwe-CPU yeemeko ze-EC2 zihlala zihamba kuluhlu lwepesenti enye," ubhala uCorey Quinn. Ngelixa i-EC2 ukuqikelela ubungakanani obufanelekileyo kunokuba sisigqibo esibiUkutshintsha eminye imibuzo yemithombo ye-Kubernetes kwifayile ye-YAML kulula kwaye kunokuzisa ukonga okukhulu.

Kodwa ngaba ngokwenene sifuna abantu batshintshe amaxabiso kwiifayile ze-YAML? Hayi, oomatshini banokwenza ngcono kakhulu! Kubernetes I-Autoscaler yePod ethe nkqo (VPA) yenza oko kanye: ihlengahlengisa izicelo zezibonelelo kunye nemiqobo ngokomthwalo womsebenzi. Nanku umzekelo wegrafu yezicelo zePrometheus CPU (umgca oluhlaza okwesibhakabhaka) olungiswe yiVPA ngokuhamba kwexesha:

IZalando isebenzisa iVPA kuwo onke amaqela ayo kumacandelo eziseko zophuhliso. Izicelo ezingabalulekanga zinokusebenzisa iVPA.

Iintsimbi zegolide ukusuka kwi-Fairwind sisixhobo esenza i-VPA kubhengezo ngalunye kwindawo yegama kwaye emva koko ibonise ingcebiso ye-VPA kwideshibhodi yayo. Inokunceda abaphuhlisi bamisele izicelo ezichanekileyo ze-CPU/zenkumbulo kwizicelo zabo:

Ndabhala encinci iblogi malunga neVPA ngo-2019, kwaye kutsha nje CNCF End User Community ixoxe ngomba weVPA.

Ukusebenzisa i-EC2 Spot Instances

Okokugqibela kodwa okungancinci, iindleko ze-AWS EC2 zinokuncitshiswa ngokusebenzisa iimeko ze-Spot njenge-Kubernetes worker nodes [3]. Iimeko zespot ziyafumaneka ngesaphulelo ukuya kuthi ga kwi-90% xa kuthelekiswa namaxabiso akwi-On-Demand. Ukubaleka i-Kubernetes kwi-EC2 Spot yindibaniselwano elungileyo: kufuneka uchaze iindidi ezininzi ezahlukeneyo zemizekelo yokufumaneka okuphezulu, okuthetha ukuba ungafumana indawo enkulu yexabiso elifanayo okanye eliphantsi, kunye nomthamo owandisiweyo unokusetyenziselwa imithwalo yemisebenzi ye-Kubernetes.

Uyiqhuba njani i-Kubernetes kwi-EC2 Spot? Kukho iinketho ezininzi: sebenzisa inkonzo yomntu wesithathu efana neSpotInst (ngoku ebizwa ngokuba yi "Spot", ungandibuzi ukuba kutheni), okanye vele udibanise iSpot AutoScalingGroup (ASG) kwiqela lakho. Umzekelo, nantsi i-CloudFormation snippet ye-"capacity-optimized" Spot ASG eneentlobo ezininzi zemizekelo:

MySpotAutoScalingGroup:
 Properties:
   HealthCheckGracePeriod: 300
   HealthCheckType: EC2
   MixedInstancesPolicy:
     InstancesDistribution:
       OnDemandPercentageAboveBaseCapacity: 0
       SpotAllocationStrategy: capacity-optimized
     LaunchTemplate:
       LaunchTemplateSpecification:
         LaunchTemplateId: !Ref LaunchTemplate
         Version: !GetAtt LaunchTemplate.LatestVersionNumber
       Overrides:
         - InstanceType: "m4.2xlarge"
         - InstanceType: "m4.4xlarge"
         - InstanceType: "m5.2xlarge"
         - InstanceType: "m5.4xlarge"
         - InstanceType: "r4.2xlarge"
         - InstanceType: "r4.4xlarge"
   LaunchTemplate:
     LaunchTemplateId: !Ref LaunchTemplate
     Version: !GetAtt LaunchTemplate.LatestVersionNumber
   MinSize: 0
   MaxSize: 100
   Tags:
   - Key: k8s.io/cluster-autoscaler/node-template/label/aws.amazon.com/spot
     PropagateAtLaunch: true
     Value: "true"

Amanye amanqaku okusebenzisa iSpot ngeKubernetes:

• Kufuneka uphathe ukupheliswa kwe-Spot, umzekelo ngokudibanisa i-node xa umzekelo umisiwe
• Zalando isebenzisa ifolokhwe i-autoscaling yeqela elisemthethweni kunye nezinto eziphambili kwi-node pool
• Amabala amabala inokunyanzeliswa yamkela "ubhaliso" lwemithwalo yomsebenzi ukuze iqhutywe kwi-Spot

Isishwankathelo

Ndiyathemba ukuba uza kufumana ezinye zezixhobo ezibonisiweyo ziluncedo ekunciphiseni ityala lakho lelifu. Ungafumana uninzi lwemixholo yenqaku naku intetho yam kwi-DevOps Gathering 2019 kuYouTube nakwizilayidi.

Zeziphi iindlela zakho ezilungileyo zokugcina iindleko zamafu kwi-Kubernetes? Nceda undazise apha Twitter (@try_ngaphandle_).

[1] Enyanisweni, ngaphantsi kwe-3 vCPUs iya kuhlala isebenziseka njengoko i-throughput ye-node iyancitshiswa yimithombo yenkqubo egciniweyo. I-Kubernetes yahlula phakathi komthamo we-node ebonakalayo kunye nezixhobo "ezibonelelweyo" (I-Node iyakwabiwa).

[2] Umzekelo wokubala: umzekelo omnye we-m5.large kunye ne-8 GiB yememori ~ $ 84 ​​ngenyanga (eu-central-1, On-Demand), i.e. Ukuvala i-1/8 node imalunga ne ~$10/ngenyanga.

[3] Kukho iindlela ezininzi zokunciphisa ityala lakho le-EC2, elifana neZimeko eziGciniweyo, iSicwangciso soSavings, njalo njalo. - Andiyi kuzigubungela ezi zihloko apha, kodwa ngokuqinisekileyo kufuneka uzijonge!

Funda ngakumbi malunga nekhosi.

umthombo: www.habr.com