Phetolelo ea sengoloa e lokisitsoe bosiung ba pele thupelo e qala .
Mokhoa oa ho boloka litšenyehelo tsa maru ha u sebetsa le Kubernetes? Ha ho na tharollo e le 'ngoe e nepahetseng, empa sehlooho sena se hlalosa lisebelisoa tse' maloa tse ka u thusang ho laola lisebelisoa tsa hau ka katleho le ho fokotsa litšenyehelo tsa hau tsa cloud computing.
Ke ngotse sengoloa sena ke nahanne ka Kubernetes bakeng sa AWS, empa se tla sebetsa (hoo e batlang e le) ka tsela e ts'oanang le ho bafani ba bang ba maru. Ke nahana hore lihlopha tsa hau li se li ntse li e-na le autoscaling e hlophisitsoeng (). Ho tlosa lisebelisoa le ho theola thepa ea hau ho tla u bolokela chelete feela haeba ho boetse ho fokotsa sehlopha sa hau sa basebetsi (maemo a EC2).
Sengoliloeng sena se tla akaretsa:
- ho hloekisa lisebelisoa tse sa sebelisoeng ()
- Fokotsa sekhahla nakong ea lihora tse sa sebetseng ()
- ho sebelisa horizontal autoscaling (HPA),
- phokotso ea ho boloka mehloli e feteletseng (, VPA)
- ho sebelisa mohlala oa Spot
Ho hloekisa lisebelisoa tse sa sebelisoeng
Ho sebetsa sebakeng se potlakileng ho monate. Re batla mekhatlo ea theknoloji . Phatlalatso e potlakileng ea software e boetse e bolela ho romelloa ho eketsehileng ho PR, tikoloho ea ponelopele, li-prototypes le litharollo tsa analytics. Tsohle li rometsoe ho Kubernetes. Ke mang ea nang le nako ea ho hloekisa lisebelisoa tsa liteko? Ho bonolo ho lebala ka ho hlakola teko ea beke. Tefiso ea leru e tla qetella e nyoloha ka lebaka la ntho eo re lebetse ho e koala:
(Henning Jacobs:
Zhiza:
(litlhaloso tse qotsitsoeng) Corey Quinn:
Khopolo-taba: Ak'haonte ea hau ea AWS ke ts'ebetso ea palo ea basebelisi bao u nang le bona.
'Nete: Lintlha tsa hau tsa AWS ke ts'ebetso ea palo ea lienjiniere tseo u nang le tsona.
Ivan Kurnosov (ka karabo):
'Nete ea' nete: Lintlha tsa hau tsa AWS ke ts'ebetso ea palo ea lintho tseo u lebetse ho li tima/ho li hlakola.)
(kube-janitor) e thusa ho hloekisa sehlopha sa hau. Tlhophiso ea janitor e fetoha habonolo bakeng sa ts'ebeliso ea lefats'e le ea lehae:
- Melao e pharalletseng ea Cluster e ka hlalosa boholo ba nako ea ho phela (TTL) bakeng sa phepelo ea PR/teko.
- Lisebelisoa tsa motho ka mong li ka hlalosoa ka janitor/ttl, mohlala ho tlosa spike/prototype ka bo eona kamora matsatsi a 7.
Melao e akaretsang e hlalositsoe faeleng ea YAML. Tsela ea eona e fetisoa ka parameter --rules-file ka kube-janitor. Mona ke molao oa mohlala oa ho tlosa libaka tsohle tsa mabitso ka -pr- ka lebitso ka mor'a matsatsi a mabeli:
- id: cleanup-resources-from-pull-requests
resources:
- namespaces
jmespath: "contains(metadata.name, '-pr-')"
ttl: 2dMohlala o latelang o laola ts'ebeliso ea leibole ea kopo ho Deployment le StatefulSet pods bakeng sa Deployments/StatefulSets tsohle tse ncha ka 2020, empa ka nako e ts'oanang e lumella ho etsoa ha liteko ntle le leibole ena bakeng sa beke:
- id: require-application-label
# удалить deployments и statefulsets без метки "application"
resources:
- deployments
- statefulsets
# см. http://jmespath.org/specification.html
jmespath: "!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'"
ttl: 7dMatha demo e lekanyelitsoeng ea nako bakeng sa metsotso e 30 sehlopheng se tsamaisang kube-janitor:
kubectl run nginx-demo --image=nginx
kubectl annotate deploy nginx-demo janitor/ttl=30mMohloli o mong oa litšenyehelo tse ntseng li eketseha ke li-persistent volumes (AWS EBS). Ho hlakola Kubernetes StatefulSet ha ho hlakole meqolo ea eona e phehellang (PVC - PersistentVolumeClaim). Meqolo ea EBS e sa sebelisoeng e ka baka litšenyehelo tsa liranta tse makholo ka khoeli. Kubernetes Janitor e na le tšobotsi ea ho hloekisa li-PVC tse sa sebelisoeng. Mohlala, molao ona o tla tlosa li-PVC tsohle tse sa kenngoeng ke mojule mme li sa boleloa ke StatefulSet kapa CronJob:
# удалить все PVC, которые не смонтированы и на которые не ссылаются StatefulSets
- id: remove-unused-pvcs
resources:
- persistentvolumeclaims
jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
ttl: 24hKubernetes Janitor e ka u thusa ho boloka sehlopha sa hau se hloekile le ho thibela litšenyehelo tsa komporo ea maru hore li se ke tsa eketseha butle. Bakeng sa litaelo tsa phepelo le tlhophiso, latela .
Fokotsa sekhahla nakong ea lihora tse sa sebetseng
Sistimi ea liteko le ea sethala hangata e hlokoa ho sebetsa feela nakong ea lihora tsa khoebo. Lisebelisoa tse ling tsa tlhahiso, joalo ka lisebelisoa tsa morao-rao tsa ofisi/taolo, le tsona li hloka ho fumaneha ho lekanyelitsoeng feela 'me li ka koalloa ka bosiu bo le bong.
(kube-downscaler) e lumella basebelisi le basebelisi ho theola sistimi nakong ea lihora tse sa sebetseng. Deployments le StatefulSets li ka fihla ho zero replicas. CronJobs e kanna ea emisoa. Kubernetes Downscaler e etselitsoe sehlopha sohle, sebaka se le seng kapa ho feta, kapa lisebelisoa tsa motho ka mong. U ka beha "nako ea ho se sebetse" kapa, ka lehlakoreng le leng, "nako ea ho sebetsa". Mohlala, ho fokotsa sekhahla ka hohle kamoo ho ka khonehang bosiu le mafelo-beke:
image: hjacobs/kube-downscaler:20.4.3
args:
- --interval=30
# не отключать компоненты инфраструктуры
- --exclude-namespaces=kube-system,infra
# не отключать kube-downscaler, а также оставить Postgres Operator, чтобы исключенными БД можно было управлять
- --exclude-deployments=kube-downscaler,postgres-operator
- --default-uptime=Mon-Fri 08:00-20:00 Europe/Berlin
- --include-resources=deployments,statefulsets,stacks,cronjobs
- --deployment-time-annotation=deployment-timeMona ke graph bakeng sa ho eketsa libaka tsa basebetsi ka mafelo-beke:
Ho theola ho tloha ho ~ 13 ho isa ho li-node tsa basebetsi tse 4 ka sebele ho etsa phapang e bonahalang bili ea hau ea AWS.
Empa ho thoe'ng haeba ke hloka ho sebetsa nakong ea "downtime" ea sehlopha? Lisebelisoa tse ling li ka qheleloa ka thoko ho ea ho sekala ka ho eketsa theolelo/ho qhelela ka thoko: tlhaloso ea 'nete. Ho romelloa ho ka qheleloa ka thoko ka nakoana ho sebelisoa sehatisi se tlase-tlase-ho fihlela tlhaloso e nang le setempe sa nako se felletseng ka sebopeho sa YYYY-MM-DD HH:MM (UTC). Haeba ho hlokahala, sehlopha sohle se ka fokotsoa ka ho kenya pod e nang le tlhaloso downscaler/force-uptime, mohlala, ka ho qala nginx e se nang letho:
kubectl run scale-up --image=nginx
kubectl annotate deploy scale-up janitor/ttl=1h # удалить развертывание через час
kubectl annotate pod $(kubectl get pod -l run=scale-up -o jsonpath="{.items[0].metadata.name}") downscaler/force-uptime=trueBona , haeba u thahasella litaelo tsa ho romelloa le likhetho tse eketsehileng.
Sebelisa autoscaling e otlolohileng
Lisebelisoa / lits'ebeletso tse ngata li sebetsana le mokhoa o matla oa ho jarolla: ka linako tse ling li-module tsa tsona ha li sebetse, 'me ka linako tse ling li sebetsa ka botlalo. Ho sebetsa ka bongata bo sa feleng ba li-pods ho sebetsana le maemo a phahameng a holimo ha ho na chelete. Kubernetes e ts'ehetsa ho pharalla ha auto-auto ho pholletsa le sesebelisoa (HPA). Tšebeliso ea CPU hangata ke sesupo se setle sa ho hola:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: my-app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
averageUtilization: 100
type: UtilizationZalando e thehile karolo ho hokahanya metrics ea tloaelo habonolo bakeng sa ho lekanya: (kube-metrics-adapter) ke adaptara ea metrics ea generic bakeng sa Kubernetes e ka bokellang le ho fana ka metrics ea tloaelo le ea kantle bakeng sa ho otloloha ha li-pods. E tšehetsa sekhahla se ipapisitseng le metrics ea Prometheus, mela ea SQS le litlhophiso tse ling. Mohlala, ho lekanya phetisetso ea hau ho metric ea tloaelo e emeloang ke sesebelisoa ka bosona joalo ka JSON ho / tšebeliso ea metrics:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa
annotations:
# metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
metric-config.pods.requests-per-second.json-path/json-key: "$.http_server.rps"
metric-config.pods.requests-per-second.json-path/path: /metrics
metric-config.pods.requests-per-second.json-path/port: "9090"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: requests-per-second
target:
averageValue: 1k
type: AverageValueHo hlophisa autoscaling e otlolohileng le HPA e lokela ho ba e 'ngoe ea liketso tsa kamehla ho ntlafatsa ts'ebetso ea lits'ebeletso tse se nang naha. Spotify e na le tlhahiso e nang le boiphihlelo le likhothaletso tsa HPA: .
Fokotsa tšebeliso e feteletseng ea lisebelisoa
Mesebetsi ea Kubernetes e khetholla litlhoko tsa bona tsa CPU / memori ka "likopo tsa lisebelisoa." Lisebelisoa tsa CPU li lekantsoe ka li-cores kapa hangata ho "millicores", mohlala 500m e bolela 50% vCPU. Lisebelisoa tsa memori li lekanngoa ka li-byte, 'me li-suffixes tse tloaelehileng li ka sebelisoa, tse kang 500Mi, e bolelang 500 megabytes. Likopo tsa lisebelisoa tsa "notlolo" ho li-node tsa basebetsi, ho bolelang hore pod e nang le kopo ea 1000m CPU sebakeng se nang le li-vCPU tse 4 e tla siea li-vCPU tse 3 feela tse fumanehang ho li-pods tse ling.
Slack (sebaka se feteletseng) ke phapang lipakeng tsa lisebelisoa tse kopiloeng le ts'ebeliso ea 'nete. Mohlala, pod e kopang 2 GiB ea mohopolo empa e sebelisa 200 MiB feela e na le ~ 1,8 GiB ea memori e "feteletseng". Ho feta tekano ho bitsa chelete. Motho a ka hakanya hore 1 GiB ea memori e sa sebetseng e bitsa ~ $ 10 ka khoeli.
(kube-resource-report) e bonts'a mehloli e mengata mme e ka u thusa ho tseba bokhoni ba ho boloka:
e bonts'a ho fetelletseng ho kopantsoeng ke ts'ebeliso le taelo. Sena se o nolofalletsa ho fumana libaka tseo litlhoko tsa lisebelisoa li ka fokotsoang. Tlaleho ea HTML e hlahisitsoeng e fana feela ka setšoantšo sa tšebeliso ea lisebelisoa. U lokela ho sheba ts'ebeliso ea CPU / memori ha nako e ntse e feta ho fumana likopo tse lekaneng tsa lisebelisoa. Mona ke chate ea Grafana bakeng sa ts'ebeletso e "tloaelehileng" ea CPU-boima: li-pods kaofela li sebelisa ka tlase ho li-cores tse 3 tse kopiloeng tsa CPU:
Ho fokotsa kopo ea CPU ho tloha ho 3000m ho isa ~ 400m ho lokolla lisebelisoa bakeng sa meroalo e meng ea mosebetsi mme ho lumella sehlopha ho ba nyane.
"Karolelano ea tšebeliso ea CPU ea maemo a EC2 hangata e tsamaea ka har'a karolo ea palo e le 'ngoe," . Nakong ea EC2 Ho fetola lipotso tsa lisebelisoa tsa Kubernetes faeleng ea YAML ho bonolo ebile ho ka tlisa chelete e ngata.
Empa na ehlile re batla hore batho ba fetole boleng ho lifaele tsa YAML? Che, mechini e ka e etsa hamolemonyana! Kubernetes (VPA) e etsa joalo feela: e fetola likopo tsa lisebelisoa le lithibelo ho latela boima ba mosebetsi. Mona ke mohlala oa likopo tsa Prometheus CPU (mohala o mosesaane o moputsoa) o fetotsoeng ke VPA ha nako e ntse e ea:
bakeng sa likarolo tsa motheo. Lisebelisoa tse seng tsa bohlokoa le tsona li ka sebelisa VPA.
ho tsoa ho Fairwind ke sesebelisoa se etsang VPA bakeng sa thomello e 'ngoe le e' ngoe sebakeng sa mabitso ebe se hlahisa khothaletso ea VPA ho dashboard ea eona. E ka thusa bahlahisi ho theha likopo tse nepahetseng tsa CPU / memori bakeng sa lits'ebetso tsa bona:
Ke ngotse e nyane ka 2019, le haufinyane tjena .
Ho sebelisa EC2 Spot Instances
Qetellong, litšenyehelo tsa AWS EC2 li ka fokotsoa ka ho sebelisa maemo a Spot joalo ka li-node tsa basebetsi ba Kubernetes. . Litheko tsa Spot li fumaneha ho fihla ho 90% litheolelo ha li bapisoa le litheko tsa On-Demand. Running Kubernetes ho EC2 Spot ke motsoako o motle: o hloka ho hlakisa mefuta e fapaneng ea mehlala bakeng sa phumaneho e phahameng, ho bolelang hore o ka fumana node e kholoanyane ka theko e tšoanang kapa e tlase, mme matla a eketsehileng a ka sebelisoa ke mojaro oa mosebetsi oa Kubernetes.
Mokhoa oa ho tsamaisa Kubernetes ho EC2 Spot? Ho na le likhetho tse 'maloa: sebelisa ts'ebeletso ea mokha oa boraro joalo ka SpotInst (eo hona joale e bitsoang "Spot", u se mpotse hore na hobaneng), kapa eketsa Spot AutoScalingGroup (ASG) sehlopheng sa hau. Mohlala, mona ke snippet ea CloudFormation bakeng sa "capacity-optimized" Spot ASG e nang le mehlala e mengata:
MySpotAutoScalingGroup:
Properties:
HealthCheckGracePeriod: 300
HealthCheckType: EC2
MixedInstancesPolicy:
InstancesDistribution:
OnDemandPercentageAboveBaseCapacity: 0
SpotAllocationStrategy: capacity-optimized
LaunchTemplate:
LaunchTemplateSpecification:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
Overrides:
- InstanceType: "m4.2xlarge"
- InstanceType: "m4.4xlarge"
- InstanceType: "m5.2xlarge"
- InstanceType: "m5.4xlarge"
- InstanceType: "r4.2xlarge"
- InstanceType: "r4.4xlarge"
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
MinSize: 0
MaxSize: 100
Tags:
- Key: k8s.io/cluster-autoscaler/node-template/label/aws.amazon.com/spot
PropagateAtLaunch: true
Value: "true"Lintlha tse ling mabapi le ho sebelisa Spot le Kubernetes:
- U hloka ho sebetsana le ho felisoa ha Spot, mohlala, ka ho kopanya node ha mohlala o emisitsoe
- Zalando e sebelisa semmuso cluster autoscaling e nang le lintho tse tlang pele ho node pool
- Spot nodes amohela "ngoliso" ea mesebetsi e mengata e tla etsoa sebakeng sa Spot
Kakaretso
Ke tšepa hore u tla fumana tse ling tsa lisebelisoa tse hlahisitsoeng li le molemo ho fokotsa sekoloto sa hau sa maru. U ka fumana boholo ba litaba tsa sengoloa hape ho .
Mekhoa ea hau e metle ea ho boloka litšenyehelo tsa maru ho Kubernetes ke efe? Ke kopa o ntsebise ho .
Ebile, li-vCPU tse ka tlase ho 3 li tla lula li sebelisoa ha ts'ebetso ea node e fokotsoa ke lisebelisoa tsa sistimi e bolokiloeng. Kubernetes e khetholla pakeng tsa matla a node ea 'mele le lisebelisoa "tse fanoeng" ().
Mohlala oa lipalo: mohlala o le mong oa m5.kholo o nang le 8 GiB ea memori ke ~$84 ka khoeli (eu-central-1, On-Demand), i.e. ho thiba 1/8 node e ka ba ~$10/khoeli.
Ho na le mekhoa e meng e mengata ea ho fokotsa sekoloto sa hau sa EC2, joalo ka Maemo a Behelitsoeng, Moralo oa Poloko, joalo-joalo - Nke ke ka bua ka lihlooho tseo mona, empa ka sebele u lokela ho li sheba!
Source: www.habr.com