Kumasulira kwa nkhaniyi kunakonzedwa madzulo a chiyambi cha maphunziro .
Momwe mungasungire pamitengo yamtambo mukamagwira ntchito ndi Kubernetes? Palibe yankho limodzi lolondola, koma nkhaniyi ikufotokoza zida zingapo zomwe zingakuthandizeni kuyendetsa bwino chuma chanu ndikuchepetsa ndalama zanu zamakompyuta.
Ndinalemba nkhaniyi ndi Kubernetes kwa AWS m'maganizo, koma idzagwira ntchito (pafupifupi) chimodzimodzi kwa ena opereka mitambo. Ndikuganiza kuti magulu anu ali kale ndi autoscaling yokonzedwa (). Kuchotsa zothandizira ndikuchepetsa ntchito yanu kungakupulumutseni ndalama ngati kumachepetsanso gulu lanu la ogwira ntchito (nthawi za EC2).
Nkhaniyi ifotokoza:
- kuyeretsa zinthu zosagwiritsidwa ntchito ()
- Kuchepetsa makulitsidwe panthawi yomwe sikugwira ntchito ()
- kugwiritsa ntchito horizontal autoscaling (HPA),
- kuchepetsa kusungitsa zinthu mopitirira muyeso (, VPA)
- pogwiritsa ntchito Spot zitsanzo
Kuyeretsa zinthu zosagwiritsidwa ntchito
Kugwira ntchito m'malo othamanga ndikwabwino. Tikufuna mabungwe aukadaulo . Kutumiza mwachangu kwa mapulogalamu kumatanthawuzanso kutumizidwa kwa PR, malo owoneratu, ma prototypes, ndi mayankho a analytics. Chilichonse chimayikidwa pa Kubernetes. Ndani ali ndi nthawi yoyeretsa pamanja zoyeserera? Ndi zophweka kuiwala za deleting sabata chaka kuyesera. Bilu yamtambo imatha kukwera chifukwa cha zomwe tayiwala kutseka:
(Henning Jacobs:
Zhiza:
(zolemba) Corey Quinn:
Bodza: Akaunti yanu ya AWS ndi ntchito ya kuchuluka kwa ogwiritsa ntchito omwe muli nawo.
Zoona zake: Kupambana kwanu kwa AWS ndi ntchito ya kuchuluka kwa mainjiniya omwe muli nawo.
Ivan Kurnosov (poyankha):
Zowona Zenizeni: Kupambana kwanu kwa AWS ndi ntchito ya kuchuluka kwa zinthu zomwe munaiwala kuzimitsa/kuzimitsa.)
(kube-janitor) imathandizira kuyeretsa gulu lanu. Kukonzekera kwa janitor ndikosavuta kugwiritsa ntchito padziko lonse lapansi komanso kwanuko:
- Malamulo a Cluster-wide amatha kufotokozera nthawi yayitali yokhala ndi moyo (TTL) ya PR/mayeso otumizidwa.
- Zothandizira zapayekha zitha kufotokozedwa ndi janitor/ttl, mwachitsanzo kuchotsa spike/prototype pakadutsa masiku 7.
Malamulo onse amafotokozedwa mufayilo ya YAML. Njira yake imadutsa parameter --rules-file mu kube-janitor. Pano pali lamulo lachitsanzo kuti muchotse malo onse a mayina -pr- m'dzina pambuyo pa masiku awiri:
- id: cleanup-resources-from-pull-requests
resources:
- namespaces
jmespath: "contains(metadata.name, '-pr-')"
ttl: 2dChitsanzo chotsatirachi chimayang'anira kagwiritsidwe ntchito ka lemba yofunsira pa Deployment and StatefulSet pods pa Deployments/StatefulSets zonse zatsopano mu 2020, koma nthawi yomweyo zimalola kuyesedwa popanda chizindikirochi kwa sabata imodzi:
- id: require-application-label
# удалить deployments и statefulsets без метки "application"
resources:
- deployments
- statefulsets
# см. http://jmespath.org/specification.html
jmespath: "!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'"
ttl: 7dYendetsani chiwonetsero chanthawi yochepa kwa mphindi 30 pagulu lomwe likuyendetsa kube-janitor:
kubectl run nginx-demo --image=nginx
kubectl annotate deploy nginx-demo janitor/ttl=30mNjira ina yowonjezeretsa ndalama ndi ma voliyumu opitilira (AWS EBS). Kuchotsa Kubernetes StatefulSet sikuchotsa ma voliyumu ake osalekeza (PVC - PersistentVolumeClaim). Ma voliyumu osagwiritsidwa ntchito a EBS atha kubweretsa ndalama zokwana madola mazana ambiri pamwezi. Kubernetes Janitor ali ndi gawo loyeretsa ma PVC osagwiritsidwa ntchito. Mwachitsanzo, lamuloli lichotsa ma PVC onse omwe sanakwezedwe ndi gawo ndipo sanatchulidwe ndi StatefulSet kapena CronJob:
# удалить все PVC, которые не смонтированы и на которые не ссылаются StatefulSets
- id: remove-unused-pvcs
resources:
- persistentvolumeclaims
jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
ttl: 24hKubernetes Janitor ikhoza kukuthandizani kuti gulu lanu likhale loyera komanso kupewa kuti mtengo wa computing wamtambo usachuluke pang'onopang'ono. Kuti mumve malangizo otumizira ndi kasinthidwe, tsatirani .
Chepetsani makulitsidwe panthawi yomwe simukugwira ntchito
Makina oyesera ndi masitepe nthawi zambiri amafunikira kuti azigwira ntchito panthawi yantchito. Ntchito zina zopanga, monga zida zobwerera kuofesi/zoyang'anira, zimangofunikanso kupezeka kochepa ndipo zitha kuzimitsidwa usiku wonse.
(kube-downscaler) imalola ogwiritsa ntchito ndi ogwiritsa ntchito kuti achepetse dongosolo munthawi yosagwira ntchito. Ma Deployments ndi StatefulSets amatha kufika ku ziro replicas. CronJobs ikhoza kuyimitsidwa. Kubernetes Downscaler imapangidwira gulu lonse, malo amodzi kapena angapo, kapena zothandizira payekha. Mukhoza kukhazikitsa "nthawi yopanda ntchito" kapena, mosiyana, "nthawi yogwira ntchito". Mwachitsanzo, kuchepetsa makulitsidwe momwe mungathere usiku ndi kumapeto kwa sabata:
image: hjacobs/kube-downscaler:20.4.3
args:
- --interval=30
# не отключать компоненты инфраструктуры
- --exclude-namespaces=kube-system,infra
# не отключать kube-downscaler, а также оставить Postgres Operator, чтобы исключенными БД можно было управлять
- --exclude-deployments=kube-downscaler,postgres-operator
- --default-uptime=Mon-Fri 08:00-20:00 Europe/Berlin
- --include-resources=deployments,statefulsets,stacks,cronjobs
- --deployment-time-annotation=deployment-timeNayi graph yokulitsa ma node ogwira ntchito m'magulu kumapeto kwa sabata:
Kutsika kuchokera ku ~ 13 mpaka 4 node za ogwira ntchito kumapangitsa kusiyana kwakukulu mu bilu yanu ya AWS.
Koma bwanji ngati ndikufunika kugwira ntchito pamagulu "nthawi yopuma"? Kutumiza kwina kutha kuchotsedwa pakukulitsa powonjezera chotsitsa/kupatula: zofotokozera zenizeni. Kutumiza kutha kuchotsedwa kwakanthawi pogwiritsa ntchito sikelo yotsika/kupatula-mpaka mawu ofotokozera okhala ndi sitampu yanthawi zonse mumtundu wa YYYY-MM-DD HH:MM (UTC). Ngati ndi kotheka, gulu lonse likhoza kuchepetsedwa poyika poto ndi mawuwo downscaler/force-uptime, mwachitsanzo, poyambitsa nginx opanda kanthu:
kubectl run scale-up --image=nginx
kubectl annotate deploy scale-up janitor/ttl=1h # удалить развертывание через час
kubectl annotate pod $(kubectl get pod -l run=scale-up -o jsonpath="{.items[0].metadata.name}") downscaler/force-uptime=trueOnani , ngati mukufuna malangizo otumizira ndi zina zowonjezera.
Gwiritsani ntchito horizontal autoscaling
Mapulogalamu / mautumiki ambiri amakumana ndi njira yosinthira: nthawi zina ma module awo amakhala opanda pake, ndipo nthawi zina amagwira ntchito mokwanira. Kugwiritsira ntchito zombo zokhazikika kuti muthe kupirira kuchuluka kwapamwamba sikuli ndalama. Kubernetes imathandizira kuwongolera molunjika pazida (HPA). Kugwiritsa ntchito kwa CPU nthawi zambiri kumakhala chizindikiro chabwino pakukulitsa:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: my-app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
averageUtilization: 100
type: UtilizationZalando yapanga gawo lolumikizira mosavuta ma metric achikhalidwe: (kube-metrics-adapter) ndi chosinthira cha generic metrics cha Kubernetes chomwe chimatha kusonkhanitsa ndikutumizira ma metric akunja ndi ma metric akunja owongolera ma pod. Imathandizira makulitsidwe potengera ma metric a Prometheus, mizere ya SQS, ndi zina. Mwachitsanzo, kuti muwongolere kutumizidwa kwanu kumetric yomwe imayimiridwa ndi pulogalamu yokhayo ngati JSON mu /magwiritsidwe ntchito:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa
annotations:
# metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
metric-config.pods.requests-per-second.json-path/json-key: "$.http_server.rps"
metric-config.pods.requests-per-second.json-path/path: /metrics
metric-config.pods.requests-per-second.json-path/port: "9090"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: requests-per-second
target:
averageValue: 1k
type: AverageValueKukonza ma autoscaling opingasa ndi HPA kuyenera kukhala chimodzi mwazinthu zosasinthika kuti muwongolere bwino ntchito zopanda malire. Spotify ali ndi ulaliki ndi zomwe adakumana nazo komanso malingaliro a HPA: .
Chepetsani kusungitsa zida
Kuchuluka kwa ntchito za Kubernetes kumatsimikizira zosowa zawo za CPU / kukumbukira kudzera "zopempha zothandizira." Zothandizira za CPU zimayesedwa m'makona enieni kapena nthawi zambiri mu "millicores", mwachitsanzo 500m ikutanthauza 50% vCPU. Zipangizo zokumbukira zimayesedwa ndi ma byte, ndipo ma suffixes wamba amatha kugwiritsidwa ntchito, monga 500Mi, kutanthauza 500 megabytes. Zofunsira zofunsira "lock" pazantchito, kutanthauza kuti pod yokhala ndi pempho la 1000m CPU pa node yokhala ndi 4 vCPU ingosiya ma vCPU atatu okha omwe amapezeka ku ma pod ena.
Slack (kusungirako mopitirira muyeso) ndiye kusiyana pakati pa zinthu zomwe zapemphedwa ndi zomwe zimagwiritsidwa ntchito. Mwachitsanzo, pod yomwe imapempha 2 GiB ya kukumbukira koma imagwiritsa ntchito 200 MiB ili ndi ~ 1,8 GiB ya "kupitirira" kukumbukira. Zowonjezera zimawononga ndalama. Mmodzi akhoza kuyerekeza kuti 1 GiB ya kukumbukira kofunikira kumawononga ~ $ 10 pamwezi.
(kube-resource-report) imawonetsa nkhokwe zambiri ndipo imatha kukuthandizani kudziwa momwe mungasungire:
zikuwonetsa kuchulukira komwe kumaphatikizidwa ndi ntchito ndi lamulo. Izi zimakupatsani mwayi wopeza malo omwe zofuna zazinthu zitha kuchepetsedwa. Lipoti la HTML lopangidwa limangopereka chithunzithunzi cha kugwiritsidwa ntchito kwazinthu. Muyenera kuyang'ana kugwiritsidwa ntchito kwa CPU / kukumbukira pakapita nthawi kuti muwone zopempha zokwanira. Nayi tchati cha Grafana cha ntchito "yolemetsa" ya CPU: ma pod onse akugwiritsa ntchito zochepa kwambiri kuposa ma cores atatu omwe adafunsidwa:
Kuchepetsa pempho la CPU kuchokera ku 3000m kupita ku ~ 400m kumamasula zida zogwirira ntchito zina ndikulola gululo kukhala laling'ono.
"Avereji ya CPU yogwiritsa ntchito nthawi za EC2 nthawi zambiri imayenda pamaperesenti amodzi," . Kwa EC2 Kusintha mafunso ena a Kubernetes mu fayilo ya YAML ndikosavuta ndipo kumatha kubweretsa ndalama zambiri.
Koma kodi tikufunadi kuti anthu asinthe ma fayilo a YAML? Ayi, makina amatha kuchita bwino kwambiri! Kubernetes (VPA) imachita izi: imasintha zopempha ndi zopinga malinga ndi kuchuluka kwa ntchito. Nayi chitsanzo cha zopempha za Prometheus CPU (mzere wopyapyala wabuluu) wosinthidwa ndi VPA pakapita nthawi:
kwa zigawo zikuluzikulu. Mapulogalamu osafunikira angagwiritsenso ntchito VPA.
kuchokera ku Fairwind ndi chida chomwe chimapanga VPA pa kutumizidwa kulikonse mu malo a mayina ndiyeno kuwonetsa malingaliro a VPA pa dashboard yake. Itha kuthandiza otukula kukhazikitsa zopempha zolondola za CPU/memory pamapulogalamu awo:
Ndinalemba pang'ono mu 2019, ndipo posachedwa .
Kugwiritsa ntchito EC2 Spot Instances
Pomaliza, mtengo wa AWS EC2 ukhoza kuchepetsedwa pogwiritsa ntchito ma Spot monga ma node antchito a Kubernetes. . Zochitika zaposachedwa zimapezeka mpaka kuchotsera 90% poyerekeza ndi mitengo ya On-Demand. Kuthamanga Kubernetes pa EC2 Spot ndikophatikiza kwabwino: muyenera kutchula mitundu ingapo yosiyana kuti mupeze kupezeka kwapamwamba, kutanthauza kuti mutha kupeza node yokulirapo pamtengo womwewo kapena wotsika, ndipo kuchuluka kwachulukidwe kungagwiritsidwe ntchito ndi katundu wa Kubernetes.
Momwe mungayendetsere Kubernetes pa EC2 Spot? Pali zosankha zingapo: gwiritsani ntchito chipani chachitatu monga SpotInst (tsopano chotchedwa "Spot", osandifunsa chifukwa chake), kapena ingowonjezerani Spot AutoScalingGroup (ASG) kugulu lanu. Mwachitsanzo, nayi mawu a CloudFormation a "capacity-optimized" Spot ASG okhala ndi mitundu ingapo:
MySpotAutoScalingGroup:
Properties:
HealthCheckGracePeriod: 300
HealthCheckType: EC2
MixedInstancesPolicy:
InstancesDistribution:
OnDemandPercentageAboveBaseCapacity: 0
SpotAllocationStrategy: capacity-optimized
LaunchTemplate:
LaunchTemplateSpecification:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
Overrides:
- InstanceType: "m4.2xlarge"
- InstanceType: "m4.4xlarge"
- InstanceType: "m5.2xlarge"
- InstanceType: "m5.4xlarge"
- InstanceType: "r4.2xlarge"
- InstanceType: "r4.4xlarge"
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
MinSize: 0
MaxSize: 100
Tags:
- Key: k8s.io/cluster-autoscaler/node-template/label/aws.amazon.com/spot
PropagateAtLaunch: true
Value: "true"Zolemba zina pakugwiritsa ntchito Spot ndi Kubernetes:
- Muyenera kuthana ndi Spot terminations, mwachitsanzo pophatikiza mfundoyi pomwe chitsanzocho chayimitsidwa
- Zalando amagwiritsa ma cluster autoscaling okhala ndi zofunikira za node pool
- Node zamawanga vomerezani "kulembetsa" kwa ntchito zambiri kuti ziyendetsedwe ku Spot
Chidule
Ndikukhulupirira kuti mupeza zida zina zomwe zaperekedwa zothandiza pochepetsa bilu yanu yamtambo. Mutha kupeza zambiri zomwe zili m'nkhaniyi komanso pa .
Kodi njira zanu zabwino zotani zopulumutsira mitengo yamtambo pa Kubernetes? Chonde ndidziwitseni pa .
M'malo mwake, ma vCPU ochepera 3 adzakhalabe ogwiritsidwa ntchito pomwe kutulutsa kwa node kumachepetsedwa ndi zida zosungidwa. Kubernetes amasiyanitsa pakati pa mphamvu ya node ndi zinthu "zoperekedwa" ().
Chitsanzo chowerengera: chitsanzo chimodzi cha m5.large chokhala ndi 8 GiB ya kukumbukira ndi ~$84 pa mwezi (eu-central-1, On-Demand), i.e. kutsekereza 1/8 node ndi pafupifupi ~$10/mwezi.
Pali njira zambiri zochepetsera bilu yanu ya EC2, monga Zosungirako Zosungidwa, Mapulani Osungira, ndi zina zotero - sindifotokoza mitu imeneyi pano, koma muyenera kuyiyang'ana!
Source: www.habr.com