ααΆαααααααα’αααααααααΌαααΆααααα
ααα
αα»αααααα
αΆααααααΎααααααααα·ααααΆ
ααΎααααΎααΌα
ααααα
ααΎααααΈαααααααΎααΆαα
αααΆαααΎααααα
αααααααΎααΆαααΆαα½α Kubernetes? αα·αααΆααααααααααΆαααααΉαααααΌααααα½ααα ααα»ααααα’αααααααααα·αααααΆα’αααΈα§αααααααΆα
αααΎααααα’αΆα
αα½αα’ααααααα»αααΆααααααααααααααΆαααααα’αααα±ααααΆααααααΆαααααα·αααααΆα αα·αααΆαααααααααΆαα
αααΆαααΎαα»αααααΌαααααΎαααααααα’αααα
αααα»αααΆααααααα’ααααααααααΆαα½α Kubernetes αααααΆαα AWS αα
αααα»αα
α·ααα ααα»ααααααΆααΉαα’αα»αααα (ααααΎααα) ααΌα
ααααΆαα
ααΉαα’ααααααααααααΆαααααααααααα αααα»αααααααααΆα
αααααααααα’αααααΆαααααααα
ααΆαααααααα autoscaling αα½α
α αΎα (
α’ααααααααααΉαααααααααααα
- αααα’αΆαααααΆαααααα·αααααΎ (
kube-janitor ) - ααΆαααααααααΆαααααΎααΆαααααααΆααααα»αα’αα‘α»αααααα·αααααΎααΆα (
kube-downscaler ) - αααααααΎααΆαααααΎααΆαααααααΆααααααααααααααααα·ααααα (HPA),
- ααΆαααΆαααααααααΆααααααααΆαααΎαααα (
kube-resource-report , VPA) - αααααααΎα§ααΆα ααα Spot
ααΆααααα’αΆαααααΆαααααα·αααααΎ
ααααΎααΆααααα»αααα·ααΆααΆααααααΆαααααΏαααΏαααΊα’ααα
αΆαααααΆααα ααΎαα
ααααΆαα’αααααΆααα
αα
αααα·ααααΆ
(Henning Jacobs:
α αααΈα αααΆα
(ααααααα) Corey Quinnα
ααααααΆα ααααΈ AWS ααααα’αααααΊααΆαα»αααΆαααα
ααα½αα’αααααααΎααααΆαααααα’αααααΆαα
ααΆααα·αα αα·αααα» AWS ααααα’αααααΊααΆαα»αααΆαααα
ααα½ααα·ααααααααα’αααααΆαα
Ivan Kurnosov (ααααΎααα)α
ααΆααα·αα αα·αααα» AWS ααααα’αααααΊααΆαα»αααΆαααα
ααα½αα’αααΈαααα’αααααααα
αα·α/αα»αα)
- α αααΆααααΌααααΌααΆαααΆαααα»αα’αΆα ααααααααααααΆα’αα·ααααΆααΎααααΈααααααααΆαα (TTL) αααααΆααααΆαααΆαααααααΆα PR/test α
- ααααΆααα»ααααα’αΆα ααααΌαααΆααααααα αααΆαααΆαα½α janitor/ttl α§ααΆα αααααΎααααΈαα spike/prototype αααααααααααααααα·αααααΆααααΈ 7 ααααα
α
αααΆααααΌαα
ααααΌαααΆαααααααα
αααα»αα―αααΆα YAML α ααααΌαααααααΆααααΌαααΆαααααααΆαααααΆαααΆαααααα --rules-file
αα
kube-janitor α αααβααΆβα
αααΆααβα§ααΆα αααβαα½αβααΎααααΈβαα»αβααβααααΆβαααααβααΆααα’ααβα
ααβααΆαα½α -pr-
αα
αααα»αααααααααααΆααααΈααΈααααα:
- id: cleanup-resources-from-pull-requests
resources:
- namespaces
jmespath: "contains(metadata.name, '-pr-')"
ttl: 2d
α§ααΆα αααααΆααααααααααΎαα·αααααααααΆαααααΎααααΆααααααΆααααααα·ααΈαα ααΎ Deployment and StatefulSet pods αααααΆαα Deployments/StatefulSets ααααΈααΆααα’αααααα»αααααΆα 2020 ααα»αααααααα»αααααααα½αα’αα»ααααΆαα±ααα’αα»ααααααΆαααΆααααααααααααΆαααααΆαααααααααααα½ααααααΆα αα
- id: require-application-label
# ΡΠ΄Π°Π»ΠΈΡΡ deployments ΠΈ statefulsets Π±Π΅Π· ΠΌΠ΅ΡΠΊΠΈ "application"
resources:
- deployments
- statefulsets
# ΡΠΌ. http://jmespath.org/specification.html
jmespath: "!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'"
ttl: 7d
ααααΎαααΆαααΆααααα αΆααααααΆαααααααααααα 30 ααΆααΈαα ααΎα αααααααααααα»αααααΎαααΆα kube-janitorα
kubectl run nginx-demo --image=nginx
kubectl annotate deploy nginx-demo janitor/ttl=30m
ααααααα½ααααααααΆαααΎαα‘αΎαααααααΎαααΊααα·ααΆαααΆααααΆαα (AWS EBS)α ααΆααα»α Kubernetes StatefulSet αα·ααα»αααΆαααααααααααΆ (PVC - PersistentVolumeClaim) ααα ααα·ααΆα EBS ααααα·αααΆαααααΎα’αΆα αααααΆαα±ααααΆαααΆαα αααΆαααΆαααααα»ααααΆααααα»ααα½ααααααΆαααΆααααα½αα Kubernetes Janitor ααΆααα»αααΆααααα’αΆα PVC ααααα·αααααΎα α§ααΆα ααα α αααΆαααααααΉααα PVCs ααΆααα’ααααααα·αααααΌαααΆαααααΆααααααααΌαα»α α αΎαααααα·αααααΌαααΆααααααα StatefulSet α¬ CronJobα
# ΡΠ΄Π°Π»ΠΈΡΡ Π²ΡΠ΅ PVC, ΠΊΠΎΡΠΎΡΡΠ΅ Π½Π΅ ΡΠΌΠΎΠ½ΡΠΈΡΠΎΠ²Π°Π½Ρ ΠΈ Π½Π° ΠΊΠΎΡΠΎΡΡΠ΅ Π½Π΅ ΡΡΡΠ»Π°ΡΡΡΡ StatefulSets
- id: remove-unused-pvcs
resources:
- persistentvolumeclaims
jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
ttl: 24h
Kubernetes Janitor α’αΆα
αα½αα’ααααααααΆα
αααααααααα’αααα±ααααα’αΆα αα·αααΆαααΆααα»αα±ααααΆαα
αααΆαααΎαααααΎαα‘αΎαααΊααα αααααΆααααΆαααααΆαα’αααΈααΆαααΆαααααααΆα αα·αααΆαααααααα
ααΆαααααααα ααΌαα’αα»ααααααΆα
ααΆαααααααααΆαααααΎααΆαααααααΆααααα»αα’αα‘α»αααααα·αααααΎααΆα
ααααααααααΆααααα αα·αααααΆααααΆαααΆααααααΆααααΌαααΆαααΆαααΆααααααΆααααααα·ααααα·ααΆααααααα»αα’αα‘α»ααααααααααααΎααΆαααα»αααααα αααααα·ααΈααα·ααααααα½αα ααα½α ααΌα ααΆ Back office/admin tools ααααααΌαααΆαααααΆαααααα α αΎαα’αΆα αα·αααΆαααααα½ααααα
image: hjacobs/kube-downscaler:20.4.3
args:
- --interval=30
# Π½Π΅ ΠΎΡΠΊΠ»ΡΡΠ°ΡΡ ΠΊΠΎΠΌΠΏΠΎΠ½Π΅Π½ΡΡ ΠΈΠ½ΡΡΠ°ΡΡΡΡΠΊΡΡΡΡ
- --exclude-namespaces=kube-system,infra
# Π½Π΅ ΠΎΡΠΊΠ»ΡΡΠ°ΡΡ kube-downscaler, Π° ΡΠ°ΠΊΠΆΠ΅ ΠΎΡΡΠ°Π²ΠΈΡΡ Postgres Operator, ΡΡΠΎΠ±Ρ ΠΈΡΠΊΠ»ΡΡΠ΅Π½Π½ΡΠΌΠΈ ΠΠ ΠΌΠΎΠΆΠ½ΠΎ Π±ΡΠ»ΠΎ ΡΠΏΡΠ°Π²Π»ΡΡΡ
- --exclude-deployments=kube-downscaler,postgres-operator
- --default-uptime=Mon-Fri 08:00-20:00 Europe/Berlin
- --include-resources=deployments,statefulsets,stacks,cronjobs
- --deployment-time-annotation=deployment-time
αααβααΆβααααΆα ααβαααααΆααβααααΎβααΆαααααααΆαβααααΆααβααααααβα αααααβαα βα α»αβαααααΆα αα
ααΆαααααΎααΆαααααααΆαα
α»αααΈ ~ 13 αα
4 ααααΆαααααααααα·αααΆααααΎα±ααααΆαααΆααα»αααααΆαα½αα±ααααααααααΆαααα
αααα»ααα·αααααααα AWS ααααα’αααα
ααα»ααααα
α»ααααΆαααΆααΎαααα»αααααΌαααααΎααΆαααα‘α»αααααααα»α "αααααααα"? ααΆαααΆαααααααΆααα½αα
ααα½αα’αΆα
ααααΌαααΆαααα
ααααΆα’α
α·ααααααααααΈααΆαααααΎααΆαααααααΆαααααααααααααααα·ααΈα
α»αααΆαααααααΆα/ααα
ααα α
αααΆααααααααα·αα ααΆαααΆαααααααΆαα’αΆα
ααααΌαααΆαααα
ααααΆααααααα’αΆαααααααααααΎαααααα·ααΈααααα/αα·αααΆαααααα
αΌα-αα αΌααααα
αααΆααααααααααααΆαααααΆαααααααΆααΆα
αααΆααααα»ααααααα YYYY-MM-DD HH:MM (UTC) α ααΎα
αΆαααΆα
α α
αααααααΆααααΌαα’αΆα
ααααΌαααΆαααααΎααΆαααααααΆααααα‘αααααα·ααααααΆαααααααΆαααααΆαα½αα
αααΆααααααα downscaler/force-uptime
α§ααΆα ααααααααΎαααααΎαααΆα nginx αααα
kubectl run scale-up --image=nginx
kubectl annotate deploy scale-up janitor/ttl=1h # ΡΠ΄Π°Π»ΠΈΡΡ ΡΠ°Π·Π²Π΅ΡΡΡΠ²Π°Π½ΠΈΠ΅ ΡΠ΅ΡΠ΅Π· ΡΠ°Ρ
kubectl annotate pod $(kubectl get pod -l run=scale-up -o jsonpath="{.items[0].metadata.name}") downscaler/force-uptime=true
ααΌαααΎα
ααααΎβααΆαβααααΎβααΆαααααααΆαβαααβαααααβαααααααα·βααααα
αααααα·ααΈ/ααααΆααααααΆα
αααΎααααααααΆαααΆαα½αααααΆααααα»αααΆαααααα ααααααααααΌαα»ααααααα½ααααα
ααααα α αΎαααααααααα½αααααααΎαααΆαααααααααααΆαα ααααα·ααααα·ααΆαααααΆααΆα’α
α·ααααααααααΎααααΈααααααααΉαααααα»αα’αα·ααααΆααΊαα·ααααααααα
αααα Kubernetes ααΆααααααΆαααααΎααΆαααααααΆααααααααααααααααα·αααααααααααΆααααααΆααα½αα
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: my-app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
averageUtilization: 100
type: Utilization
Zalando ααΆααααααΎααααΆαααΆαα»αα½αααΎααααΈααΆααααα½αααααΆααααααααααααΆαααααα½ααααααΆααααΆαααααΎααΆαααααααΆαα
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa
annotations:
# metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
metric-config.pods.requests-per-second.json-path/json-key: "$.http_server.rps"
metric-config.pods.requests-per-second.json-path/path: /metrics
metric-config.pods.requests-per-second.json-path/port: "9090"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: requests-per-second
target:
averageValue: 1k
type: AverageValue
ααΆαααααααα
ααΆααααααααααααααααααααα·αααααααααααΆαα½α HPA αα½αααααΆαααααααΆαααααΆαααΎααα½α ααΎααααΈαααααΎαααααα·αααααΆααααααΆααααααΆααααααααΆαααααα Spotify ααΆααααααα αΆαααΆαα½αααΉααααα·ααααα αα·αααΆαααααΆααααααα½ααααααααΆαα HPAα
ααΆαααααααααΆααααααααΆαααΎα
ααααα»αααΆαααΆα Kubernetes ααααααααααΌαααΆα CPU/memory αααααα½αααααΆαααα "ααααΎααααΆα" α ααααΆααααΈααΈααΌααααΌαααΆαααΆααααΆααααΌααα·αααα·α α¬ααΆααΌαα
αα
αααα»α "ααΈααΈααα" α§ααΆα ααα 500m ααααααααα 50% vCPU α ααααΆαα’αααα
αα
αΆαααααΌαααΆαααΆααααΆαα α αΎααα
αα
ααααΌαα
α’αΆα
ααααΌαααΆαααααΎααΌα
ααΆ 500Mi αααααΆααααααΆ 500 ααααΆααα ααααΆαααααΎαα»ααααααααΆα "α
αΆαααα" αα
ααΎααααΆαααααααα ααΆααααααΆ αααααααΆαααααΎαααΈααΈααΌ 1000 α αα
ααΎααααΆαααααααΆα vCPU α
ααα½α 4 ααΉααα»ααα vCPU 3 ααα»ααααααααααΆααααααΆααααααααααα
Slack (ααααα»αααΎα) ααΊααΆααΆααα»αααααΆαααΆαααααΆααααααΆαααααΎαα»α αα·αααΆαααααΎααααΆααααΆαααααααα α§ααΆα ααα αααααααααΎαα»αα’αααα
αα
αΆα 2 GiB ααα»ααααααααΎαα 200 MiB ααΆαα’αααα
αα
αΆα "ααΎα" ~ 1,8 GiB α ααΎαααααΌαα
αααΆαααααΆααα ααα»αααααααΆααα’αΆα
αααΆααααααΆαααΆαααΆ 1 GiB ααα’αααα
αα
αΆαααααααααααΌαααΆααααααΆαααααα ~ $10 αααα»ααα½αααα
ααΆαααΆαααααααααΆαααααΎαα»α CPU ααΈ 3000m αα
~400m αααααΎαααααΆααααααΆααααααα»αααΆαααΆααααααααα αα·αα’αα»ααααΆαα±ααα
αααααααΌα
ααΆαα
"ααΆαααααΎααααΆαααααΈααΈααΌααΆαααααααααααα» EC2 ααΆααΉαααΆααααΆαααααα»ααα½αααΆααααα½αααααα"
ααα»ααααααΎααΎααα·αααΆα
ααα±ααααα»αααααααΆααααααΌαααααααα
αααα»αα―αααΆα YAML ααααα? αα αααΆαααΈαα’αΆα
ααααΎααΆααΆααααααΎαααΆα! Kubernetes
αααα»αααΆααααααααΌα
αα½α
ααΆαααααΎααααΆαα EC2 Spot Instances
ααΆα
α»αααααα ααΆαα
αααΆαααΎ AWS EC2 α’αΆα
ααααΌαααΆαααΆααααααααααααααΎ Spot instances ααΆααααΆαααααααα Kubernetes
ααΎααααΎααΌα ααααα ααΎααααΈααααΎαααΆα Kubernetes αα ααΎ EC2 Spot? ααΆααααααΎαααΆα αααΎαα ααααΎααααΆααααααΆααΈααΈααΈααΌα ααΆ SpotInst (α₯α‘αΌαα α ααΆ "Spot" αα»ααα½ααααα»αααΈααΌαα ααα») α¬ααααΆαααααααααα Spot AutoScalingGroup (ASG) αα αααα»αα αααααααααα’αααα α§ααΆα ααα αααβααΆβααααααΆαβαααααα CloudFormation αααααΆαα "capacity-optimized" Spot ASG ααΆαα½αβααΉαβααααααβα§ααΆα αααβα αααΎαα
MySpotAutoScalingGroup:
Properties:
HealthCheckGracePeriod: 300
HealthCheckType: EC2
MixedInstancesPolicy:
InstancesDistribution:
OnDemandPercentageAboveBaseCapacity: 0
SpotAllocationStrategy: capacity-optimized
LaunchTemplate:
LaunchTemplateSpecification:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
Overrides:
- InstanceType: "m4.2xlarge"
- InstanceType: "m4.4xlarge"
- InstanceType: "m5.2xlarge"
- InstanceType: "m5.4xlarge"
- InstanceType: "r4.2xlarge"
- InstanceType: "r4.4xlarge"
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
MinSize: 0
MaxSize: 100
Tags:
- Key: k8s.io/cluster-autoscaler/node-template/label/aws.amazon.com/spot
PropagateAtLaunch: true
Value: "true"
αααααα αααΆααα½αα ααα½αα’αααΈααΆαααααΎααααΆαα Spot ααΆαα½α Kubernetesα
- α’αααβααααΌαβαααααααΆαβααΆαβαααα αα Spot ααΆβα§ααΆα αααβαααβααΆαβαααα αΌαβααααΆααβαα βαααβααααα»βααααΌαβααΆαβαααααα
- Zalando ααααΎ
αα ααΆαααααΎααΆαααααααΆααααααααααααααααα·ααΌαα αααααααΆααααΌαααΆαααΆαα½αααΉαα’αΆαα·ααΆααααα»αααααΆαα - ααααΆααα
ααα»α
α’αΆα ααααΌαααΆαααααα ααα½ααα "ααΆαα α»αααααα" ααααααα»αααΆαααΆαααΎααααΈααααΎαααΆααα αααα»α Spot
αααααα
αααα»ααααααΉαααΆα’αααααααΎαα§ααααααα½αα
ααα½ααααααΆααααα αΆαααΆααααααααααααα»αααΆαααΆααααααααα·ααααααααααΎαααααααα’αααα α’αααβα’αΆα
βαααααβααβααΆαα·ααΆβααΆαβα
αααΎαβααβα’αααααβααβαααβαα
ααΎα’αααΈααΆααΆαα’αα»ααααααα’αααα»αααααα’ααααααααΆααααΆααααααααΆαα
αααΆαααΎααααα
ααΎ Kubernetes? ααΌαααααΆαααααα»ααα
ααααα: www.habr.com