ProHoster > Pūnaewele > Nā Administration > Nā polokalamu Autoscaling Kubernetes me Prometheus a me KEDA

Nā polokalamu Autoscaling Kubernetes me Prometheus a me KEDA

Nā polokalamu Autoscaling Kubernetes me Prometheus a me KEDAKanaka Baluna na Cimuanos

He koi nui ka Scalability no nā noi ao. Me nā Kubernetes, maʻalahi ka hoʻonui ʻana i kahi noi e like me ka hoʻonui ʻana i ka nui o nā replicas no ka hoʻonohonoho kūpono a i ʻole. ReplicaSet - akā, he kaʻina hana lima.

Hāʻawi ʻo Kubernetes i nā noi e hoʻonui ʻia (ʻo ia hoʻi ReplicaSet) ma ke ʻano hoʻolaha me ka hoʻohana ʻana i ka kikoʻī Horizontal Pod Autoscaler. ʻO ka helu paʻamau no ka hoʻonui ʻana i ka maʻamau ʻo ia nā metric hoʻohana CPU (nā kumu waiwai), akā hiki iā ʻoe ke hoʻohui i nā metric maʻamau a hāʻawi ʻia i waho.

hui Kubernetes aaS mai Mail.ru unuhi i ka ʻatikala e pili ana i ka hoʻohana ʻana i nā ana waho e hoʻonui ʻakomi i kahi noi Kubernetes. No ka hōʻike ʻana i ka hana ʻana o nā mea āpau, hoʻohana ka mea kākau i nā metric noi noi komo HTTP, i hōʻiliʻili ʻia me ka hoʻohana ʻana iā Prometheus.

Ma kahi o ka hoʻokaʻawale autoscaling o nā pods, hoʻohana ʻia ʻo Kubernetes Event Driven Autoscaling (KEDA), kahi mea hoʻohana Kubernetes open source. Hoʻohui maoli ia me ka Horizontal Pod Autoscaler e hāʻawi i ka autoscaling maʻemaʻe (me ka / mai ka zero) no nā haʻahaʻa hana. Loaʻa ke code ma GitHub.

ʻIke pōkole o ka ʻōnaehana

Nā polokalamu Autoscaling Kubernetes me Prometheus a me KEDA

Hōʻike ke kiʻikuhi i kahi wehewehe pōkole o ka hana ʻana o nā mea a pau:

  1. Hāʻawi ka palapala noi i nā helu helu helu HTTP ma Prometheus format.
  2. Hoʻonohonoho ʻia ʻo Prometheus e hōʻiliʻili i kēia mau ana.
  3. Hoʻonohonoho ʻia ka scaler Prometheus ma KEDA e hoʻonui i ka noi ma muli o ka helu o HTTP hits.

I kēia manawa e haʻi aku wau iā ʻoe i nā kikoʻī e pili ana i kēlā me kēia mea.

KEDA a me Prometheus

ʻO Prometheus kahi ʻōnaehana wehe ʻana i ka ʻōnaehana mālama a me ka mea hana makaʻala, ʻāpana Ka Papahana Hoʻonohonoho ʻInikua Kapua. E hōʻiliʻili i nā metric mai nā kumu like ʻole a mālama iā lākou ma ke ʻano he ʻikepili manawa. No ka nānā ʻana i ka ʻikepili hiki iā ʻoe ke hoʻohana grafana a i ʻole nā ​​mea hana ʻike ʻē aʻe e hana pū me ka Kubernetes API.

Kākoʻo ʻo KEDA i ka manaʻo o ka scaler - he alahaka ma waena o KEDA a me ka ʻōnaehana waho. He kiko'ī ka hoʻokō scaler i kēlā me kēia ʻōnaehana i hoʻopaʻa ʻia a unuhi i ka ʻikepili mai ia mea. A laila hoʻohana ʻo KEDA iā lākou e hoʻomalu i ka scaling maʻalahi.

Kākoʻo nā Scalers i nā kumu ʻikepili he nui, no ka laʻana, Kafka, Redis, Prometheus. ʻO ia hoʻi, hiki ke hoʻohana ʻia ke KEDA no ka hoʻonui ʻana i nā hoʻolaha Kubernetes me ka hoʻohana ʻana i nā metric Prometheus ma ke ʻano he pae hoʻohālike.

Noi hoao

Hāʻawi ka palapala hoʻokolohua Golang i ke komo ma o HTTP a hana i ʻelua mau hana koʻikoʻi:

  1. Hoʻohana i ka waihona mea kūʻai Prometheus Go e hoʻohana i ka palapala noi a hāʻawi i ka metric http_requests, aia kahi helu pā. Aia ka helu hope i loaʻa ai nā metric Prometheus ma ka URI /metrics. 
    var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests",
       Help: "number of http requests",
   })
  2. I ka pane ʻana i kahi noi GET hoʻonui ka noi i ka waiwai o ke kī (access_count) ma Redis. He ala maʻalahi kēia e hana ai i ka hana ma ke ʻano he mea lawelawe HTTP a nānā pū i nā metric Prometheus. Pono e like ka waiwai me ka waiwai access_count ma Redis. 
    func main() {
       http.Handle("/metrics", promhttp.Handler())
       http.HandleFunc("/test", func(w http.ResponseWriter, r 
*http.Request) {
           defer httpRequestsCounter.Inc()
           count, err := client.Incr(redisCounterName).Result()
           if err != nil {
               fmt.Println("Unable to increment redis counter", err)
               os.Exit(1)
           }
           resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count))
           w.Write([]byte(resp))
       })
       http.ListenAndServe(":8080", nil)
   }

Hoʻouna ʻia ka noi iā Kubernetes ma o Deployment. Hana ʻia kahi lawelawe ClusterIP, ʻae ia i ka server Prometheus e loaʻa nā metric noi.

ʻaneʻi hōʻike hoʻolaha no ka noi.

Kahua Prometheus

ʻO ka hōʻike hoʻolaha Prometheus aia nā:

  • ConfigMap - e hoʻololi i ka Prometheus config;
  • Deployment - no ka lawe ʻana iā Prometheus i kahi hui Kubernetes;
  • ClusterIP - lawelawe no ke komo ʻana iā UI Prometheus;
  • ClusterRole, ClusterRoleBinding и ServiceAccount — no ka ʻike ʻana i nā lawelawe ma Kubernetes (Auto-discovery).

ʻaneʻi hōʻike no ka holo ʻana iā Prometheus.

KEDA Prometheus ScaledObject

ʻO ka scaler ke alahaka ma waena o KEDA a me ka ʻōnaehana waho kahi e loaʻa ai nā ana. ScaledObject he kumu waiwai maʻamau e pono e hoʻokaʻawale ʻia e hoʻonohonoho i ka hoʻoili ʻana me ke kumu hanana, i kēia hihia Prometheus.

ScaledObject Loaʻa i ka ʻike hoʻolalelale ʻana, ka metadata kumu hanana (e like me nā mea huna pili, ka inoa queue), ka wā koho, ka manawa hoʻihoʻi, a me nā ʻikepili ʻē aʻe. Loaʻa ia i ka waiwai autoscaling e pili ana (ka wehewehe HPA) e hoʻonui i ka hoʻolālā.

Ina he mea ScaledObject holoi ʻia, holoi ʻia ka wehewehe HPA pili.

Eia ka wehewehe ScaledObject no kā mākou laʻana, hoʻohana ia i kahi scaler Prometheus:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
 name: prometheus-scaledobject
 namespace: default
 labels:
   deploymentName: go-prom-app
spec:
 scaleTargetRef:
   deploymentName: go-prom-app
 pollingInterval: 15
 cooldownPeriod:  30
 minReplicaCount: 1
 maxReplicaCount: 10
 triggers:
 - type: prometheus
   metadata:
     serverAddress: 
http://prometheus-service.default.svc.cluster.local:9090
     metricName: access_frequency
     threshold: '3'
     query: sum(rate(http_requests[2m]))

E noʻonoʻo i kēia mau mea:

  1. Kuhikuhi ʻo ia iā Deployment Me ka inoa go-prom-app.
  2. ʻAno hoʻāla - Prometheus. Ua ʻōlelo ʻia ka helu kikowaena Prometheus me ka inoa metric, paepae a me Nīnau PromQL, e hoʻohana ʻia. Nīnau PromQL - sum(rate(http_requests[2m])).
  3. Wahi a pollingInterval, Noi ʻo KEDA i ka pahuhopu mai Prometheus i kēlā me kēia ʻumikumamālima kekona. Ma lalo o hoʻokahi (minReplicaCount), a ʻaʻole i ʻoi aku ka nui o nā pods maxReplicaCount (i kēia laʻana - ʻumi).

Hiki ke hoʻouka minReplicaCount like me ka ʻole. I kēia hihia, hoʻāla ʻo KEDA i ka hoʻolaha ʻana i ka zero-a-one a laila e hōʻike i ka HPA no ka hoʻonui ʻana i ka ʻakomi. Hiki nō hoʻi ke kaʻina hoʻohuli, ʻo ia hoʻi, ka hoʻonui ʻana mai kahi a i ʻole. I ka laʻana, ʻaʻole mākou i koho i ka zero no ka mea he lawelawe HTTP kēia a ʻaʻole kahi ʻōnaehana koi.

ʻO ke kilokilo i loko o ka autoscaling

Hoʻohana ʻia ka paepae ma ke ʻano he kumu e hoʻonui ai i ka hoʻolaha. Ma kā mākou hiʻohiʻona, ka nīnau PromQL sum(rate (http_requests [2m])) hoʻihoʻi i ka helu noi HTTP aggregated (noi i kekona), ana ʻia i nā minuke ʻelua i hala.

No ka mea he ʻekolu ka waiwai o ka paepae, ʻo ia hoʻi, aia kekahi ma lalo o ka waiwai sum(rate (http_requests [2m])) emi mai ka ekolu. Inā piʻi ka waiwai, hoʻohui ʻia kahi sub i kēlā me kēia manawa sum(rate (http_requests [2m])) piʻi ʻekolu. No ka laʻana, inā mai 12 a 14 ka waiwai, a laila ʻehā ka helu o nā pods.

I kēia manawa e hoʻāʻo kāua e hoʻonohonoho!

presetting

ʻO nā mea a pau āu e makemake ai he pūʻulu Kubernetes a me kahi pono i hoʻonohonoho ʻia kubectl. Hoʻohana kēia laʻana i kahi pūʻulu minikube, akā hiki iā ʻoe ke lawe i kekahi. No ka hoʻokomo ʻana i kahi pūʻulu aia alakaʻi.

E hoʻouka i ka mana hou loa ma Mac:

curl -Lo minikube 
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Kau kubectle komo i ka hui Kubernetes.

E hoʻouka i ka mana hou loa ma Mac:

curl -LO 
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

Hoʻokomo ʻia ʻo KEDA

Hiki iā ʻoe ke kau iā KEDA ma nā ʻano he nui, ua helu ʻia lākou ma palapala. Ke hoʻohana nei au i ka YAML monolithic:

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml

Hoʻokomo ʻia ʻo KEDA a me kāna mau ʻāpana i ka inoa inoa keda. Kauoha e nānā:

kubectl get pods -n keda

E kali iā KEDA Operator e hoʻomaka a hele i Running State. A ma hope o kēlā, e hoʻomau.

Ke hoʻouka nei iā Redis me ka hoʻohana ʻana iā Helm

Inā ʻaʻole i hoʻokomo ʻia ʻo Helm, e hoʻohana i kēia alakaʻi. Kauoha e hoʻouka ma Mac:

brew install kubernetes-helm
helm init --history-max 200

helm init hoʻomaka i ke kikowaena laina kauoha kūloko a hoʻokomo pū Tiller i ka hui Kubernetes.

kubectl get pods -n kube-system | grep tiller

E kali no ke komo ʻana o ka Pod Tiller i ka mokuʻāina holo.

Palapala unuhi: Hoʻohana ka mea kākau iā Helm@2, kahi e pono ai e hoʻokomo i ka ʻāpana kikowaena Tiller. I kēia manawa kūpono ʻo Helm@3, ʻaʻole ia e koi i kahi ʻāpana kikowaena.

Ma hope o ka hoʻokomo ʻana iā Helm, ua lawa kahi kauoha e hoʻomaka iā Redis:

helm install --name redis-server --set cluster.enabled=false --set 
usePassword=false stable/redis

E hōʻoia ua hoʻomaka pono ʻo Redis:

kubectl get pods/redis-server-master-0

E kali iā Redis e hele i ka moku'āina Running.

Hoʻolaha Noi

Kauoha hoʻolaha:

kubectl apply -f go-app.yaml

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

E nānā ua hoʻomaka nā mea a pau:

kubectl get pods -l=app=go-prom-app

E kali iā Redis e komo i ka mokuʻāina Running.

Ke hoʻolālā nei i kahi kikowaena Prometheus

Hoʻohana ka Prometheus manifest ʻIke lawelawe Kubernetes no Prometheus. Hiki iā ia ke ʻike ikaika i nā pods noi e pili ana i ka lepili lawelawe.

kubernetes_sd_configs:
   - role: service
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_label_run]
     regex: go-prom-app-service
     action: keep

No ke kau ʻana:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

E nānā ua hoʻomaka nā mea a pau:

kubectl get pods -l=app=prometheus-server

E kali iā Prometheus e hele i ka moku'āina Running.

E hoʻohana kubectl port-forward e komo i ka Prometheus mea hoʻohana (a i ʻole API server) ma http://localhost:9090.

kubectl port-forward service/prometheus-service 9090

Ke hoʻohana nei i ka hoʻonohonoho ʻana i ka hoʻonohonoho ʻokoʻa o KEDA

Kauoha e hana ScaledObject:

kubectl apply -f keda-prometheus-scaledobject.yaml

E nānā i nā moʻolelo mea hoʻohana KEDA:

KEDA_POD_NAME=$(kubectl get pods -n keda 
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

ʻO ka hopena e like me kēia:

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with 
namespace default and name keda-hpa-go-prom-app"

E nānā ma lalo o nā noi. Pono e holo kekahi laʻana no ka mea minReplicaCount like 1:

kubectl get pods -l=app=go-prom-app

E hōʻoia i ka hana pono ʻana o ka punawai HPA:

kubectl get hpa

Pono ʻoe e ʻike i kekahi mea e like me:

NAME                   REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   0/3 (avg)   1         10        1          45s

Nānā olakino: komo i ka noi

No ke kiʻi ʻana i ka palena hope REST o kā mākou noi, holo:

kubectl port-forward service/go-prom-app-service 8080

Hiki iā ʻoe ke komo i kāu polokalamu Go me ka hoʻohana ʻana i ka helu wahi http://localhost:8080. No ka hana ʻana i kēia, e holo i ke kauoha:

curl http://localhost:8080/test

ʻO ka hopena e like me kēia:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC 
m=+406004.817901246
Access count 1

I kēia manawa e nānā pū iā Redis. E ʻike ʻoe i ke kī access_count hoʻonui i ka 1:

kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"

E hōʻoia i ka waiwai metric http_requests ka mea like:

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Hoʻouka ʻia

E hoʻohana mākou Hey — mea pono no ka hana ana i ka ukana:

curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64 
&& chmod a+x hey

Hiki iā ʻoe ke hoʻoiho i ka pono no Linux ai ole ia, Windows.

Holo ia:

./hey http://localhost:8080/test

Ma ka maʻamau, hoʻouna ka pono i nā noi 200. Hiki iā ʻoe ke hōʻoia i kēia me ka hoʻohana ʻana i nā metric Prometheus a me Redis.

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201

E hōʻoia i ka waiwai o ka metric maoli (hoʻihoʻi ʻia e ka nīnau PromQL):

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

I kēia hihia ka hopena maoli 1,686057971014493 a hōʻike ʻia ma ke kahua value. ʻAʻole lawa kēia no ka hoʻonui ʻana, ʻoiai ʻo ka paepae a mākou i hoʻonohonoho ai he 3.

Hoʻouka hou aku!

Ma ka pahu hou, e nānā i ka helu o nā pods noi:

kubectl get pods -l=app=go-prom-app -w

E hoʻonui i ka ukana me ke kauoha: 

./hey -n 2000 http://localhost:8080/test

Ma hope o kekahi manawa, e ʻike ʻoe iā HPA e hoʻonui ana i ka hoʻolaha ʻana a me ka hoʻokuʻu ʻana i nā pods hou. E nānā i kāu HPA no ka ʻike:

kubectl get hpa
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   1830m/3 (avg)   1         10        6          4m22s

Inā ʻaʻole kūlike ka ukana, e hoʻemi ʻia ka hoʻolaha ʻana i kahi e holo ana hoʻokahi pod. Inā makemake ʻoe e nānā i ka metric maoli (hoʻihoʻi ʻia e ka nīnau PromQL), a laila e hoʻohana i ke kauoha:

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'

Очистка

//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server

hopena

Hiki iā KEDA ke ho'onui 'akomi i kāu ho'olālā Kubernetes (i/mai ka 'ole) ma muli o ka 'ikepili mai nā ana o waho. No ka laʻana, e pili ana i nā metric Prometheus, ka lōʻihi o ka pila ma Redis, ka latency o nā mea kūʻai aku ma ke kumuhana Kafka.

Hoʻohui pū ʻo KEDA me kahi kumu waho a hāʻawi pū i kāna mau ana ma o Metrics Server i Horizontal Pod Autoscaler.

Pomaikai!

He aha hou aʻe e heluhelu ai:

  1. ʻO nā hana maikaʻi loa a me nā hana maikaʻi loa no ka holo ʻana i nā pahu a me nā Kubernetes i nā kaiapuni hana.
  2. 90+ Mea Hana Pono no nā Kubernetes: Hoʻolālā, Hoʻokele, Nānā, Palekana a me nā mea hou aku.
  3. ʻO kā mākou kahawai a puni Kubernetes ma Telegram.

Source: www.habr.com

Pākuʻi i ka manaʻo hoʻopuka