ProHoster > Blog > Pulega > Autoscaling Kubernetes talosaga e fa'aaoga ai le Prometheus ma le KEDA

Autoscaling Kubernetes talosaga e fa'aaoga ai le Prometheus ma le KEDA

Autoscaling Kubernetes talosaga e fa'aaoga ai le Prometheus ma le KEDATagata Baloona saunia e Cimuanos

Scalability ose mana'oga autu mo fa'aoga ao. Faatasi ai ma Kubernetes, o le faʻatupuina o se talosaga e faigofie pei o le faʻateleina o le numera o faʻataʻitaʻiga mo le faʻatulagaina talafeagai poʻo ReplicaSet - ae o se faiga tusilima.

Kubernetes fa'atagaina talosaga e otometi ona fa'afuaina (ie Pods i se fa'agaioiga po'o ReplicaSet) i se auala faʻaalia e faʻaaoga ai le faʻamatalaga Horizontal Pod Autoscaler. Ole fa'ata'ita'iga fa'aletonu mo le fa'avasegaina otometi o le fa'aogaina o fua ole CPU (mea fa'atatau), ae e mafai ona e tu'ufa'atasia ma tu'ufa'atasiga tu'ufa'atasiga mai fafo.

au Kubernetes aaS mai Mail.ru fa'aliliuina se tala ile fa'aogaina o fua fa'atatau i fafo e otometi ai ona fua le talosaga Kubernetes. Ina ia faʻaalia pe faʻafefea ona faʻaogaina mea uma, e faʻaogaina e le tusitala le HTTP access request metrics, lea e aoina e faʻaaoga ai le Prometheus.

Nai lo le fa'ata'ita'i fa'ata'amilosaga o pods, o lo'o fa'aogaina le Kubernetes Event Driven Autoscaling (KEDA), o se fa'aoga Kubernetes fa'apitoa. O lo'o tu'ufa'atasia fa'atasi ma le Horizontal Pod Autoscaler e tu'uina atu ai le fa'aogaina o le ta'avale (e aofia ai i/mai le leai) mo galuega fa'aosoina mea. Code e maua ile GitHub.

Fa'amatalaga puupuu o le faiga

Autoscaling Kubernetes talosaga e fa'aaoga ai le Prometheus ma le KEDA

O le ata o loʻo faʻaalia ai se faʻamatalaga puupuu o le auala e galue ai mea uma:

  1. O le talosaga e tuʻuina atu fua faʻatatau HTTP hit i le Prometheus format.
  2. Prometheus ua fa'atulagaina e aoina nei fua fa'atatau.
  3. O le Prometheus scaler i KEDA ua faʻatulagaina e otometi le fuaina o le talosaga e faʻatatau i le numera o HTTP hits.

O lenei o le a ou taʻuina atu ia te oe auiliiliga e uiga i elemene taʻitasi.

KEDA ma Prometheus

O le Prometheus o se faʻasalalauga faʻapitoa e mataʻituina ma mataala meafaigaluega, vaega Cloud Native Computing Foundation. Aoina metric mai punaoa eseese ma teu i latou e avea ma faʻamaumauga o taimi. Ina ia va'ai fa'amatalaga e mafai ona e fa'aogaina tusifana po'o isi mea faigaluega fa'aaliga e galulue ma le Kubernetes API.

E lagolagoina e le KEDA le manatu o le sikola - e fai ma alalaupapa i le va o le KEDA ma le faiga i fafo. Ole fa'atinoga ole scaler e fa'apitoa i faiga fa'atatau ta'itasi ma maua mai fa'amaumauga mai ai. Ona fa'aogaina lea e le KEDA e fa'atonutonu ai le fa'avasegaina otometi.

E lagolagoina e tagata sikola le tele o faʻamaumauga, mo se faʻataʻitaʻiga, Kafka, Redis, Prometheus. O lona uiga, e mafai ona fa'aogaina le KEDA e fa'afua ai le fa'aogaina o Kubernetes e fa'aaoga ai le Prometheus metrics e fai ma ta'iala.

Su'ega talosaga

O le su'ega a le Golang e maua ai le avanoa e ala i le HTTP ma faia ni galuega taua se lua:

  1. Fa'aoga le faletusi a le tagata o tausia o le Prometheus Go e fa'aoga ai le talosaga ma tu'u mai le metric http_requests, o lo'o i ai se numera lavea. O le fa'ai'uga o lo'o maua ai le Prometheus metrics o lo'o i le URI /metrics. 
    var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests",
       Help: "number of http requests",
   })
  2. I le tali atu i se talosaga GET o le talosaga e faʻaopoopo le tau o le ki (access_count) i Redis. O se auala faigofie lea e fai ai le galuega o se vaega o le HTTP handler ma siaki Prometheus metrics. E tatau ona tutusa le tau metric ma le tau access_count i Redis. 
    func main() {
       http.Handle("/metrics", promhttp.Handler())
       http.HandleFunc("/test", func(w http.ResponseWriter, r 
*http.Request) {
           defer httpRequestsCounter.Inc()
           count, err := client.Incr(redisCounterName).Result()
           if err != nil {
               fmt.Println("Unable to increment redis counter", err)
               os.Exit(1)
           }
           resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count))
           w.Write([]byte(resp))
       })
       http.ListenAndServe(":8080", nil)
   }

O loʻo tuʻuina atu le talosaga i Kubernetes e ala i Deployment. Ua faia foi se auaunaga ClusterIP, e mafai ai e le Prometheus server ona maua fua faʻatatau.

iinei fa'aaliga fa'apipi'i mo le talosaga.

Prometheus Server

O le Prometheus deployment manifest e aofia ai:

  • ConfigMap - ia tuʻuina atu le Prometheus config;
  • Deployment - mo le faʻaaogaina o Prometheus i totonu o le Kubernetes cluster;
  • ClusterIP - auaunaga mo le avanoa i le UI Prometheus;
  • ClusterRole, ClusterRoleBinding и ServiceAccount — mo le su'esu'eina otometi o auaunaga i Kubernetes (Auto-discovery).

iinei faʻaalia mo le faʻatinoina o Prometheus.

KEDA Prometheus ScaledObject

O le sikola e fai ma alalaupapa i le va o le KEDA ma le faiga i fafo e mana'omia ona maua mai ai fua. ScaledObject o se aganuʻu aganuʻu e manaʻomia ona faʻapipiʻiina e faʻafetaui le faʻapipiʻiina ma le mea na tupu, i lenei tulaga Prometheus.

ScaledObject o lo'o iai fa'amatalaga fa'ata'atiaga, fa'amatalaga fa'apogai o mea na tutupu (e pei o feso'ota'iga mealilo, igoa fa'asologa), va'aiga palota, taimi toe fa'aleleia, ma isi fa'amaumauga. E i'u ai i le fa'atatauina o le autoscaling resource (HPA definition) e fa'afua ai le fa'atinoina.

A o se mea faitino ScaledObject ua tapeina, ua kilia le faauigaga tutusa o le HPA.

O le fa'amatalaga lea ScaledObject mo la tatou faʻataʻitaʻiga, e faʻaaogaina ai se sikola Prometheus:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
 name: prometheus-scaledobject
 namespace: default
 labels:
   deploymentName: go-prom-app
spec:
 scaleTargetRef:
   deploymentName: go-prom-app
 pollingInterval: 15
 cooldownPeriod:  30
 minReplicaCount: 1
 maxReplicaCount: 10
 triggers:
 - type: prometheus
   metadata:
     serverAddress: 
http://prometheus-service.default.svc.cluster.local:9090
     metricName: access_frequency
     threshold: '3'
     query: sum(rate(http_requests[2m]))

Faamolemole ia matau manatu nei:

  1. Na ia faasino i Deployment Ma igoa go-prom-app.
  2. Ituaiga fa'aoso - Prometheus. O le tuatusi server Prometheus o loʻo taʻua faʻatasi ma le igoa metric, paepae ma ole fesili ole PromQL, lea o le a faʻaaogaina. Fesili PromQL - sum(rate(http_requests[2m])).
  3. E tusa ai pollingInterval,E mana'omia e KEDA se fa'amoemoe mai le Prometheus i le sefululima sekone. Le itiiti ifo ma le tasi i lalo (minReplicaCount), ma le numera maualuga o pods e le sili atu maxReplicaCount (i lenei faʻataʻitaʻiga - sefulu).

E mafai ona fa'apipi'i minReplicaCount tutusa ma le leai. I le tulaga lea, e fa'agaoioi e KEDA le fa'aogaina o le zero-to-one ona fa'aalia lea o le HPA mo le fa'ateleina otometi. O le fa'asologa fa'asolosolo e mafai fo'i, o lona uiga, fa'asolo mai le tasi i le zero. I le faʻataʻitaʻiga, matou te leʻi filifilia se zero ona o se auaunaga HTTP ae le o se faiga faʻapitoa.

O le togafiti i totonu o le autoscaling

E faʻaaogaina le faʻailoga e fai ma faʻaoso e faʻateleina le faʻapipiʻiina. I la matou faʻataʻitaʻiga, o le PromQL fesili sum(rate (http_requests [2m])) toe fa'afo'i le aofa'i o talosaga HTTP fa'aopoopo (talosaga i le sekone), fua i le lua minute mulimuli.

Talu ai o le tau fa'ailoga e tolu, o lona uiga o le ai ai le tasi i lalo ae o le tau sum(rate (http_requests [2m])) itiiti ifo i le tolu. Afai e fa'atupula'ia le tau, e fa'aopoopoina i taimi ta'itasi se fa'aopoopoga fa'aopoopo sum(rate (http_requests [2m])) faateleina i le tolu. Mo se faʻataʻitaʻiga, afai o le tau mai le 12 i le 14, o le numera o pods e fa.

Se'i o tatou taumafai nei e fa'atulaga!

muai seti

Pau lava le mea e te manaʻomia o le Kubernetes cluster ma se faʻaoga faʻaoga kubectl. O lenei faʻataʻitaʻiga e faʻaaogaina se fuifui minikube, ae e mafai ona e aveina se isi. E fa'apipi'i se fuifui e iai taʻitaʻi.

Fa'apipi'i le lomiga fou i le Mac:

curl -Lo minikube 
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Faʻamau kubectle maua ai le vaega Kubernetes.

Fa'apipi'i le lomiga fou i le Mac:

curl -LO 
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

fa'apipi'i KEDA

E mafai ona e faʻaogaina KEDA i ni auala se tele, o loʻo lisiina i totonu fa'amaumauga. O loʻo ou faʻaaogaina monolithic YAML:

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml

KEDA ma ona vaega o lo'o fa'apipi'iina i totonu ole igoa avanoa keda. Poloaiga e siaki:

kubectl get pods -n keda

Faatali mo le KEDA Operator e amata ma alu i Running State. Ma a uma lena, faaauau.

Fa'apipi'i Redis e fa'aaoga ai le Helm

Afai e le'i fa'apipi'iina sau Helm, fa'aoga lea pulega. Poloaiga e fa'apipi'i ile Mac:

brew install kubernetes-helm
helm init --history-max 200

helm init amataina le fa'aoga laina laina fa'apitonu'u ma fa'apipi'i fo'i Tiller i le fuifui Kubernetes.

kubectl get pods -n kube-system | grep tiller

Faatali mo le Tiller pod e ulufale i le Running state.

Fa'amatalaga a le faaliliu: E faʻaaogaina e le tusitala Helm@2, lea e manaʻomia ai le faʻapipiʻiina o le Tiller server. Ole Helm@3 e talafeagai, e le manaʻomia se vaega o le server.

A maeʻa ona faʻapipiʻi Helm, e tasi le poloaiga e lava e amata ai Redis:

helm install --name redis-server --set cluster.enabled=false --set 
usePassword=false stable/redis

Fa'amaonia na amata manuia le Redis:

kubectl get pods/redis-server-master-0

Faatali mo Redis e alu i le setete Running.

Talosaga Fa'atonu

Poloaiga fa'ata'oto:

kubectl apply -f go-app.yaml

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

Siaki ua amata mea uma:

kubectl get pods -l=app=go-prom-app

Faatali mo Redis e ulufale i le setete Running.

Fa'atonuina o le Prometheus Server

O loʻo faʻaaogaina e le Prometheus manifest Kubernetes Service Discovery mo Prometheus. E fa'ataga ai le su'esu'eina malosi o pusa talosaga e fa'atatau i le igoa o le tautua.

kubernetes_sd_configs:
   - role: service
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_label_run]
     regex: go-prom-app-service
     action: keep

E fa'apipi'i:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

Siaki ua amata mea uma:

kubectl get pods -l=app=prometheus-server

Faatali mo Prometheus e alu i le setete Running.

Faaaoga kubectl port-forward e maua ai le Prometheus user interface (po'o le API server) i http://localhost:9090.

kubectl port-forward service/prometheus-service 9090

Fa'aogaina le KEDA Autoscaling Configuration

Poloaiga e fai ScaledObject:

kubectl apply -f keda-prometheus-scaledobject.yaml

Siaki fa'amaumauga a le tagata fa'afoe KEDA:

KEDA_POD_NAME=$(kubectl get pods -n keda 
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

O le taunuuga e foliga mai e pei o lenei:

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with 
namespace default and name keda-hpa-go-prom-app"

Siaki i lalo o talosaga. E tasi le faʻataʻitaʻiga e tatau ona tamoe aua minReplicaCount tutusa 1:

kubectl get pods -l=app=go-prom-app

Fa'amaonia na fa'atupuina ma le manuia le punaoa HPA:

kubectl get hpa

E tatau ona e vaʻai i se mea e pei o:

NAME                   REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   0/3 (avg)   1         10        1          45s

Siakiga ole soifua maloloina: avanoa fa'aoga

Ina ia maua le fa'ai'uga a le matou talosaga, tamo'e:

kubectl port-forward service/go-prom-app-service 8080

Ua mafai nei ona e mauaina lau Go app e fa'aoga ai le tuatusi http://localhost:8080. Ina ia faia lenei mea, fai le poloaiga:

curl http://localhost:8080/test

O le taunuuga e foliga mai e pei o lenei:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC 
m=+406004.817901246
Access count 1

I le taimi lea, siaki foi Redis. O le ae vaai i lena ki access_count faateleina i le 1:

kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"

Ia mautinoa o le tau metric http_requests tutusa lava:

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Foafoaga uta

O le a matou faʻaaogaina Ei - aoga mo le gaosia o uta:

curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64 
&& chmod a+x hey

E mafai foi ona e sii maia le aoga mo Linux poʻo pupuni.

Fa'atino:

./hey http://localhost:8080/test

Ona o le faaletonu, e auina atu e le aoga 200 talosaga. E mafai ona e faʻamaonia lenei mea e faʻaaoga ai le Prometheus metrics faʻapea foʻi ma Redis.

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201

Fa'amaonia le tau o le fua fa'atatau (toe fa'afo'i e le fesili a le PromQL):

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

I lenei tulaga o le taunuuga moni o le 1,686057971014493 ma o loʻo faʻaalia i le fanua value. E le lava lea mo le fa'afuaina, talu ai o le fa'ailoga matou te setiina e 3.

Tele uta!

I totonu o le laina fou, mataʻituina le numera o pusa talosaga:

kubectl get pods -l=app=go-prom-app -w

Sei o tatou faʻateleina le uta e faʻaaoga ai le poloaiga: 

./hey -n 2000 http://localhost:8080/test

A maeʻa sina taimi, o le a e vaʻai i le HPA o loʻo faʻapipiʻiina le faʻapipiʻiina ma faʻalauiloa fou pods. Siaki lau HPA ina ia mautinoa:

kubectl get hpa
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   1830m/3 (avg)   1         10        6          4m22s

Afai e le fetaui le uta, o le a faʻaitiitia le faʻapipiʻiina i le tulaga e naʻo le tasi le pod o loʻo taʻavale. Afai e te manaʻo e siaki le metric moni (toe faʻafoʻi e le PromQL fesili), ona faʻaaoga lea o le poloaiga:

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'

Faamama

//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server

iʻuga

KEDA fa'atagaina oe e otometi le fuaina o au fa'agaioiga Kubernetes (i/mai le zero) fa'atatau i fa'amaumauga mai metric mai fafo. Mo se faʻataʻitaʻiga, faʻavae i luga o le Prometheus metrics, laina umi i Redis, tagata faʻatau i le autu Kafka.

O lo'o tu'ufa'atasia le KEDA ma se puna mai fafo ma tu'uina atu fo'i ana fua fa'atatau e ala ile Metrics Server ile Horizontal Pod Autoscaler.

Suʻega!

O le a se isi mea e faitau:

  1. Fa'ata'ita'iga Sili ma Fa'ata'ita'iga Sili mo Container Tafe ma Kubernetes i Si'osi'omaga o Gaosia.
  2. 90+ Meafaigaluega Fa'aoga mo Kubernetes: Fa'atonu, Pulega, Mata'ituina, Saogalemu ma isi mea.
  3. O la matou alalaupapa Around Kubernetes i Telegram.

puna: www.habr.com

Faaopoopo i ai se faamatalaga