🥇Ukukala okuzenzakalelayo kwezinhlelo zokusebenza ze-Kubernetes usebenzisa i-Prometheus ne-KEDA

Ibhaluni Indoda nguCimuanos

I-scalability iyisidingo esiyinhloko sezinhlelo zokusebenza zamafu. Nge-Kubernetes, ukukala uhlelo lokusebenza kulula njengokukhulisa inani lezifaniso zokuthunyelwa okufanele noma ReplicaSet - kodwa inqubo eyenziwa mathupha.

I-Kubernetes ivumela izinhlelo zokusebenza ukuthi zikalwe ngokuzenzakalelayo (okungukuthi. Ama-Pods ekusetshenzisweni noma ReplicaSet) ngendlela yokumemezela kusetshenziswa ukucaciswa kwe-Horizontal Pod Autoscaler. Umbandela ozenzakalelayo wokukala okuzenzakalelayo amamethrikhi okusetshenziswa kwe-CPU (amamethrikhi ensiza), kodwa ungakwazi ukuhlanganisa amamethrikhi angokwezifiso nahlinzekwe ngaphandle.

Ithimba I-Kubernetes aaS evela ku-Mail.ru uhumushe isiqephu sendatshana sendlela yokusebenzisa amamethrikhi angaphandle ukukala ngokuzenzakalelayo uhlelo lokusebenza lwe-Kubernetes. Ukukhombisa ukuthi yonke into isebenza kanjani, umbhali usebenzisa amamethrikhi esicelo sokufinyelela ku-HTTP, aqoqwa kusetshenziswa i-Prometheus.

Esikhundleni sokulinganisa okuzenzakalelayo kwama-pods, kusetshenziswa i-Kubernetes Event Driven Autoscaling (KEDA), umthombo ovulekile we-Kubernetes opharetha. Ihlanganisa ngokomdabu ne-Horizontal Pod Autoscaler ukuze inikeze ukukala okuzenzakalelayo okungenamthungo (okuhlanganisa ukuya/kusuka ku-zero) kwemithwalo yomsebenzi eqhutshwa umcimbi. Ikhodi iyatholakala ku GitHub.

Uhlolojikelele olufushane lwesistimu

Umdwebo ubonisa incazelo emfushane yokuthi yonke into isebenza kanjani:

Uhlelo lokusebenza luhlinzeka ngamamethrikhi wokubala we-HTTP ngefomethi ye-Prometheus.
I-Prometheus ilungiselelwe ukuthi iqoqe lawa mamethrikhi.
Isikali se-Prometheus ku-KEDA silungiselelwe ukukala ngokuzenzakalelayo uhlelo lokusebenza ngokusekelwe enanini lamahithi e-HTTP.

Manje ngizokutshela ngokuningiliziwe mayelana nesici ngasinye.

I-KEDA kanye ne-Prometheus

I-Prometheus iyikhithi yokuqapha yesistimu yomthombo ovulekile kanye nesexwayiso, ingxenye Isisekelo se-Cloud Native Computing. Iqoqa amamethrikhi emithonjeni ehlukahlukene futhi iwagcine njengedatha yochungechunge lwesikhathi. Ukuze ubone ngeso lengqondo idatha ungasebenzisa UGrafana noma amanye amathuluzi okubuka asebenza ne-Kubernetes API.

I-KEDA isekela umqondo we-scaler - isebenza njengebhuloho phakathi kwe-KEDA nohlelo lwangaphandle. Ukuqaliswa kwe-scaler kucacile ohlelweni ngalunye oluqondiwe futhi lukhipha idatha kulo. I-KEDA bese iwasebenzisela ukulawula ukukala okuzenzakalelayo.

Ama-Scalers asekela imithombo eminingi yedatha, isibonelo, i-Kafka, i-Redis, i-Prometheus. Okusho ukuthi, i-KEDA ingasetshenziswa ukukala ngokuzenzakalelayo ukuthunyelwa kwe-Kubernetes kusetshenziswa amamethrikhi e-Prometheus njengemibandela.

Isicelo sokuhlola

Uhlelo lokusebenza lokuhlola lwe-Golang lunikeza ukufinyelela nge-HTTP futhi lenza imisebenzi emibili ebalulekile:

Isebenzisa umtapo wolwazi weklayenti le-Prometheus Go ukuze isebenze uhlelo lokusebenza futhi inikeze i-http_requests metric, equkethe ukubala okushaywayo. Indawo yokugcina lapho kutholakala khona amamethrikhi e-Prometheus itholakala ku-URI /metrics.
```
var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests",
       Help: "number of http requests",
   })
```

Ephendula isicelo GET uhlelo lokusebenza lunyusa inani lokhiye (access_count) eRedis. Lena indlela elula yokwenza umsebenzi njengengxenye yesibambi se-HTTP futhi uhlole futhi amamethrikhi e-Prometheus. Inani lemethrikhi kufanele lifane nevelu access_count eRedis.

func main() {
       http.Handle("/metrics", promhttp.Handler())
       http.HandleFunc("/test", func(w http.ResponseWriter, r 
*http.Request) {
           defer httpRequestsCounter.Inc()
           count, err := client.Incr(redisCounterName).Result()
           if err != nil {
               fmt.Println("Unable to increment redis counter", err)
               os.Exit(1)
           }
           resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count))
           w.Write([]byte(resp))
       })
       http.ListenAndServe(":8080", nil)
   }

Isicelo sisatshalaliswa ku-Kubernetes nge Deployment. Kuphinde kudalwe isevisi ClusterIP, ivumela iseva ye-Prometheus ukuthi ithole amamethrikhi ohlelo lokusebenza.

Lapha i-manifest yokuphakelwa kohlelo lokusebenza.

Iseva ye-Prometheus

I-Prometheus deployment manifest iqukethe:

ConfigMap - ukudlulisa ukucushwa kwe-Prometheus;
Deployment - ngokuthumela i-Prometheus kuqoqo le-Kubernetes;
ClusterIP - isevisi yokufinyelela ku-UI Prometheus;
ClusterRole, ClusterRoleBinding и ServiceAccount — ukuze kutholwe ngokuzenzakalela izinsiza ku-Kubernetes (I-Auto-discovery).

Lapha i-manifest yokusebenzisa i-Prometheus.

I-KEDA Prometheus ScaledObject

I-scaler isebenza njengebhuloho phakathi kwe-KEDA nohlelo lwangaphandle lapho amamethrikhi adinga ukutholwa khona. ScaledObject iwumthombo wangokwezifiso odinga ukuthunyelwa ukuze uvumelanise ukusetshenziswa nomthombo womcimbi, kulokhu i-Prometheus.

ScaledObject iqukethe ulwazi lwesikali sokuphakwa, imethadatha yomthombo womcimbi (njengezimfihlo zokuxhuma, igama lomugqa), isikhathi sokuvota, isikhathi sokutakula, nenye idatha. Kuphumela kusisetshenziswa esihambisanayo sokulinganisa okuzenzakalelayo (incazelo ye-HPA) ukuze kukale ukuthunyelwa.

Lapho into ScaledObject iyasuswa, incazelo ye-HPA ehambisanayo isuliwe.

Nansi incazelo ScaledObject isibonelo sethu, isebenzisa i-scaler Prometheus:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
 name: prometheus-scaledobject
 namespace: default
 labels:
   deploymentName: go-prom-app
spec:
 scaleTargetRef:
   deploymentName: go-prom-app
 pollingInterval: 15
 cooldownPeriod:  30
 minReplicaCount: 1
 maxReplicaCount: 10
 triggers:
 - type: prometheus
   metadata:
     serverAddress: 
http://prometheus-service.default.svc.cluster.local:9090
     metricName: access_frequency
     threshold: '3'
     query: sum(rate(http_requests[2m]))

Sicela uqaphele amaphuzu alandelayo:

Ukhomba Deployment Ngegama go-prom-app.
Uhlobo lwe-trigger - Prometheus. Ikheli leseva ye-Prometheus lishiwo kanye negama lemethrikhi, umkhawulo kanye Umbuzo we-PromQL, ezosetshenziswa. Umbuzo we-PromQL - sum(rate(http_requests[2m])).
Ngokusho pollingInterval,I-KEDA icela ithagethi ku-Prometheus njalo ngemizuzwana eyishumi nanhlanu. Okungenani okukodwa ngaphansi (minReplicaCount), futhi inani eliphezulu lama-pods alidluli maxReplicaCount (kulesi sibonelo - ishumi).

Ingafakwa minReplicaCount ilingana noziro. Kulesi simo, i-KEDA yenza kusebenze ukusetshenziswa kweziro-to-one bese iveza i-HPA ukuze iqhubeke nokukala okuzenzakalelayo. Ukuhleleka okuphambene nakho kuyenzeka, okungukuthi, ukukala kusuka kokukodwa kuye kuziro. Esibonelweni, asikhethanga uziro ngoba lena yisevisi ye-HTTP hhayi isistimu efunwa kakhulu.

Umlingo ongaphakathi kwe-autoscaling

I-threshold isetshenziswa njenge-trigger ukukala ukuthunyelwa. Esibonelweni sethu, umbuzo we-PromQL sum(rate (http_requests [2m])) ibuyisela isilinganiso sesicelo se-HTTP esihlanganisiwe (izicelo ngomzuzwana), esilinganiswa ngemizuzu emibili edlule.

Njengoba inani le-threshold lithathu, kusho ukuthi kuzoba neyodwa ngaphansi ngenkathi inani sum(rate (http_requests [2m])) ngaphansi kokuthathu. Uma inani likhuphuka, i-sub eyengeziwe yengezwa isikhathi ngasinye sum(rate (http_requests [2m])) ukwanda kathathu. Isibonelo, uma inani lisuka ku-12 kuya ku-14, inani lama-pods limane.

Manje ake sizame ukuyisetha!

ukuhlela kusengaphambili

Okudingayo iqoqo le-Kubernetes kanye nensiza emisiwe kubectl. Lesi sibonelo sisebenzisa iqoqo minikube, kodwa ungathatha noma iyiphi enye. Ukufaka iqoqo kukhona umhlahlandlela.

Faka inguqulo yakamuva ku-Mac:

curl -Lo minikube 
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Faka kubctlukufinyelela iqoqo le-Kubernetes.

Faka inguqulo yakamuva ku-Mac:

curl -LO 
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

Ukufakwa kwe-KEDA

Ungaphakela i-KEDA ngezindlela ezimbalwa, ezifakwe kuhlu imibhalo. Ngisebenzisa i-YAML ye-monolithic:

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml

I-KEDA kanye nezingxenye zayo zifakwe endaweni yamagama keda. Umyalo okufanele uhlolwe:

kubectl get pods -n keda

Linda i-KEDA Operator ukuthi iqale futhi uye ku Running State. Futhi emva kwalokho, qhubeka.

Ukufaka i-Redis usebenzisa i-Helm

Uma ungenayo i-Helm efakiwe, sebenzisa lokhu ubuholi. Umyalo ongawufaka ku-Mac:

brew install kubernetes-helm
helm init --history-max 200

helm init iqalisa isixhumi esibonakalayo somugqa womyalo wendawo futhi iphinde ifake Tiller kwiqoqo leKubernetes.

kubectl get pods -n kube-system | grep tiller

Linda i-Tiller pod ukuze ingene kusimo sokuSebenza.

Inothi lomhumushi: Umbhali usebenzisa i-Helm@2, edinga ukuthi ingxenye yeseva yeTiller ifakwe. Manje i-Helm@3 iyasebenza, ayidingi ingxenye yeseva.

Ngemuva kokufaka i-Helm, umyalo owodwa wanele ukuqala i-Redis:

helm install --name redis-server --set cluster.enabled=false --set 
usePassword=false stable/redis

Qinisekisa ukuthi i-Redis iqale ngempumelelo:

kubectl get pods/redis-server-master-0

Linda i-Redis ukuthi ingene esimweni Running.

Ukuthunyelwa Kohlelo

Umyalo wokuthunyelwa:

kubectl apply -f go-app.yaml

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

Hlola ukuthi konke sekuqalile:

kubectl get pods -l=app=go-prom-app

Linda i-Redis ukuthi ingene esifundazweni Running.

Kusetshenziswa iseva ye-Prometheus

I-Prometheus manifest isebenzisa I-Kubernetes Service Discovery ye-Prometheus. Ivumela ukutholwa okuguquguqukayo kwama-pod ohlelo lokusebenza ngokusekelwe kulebula yesevisi.

kubernetes_sd_configs:
   - role: service
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_label_run]
     regex: go-prom-app-service
     action: keep

Ukuze usebenzise:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

Hlola ukuthi konke sekuqalile:

kubectl get pods -l=app=prometheus-server

Lindela u-Prometheus ukuthi angene esimweni Running.

Sebenzisa kubectl port-forward ukufinyelela ku-interface yomsebenzisi ye-Prometheus (noma iseva ye-API) ku http://localhost:9090.

kubectl port-forward service/prometheus-service 9090

Isebenzisa ukucushwa kwe-KEDA Autoscaling

Yala ukuze udale ScaledObject:

kubectl apply -f keda-prometheus-scaledobject.yaml

Hlola amalogi o-opharetha we-KEDA:

KEDA_POD_NAME=$(kubectl get pods -n keda 
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

Umphumela ubukeka kanjena:

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with 
namespace default and name keda-hpa-go-prom-app"

Hlola ngaphansi kwezicelo. Isibonelo esisodwa kufanele sisebenze ngoba minReplicaCount kulingana no-1:

kubectl get pods -l=app=go-prom-app

Qinisekisa ukuthi insiza ye-HPA idalwe ngempumelelo:

kubectl get hpa

Kufanele ubone okuthile okufana nalokhu:

NAME                   REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   0/3 (avg)   1         10        1          45s

Ukuhlolwa kwezempilo: ukufinyelela kwesicelo

Ukuze ufinyelele indawo yokugcina ye-REST yohlelo lwethu lokusebenza, sebenzisa:

kubectl port-forward service/go-prom-app-service 8080

Manje usungakwazi ukufinyelela kuhlelo lwakho lokusebenza lwe-Go usebenzisa ikheli http://localhost:8080. Ukuze wenze lokhu, sebenzisa umyalo:

curl http://localhost:8080/test

Umphumela ubukeka kanjena:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC 
m=+406004.817901246
Access count 1

Kuleli phuzu futhi hlola i-Redis. Uzobona ukuthi ukhiye access_count ikhuphuke yaba ngu-1:

kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"

Qiniseka ukuthi inani lemethrikhi liyiyo http_requests okufanayo:

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Layisha Indalo

Sizosebenzisa hey - Usizo lokukhiqiza umthwalo:

curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64 
&& chmod a+x hey

Ungakwazi futhi ukulanda uhlelo lokusebenza lwe Linux noma Windows.

Yiqalise:

./hey http://localhost:8080/test

Ngokuzenzakalelayo, insiza ithumela izicelo ezingama-200. Ungaqinisekisa lokhu usebenzisa amamethrikhi e-Prometheus kanye ne-Redis.

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201

Qinisekisa inani lemethrikhi yangempela (ibuyiswe umbuzo we-PromQL):

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

Kulokhu umphumela wangempela 1,686057971014493 futhi iboniswa ebaleni value. Lokhu akwanele ukukala, njengoba umkhawulo esiwusethile ungu-3.

Umthwalo owengeziwe!

Kutheminali entsha, qapha inani lamaphodi ohlelo lokusebenza:

kubectl get pods -l=app=go-prom-app -w

Masikhulise umthwalo sisebenzisa umyalo:

./hey -n 2000 http://localhost:8080/test

Ngemuva kwesikhashana, uzobona i-HPA ikhulisa ukuthunyelwa futhi yethula ama-pods amasha. Hlola i-HPA yakho ukuze uqiniseke:

kubectl get hpa
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   1830m/3 (avg)   1         10        6          4m22s

Uma umthwalo ungahambisani, ukuthunyelwa kuzoncishiswa kuze kufike lapho kusebenza khona i-pod eyodwa kuphela. Uma ufuna ukuhlola imethrikhi yangempela (ebuyiswe umbuzo we-PromQL), bese usebenzisa umyalo:

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'

Ukuhlanza

//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server

isiphetho

I-KEDA ikuvumela ukuthi ukale ngokuzenzakalelayo ukusetshenziswa kwakho kwe-Kubernetes (kuya/kusuka kuqanda) ngokusekelwe kudatha evela kumamethrikhi angaphandle. Isibonelo, ngokusekelwe kumamethrikhi e-Prometheus, ubude bomugqa e-Redis, ukubambezeleka komthengi esihlokweni se-Kafka.

I-KEDA ihlanganisa nomthombo wangaphandle futhi ihlinzeka ngamamethrikhi ayo nge-Metrics Server kuya ku-Horizontal Pod Autoscaler.

Good luck!

Okunye ongakufunda:

Source: www.habr.com

Autoscaling Kubernetes izicelo usebenzisa Prometheus kanye KEDA