🥇Autoscaling Kubernetes application uchishandisa Prometheus uye KEDA

Balloon Man naCimuanos

Scalability chinhu chakakosha chinodiwa kune Cloud application. NaKubernetes, kuyera application kuri nyore sekuwedzera huwandu hwemakopi ekutumirwa kwakakodzera kana. ReplicaSet - asi inzira yemanyorero.

Kubernetes inobvumira maapplication kuti azviyerwe otomatiki (kureva maPods mune deployment kana ReplicaSet) nenzira yekuzivisa uchishandisa Horizontal Pod Autoscaler yakatarwa. Iyo yekumisikidza chiyero chekuyera otomatiki ndeye CPU yekushandisa metrics (resource metrics), asi iwe unogona kubatanidza tsika uye kunze metrics yakapihwa.

chikwata Kubernetes aaS kubva kuMail.ru yakashandura chinyorwa che mashandisiro ekunze metrics kuyera otomatiki Kubernetes application. Kuratidza kuti zvese zvinoshanda sei, munyori anoshandisa HTTP yekuwana chikumbiro metrics, iyo inounganidzwa uchishandisa Prometheus.

Panzvimbo peyakachinjika autoscaling yemapods, Kubernetes Chiitiko Inotyairwa Autoscaling (KEDA) inoshandiswa, yakavhurika sosi Kubernetes opareta. Iyo inosanganisa natively neHorizontal Pod Autoscaler kuti ipe isina musono autoscaling (inosanganisira kusvika/kubva ku zero) yemabasa anofambiswa nechiitiko. Code inowanikwa pa GitHub.

Mhedziso pfupi yehurongwa

Dhiagiramu inoratidza tsananguro pfupi yekuti zvese zvinoshanda sei:

Chishandiso chinopa HTTP hit count count metrics muPrometheus fomati.
Prometheus inogadzirirwa kuunganidza aya metrics.
Iyo Prometheus scaler muKEDA inogadzirirwa kuyera otomatiki application zvichienderana nehuwandu hweHTTP hits.

Zvino ini ndichakuudza zvakadzama nezve chinhu chimwe nechimwe.

KEDA uye Prometheus

Prometheus ndeye yakavhurika sosi system yekutarisa uye yekunyevera toolkit, chikamu Cloud Native Computing Nheyo. Inounganidza metrics kubva kwakasiyana masosi uye inoachengeta senge data data. Kuti uone data yaunogona kushandisa grafana kana mamwe maturusi ekuona anoshanda neKubernetes API.

KEDA inotsigira pfungwa ye scaler - inoita sebhiriji pakati peKEDA nekunze system. Kuitwa kwe scaler kwakananga kune yega yega chinangwa system uye inobvisa data kubva mairi. KEDA yobva yavashandisa kudzora otomatiki kuyera.

Scalers inotsigira akawanda data masosi, semuenzaniso, Kafka, Redis, Prometheus. Ndokunge, KEDA inogona kushandiswa kuyera otomatiki Kubernetes deployments uchishandisa Prometheus metrics semaitiro.

Test application

Iyo Golang test application inopa mukana kuburikidza neHTTP uye inoita maviri akakosha mabasa:

Inoshandisa iyo Prometheus Go mutengi raibhurari kuridza chishandiso uye kupa iyo http_requests metric, ine hit count. Iyo yekupedzisira iyo Prometheus metrics inowanikwa inowanikwa kuURI /metrics.
```
var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests",
       Help: "number of http requests",
   })
```

Mukupindura chikumbiro GET iyo application inowedzera kukosha kwekiyi (access_count) muRedis. Iyi inzira iri nyore yekuita basa sechikamu cheHTTP inobata uye zvakare tarisa Prometheus metrics. Huwandu hwemetric hunofanira kufanana neukoshi access_count muRedis.

func main() {
       http.Handle("/metrics", promhttp.Handler())
       http.HandleFunc("/test", func(w http.ResponseWriter, r 
*http.Request) {
           defer httpRequestsCounter.Inc()
           count, err := client.Incr(redisCounterName).Result()
           if err != nil {
               fmt.Println("Unable to increment redis counter", err)
               os.Exit(1)
           }
           resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count))
           w.Write([]byte(resp))
       })
       http.ListenAndServe(":8080", nil)
   }

Iyo application inotumirwa kuKubernetes kuburikidza Deployment. Sevhisi inogadzirwa zvakare ClusterIP, inobvumira sevha yePrometheus kuwana mametric ekushandisa.

pano deployment manifest yekushandiswa.

Prometheus Server

Iyo Prometheus deployment manifest inosanganisira:

ConfigMap - kuendesa iyo Prometheus config;
Deployment - yekuendesa Prometheus muboka reKubernetes;
ClusterIP - sevhisi yekuwana UI Prometheus;
ClusterRole, ClusterRoleBinding и ServiceAccount - yekutarisa otomatiki masevhisi muKubernetes (Auto-discovery).

pano manifest yekumhanyisa Prometheus.

KEDA Prometheus ScaledObject

Iyo scaler inoita sebhiriji pakati peKEDA uye yekunze system kubva kune iyo metrics inoda kuwanikwa. ScaledObject itsika sosi inoda kuisirwa kuwiriranisa kutumirwa neyechiitiko sosi, mune iyi kesi Prometheus.

ScaledObject ine ruzivo rwekuyeresa, chiitiko sosi metadata (sezvakavanzika zvekubatanidza, zita remutsetse), nguva yekuvhota, nguva yekudzoreredza, uye imwe data. Iyo inoguma mune inoenderana autoscaling sosi (HPA tsananguro) kuyera kutumirwa.

Kana chinhu ScaledObject inodzimwa, iyo inoenderana neHPA tsananguro inobviswa.

Heino tsanangudzo ScaledObject semuenzaniso wedu, inoshandisa scaler Prometheus:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
 name: prometheus-scaledobject
 namespace: default
 labels:
   deploymentName: go-prom-app
spec:
 scaleTargetRef:
   deploymentName: go-prom-app
 pollingInterval: 15
 cooldownPeriod:  30
 minReplicaCount: 1
 maxReplicaCount: 10
 triggers:
 - type: prometheus
   metadata:
     serverAddress: 
http://prometheus-service.default.svc.cluster.local:9090
     metricName: access_frequency
     threshold: '3'
     query: sum(rate(http_requests[2m]))

Chimbofunga pfungwa dzinotevera:

Anonongedzera Deployment Nezita go-prom-app.
Trigger type - Prometheus. Iyo Prometheus server kero inotaurwa pamwe chete nezita remetric, chikumbaridzo uye PromQL mubvunzo, iyo ichashandiswa. PromQL Mubvunzo - sum(rate(http_requests[2m])).
Maererano ne pollingInterval,KEDA inokumbira chinangwa kubva kuPrometheus masekonzi gumi neshanu ega ega. Inenge imwe pasi (minReplicaCount), uye nhamba yepamusoro yepods haipfuuri maxReplicaCount (mumuenzaniso uyu - gumi).

Inogona kuisa minReplicaCount zvakaenzana ne zero. Muchiitiko ichi, KEDA inomisa zero-kune-imwe kutumirwa uye yobva yafumura iyo HPA kuti iwedzere kuyera otomatiki. Kurongeka kwemashure kunogonekawo, kureva, kuyera kubva kune imwe kuenda ku zero. Mumuenzaniso, isu hatina kusarudza zero nekuti iyi ibasa reHTTP uye kwete pane-inodiwa system.

Iwo mashiripiti mukati autoscaling

Chikumbaridzo chinoshandiswa sechinhu chinokonzeresa kuyera kutumirwa. Mumuenzaniso wedu, iyo PromQL mubvunzo sum(rate (http_requests [2m])) inodzosa iyo yakaunganidzwa yeHTTP yekukumbira mwero (zvikumbiro pasekondi), yakayerwa pamaminetsi maviri apfuura.

Sezvo kukosha kwechikumbaridzo kuri matatu, zvinoreva kuti pachave neimwe pasi apo kukosha sum(rate (http_requests [2m])) asingasviki matatu. Kana kukosha kwacho kuchiwedzera, imwe sub inowedzerwa nguva imwe neimwe sum(rate (http_requests [2m])) inowedzera katatu. Semuenzaniso, kana kukosha kuri kubva pa12 kusvika pa14, ipapo nhamba yemapodhi mana.

Zvino ngatiedze kuimisa!

Pre-setting

Zvese zvaunoda iKubernetes cluster uye yakagadziriswa utility kubectl. Uyu muenzaniso unoshandisa cluster minikube, asi unogona kutora chero imwe ipi zvayo. Kuisa cluster iripo gwara.

Isa iyo yazvino vhezheni paMac:

curl -Lo minikube 
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Set kubectlkuti uwane iyo Kubernetes cluster.

Isa iyo yazvino vhezheni paMac:

curl -LO 
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

KEDA kuiswa

Unogona kuendesa KEDA munzira dzinoverengeka, dzakanyorwa mukati zvinyorwa. Ndiri kushandisa monolithic YAML:

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml

KEDA uye zvikamu zvayo zvakaiswa munzvimbo yezita keda. Raira kutarisa:

kubectl get pods -n keda

Mirira KEDA Operator kutanga uye kuenda Running State. Uye shure kwaizvozvo, enderera.

Kuisa Redis uchishandisa Helm

Kana iwe usina Helm yakaiswa, shandisa izvi utungamiri. Raira kuisa paMac:

brew install kubernetes-helm
helm init --history-max 200

helm init inotanga iyo yemunharaunda yekuraira mutsara interface uye zvakare inoisa Tiller kune Kubernetes cluster.

kubectl get pods -n kube-system | grep tiller

Mirira iyo Tiller pod kuti ipinde muRunning state.

Chiziviso chemushanduri: Munyori anoshandisa Helm @ 2, iyo inoda kuti Tiller server chikamu kuti iiswe. Iye zvino Helm @ 3 yakakosha, haidi sevha chikamu.

Mushure mekuisa Helm, murairo mumwe wakakwana kutanga Redis:

helm install --name redis-server --set cluster.enabled=false --set 
usePassword=false stable/redis

Tarisa kuti Redis yatanga zvinobudirira:

kubectl get pods/redis-server-master-0

Mirira kuti Redis iende muhurumende Running.

Application Deployment

Deployment command:

kubectl apply -f go-app.yaml

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

Tarisa kuti zvese zvatanga:

kubectl get pods -l=app=go-prom-app

Mirira kuti Redis ipinde mudunhu Running.

Kutumira Prometheus Server

Iyo Prometheus inoratidza inoshandisa Kubernetes Service Discovery yePrometheus. Iyo inobvumira kuwanikwa kwemaitiro ekushandisa pods zvichibva pane iyo sevhisi label.

kubernetes_sd_configs:
   - role: service
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_label_run]
     regex: go-prom-app-service
     action: keep

Kutumira:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

Tarisa kuti zvese zvatanga:

kubectl get pods -l=app=prometheus-server

Mirira kuti Prometheus apinde muhurumende Running.

Shandisa kubectl port-forward kuwana iyo Prometheus mushandisi interface (kana API server) pa http://localhost:9090.

kubectl port-forward service/prometheus-service 9090

Kutumira KEDA Autoscaling Configuration

Raira kugadzira ScaledObject:

kubectl apply -f keda-prometheus-scaledobject.yaml

Tarisa KEDA opareta matanda:

KEDA_POD_NAME=$(kubectl get pods -n keda 
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

Mhedzisiro inotaridzika seizvi:

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with 
namespace default and name keda-hpa-go-prom-app"

Tarisa pasi pemaapplication. Imwe muenzaniso inofanira kunge ichimhanya nekuti minReplicaCount zvakaenzana ne1:

kubectl get pods -l=app=go-prom-app

Tarisa kuti HPA sosi yakagadzirwa zvinobudirira:

kubectl get hpa

Iwe unofanirwa kuona chimwe chinhu chakadai:

NAME                   REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   0/3 (avg)   1         10        1          45s

Cheki yehutano: kuwanikwa kwekushandisa

Kuti uwane yedu application's REST endpoint, mhanya:

kubectl port-forward service/go-prom-app-service 8080

Iwe unogona ikozvino kuwana yako Go app uchishandisa kero http://localhost:8080. Kuti uite izvi, shandisa murairo:

curl http://localhost:8080/test

Mhedzisiro inotaridzika seizvi:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC 
m=+406004.817901246
Access count 1

Panguva ino zvakare tarisa Redis. Uchaona kuti kiyi yacho access_count yakawedzera kusvika ku1:

kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"

Ita shuwa kuti metric kukosha ndeye http_requests zvimwe chete:

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Load Creation

Tichashandisa Hey - chishandiso chekugadzira mutoro:

curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64 
&& chmod a+x hey

Iwe unogona zvakare kudhawunirodha chishandiso che Linux kana Windows.

Imhanye:

./hey http://localhost:8080/test

Nekusagadzikana, iyo yekushandisa inotumira mazana maviri zvikumbiro. Unogona kuona izvi uchishandisa Prometheus metrics pamwe neRedis.

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201

Simbisa kukosha kweiyo metric chaiyo (yakadzoserwa nemubvunzo wePromQL):

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

Muchiitiko ichi mhedzisiro chaiyo ndeye 1,686057971014493 uye inoratidzwa mumunda value. Izvi hazvina kukwana kuyera, sezvo chikumbaridzo chatinoisa chiri 3.

More mutoro!

Mune itsva terminal, tarisa huwandu hwemapodhi ekushandisa:

kubectl get pods -l=app=go-prom-app -w

Ngatiwedzerei mutoro tichishandisa murairo:

./hey -n 2000 http://localhost:8080/test

Mushure mechinguva, iwe uchaona HPA ichiyera kutumirwa uye kuvhura mapodhi matsva. Tarisa HPA yako kuti uve nechokwadi:

kubectl get hpa
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   1830m/3 (avg)   1         10        6          4m22s

Kana mutoro wacho usingaenderani, kutumirwa kunoderedzwa kusvika pane imwe chete pod iri kushanda. Kana iwe uchida kutarisa iyo chaiyo metric (yakadzoserwa nemubvunzo wePromQL), wobva washandisa murairo:

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'

Kuchenesa

//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server

mhedziso

KEDA inokutendera kuti uzviyere otomatiki yako Kubernetes deployments (ku/kubva zero) zvichibva pane data kubva kunze metrics. Semuenzaniso, zvichibva paPrometheus metrics, kureba kwemutsetse muRedis, mutengi latency muKafka musoro.

KEDA inosanganisirwa neyekunze sosi uye inopawo metrics yayo kuburikidza neMetrics Server kune Horizontal Pod Autoscaler.

Good luck!

Zvimwe zvekuverenga:

Source: www.habr.com

Autoscaling Kubernetes application uchishandisa Prometheus uye KEDA