ProHoster > Blog > Tsamaiso > Autoscaling Kubernetes lits'ebetso tse sebelisang Prometheus le KEDA

Autoscaling Kubernetes lits'ebetso tse sebelisang Prometheus le KEDA

Autoscaling Kubernetes lits'ebetso tse sebelisang Prometheus le KEDABalloon Man ka Cimuanos

Scalability ke tlhokahalo ea mantlha bakeng sa lits'ebetso tsa leru. Ka Kubernetes, ho lekola kopo ho bonolo joalo ka ho eketsa palo ea likopi bakeng sa thomello e nepahetseng kapa ReplicaSet - empa ke ts'ebetso ea matsoho.

Kubernetes e lumella lits'ebetso hore li ntlafatsoe ka bo eona (ke hore, li-Pods sebakeng sa thomello kapa ReplicaSet) ka mokhoa oa ho phatlalatsa o sebelisa litlhaloso tsa Horizontal Pod Autoscaler. Maemo a kamehla a ho lekanya ka bohona ke metrics ea ts'ebeliso ea CPU (metrics ea lisebelisoa), empa o ka kopanya metrics ea tloaelo le e fanoeng kantle.

sehlopha Kubernetes aaS ho tsoa ho Mail.ru e fetoletse sengoloa se mabapi le mokhoa oa ho sebelisa metrics ea kantle ho iteka ka bohona ts'ebeliso ea Kubernetes. Ho bontša kamoo ntho e 'ngoe le e' ngoe e sebetsang kateng, mongoli o sebelisa metrics ea kopo ea ho fumana HTTP, e bokelloang ho sebelisoa Prometheus.

Sebakeng sa autoscaling e otlolohileng ea li-pods, ho sebelisoa Kubernetes Event Driven Autoscaling (KEDA), mochine o bulehileng oa Kubernetes. E kopanya natively le Horizontal Pod Autoscaler ho fana ka autoscaling e se nang moeli (ho kenyeletsoa ho / ho tloha ho zero) bakeng sa meroalo e tsamaisoang ke ketsahalo. Khoutu e fumaneha ho GitHub.

Tlhaloso e khutšoanyane ea tsamaiso

Autoscaling Kubernetes lits'ebetso tse sebelisang Prometheus le KEDA

Setšoantšo se bontša tlhaloso e khutšoanyane ea hore na ntho e 'ngoe le e' ngoe e sebetsa joang:

  1. Sesebelisoa se fana ka metrics ea HTTP hit count ka sebopeho sa Prometheus.
  2. Prometheus e lokiselitsoe ho bokella metrics ena.
  3. Prometheus scaler ho KEDA e lokiselitsoe ho lekanya ts'ebeliso ka bo eona ho latela palo ea HTTP hits.

Joale ke tla u joetsa ka botlalo ka elemente ka 'ngoe.

KEDA le Prometheus

Prometheus ke karolo e bulehileng ea tlhahlobo ea sistimi le tlhokomeliso ea lisebelisoa Cloud Native Computing Foundation. E bokella metrics ho tsoa mehloling e fapaneng 'me e li boloke e le data letoto la linako. Ho bona data eo u ka e sebelisang grafana kapa lisebelisoa tse ling tsa pono tse sebetsang le Kubernetes API.

KEDA e ts'ehetsa mohopolo oa sekala - e sebetsa joalo ka borokho lipakeng tsa KEDA le sistimi ea kantle. Ts'ebetsong ea scaler e totobetse tsamaisong e 'ngoe le e' ngoe ea sepheo mme e ntša lintlha ho eona. KEDA e ntan'o li sebelisa ho laola sekhahla sa othomathike.

Scalers e tšehetsa mehloli e mengata ea data, mohlala, Kafka, Redis, Prometheus. Ka mantsoe a mang, KEDA e ka sebelisoa ho ikemela ka bo eona ho tsamaisoa ha Kubernetes ho sebelisa metrics ea Prometheus joalo ka litekanyetso.

Tshebediso ya teko

Sesebelisoa sa tlhahlobo ea Golang se fana ka phihlello ka HTTP mme se etsa mesebetsi e 'meli ea bohlokoa:

  1. E sebelisa laeborari ea bareki ea Prometheus Go ho sebelisa sesebelisoa le ho fana ka metric ea http_requests, e nang le palo e otlang. Qetello moo metrics ea Prometheus e fumanehang e fumaneha URI /metrics. 
    var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests",
       Help: "number of http requests",
   })
  2. Ho arabela kopo GET kopo e eketsa boleng ba senotlolo (access_count) ho Redis. Ena ke tsela e bonolo ea ho etsa mosebetsi e le karolo ea mohlokomeli oa HTTP hape u hlahlobe metrics ea Prometheus. Boleng ba metric bo tlameha ho tšoana le boleng access_count ho Redis. 
    func main() {
       http.Handle("/metrics", promhttp.Handler())
       http.HandleFunc("/test", func(w http.ResponseWriter, r 
*http.Request) {
           defer httpRequestsCounter.Inc()
           count, err := client.Incr(redisCounterName).Result()
           if err != nil {
               fmt.Println("Unable to increment redis counter", err)
               os.Exit(1)
           }
           resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count))
           w.Write([]byte(resp))
       })
       http.ListenAndServe(":8080", nil)
   }

Sesebelisoa se romelloa ho Kubernetes ka Deployment. Ho boetse ho etsoa tšebeletso ClusterIP, e lumella seva sa Prometheus ho fumana metrics ea kopo.

mona ponahalo ea thomello bakeng sa ts'ebeliso.

Seva ea Prometheus

Pontšo ea tlhahiso ea Prometheus e na le:

  • ConfigMap - ho fetisetsa setlhophiso sa Prometheus;
  • Deployment - bakeng sa ho tsamaisa Prometheus sehlopheng sa Kubernetes;
  • ClusterIP - tšebeletso ea ho fihlella UI Prometheus;
  • ClusterRole, ClusterRoleBinding и ServiceAccount - bakeng sa ho iphumanela lits'ebeletso ho Kubernetes (Auto-discovery).

mona manifest bakeng sa ho matha Prometheus.

KEDA Prometheus ScaledObject

Scaler e sebetsa e le borokho pakeng tsa KEDA le tsamaiso ea ka ntle eo ho eona ho hlokahalang hore ho fumanoe metrics. ScaledObject ke sesebelisoa sa tloaelo se hlokang ho sebelisoa ho hokahanya thomello le mohloli oa ketsahalo, tabeng ena Prometheus.

ScaledObject e na le lintlha tsa phalliso, metadata ea mohloli oa liketsahalo (joalo ka makunutu a khokahano, lebitso la mokoloko), nako ea likhetho, nako ea ho hlaphoheloa, le lintlha tse ling. E fella ka sesebelisoa se tsamaisanang sa autoscaling (tlhaloso ea HPA) ho lekanya phepelo.

Ha ntho ScaledObject e hlakotsoe, tlhaloso e lumellanang ea HPA e hlakotsoe.

Tlhaloso ke ena ScaledObject mohlala oa rona, e sebelisa sekala Prometheus:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
 name: prometheus-scaledobject
 namespace: default
 labels:
   deploymentName: go-prom-app
spec:
 scaleTargetRef:
   deploymentName: go-prom-app
 pollingInterval: 15
 cooldownPeriod:  30
 minReplicaCount: 1
 maxReplicaCount: 10
 triggers:
 - type: prometheus
   metadata:
     serverAddress: 
http://prometheus-service.default.svc.cluster.local:9090
     metricName: access_frequency
     threshold: '3'
     query: sum(rate(http_requests[2m]))

Nahana ka lintlha tse latelang:

  1. O supa ho Deployment Ka lebitso go-prom-app.
  2. Mofuta oa ho qala - Prometheus. Aterese ea seva ea Prometheus e boleloa hammoho le lebitso la metric, moeli le PromQL potso, e tla sebelisoa. Potso ea PromQL - sum(rate(http_requests[2m])).
  3. Ho ea ka pollingInterval,KEDA e kopa sepheo ho tsoa ho Prometheus metsotsoana e meng le e meng e leshome le metso e mehlano. Bonyane ka tlase ho (minReplicaCount), 'me palo e kholo ea li-pods ha e fete maxReplicaCount (mohlaleng ona - leshome).

E ka kenngoa minReplicaCount lekana le zero. Tabeng ena, KEDA e kenya ts'ebetsong phetiso ea zero-to-one ebe e pepesa HPA bakeng sa ho ikemela ho eketsehileng. Taelo e ka morao e ka khoneha, ke hore, ho tloha ho e 'ngoe ho ea ho zero. Mohlala, ha rea ​​khetha zero hobane ena ke ts'ebeletso ea HTTP eseng sistimi e batloang feela.

Boselamose ka hare ho autoscaling

Sekhahla se sebelisoa e le sesupo sa ho phahamisa phallo. Mohlala oa rona, potso ea PromQL sum(rate (http_requests [2m])) e khutlisa sekhahla sa kopo e kopaneng ea HTTP (likopo ka motsotsoana), tse lekantsoeng metsotsong e 'meli e fetileng.

Kaha threshold value ke tse tharo, ho bolela hore ho tla ba le e 'ngoe tlas'a ha boleng sum(rate (http_requests [2m])) ka tlase ho boraro. Haeba boleng bo eketseha, ho eketsoa sub e eketsehileng nako le nako sum(rate (http_requests [2m])) e eketseha ka tse tharo. Ka mohlala, haeba boleng bo tloha ho 12 ho isa ho 14, joale palo ea li-pods ke tse 'nè.

Joale ha re lekeng ho e beha!

ho seta pele

Seo u se hlokang feela ke sehlopha sa Kubernetes le sesebelisoa se hlophisitsoeng kubectl. Mohlala ona o sebelisa sehlopha minikube, empa u ka nka leha e le efe e 'ngoe. Ho kenya sehlopha ho na le tsamaiso.

Kenya mofuta oa morao-rao ho Mac:

curl -Lo minikube 
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Beha kubctlho fihlella sehlopha sa Kubernetes.

Kenya mofuta oa morao-rao ho Mac:

curl -LO 
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

Ho kenya KEDA

O ka tsamaisa KEDA ka mekhoa e mengata, e thathamisitsoe ho litokomane. Ke sebelisa monolithic YAML:

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml

KEDA le likarolo tsa eona li kentsoe sebakeng sa mabitso keda. Taelo ea ho hlahloba:

kubectl get pods -n keda

Emela hore KEDA Operator e qale ebe o ea ho Running State. 'Me ka mor'a moo, tsoela pele.

Ho kenya Redis ho sebelisa Helm

Haeba u sena Helm e kentsoeng, sebelisa sena boetapele. Taelo ea ho kenya Mac:

brew install kubernetes-helm
helm init --history-max 200

helm init e qala segokanyimmediamentsi sa selehae sa mola oa taelo hape e kenya Tiller ho sehlopha sa Kubernetes.

kubectl get pods -n kube-system | grep tiller

Emela hore Tiller pod e kene sebakeng sa Running.

Molaetsa oa mofetoleli: Mongoli o sebelisa Helm@2, e hlokang hore karolo ea seva sa Tiller e kenngoe. Hona joale Helm@3 e bohlokoa, ha e hloke karolo ea seva.

Kamora ho kenya Helm, taelo e le 'ngoe e lekane ho qala Redis:

helm install --name redis-server --set cluster.enabled=false --set 
usePassword=false stable/redis

Netefatsa hore Redis e qalile ka katleho:

kubectl get pods/redis-server-master-0

Emela hore Redis e kene maemong Running.

Phetiso ea Kopo

Taelo ea ho tsamaisa:

kubectl apply -f go-app.yaml

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

Etsa bonnete ba hore tsohle li qalile:

kubectl get pods -l=app=go-prom-app

Emela hore Redis e kene naheng Running.

Ho tsamaisa Seva ea Prometheus

Prometheus manifest e sebelisa Ho sibolloa ha Tšebeletso ea Kubernetes bakeng sa Prometheus. E lumella ho sibolloa ho matla ha li-pods tsa ts'ebeliso ho ipapisitse le leibole ea lits'ebeletso.

kubernetes_sd_configs:
   - role: service
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_label_run]
     regex: go-prom-app-service
     action: keep

Ho tsamaisa:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

Etsa bonnete ba hore tsohle li qalile:

kubectl get pods -l=app=prometheus-server

Emela hore Prometheus a kene mmusong Running.

Sebelisa kubectl port-forward ho fihlella sebopeho sa mosebelisi sa Prometheus (kapa seva sa API) ho http://localhost:9090.

kubectl port-forward service/prometheus-service 9090

Ho sebelisa KEDA Autoscaling Configuration

Laela ho bopa ScaledObject:

kubectl apply -f keda-prometheus-scaledobject.yaml

Sheba lintlha tsa opereishene tsa KEDA:

KEDA_POD_NAME=$(kubectl get pods -n keda 
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

Sephetho se shebahala tjena:

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with 
namespace default and name keda-hpa-go-prom-app"

Sheba tlas'a lisebelisoa. Mohlala o mong o tlameha ho sebetsa hobane minReplicaCount e lekana le 1:

kubectl get pods -l=app=go-prom-app

Netefatsa hore sesebelisoa sa HPA se entsoe ka katleho:

kubectl get hpa

U lokela ho bona ntho e kang:

NAME                   REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   0/3 (avg)   1         10        1          45s

Tlhahlobo ea bophelo bo botle: phihlello ea kopo

Ho fihlella pheletso ea ts'ebeliso ea rona ea REST, matha:

kubectl port-forward service/go-prom-app-service 8080

Hona joale u ka kena ho app ea hau ea Go u sebelisa aterese http://localhost:8080. Ho etsa sena, tsamaisa taelo:

curl http://localhost:8080/test

Sephetho se shebahala tjena:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC 
m=+406004.817901246
Access count 1

Ka nako ena hape hlahloba Redis. O tla bona hore senotlolo access_count e eketsehile ho 1:

kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"

Etsa bonnete ba hore boleng ba metric bo http_requests e tšoanang:

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Loaro Pōpo

Re tla sebelisa Hey - lisebelisoa tsa ho hlahisa thepa:

curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64 
&& chmod a+x hey

U ka boela ua khoasolla utility bakeng sa Linux kapa Windows.

E mathe:

./hey http://localhost:8080/test

Ka tloaelo, sesebelisoa se romella likopo tse 200. U ka netefatsa sena u sebelisa metrics ea Prometheus hammoho le Redis.

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201

Netefatsa boleng ba metric ea sebele (e khutlisitsoeng ke potso ea PromQL):

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

Tabeng ena sephetho sa sebele ke 1,686057971014493 mme e bontshwa lebaleng value. Sena ha sea lekana bakeng sa ho lekanya, kaha moeli oo re o behileng ke 3.

Mojaro o mong!

Sebakeng se secha, hlokomela palo ea li-pods tsa kopo:

kubectl get pods -l=app=go-prom-app -w

Ha re eketse mojaro ka ho sebelisa taelo: 

./hey -n 2000 http://localhost:8080/test

Kamora nakoana, o tla bona HPA e eketsa phallo le ho qala li-pods tse ncha. Sheba HPA ea hau ho etsa bonnete ba hore:

kubectl get hpa
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   1830m/3 (avg)   1         10        6          4m22s

Haeba mojaro o sa lumellane, ho tsamaisoa ho tla fokotsoa ho fihlela moo pod e le 'ngoe e sebetsang teng. Haeba u batla ho lekola metric ea 'nete (e khutlisitsoeng ke potso ea PromQL), sebelisa taelo:

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'

Ho hloekisa

//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server

fihlela qeto e

KEDA e u lumella ho iteka ka boits'oaro li-deployments tsa Kubernetes (ho ea/ho tloha ho zero) ho ipapisitse le data e tsoang ho metrics ea kantle. Mohlala, ho ipapisitsoe le metrics ea Prometheus, bolelele ba queue ho Redis, latency ea bareki ka sehlooho sa Kafka.

KEDA e hokahana le mohloli o kantle hape e fana ka metrics ea eona ka Metrics Server ho Horizontal Pod Autoscaler.

Katleho!

Ke eng hape eo u ka e balang:

  1. Mekhoa e metle le mekhoa e metle ea ho tsamaisa lijana le Kubernetes libakeng tsa tlhahiso.
  2. 90+ lisebelisoa tse sebetsang bakeng sa Kubernetes: ho tsamaisoa, tsamaiso, ho beha leihlo, ts'ireletso le tse ling.
  3. Seteishene sa rona se Around Kubernetes ho Telegraph.

Source: www.habr.com

Eketsa ka tlhaloso