ProHoster > Π‘Π»ΠΎΠ³ > Kev tswj hwm > Autoscaling Kubernetes Daim Ntawv Thov nrog Prometheus thiab KEDA

Autoscaling Kubernetes Daim Ntawv Thov nrog Prometheus thiab KEDA

Autoscaling Kubernetes Daim Ntawv Thov nrog Prometheus thiab KEDABalloon Man by Cimuanos

Scalability yog qhov xav tau tseem ceeb rau kev siv huab. Nrog Kubernetes, scaling ib daim ntawv thov yog yooj yim li nce tus naj npawb ntawm replicas rau qhov tsim nyog xa mus los yog ReplicaSet - tab sis nws yog tus txheej txheem phau ntawv.

Kubernetes tso cai rau cov ntawv thov kom tau txais kev ntsuas (piv txwv li Pods hauv kev xa tawm lossis ReplicaSet) nyob rau hauv ib qho kev tshaj tawm siv cov kab rov tav Pod Autoscaler specification. Lub ntsiab lus tseem ceeb rau kev ntsuas tsis siv neeg yog CPU siv cov ntsuas ntsuas (peev txheej ntsuas), tab sis koj tuaj yeem sib xyaw cov kev cai thiab kev ntsuas sab nraud.

pab neeg Kubernetes aaS los ntawm Mail.ru tau txhais ib tsab xov xwm hais txog yuav ua li cas siv cov ntsuas sab nraud kom tau txais daim ntawv thov Kubernetes. Los qhia tias txhua yam ua haujlwm li cas, tus kws sau ntawv siv HTTP nkag mus thov ntsuas, uas tau sau los ntawm Prometheus.

Hloov chaw kab rov tav autoscaling ntawm pods, Kubernetes Event Driven Autoscaling (KEDA) yog siv, qhib qhov chaw Kubernetes tus neeg teb xov tooj. Nws koom nrog ib txwm nyob nrog Kab rov tav Pod Autoscaler los muab seamless autoscaling (suav nrog rau / los ntawm xoom) rau cov xwm txheej-tsav kev ua haujlwm. Code muaj nyob ntawm GitHub.

Cov ntsiab lus luv luv ntawm qhov system

Autoscaling Kubernetes Daim Ntawv Thov nrog Prometheus thiab KEDA

Daim duab qhia qhia luv luv ntawm txhua yam ua haujlwm li cas:

  1. Daim ntawv thov muab HTTP ntaus cov ntsuas ntsuas hauv Prometheus hom.
  2. Prometheus tau teeb tsa los sau cov ntsuas no.
  3. Prometheus scaler hauv KEDA tau teeb tsa kom tau txais daim ntawv teev npe raws li tus lej HTTP hits.

Tam sim no kuv yuav qhia rau koj paub meej txog txhua yam.

KEDA thiab Prometheus

Prometheus yog qhov qhib qhov system saib xyuas thiab ceeb toom cov cuab yeej, ib feem Huab hwm coj xam huab. Sau cov metrics los ntawm ntau qhov chaw thiab khaws cia ua cov ntaub ntawv teev sijhawm. Txhawm rau pom cov ntaub ntawv koj tuaj yeem siv ua grafana lossis lwm yam cuab yeej pom kev pom uas ua haujlwm nrog Kubernetes API.

KEDA txhawb nqa lub tswv yim ntawm tus nplaig - nws ua tus choj ntawm KEDA thiab cov txheej txheem sab nraud. Kev siv scaler yog tshwj xeeb rau txhua lub hom phiaj thiab muab cov ntaub ntawv los ntawm nws. KEDA ces siv lawv los tswj tsis siv neeg scaling.

Scalers txhawb ntau cov ntaub ntawv, piv txwv li, Kafka, Redis, Prometheus. Ntawd yog, KEDA tuaj yeem siv los txiav txim ntsuas Kubernetes kev xa tawm siv Prometheus metrics raws li cov qauv.

Test daim ntawv thov

Daim ntawv thov Golang xeem muab nkag los ntawm HTTP thiab ua ob txoj haujlwm tseem ceeb:

  1. Siv lub tsev qiv ntawv Prometheus Go cov neeg siv khoom los ntsuas daim ntawv thov thiab muab qhov http_requests metric, uas muaj cov suav ntaus. Qhov kawg qhov twg Prometheus metrics muaj nyob ntawm URI /metrics. 
    var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests",
       Help: "number of http requests",
   })
  2. Teb rau qhov kev thov GET daim ntawv thov nce tus nqi ntawm tus yuam sij (access_count) hauv Redis. Qhov no yog ib txoj hauv kev yooj yim los ua haujlwm raws li ib feem ntawm HTTP handler thiab kuj tshawb xyuas Prometheus metrics. Tus nqi metric yuav tsum yog tib yam li tus nqi access_count hauv Redis. 
    func main() {
       http.Handle("/metrics", promhttp.Handler())
       http.HandleFunc("/test", func(w http.ResponseWriter, r 
*http.Request) {
           defer httpRequestsCounter.Inc()
           count, err := client.Incr(redisCounterName).Result()
           if err != nil {
               fmt.Println("Unable to increment redis counter", err)
               os.Exit(1)
           }
           resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count))
           w.Write([]byte(resp))
       })
       http.ListenAndServe(":8080", nil)
   }

Daim ntawv thov raug xa mus rau Kubernetes ntawm Deployment. Ib qho kev pabcuam kuj raug tsim ClusterIP, nws tso cai rau Prometheus server kom tau txais daim ntawv thov ntsuas.

no deployment manifest rau daim ntawv thov.

Prometheus Server

Prometheus deployment manifest muaj xws li:

  • ConfigMap - hloov lub Prometheus config;
  • Deployment - rau deploying Prometheus hauv Kubernetes pawg;
  • ClusterIP - kev pabcuam rau kev nkag mus rau UI Prometheus;
  • ClusterRole, ClusterRoleBinding ΠΈ ServiceAccount - rau kev tshawb nrhiav nws pib ntawm cov kev pabcuam hauv Kubernetes (Auto-discovery).

no manifest rau khiav Prometheus.

KEDA Prometheus ScaledObject

Lub scaler ua raws li tus choj ntawm KEDA thiab cov txheej txheem sab nraud los ntawm qhov kev ntsuas yuav tsum tau txais. ScaledObject yog ib qho kev cai uas yuav tsum tau muab xa mus rau synchronize kev xa mus nrog qhov xwm txheej, qhov no Prometheus.

ScaledObject muaj cov ntaub ntawv qhia txog kev xa tawm, cov xwm txheej metadata (xws li kev sib txuas lus zais cia, npe hu), lub sijhawm xaiv tsa, lub sijhawm rov qab los, thiab lwm yam ntaub ntawv. Nws ua rau qhov sib thooj autoscaling resource (HPA txhais) los ntsuas qhov kev xa tawm.

Thaum ib yam khoom ScaledObject raug tshem tawm, qhov sib thooj HPA txhais tau raug tshem tawm.

Nov yog qhov txhais ScaledObject rau peb piv txwv, nws siv lub scaler Prometheus:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
 name: prometheus-scaledobject
 namespace: default
 labels:
   deploymentName: go-prom-app
spec:
 scaleTargetRef:
   deploymentName: go-prom-app
 pollingInterval: 15
 cooldownPeriod:  30
 minReplicaCount: 1
 maxReplicaCount: 10
 triggers:
 - type: prometheus
   metadata:
     serverAddress: 
http://prometheus-service.default.svc.cluster.local:9090
     metricName: access_frequency
     threshold: '3'
     query: sum(rate(http_requests[2m]))

Xav txog cov ntsiab lus hauv qab no:

  1. Nws taw rau Deployment Nrog lub npe go-prom-app.
  2. Trigger hom - Prometheus. Prometheus server chaw nyob tau hais nrog rau lub npe metric, pib thiab PromQL lus nug, uas yuav siv tau. PromQL Query - sum(rate(http_requests[2m])).
  3. Raws li pollingInterval, KEDA thov lub hom phiaj los ntawm Prometheus txhua kaum tsib vib nas this. Tsawg kawg yog ib qho hauv qab (minReplicaCount), thiab qhov siab tshaj plaws ntawm cov pods tsis tshaj maxReplicaCount (hauv qhov piv txwv no - kaum).

Yuav nruab minReplicaCount sib npaug rau xoom. Nyob rau hauv rooj plaub no, KEDA qhib lub xoom-rau-ib xa tawm thiab tom qab ntawd nthuav tawm HPA rau kev ntsuas tsis siv neeg ntxiv. Qhov kev txiav txim rov qab kuj tseem ua tau, uas yog, ntsuas los ntawm ib mus rau xoom. Hauv qhov piv txwv, peb tsis tau xaiv xoom vim qhov no yog qhov kev pabcuam HTTP thiab tsis yog qhov kev xav tau.

Cov khawv koob hauv autoscaling

Qhov chaw pib yog siv los ua ib qho txiaj ntsig los ntsuas qhov kev xa tawm. Hauv peb qhov piv txwv, PromQL cov lus nug sum(rate (http_requests [2m])) xa rov qab qhov sib sau HTTP thov tus nqi (thov ib ob), ntsuas ob feeb dhau los.

Txij li thaum tus nqi pib yog peb, nws txhais tau tias yuav muaj ib qho qis dua thaum tus nqi sum(rate (http_requests [2m])) tsawg dua peb. Yog tias tus nqi nce, ib qho ntxiv sub ntxiv txhua zaus sum(rate (http_requests [2m])) nce los ntawm peb. Piv txwv li, yog tias tus nqi yog los ntawm 12 mus rau 14, ces tus naj npawb ntawm cov pods yog plaub.

Tam sim no cia sim teeb nws!

preseting

Txhua yam koj xav tau yog Kubernetes pawg thiab cov khoom siv teeb tsa kubectl. Qhov piv txwv no siv ib pawg minikube, tab sis koj tuaj yeem nqa lwm tus. Txhawm rau nruab ib pawg muaj kev ua thawj coj.

Nruab qhov tseeb version ntawm Mac:

curl -Lo minikube 
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Teeb kubtl uankag mus rau Kubernetes pawg.

Nruab qhov tseeb version ntawm Mac:

curl -LO 
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

KEDA installation

Koj tuaj yeem xa KEDA rau ntau txoj hauv kev, lawv tau teev nyob rau hauv cov ntaub ntawv. Kuv siv monolithic YAML:

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml

KEDA thiab nws cov khoom tau nruab rau hauv lub npe keda. Hais kom kuaj:

kubectl get pods -n keda

Tos rau KEDA Operator pib thiab mus rau Running State. Thiab tom qab ntawd, txuas ntxiv mus.

Txhim kho Redis siv Helm

Yog tias koj tsis muaj Helm ntsia, siv qhov no kev coj noj coj ua. Hais kom nruab rau Mac:

brew install kubernetes-helm
helm init --history-max 200

helm init initializes lub zos hais kom ua kab interface thiab kuj installs Tiller mus rau Kubernetes pawg.

kubectl get pods -n kube-system | grep tiller

Tos kom Tiller pod nkag mus rau lub xeev khiav.

Tus neeg txhais lus sau tseg: Tus kws sau ntawv siv Helm@2, uas yuav tsum tau nruab Tiller server tivthaiv. Tam sim no Helm@3 yog qhov tseem ceeb, nws tsis tas yuav muaj ib feem ntawm cov neeg rau zaub mov.

Tom qab txhim kho Helm, ib qho lus txib txaus los pib Redis:

helm install --name redis-server --set cluster.enabled=false --set 
usePassword=false stable/redis

Xyuas kom tseeb tias Redis tau pib ua tiav:

kubectl get pods/redis-server-master-0

Tos Redis mus rau hauv lub xeev Running.

Daim ntawv thov Deployment

Deployment command:

kubectl apply -f go-app.yaml

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

Xyuas tias txhua yam tau pib:

kubectl get pods -l=app=go-prom-app

Tos Redis nkag mus rau hauv xeev Running.

Deploying Prometheus Server

Prometheus manifest siv Kubernetes Service Discovery rau Prometheus. Nws tso cai rau dynamic nrhiav tau ntawm daim ntawv thov pods raws li cov kev pab cuam daim ntawv lo.

kubernetes_sd_configs:
   - role: service
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_label_run]
     regex: go-prom-app-service
     action: keep

Yuav siv tau:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

Xyuas tias txhua yam tau pib:

kubectl get pods -l=app=prometheus-server

Tos kom Prometheus mus rau hauv lub xeev Running.

Siv kubectl port-forward nkag mus rau Prometheus tus neeg siv interface (lossis API server) ntawm http://localhost:9090.

kubectl port-forward service/prometheus-service 9090

Deploying KEDA Autoscaling Configuration

Hais kom tsim ScaledObject:

kubectl apply -f keda-prometheus-scaledobject.yaml

Txheeb xyuas KEDA tus neeg teb xov tooj cav:

KEDA_POD_NAME=$(kubectl get pods -n keda 
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

Cov txiaj ntsig zoo li no:

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with 
namespace default and name keda-hpa-go-prom-app"

Xyuas hauv qab daim ntawv thov. Ib qho piv txwv yuav tsum tau khiav vim minReplicaCount sib npaug 1:

kubectl get pods -l=app=go-prom-app

Xyuas kom tseeb tias HPA cov peev txheej tau tsim tiav:

kubectl get hpa

Koj yuav tsum pom qee yam xws li:

NAME                   REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   0/3 (avg)   1         10        1          45s

Kev kuaj mob: daim ntawv thov nkag

Txhawm rau nkag mus rau peb daim ntawv thov REST kawg, khiav:

kubectl port-forward service/go-prom-app-service 8080

Tam sim no koj tuaj yeem nkag mus rau koj lub Go app siv qhov chaw nyob http://localhost:8080. Txhawm rau ua qhov no, khiav cov lus txib:

curl http://localhost:8080/test

Cov txiaj ntsig zoo li no:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC 
m=+406004.817901246
Access count 1

Nyob rau ntawm no tseem kos Redis. Koj yuav pom tus yuam sij ntawd access_count nce mus rau 1:

kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"

Xyuas kom tseeb tias tus nqi metric yog http_requests tib yam:

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Load Creation

Peb yuav siv hav - Kev siv hluav taws xob tsim hluav taws xob:

curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64 
&& chmod a+x hey

Koj tseem tuaj yeem rub tawm cov nqi hluav taws xob rau Linux los yog lub qhov rais.

Khiav nws:

./hey http://localhost:8080/test

Los ntawm lub neej ntawd, qhov hluav taws xob xa 200 thov. Koj tuaj yeem txheeb xyuas qhov no siv Prometheus metrics thiab Redis.

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201

Txheeb xyuas tus nqi ntawm qhov ntsuas qhov tseeb (rov qab los ntawm PromQL cov lus nug):

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

Hauv qhov no qhov tshwm sim tiag tiag yog 1,686057971014493 thiab tso tawm kom pom hauv daim teb value. Qhov no tsis txaus rau kev ntsuas, txij li qhov pib peb teev yog 3.

Ntau load!

Hauv lub davhlau ya nyob twg tshiab, saib xyuas tus naj npawb ntawm daim ntawv thov pods:

kubectl get pods -l=app=go-prom-app -w

Cia peb nce lub load siv cov lus txib: 

./hey -n 2000 http://localhost:8080/test

Tom qab ib pliag, koj yuav pom HPA scaling the deployment and launching new pods. Xyuas koj HPA kom paub tseeb tias:

kubectl get hpa
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   1830m/3 (avg)   1         10        6          4m22s

Yog tias qhov kev thauj khoom tsis sib haum, qhov kev xa tawm yuav raug txo mus rau qhov chaw uas tsuas yog ib lub pod tab tom khiav. Yog tias koj xav tshawb xyuas qhov ntsuas qhov tseeb (rov qab los ntawm PromQL cov lus nug), ces siv cov lus txib:

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'

Tu

//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server

xaus

KEDA tso cai rau koj tuaj yeem ntsuas koj lub Kubernetes kev xa tawm (rau / los ntawm xoom) raws li cov ntaub ntawv los ntawm kev ntsuas sab nraud. Piv txwv li, raws li Prometheus metrics, queue ntev nyob rau hauv Redis, neeg siv latency nyob rau hauv Kafka lub ncauj lus.

KEDA koom nrog ib qho chaw sab nraud thiab tseem muab nws cov kev ntsuas los ntawm Metrics Server rau Kab rov tav Pod Autoscaler.

Hmoov zoo!

Yuav nyeem dab tsi ntxiv:

  1. Cov kev coj ua zoo tshaj plaws thiab cov kev coj ua zoo tshaj plaws rau kev khiav ntim thiab Kubernetes hauv kev tsim khoom ib puag ncig.
  2. 90+ Cov cuab yeej siv tau zoo rau Kubernetes: Kev xa tawm, kev tswj hwm, saib xyuas, kev ruaj ntseg thiab ntau dua.
  3. Peb channel ncig Kubernetes hauv Telegram.

Tau qhov twg los: www.hab.com

Ntxiv ib saib