Pulogalamu ya ProHoster > Blog > Ulamuliro > Autoscaling Kubernetes ntchito pogwiritsa ntchito Prometheus ndi KEDA

Autoscaling Kubernetes ntchito pogwiritsa ntchito Prometheus ndi KEDA

Autoscaling Kubernetes ntchito pogwiritsa ntchito Prometheus ndi KEDABalloon Man by Cimuanos

Scalability ndichinthu chofunikira kwambiri pamapulogalamu amtambo. Ndi Kubernetes, kukulitsa pulogalamu ndikosavuta monga kuwonjezera kuchuluka kwa zofananira zomwe zikuyenera kutumizidwa kapena ReplicaSet - koma ndi ndondomeko yamanja.

Kubernetes imalola kuti mapulogalamu azingodziwongoleredwa okha (ie Pods mu kutumiza kapena ReplicaSet) m'njira yolengeza pogwiritsa ntchito mafotokozedwe a Horizontal Pod Autoscaler. Mulingo wokhazikika pakukulitsa kokha ndi ma metric kagwiritsidwe ntchito a CPU (ma metrics azinthu), koma mutha kuphatikiza ma metric omwe amaperekedwa kunja.

timu Kubernetes aaS kuchokera ku Mail.ru adamasulira nkhani yamomwe mungagwiritsire ntchito ma metric akunja kuti muwonjezere pulogalamu ya Kubernetes. Kuti awonetse momwe zonse zimagwirira ntchito, wolemba amagwiritsa ntchito ma metrics ofunsira HTTP, omwe amasonkhanitsidwa pogwiritsa ntchito Prometheus.

M'malo mongoyang'ana ma pod, Kubernetes Event Driven Autoscaling (KEDA) imagwiritsidwa ntchito, wotsegulira gwero la Kubernetes. Imaphatikizana mwachilengedwe ndi Horizontal Pod Autoscaler kuti ipereke mawonekedwe osasunthika (kuphatikiza mpaka/kuchokera ku ziro) pazantchito zoyendetsedwa ndi zochitika. Kodi ikupezeka pa GitHub.

Chidule cha dongosolo

Autoscaling Kubernetes ntchito pogwiritsa ntchito Prometheus ndi KEDA

Chithunzichi chikuwonetsa kufotokozera mwachidule momwe zonse zimagwirira ntchito:

  1. Pulogalamuyi imapereka ma metric owerengera a HTTP mumtundu wa Prometheus.
  2. Prometheus idakonzedwa kuti itenge ma metrics awa.
  3. Prometheus scaler mu KEDA imakonzedwa kuti izingowonjezera pulogalamuyo potengera kuchuluka kwa kugunda kwa HTTP.

Tsopano ndikuwuzani mwatsatanetsatane za chinthu chilichonse.

KEDA ndi Prometheus

Prometheus ndi njira yotseguka yowunikira ndi kuchenjeza zida, gawo Cloud Native Computing Foundation. Imasonkhanitsa ma metrics kuchokera kuzinthu zosiyanasiyana ndikusunga ngati data yanthawi. Kuti muwone zithunzi zomwe mungagwiritse ntchito grafana kapena zida zina zowonera zomwe zimagwira ntchito ndi Kubernetes API.

KEDA imathandizira lingaliro la scaler - imakhala ngati mlatho pakati pa KEDA ndi dongosolo lakunja. Kukhazikitsa kwa scaler kumakhala kwachindunji pa dongosolo lililonse lachindunji ndikuchotsamo deta. KEDA ndiye amawagwiritsa ntchito kuwongolera makulitsidwe.

Ma Scalers amathandizira magwero angapo a data, mwachitsanzo, Kafka, Redis, Prometheus. Ndiye kuti, KEDA itha kugwiritsidwa ntchito kukulitsa kutumizidwa kwa Kubernetes pogwiritsa ntchito ma metric a Prometheus ngati njira.

Kuyesa ntchito

Ntchito yoyeserera ya Golang imapereka mwayi wofikira kudzera pa HTTP ndipo imagwira ntchito ziwiri zofunika:

  1. Amagwiritsa ntchito laibulale ya kasitomala ya Prometheus Go kuti agwiritse ntchito ndikupereka http_requests metric, yomwe ili ndi kugunda. Mapeto pomwe ma metric a Prometheus amapezeka ali ku URI /metrics. 
    var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests",
       Help: "number of http requests",
   })
  2. Poyankha pempho GET kugwiritsa ntchito kumawonjezera mtengo wa kiyi (access_count) ku Redis. Iyi ndi njira yosavuta yochitira ntchitoyo ngati gawo la wothandizira HTTP komanso onaninso ma metric a Prometheus. Mtengo wa metric uyenera kukhala wofanana ndi mtengowo access_count mu Redis. 
    func main() {
       http.Handle("/metrics", promhttp.Handler())
       http.HandleFunc("/test", func(w http.ResponseWriter, r 
*http.Request) {
           defer httpRequestsCounter.Inc()
           count, err := client.Incr(redisCounterName).Result()
           if err != nil {
               fmt.Println("Unable to increment redis counter", err)
               os.Exit(1)
           }
           resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count))
           w.Write([]byte(resp))
       })
       http.ListenAndServe(":8080", nil)
   }

Ntchitoyi imatumizidwa ku Kubernetes kudzera Deployment. Ntchito imapangidwanso ClusterIP, imalola seva ya Prometheus kupeza ma metrics ogwiritsira ntchito.

pano chiwonetsero cha kutumizidwa kwa pulogalamuyo.

Seva ya Prometheus

Chiwonetsero cha kutumizidwa kwa Prometheus chili ndi:

  • ConfigMap - kusamutsa Prometheus config;
  • Deployment - potumiza Prometheus mgulu la Kubernetes;
  • ClusterIP - ntchito yopezera UI Prometheus;
  • ClusterRole, ClusterRoleBinding ΠΈ ServiceAccount - podzizindikiritsa okha ntchito ku Kubernetes (Auto-discovery).

pano chiwonetsero chakuchita Prometheus.

KEDA Prometheus ScaledObject

Scaler imagwira ntchito ngati mlatho pakati pa KEDA ndi dongosolo lakunja lomwe ma metric amafunikira. ScaledObject ndi chida chachizolowezi chomwe chiyenera kutumizidwa kuti chigwirizanitse kutumizidwa ndi gwero la chochitika, pankhaniyi Prometheus.

ScaledObject ili ndi zambiri zochulukirachulukira, metadata yoyambira zochitika (monga zinsinsi zamalumikizidwe, dzina la pamzere), nthawi yoponya mavoti, nthawi yochira, ndi data ina. Zimabweretsa gwero lofananira la autoscaling (kutanthauzira kwa HPA) kuti muwonjezere kutumizidwa.

Pamene chinthu ScaledObject imachotsedwa, tanthauzo lofananira la HPA limachotsedwa.

Apa pali tanthauzo ScaledObject kwa chitsanzo chathu, amagwiritsa ntchito scaler Prometheus:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
 name: prometheus-scaledobject
 namespace: default
 labels:
   deploymentName: go-prom-app
spec:
 scaleTargetRef:
   deploymentName: go-prom-app
 pollingInterval: 15
 cooldownPeriod:  30
 minReplicaCount: 1
 maxReplicaCount: 10
 triggers:
 - type: prometheus
   metadata:
     serverAddress: 
http://prometheus-service.default.svc.cluster.local:9090
     metricName: access_frequency
     threshold: '3'
     query: sum(rate(http_requests[2m]))

Ganizirani mfundo zotsatirazi:

  1. Amaloza ku Deployment Ndi dzina go-prom-app.
  2. Mtundu woyambitsa - Prometheus. Adilesi ya seva ya Prometheus imatchulidwa pamodzi ndi dzina la metric, poyambira ndi Funso la PromQL, zomwe zidzagwiritsidwe ntchito. Funso la PromQL - sum(rate(http_requests[2m])).
  3. Malingana ndi pollingInterval,KEDA imapempha chandamale kuchokera kwa Prometheus masekondi khumi ndi asanu aliwonse. Mmodzi pansi (minReplicaCount), ndipo kuchuluka kwa ma pod sikudutsa maxReplicaCount (mu chitsanzo ichi - khumi).

Ikhoza kukhazikitsidwa minReplicaCount zofanana ndi ziro. Pamenepa, KEDA imayendetsa kutumizidwa kwa ziro-to-mmodzi ndikuwulula HPA kuti ichulukitsenso. Dongosolo lakumbuyo ndilothekanso, ndiye kuti, kukweza kuchokera pa imodzi kupita ku ziro. Mu chitsanzo, sitinasankhe zero chifukwa iyi ndi ntchito ya HTTP osati yofunidwa.

Matsenga mkati mwa autoscaling

Chidacho chimagwiritsidwa ntchito ngati chiwongolero kuti chiwonjezeke. Mu chitsanzo chathu, funso la PromQL sum(rate (http_requests [2m])) imabweretsanso kuchuluka kwa pempho la HTTP (zopempha pamphindikati), zoyezedwa pamphindi ziwiri zapitazi.

Popeza mtengo wapakati ndi atatu, zikutanthauza kuti padzakhala imodzi pansi pomwe mtengowo sum(rate (http_requests [2m])) osakwana atatu. Ngati mtengo ukuwonjezeka, gawo lowonjezera limawonjezeredwa nthawi iliyonse sum(rate (http_requests [2m])) kuchuluka katatu. Mwachitsanzo, ngati mtengo umachokera ku 12 mpaka 14, ndiye kuti chiwerengero cha pods ndi zinayi.

Tsopano tiyeni tiyese kuyikhazikitsa!

kukhazikitsa

Zomwe mukufunikira ndi gulu la Kubernetes ndi chida chokhazikitsidwa kubectl. Chitsanzo ichi chimagwiritsa ntchito masango minikube, koma mukhoza kutenga wina aliyense. Kukhazikitsa cluster pali kalozera.

Kukhazikitsa Baibulo atsopano pa Mac:

curl -Lo minikube 
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Khazikitsani kubctlkuti mupeze gulu la Kubernetes.

Kukhazikitsa Baibulo atsopano pa Mac:

curl -LO 
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

Kusintha kwa KEDA

Mutha kutumiza KEDA m'njira zingapo, zomwe zalembedwamo zolemba. Ndikugwiritsa ntchito monolithic YAML:

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml

KEDA ndi zigawo zake zimayikidwa mu namespace keda. Lamula kuti muwone:

kubectl get pods -n keda

Yembekezerani kuti KEDA Operator ayambe ndikupita Running State. Ndipo zitatha izi, pitirizani.

Kuyika Redis pogwiritsa ntchito Helm

Ngati mulibe Helm yoyika, gwiritsani ntchito izi utsogoleri. Lamulo kukhazikitsa pa Mac:

brew install kubernetes-helm
helm init --history-max 200

helm init imayambitsa mawonekedwe a mzere wa malamulo amderalo ndikuyikanso Tiller ku gulu la Kubernetes.

kubectl get pods -n kube-system | grep tiller

Yembekezerani kuti Tiller pod alowe mu Running state.

Ndemanga za womasulira: Wolemba amagwiritsa ntchito Helm@2, zomwe zimafuna kuti gawo la seva la Tiller likhazikitsidwe. Tsopano Helm @ 3 ndiyofunikira, sifunika gawo la seva.

Mukayika Helm, lamulo limodzi ndilokwanira kuyambitsa Redis:

helm install --name redis-server --set cluster.enabled=false --set 
usePassword=false stable/redis

Onetsetsani kuti Redis yayamba bwino:

kubectl get pods/redis-server-master-0

Yembekezerani kuti Redis alowe m'malo Running.

Kutumiza Ntchito

Lamulo lotumiza:

kubectl apply -f go-app.yaml

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

Onetsetsani kuti zonse zayamba:

kubectl get pods -l=app=go-prom-app

Yembekezerani Redis kulowa boma Running.

Kutumiza Seva ya Prometheus

Mawonekedwe a Prometheus amagwiritsa ntchito Kubernetes Service Discovery ya Prometheus. Zimalola kupezeka kwamphamvu kwa ma pods ogwiritsira ntchito kutengera chizindikiro chautumiki.

kubernetes_sd_configs:
   - role: service
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_label_run]
     regex: go-prom-app-service
     action: keep

Kutumiza:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

Onetsetsani kuti zonse zayamba:

kubectl get pods -l=app=prometheus-server

Dikirani kuti Prometheus apite ku boma Running.

Gwiritsani ntchito kubectl port-forward kuti mupeze mawonekedwe a Prometheus (kapena seva ya API) pa http://localhost:9090.

kubectl port-forward service/prometheus-service 9090

Kutumiza KEDA Autoscaling Configuration

Lamula kupanga ScaledObject:

kubectl apply -f keda-prometheus-scaledobject.yaml

Onani zolemba za KEDA:

KEDA_POD_NAME=$(kubectl get pods -n keda 
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

Zotsatira zake zikuwoneka motere:

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with 
namespace default and name keda-hpa-go-prom-app"

Onani pansi pa mapulogalamu. Chitsanzo chimodzi chiyenera kukhala chikuyenda chifukwa minReplicaCount zikufanana ndi 1:

kubectl get pods -l=app=go-prom-app

Tsimikizirani kuti zida za HPA zidapangidwa bwino:

kubectl get hpa

Muyenera kuwona chinthu chonga:

NAME                   REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   0/3 (avg)   1         10        1          45s

Kuwona zaumoyo: mwayi wogwiritsa ntchito

Kuti mupeze mapeto a pulogalamu yathu ya REST, thamangani:

kubectl port-forward service/go-prom-app-service 8080

Tsopano mutha kupeza pulogalamu yanu ya Go pogwiritsa ntchito adilesi http://localhost:8080. Kuti muchite izi, yendetsani lamulo:

curl http://localhost:8080/test

Zotsatira zake zikuwoneka motere:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC 
m=+406004.817901246
Access count 1

Pakadali pano onaninso Redis. Mudzawona kuti fungulo access_count kuchuluka kwa 1:

kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"

Onetsetsani kuti mtengo wa metric ndi http_requests momwemonso:

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Katundu Kulenga

Tidzagwiritsa ntchito Hei - Zothandizira kupanga katundu:

curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64 
&& chmod a+x hey

Mukhozanso kukopera zofunikira kwa Linux kapena Windows.

Yendetsani:

./hey http://localhost:8080/test

Mwachikhazikitso, pulogalamuyi imatumiza zopempha 200. Mutha kutsimikizira izi pogwiritsa ntchito ma metric a Prometheus komanso Redis.

curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201

Tsimikizirani mtengo wa metric weniweni (yobwezeredwa ndi funso la PromQL):

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

Pankhaniyi zotsatira zenizeni ndi 1,686057971014493 ndipo ikuwonetsedwa m'nkhaniyi value. Izi sizokwanira kukulitsa, chifukwa malire omwe timayika ndi 3.

More katundu!

Mu terminal yatsopano, yang'anani kuchuluka kwa ma pods ogwiritsira ntchito:

kubectl get pods -l=app=go-prom-app -w

Tiyeni tiwonjezere katunduyo pogwiritsa ntchito lamulo: 

./hey -n 2000 http://localhost:8080/test

Patapita kanthawi, mudzawona HPA ikukulitsa kutumizidwa ndikuyambitsa ma pod atsopano. Yang'anani HPA yanu kuti muwonetsetse:

kubectl get hpa
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-go-prom-app   Deployment/go-prom-app   1830m/3 (avg)   1         10        6          4m22s

Ngati katunduyo ndi wosagwirizana, kutumizidwako kudzachepetsedwa mpaka pomwe pod imodzi yokha ikugwira ntchito. Ngati mukufuna kuwona metric yeniyeni (yobwezeredwa ndi funso la PromQL), ndiye gwiritsani ntchito lamulo:

curl -g 
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'

Kuyeretsa

//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server

Pomaliza

KEDA imakulolani kuti muzitha kukulitsa ma Kubernetes (kuyambira/kuchokera ku ziro) kutengera ma metric akunja. Mwachitsanzo, kutengera ma metric a Prometheus, kutalika kwa mzere ku Redis, kuchedwa kwa ogula pamutu wa Kafka.

KEDA imaphatikizana ndi gwero lakunja komanso imaperekanso ma metrics ake kudzera pa Metrics Server kupita ku Horizontal Pod Autoscaler.

Kupambana!

Zomwe mungawerenge:

  1. Njira zabwino kwambiri zoyendetsera zotengera ndi Kubernetes m'malo opanga.
  2. Zida 90+ zothandiza za Kubernetes: kutumiza, kasamalidwe, kuwunika, chitetezo ndi zina zambiri.
  3. Njira yathu Yozungulira Kubernetes mu Telegraph.

Source: www.habr.com

Kuwonjezera ndemanga