ืืืจืืืืช ืืื ืืจืืฉืช ืืคืชื ืขืืืจ ืืืฉืืื ืขื ื. ืขื Kubernetes, ืฉืื ืื ืงื ื ืืืื ืฉื ืืืฉืื ืืื ืคืฉืื ืืื ืืืืืช ืืกืคืจ ืืขืชืงืื ืืคืจืืกื ืืืชืืืื ืื ReplicaSet
- ืืื ืื ืชืืืื ืืื ื.
Kubernetes ืืืคืฉืจ ืฉืื ืื ืงื ื ืืืื ืืืืืืื ืฉื ืืืฉืืืื (ืืืืืจ Pods ืืคืจืืกื ืื ReplicaSet
) ืืฆืืจื ืืฆืืจืชืืช ืชืื ืฉืืืืฉ ืืืคืจื Horizontal Pod Autoscaler. ืงืจืืืจืืื ืืจืืจืช ืืืืื ืขืืืจ ืงื ื ืืืื ืืืืืืื ืืื ืืืื ืฉืืืืฉ ืืืขืื (ืืืื ืืฉืืืื), ืื ื ืืชื ืืฉืื ืืืืื ืืืชืืืื ืืืฉืืช ืืืกืืคืงืื ืืืฆืื ืืช.
ืงืืืฆื
ืืืงืื ืงื ื ืืืื ืืืืืืื ืืืคืงื ืฉื ืชืจืืืืื, ื ืขืฉื ืฉืืืืฉ ื-Kubernetes Event Driven Autoscaling (KEDA), ืืคืขืื ืงืื ืคืชืื ืฉื Kubernetes. ืืื ืืฉืชืื ืืืืคื ืืงืืจื ืขื Horizontal Pod Autoscaler ืืื ืืกืคืง ืงื ื ืืืื ืืืืืืื ืืืง (ืืืื ืขื/ืืืคืก) ืืขืืืกื ืขืืืื ืืื ืขื ืืืจืืขืื. ืงืื ืืืื ื
ืกืงืืจื ืงืฆืจื ืฉื ืืืขืจืืช
ืืชืจืฉืื ืืฆืื ืชืืืืจ ืงืฆืจ ืฉื ืืื ืืื ืขืืื:
- ืืืคืืืงืฆืื ืืกืคืงืช ืืืื ืกืคืืจืช ืื ืืกืืช HTTP ืืคืืจืื Prometheus.
- ืคืจืืืชืืืก ืืืืืจ ืืืกืืฃ ืืืืื ืืื.
- ื-Prometheus scaler ื-KEDA ืืืืืจ ืืฉืื ืื ืงื ื ืืืื ืืืืืืื ืฉื ืืืคืืืงืฆืื ืืืชืืกืก ืขื ืืกืคืจ ืื ืืกืืช HTTP.
ืขืืฉืื ืื ื ืืกืคืจ ืื ืืคืืจืื ืขื ืื ืืืื ื.
KEDA ืืคืจืืืชืืืก
Prometheus ืืื ืขืจืืช ืืืื ืื ืืืืจ ืืืชืจืื ืฉื ืืขืจืืช ืงืื ืคืชืื, ืืืง
KEDA ืชืืื ืืจืขืืื ืฉื scaler - ืืื ืืฉืืฉ ืืืฉืจ ืืื KEDA ืืืขืจืืช ืืืืฆืื ืืช. ืืืืขืช scaler ืกืคืฆืืคืืช ืืื ืืขืจืืช ืืขื ืืืืืฆืช ืืื ื ื ืชืื ืื. ืืืืจ ืืื KEDA ืืฉืชืืฉ ืืื ืืื ืืฉืืื ืืงื ื ืืืื ืืืืืืื.
ืกืงืืืจืื ืชืืืืื ืืืงืืจืืช ื ืชืื ืื ืืจืืืื, ืืืฉื, Kafka, Redis, Prometheus. ืืืืืจ, ื ืืชื ืืืฉืชืืฉ ื-KEDA ืืฉืื ืื ืงื ื ืืืื ืืืืืืื ืฉื ืคืจืืกืืช Kubernetes ืืืืฆืขืืช ืืืื Prometheus ืืงืจืืืจืืื ืื.
ืืืฉืื ืืืืงื
ืืคืืืงืฆืืืช ืืืืืงื ืฉื Golang ืืกืคืงืช ืืืฉื ืืืืฆืขืืช HTTP ืืืืฆืขืช ืฉืชื ืคืื ืงืฆืืืช ืืฉืืืืช:
- ืืฉืชืืฉ ืืกืคืจืืืช ืืืงืืืืช ืฉื Prometheus Go ืืื ืืืืฉืืจ ืืช ืืืคืืืงืฆืื ืืืกืคืง ืืช ืืืื http_requests, ืืืืื ืกืคืืจืช ืืชืืืืช. ื ืงืืืช ืืงืฆื ืฉืื ืืืื ืื ืืืื ืคืจืืืชืืืก ืืืืงืืช ื-URI
/metrics
.var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{ Name: "http_requests", Help: "number of http requests", })
- ืืชืืืื ืืืงืฉื
GET
ืืืืฉืื ืืืืื ืืช ืืขืจื ืฉื ืืืคืชื (access_count
) ืืจืืืก. ืืืื ืืจื ืงืื ืืืฆืข ืืช ืืขืืืื ืืืืง ืืืืคื HTTP ืืื ืืืืืง ืืืื Prometheus. ืืขืจื ืืืืื ืืืื ืืืืืช ืืื ืืขืจืaccess_count
ืืจืืืก.func main() { http.Handle("/metrics", promhttp.Handler()) http.HandleFunc("/test", func(w http.ResponseWriter, r *http.Request) { defer httpRequestsCounter.Inc() count, err := client.Incr(redisCounterName).Result() if err != nil { fmt.Println("Unable to increment redis counter", err) os.Exit(1) } resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count)) w.Write([]byte(resp)) }) http.ListenAndServe(":8080", nil) }
ืืืคืืืงืฆืื ื ืคืจืกืช ื-Kubernetes ืืืืฆืขืืช Deployment
. ื ืืฆืจ ืื ืฉืืจืืช ClusterIP
, ืืื ืืืคืฉืจ ืืฉืจืช Prometheus ืืืฉืื ืืืื ืืืฉืืืื.
ืืื
ืฉืจืช ืคืจืืืชืืืก
ืื ืืคืกื ืืคืจืืกื ืฉื ืคืจืืืชืืืก ืืืจืื ื:
ConfigMap
- ืืืขืืืจ ืืช ืชืฆืืจืช ืคืจืืืชืืืก;Deployment
- ืืคืจืืกืช Prometheus ืืืฉืืื Kubernetes;ClusterIP
- ืฉืืจืืช ืืืืฉื ืืืืฉืง ืืืฉืชืืฉ Prometheus;ClusterRole
,ClusterRoleBinding
ะธServiceAccount
- ืืืืืื ืืืืืืื ืฉื ืฉืืจืืชืื ื-Kubernetes (ืืืืื ืืืืืืื).
ืืื
KEDA Prometheus ScaledObject
ื-scaler ืืฉืืฉ ืืืฉืจ ืืื KEDA ืืืขืจืืช ืืืืฆืื ืืช ืฉืืื ื ืฆืจืื ืืงืื ืืืืื. ScaledObject
ืืื ืืฉืื ืืืชืื ืืืฉืืช ืฉืฆืจืื ืืคืจืืก ืืื ืืกื ืืจื ืืช ืืคืจืืกื ืขื ืืงืืจ ืืืืจืืข, ืืืงืจื ืื Prometheus.
ScaledObject
ืืืื ืืืืข ืขื ืงื ื ืืืื ืฉื ืคืจืืกื, ืืื ื ืชืื ืื ืฉื ืืงืืจ ืืืจืืขืื (ืืืื ืกืืืืช ืืืืืจ, ืฉื ืชืืจ), ืืจืืื ืกืงืจ, ืชืงืืคืช ืฉืืืืจ ืื ืชืื ืื ืืืจืื. ืื ืืืจื ืืืฉืื ืงื ื ืืืืื ืืืืืืืื ืืืชืืื (ืืืืจืช HPA) ืืื ืืฉื ืืช ืืช ืงื ื ืืืืื ืฉื ืืคืจืืกื.
ืืืฉืจ ืืคืฅ ScaledObject
ื ืืืง, ืืืืืจื ืืืชืืืื ืฉื HPA ื ืืืงืช.
ืื ื ืืืืืจื ScaledObject
ืขืืืจ ืืืืืื ืฉืื ื, ืืื ืืฉืชืืฉ ื-scaler Prometheus
:
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: prometheus-scaledobject
namespace: default
labels:
deploymentName: go-prom-app
spec:
scaleTargetRef:
deploymentName: go-prom-app
pollingInterval: 15
cooldownPeriod: 30
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
metadata:
serverAddress:
http://prometheus-service.default.svc.cluster.local:9090
metricName: access_frequency
threshold: '3'
query: sum(rate(http_requests[2m]))
ืฉืงืื ืืช ืื ืงืืืืช ืืืืืช:
- ืืื ืืฆืืืข ืขื
Deployment
ืขื ืฉืgo-prom-app
. - ืกืื ืืจืืืจ -
Prometheus
. ืืชืืืช ืฉืจืช Prometheus ืืืืืจืช ืืื ืขื ืืฉื ืืืืจื, ืืกืฃ ืืฉืืืืชืช PromQL , ืืฉืจ ืืฉืืฉ. ืฉืืืืชืช PromQL -sum(rate(http_requests[2m]))
. - ืขื ืคื
pollingInterval
,KEDA ืืืงืฉ ืืืจื ืืคืจืืืชืืืก ืื ืืืฉ ืขืฉืจื ืฉื ืืืช. ืืคืืืช ืืื ืืชืืช (minReplicaCount
), ืืืกืคืจ ืืชืจืืืืื ืืืจืื ืืื ื ืืืจืmaxReplicaCount
(ืืืืืื ืื - ืขืฉืจ).
ื ืืชื ืืืชืงืื minReplicaCount
ืฉืืื ืืืคืก. ืืืงืจื ืื, KEDA ืืคืขืื ืืช ืืคืจืืกื ืฉื ืืคืก ืืืื ืืืืืจ ืืื ืืืฉืฃ ืืช ื-HPA ืืฉืื ืื ืงื ื ืืืื ืืืืืืื ื ืืกืฃ. ืืกืืจ ืืืคืื ืืคืฉืจื ืื ืืื, ืืืืืจ ืฉืื ืื ืงื ื ืืืื ืืืื ืืืคืก. ืืืืืื ืื ืืืจื ื ืืคืก ืื ืืืืืจ ืืฉืืจืืช HTTP ืืื ืืืขืจืืช ืืคื ืืจืืฉื.
ืืงืกื ืฉืืชืื ืงื ื ืืืืื ืืืืืืืื
ืืกืฃ ืืฉืืฉ ืืืจืืืจ ืืืืืืช ืืคืจืืกื. ืืืืืื ืฉืื ื, ืฉืืืืชืช PromQL sum(rate (http_requests [2m]))
ืืืืืจื ืืช ืฉืืขืืจ ืืงืฉืช ื-HTTP ืืืฆืืืจ (ืืงืฉืืช ืืฉื ืืื), ืฉื ืืื ืืฉืชื ืืืงืืช ืืืืจืื ืืช.
ืืืืืื ืฉืขืจื ืืกืฃ ืืื ืฉืืืฉ, ืื ืืืืจ ืฉืืืื ืืื ืืชืืช ืืขืื ืืขืจื sum(rate (http_requests [2m]))
ืคืืืช ืืฉืืืฉื. ืื ืืขืจื ืืื, ืชืชืืืกืฃ ืืฉื ื ื ืืกืฃ ืืื ืคืขื sum(rate (http_requests [2m]))
ืขืืื ืืฉืืืฉ. ืืืืืื, ืื ืืขืจื ืืื ืืื 12 ื-14, ืื ืืกืคืจ ืืชืจืืืืื ืืื ืืจืืขื.
ืขืืฉืื ืืืื ื ื ืกื ืืืืืืจ ืืช ืื!
ืืืืจื ืืจืืฉ
ืื ืื ืฉืืชื ืฆืจืื ืื ืืฉืืื Kubernetes ืืืื ืฉืืจืืช ืืืืืจ kubectl
. ืืืืื ืื ืืฉืชืืฉืช ืืืฉืืื minikube
, ืืื ืืชื ืืืื ืืงืืช ืื ืืื ืืืจ. ืืื ืืืชืงืื ืืฉืืื ืืฉ
ืืชืงื ืืช ืืืจืกื ืืืืจืื ื ื-Mac:
curl -Lo minikube
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/
ืืืชืงืื
ืืชืงื ืืช ืืืจืกื ืืืืจืื ื ื-Mac:
curl -LO
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version
ืืชืงื ืช KEDA
ืืชื ืืืื ืืคืจืืก ืืช KEDA ืืืื ืืจืืื, ืื ืจืฉืืืื ื
kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml
KEDA ืืจืืืืื ืืืชืงื ืื ืืืจืื ืืฉืืืช keda
. ืคืงืืื ืืืืืงื:
kubectl get pods -n keda
ืืืชื ื-KEDA Operator ืฉืืชืืื ืืืขืืืจ ืื Running State
. ืืืืจื ืื ืชืืฉืื.
ืืชืงื ืช Redis ืืืืฆืขืืช Helm
ืื ืื ืืืชืงื ืืฆืื Helm, ืืฉืชืืฉ ืืื
brew install kubernetes-helm
helm init --history-max 200
helm init
ืืืชืื ืืช ืืืฉืง ืฉืืจืช ืืคืงืืื ืืืงืืืืช ืืื ืืชืงืื Tiller
ืืืฉืืื Kubernetes.
kubectl get pods -n kube-system | grep tiller
ืืืชื ืืชืจืืื Tiller ืืืืื ืก ืืืฆื Running.
ืืขืจืช ืืืชืจืื: ืืืืืจ ืืฉืชืืฉ ื- Helm@2, ืืืืืื ืืช ืืชืงื ืช ืจืืื ืฉืจืช Tiller. ืขืืฉืื Helm@3 ืจืืืื ืื, ืื ืื ืืืจืฉ ืืืง ืฉืจืช.
ืืืืจ ืืชืงื ืช Helm, ืืกืคืืงื ืคืงืืื ืืืช ืืื ืืืคืขืื ืืช Redis:
helm install --name redis-server --set cluster.enabled=false --set
usePassword=false stable/redis
ืืื ืฉ- Redis ืืชืืืื ืืืฆืืื:
kubectl get pods/redis-server-master-0
ืืื ืขื ืฉืจืืืก ืืื ืก ืืืืื ื Running
.
ืคืจืืกืช ืืืฉืืืื
ืคืงืืืช ืคืจืืกื:
kubectl apply -f go-app.yaml
//output
deployment.apps/go-prom-app created
service/go-prom-app-service created
ืืืืง ืฉืืื ืืชืืื:
kubectl get pods -l=app=go-prom-app
ืืืชื ืขื ืฉ-Redis ืืื ืก ืืืฆื Running
.
ืคืจืืกืช ืฉืจืช ืคืจืืืชืืืก
ืื ืืคืกื ืคืจืืืชืืืก ืืฉืชืืฉ
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_run]
regex: go-prom-app-service
action: keep
ืืื ืืคืจืืก:
kubectl apply -f prometheus.yaml
//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created
ืืืืง ืฉืืื ืืชืืื:
kubectl get pods -l=app=prometheus-server
ืืื ืฉืคืจืืืชืืืก ืืืื ืก ืืืืื ื Running
.
ืืฉืชืืฉ kubectl port-forward
ืืื ืืืฉืช ืืืืฉืง ืืืฉืชืืฉ ืฉื Prometheus (ืื ืฉืจืช API) ืืืชืืืช
kubectl port-forward service/prometheus-service 9090
ืคืจืืกืช ืชืฆืืจืช ืงื ื ืืืื ืืืืืืื ืฉื KEDA
ืคืงืืื ืืืฆืืจื ScaledObject
:
kubectl apply -f keda-prometheus-scaledobject.yaml
ืืืืง ืืช ืืืื ื ืืืคืขืื ืฉื KEDA:
KEDA_POD_NAME=$(kubectl get pods -n keda
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda
ืืชืืฆืื ื ืจืืืช ืืขืจื ืื:
time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with
namespace default and name keda-hpa-go-prom-app"
ืืืืง ืชืืช ืืืฉืืืื. ืืืคืข ืืื ืืืื ืืคืขืื ืื minReplicaCount
ืฉืืื 1:
kubectl get pods -l=app=go-prom-app
ืืื ืื ืืฉืื HPA ื ืืฆืจ ืืืฆืืื:
kubectl get hpa
ืืชื ืืืืจ ืืจืืืช ืืฉืื ืืื:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
keda-hpa-go-prom-app Deployment/go-prom-app 0/3 (avg) 1 10 1 45s
ืืืืงืช ืชืงืื ืืช: ืืืฉื ืืืคืืืงืฆืื
ืืื ืืืฉืช ืื ืงืืืช ืืงืฆื REST ืฉื ืืืคืืืงืฆืื ืฉืื ื, ืืคืขื:
kubectl port-forward service/go-prom-app-service 8080
ืืขืช ืชืืื ืืืฉืช ืืืคืืืงืฆืืืช Go ืฉืื ืืืืฆืขืืช ืืืชืืืช
curl http://localhost:8080/test
ืืชืืฆืื ื ืจืืืช ืืขืจื ืื:
Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC
m=+406004.817901246
Access count 1
ืืฉืื ืื ืืืืง ืื ืืช Redis. ืืชื ืชืจืื ืฉืื ืืืคืชื access_count
ืืื ื-1:
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"
ืืื ืฉืืขืจื ืืืื ืืื http_requests
ืืืชื ืืืืจ:
curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1
ืืฆืืจืช ืขืืืกืื
ืื ืื ื ื ืฉืชืืฉ
curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64
&& chmod a+x hey
ืืชื ืืืื ืื ืืืืจืื ืืช ืืื ืืฉืืจืืช ืขืืืจ
ืชืจืืฅ ืืช ืื:
./hey http://localhost:8080/test
ืืืจืืจืช ืืืื, ืืฉืืจืืช ืฉืืื 200 ืืงืฉืืช. ืืชื ืืืื ืืืืช ืืืช ืืืืฆืขืืช ืืืื Prometheus ืืื ืื Redis.
curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201
ืืืช ืืช ืืขืจื ืฉื ืืืื ืืคืืขื (ืืืืืจ ืขื ืืื ืฉืืืืชืช PromQL):
curl -g
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}
ืืืงืจื ืื ืืชืืฆืื ืืคืืขื ืืื 1,686057971014493
ืืืืฆื ืืฉืื value
. ืื ืื ืืกืคืืง ืขืืืจ ืงื ื ืืืื, ืฉืื ืืกืฃ ืฉืืืืจื ื ืืื 3.
ืขืื ืขืืืก!
ืืืจืืื ื ืืืืฉ, ืขืงืื ืืืจ ืืกืคืจ ืชืจืืืื ืืืืฉืืืื:
kubectl get pods -l=app=go-prom-app -w
ืืืื ืืืืืื ืืช ืืขืืืก ืืืืฆืขืืช ืืคืงืืื:
./hey -n 2000 http://localhost:8080/test
ืืืืจ ืืื ืื, ืชืจืื ืืช HPA ืืฉื ื ืืช ืืคืจืืกื ืืืฉืืงื ืคืืืื ืืืฉืื. ืืืืง ืืช ื-HPA ืฉืื ืืื ืืืืื:
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
keda-hpa-go-prom-app Deployment/go-prom-app 1830m/3 (avg) 1 10 6 4m22s
ืื ืืขืืืก ืืื ื ืขืงืื, ืืคืจืืกื ืชืฆืืืฆื ืขื ืื ืงืืื ืฉืื ืคืืขื ืจืง ืคืื ืืื. ืื ืืชื ืจืืฆื ืืืืืง ืืช ืืืื ืืคืืขื (ืืืืืจ ืขื ืืื ืฉืืืืชืช PromQL), ืืฉืชืืฉ ืืคืงืืื:
curl -g
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
ื ืืงืื
//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server
ืืกืงื ื
KEDA ืืืคืฉืจ ืื ืืฉื ืืช ืืช ืงื ื ืืืืื ืืืืืืืืช ืฉื ืคืจืืกืืช Kubernetes ืฉืื (ืืืคืก/ืืืคืก) ืขื ืกืื ื ืชืื ืื ืืืืื ืืืฆืื ืืื. ืืืืืื, ืืืชืืกืก ืขื ืืืื Prometheus, ืืืจื ืชืืจ ืืจืืืก, ืืื ืืืืื ืฆืจืื ื ืื ืืฉื ืงืคืงื.
KEDA ืืฉืชืื ืขื ืืงืืจ ืืืฆืื ื ืืื ืืกืคืง ืืช ืืืืืื ืฉืื ืืจื ืฉืจืช ืืืืื ื- Horizontal Pod Autoscaler.
ืืื ืืื!
ืื ืขืื ืืงืจืื:
ืฉืืืืช ืขืืืื ืืืืืฆืืช ืืฉืืืืช ืขืืืื ืืืืืฆืืช ืืืคืขืืช ืงืื ืืืื ืจืื ื-Kubernetes ืืกืืืืืช ืืืฆืืจ .90+ ืืืื ืฉืืืืฉืืื ืขืืืจ Kubernetes: ืคืจืืกื, ื ืืืื, ื ืืืืจ, ืืืืื ืืขืื .ืืขืจืืฅ ืฉืื ื Around Kubernetes ืืืืืจื .
ืืงืืจ: www.habr.com