ααΆαααααΎααΆαααααααΆαααΊααΆαααααΌαααΆαααααΆαααααααΆαααααααα·ααΈαααα ααΆαα½α Kubernetes ααΆαααααΎααΆαααααααΆααααααα·ααΈααΊααΆααααααΌα
ααΆααΆααααααΎαα
ααα½αα
αααααααααΆααααΆαααΆααα±ααααααΎααααΆαααααααα α¬ ReplicaSet
- ααα»ααααααΆααΆααααΎαααΆααααααα
Kubernetes α’αα»ααααΆαα±αααααααα·ααΈααααΌαααΆαααααΎααΆαααααααΆααααααααααααααααα· (α§ααΆα ααα Pods αα
αααα»αααΆαααΆαααααααΆα α¬ ReplicaSet
) αααα»αβααααααβαααααΆαβαααβααααΎβααΆαβαααααΆααβαααα Horizontal Pod Autoscalerα αααααααα·αα·α
ααααααααΆαααΎααααααΆααααΆαααααΎααΆαααααααΆααααααααααααααααα·ααΊααΆαααααΆααααααΆαααααΎααααΆαααααΈααΈααΌ (αααααΆααααααΆα) ααα»ααααα’αααα’αΆα
αα½ααααα
αΌααααααΆααααααΆαααααα½α αα·ααααααααΆααααα
αααααΆααααααα±ααα
αααα»αααΆαααΆα
αααα½αα±ααααΆαααααΎααΆαααααααΆααααααααααααααα·ααααααα ααΆαααααΎααααΆαααααααα·ααΈ Kubernetes Event Driven Autoscaling (KEDA) αααααΆααααα·ααααα·αα Kubernetes αααααααΎαα
αα α ααΆαα½ααααα
αΌαααΎαααααΎαααΆαα½α Horizontal Pod Autoscaler ααΎααααΈαααααααΌαααΆαααααΎααΆαααααααΆααααααααααααααααα·αααααααΆαααααα (αα½αααΆαααα
/ααΈααΌααα) αααααΆααααααα»αααΆαααΆαααααααα»ααααααααΉαααα·ααΆαααα αααααΌαααΆααα
αα·αααααΆααααααααααααααααα
ααααΆααααΆααααα αΆαααΆααα·αααααΆααααααα’αααΈαααααααα’αααΈαααααΎαααΆαα
- αααααα·ααΈααααααααααΌαααΆαααΆααα ααα½α HTTP αααα»ααααααα Prometheus α
- Prometheus ααααΌαααΆαααααααα ααΆααααααααααΎααααΈαααααΌαααααααααΆαααααα
- α§αααααααααΎααΆαααααααΆα Prometheus αα αααα»α KEDA ααααΌαααΆαααα‘αΎαααΎααααΈααααΎααΆαααααααΆααααααα·ααΈαααααααααααααααα·αααααα’ααααΎα ααα½α HTTP α
α₯α‘αΌαααααααα»αααΉαααααΆααα’αααα±αααααα’α·αα’αααΈααΆαα»ααΈαα½ααα
KEDA αα·α Prometheus
Prometheus ααΊααΆαααααα·ααΈαααα½ααα·αα·ααααααααααααααααααΎαα
αα α αα·ααααα’ααα§αααααααΌαααααΉαααΆααααα
KEDA ααΆααααααααααα·αααααΆαααααααΆα - ααΆααΎααα½ααΆααααΆααααΆα KEDA αα·αααααααααααΆααααα α ααΆαα’αα»ααααααΆαααααααΆαααΊααΆααααΆααα ααααααααααααααααα ααΈαα½αα α αΎαααΆααααα·ααααααααΈααΆα αααααΆαααα KEDA ααααΎαα½αααΆααΎααααΈαααααααααααΆαααααΎααΆαααααααΆααααααααααααααααα·α
Scalers ααΆααααααααααα·ααααααααΆα αααΎαα§ααΆα ααα Kafka, Redis, Prometheus α αααααΊ KEDA α’αΆα ααααΌαααΆαααααΎααΎααααΈααααΎααΆαααααααΆαααΆαααΆαααααααΆα Kubernetes αααααααααααααααα·αααααααΎαααααΆαα Prometheus ααΆαααααααα·αα·α ααααα
αααααα·ααΈααΆααααα
αααααα·ααΈααΆααααα Golang αααααααΌαααΆαα αΌαααααΎααΆαααα HTTP αα·αα’αα»αααααα»αααΆαααααΆααααΈαα
- ααααΎαααααΆααααααΆαααΈαααααα Prometheus Go ααΎααααΈαααααα§ααααααααααα·ααΈ αα·αααααααααααα http_requests αααααΆαα
ααα½ααα»αα α
ααα»α
αααα
αααααα§αααααααΆααααααα Prometheus ααΆαααΆαααΈααΆαααα
URI
/metrics
.var httpRequestsCounter = promauto.NewCounter(prometheus.CounterOpts{ Name: "http_requests", Help: "number of http requests", })
- ααΆααΆαααααΎααααα
ααΉαααααΎ
GET
αααααα·ααΈαααααΎαααααααααα (access_count
) αα αααα»α Redis α αααααΊααΆαα·ααΈααΆααααα½ααααα»αααΆαααααΎααΆαααΆαααΆααααααααααααα·ααΈαααααααΆα HTTP αα·ααα·αα·αααααΎααααααααααα Prometheus αααααα αααααααααααααααΌαααααΌα ααααΆαα ααΉααααααaccess_count
αα Redis αfunc main() { http.Handle("/metrics", promhttp.Handler()) http.HandleFunc("/test", func(w http.ResponseWriter, r *http.Request) { defer httpRequestsCounter.Inc() count, err := client.Incr(redisCounterName).Result() if err != nil { fmt.Println("Unable to increment redis counter", err) os.Exit(1) } resp := "Accessed on " + time.Now().String() + "nAccess count " + strconv.Itoa(int(count)) w.Write([]byte(resp)) }) http.ListenAndServe(":8080", nil) }
αααααα·ααΈααααΌαααΆαααΆαααααααΆααα
ααΆαα Kubernetes ααΆαααα Deployment
. ααααΆααααααααααΌαααΆααααααΎαα‘αΎααααααα ClusterIP
ααΆα’αα»ααααΆαα±αααααΆαααΈααα Prometheus ααα½αααΆααααααααααααα·ααΈα
αα
ααΈααα
αααΆαααΈααα Prometheus
ααΆααααα αΆαα’αααΈααΆαααΆαααααααΆα Prometheus αα½αααΆαα
ConfigMap
- ααΎααααΈαααααααΆαααααααα ααΆαααααααα Prometheus;Deployment
- αααααΆααααΆαααΆαααααααΆα Prometheus αα αααα»αα ααααα Kubernetes;ClusterIP
- ααααΆαααααααααΆααααΆαα αΌαααααΎ UI Prometheus;ClusterRole
,ClusterRoleBinding
ΠΈServiceAccount
β αααααΆααααΆαααααΎααααααααααααααααα·ααααααΆαααααα αααα»α Kubernetes (ααΆαααααΎααααααααααααααααα·)α
αα
ααΈααα
ααααα» KEDA Prometheus ScaledObject
α§αααααααΆαααααααααΎααα½ααΆααΈααΆααααΆααααΆα KEDA αα·αααααααααααΆααααα
αααααααααααααΌαααΆαααα½αααΆαα ScaledObject
ααΊααΆααααΆαααααΆαααααα½ααααααααΌαααΆαααΆαααααααΆαααΎααααΈααααΎααααΆαααααααΆαααΆαααααααΆαααΆαα½ααααααααααΉαααα·ααΆααα αααα»αααααΈααα Prometheus α
ScaledObject
ααΆαααααααΆαα’αααΈααΆαααααΎααΆαααααααΆαααΆαααααααΆα αα·ααααααααααΆαααααααααΉαααα·ααΆααα (ααΌα
ααΆααΆααααααΆααααααΆααααααΆαα ααααααα½α) α
αααααααααααααααα ααααααααααΆαααααΆαα‘αΎααα·α αα·ααα·ααααααααααααααα ααΆααααααααααααα
αααα»αααααΆα autoscaling αααααααΌαααααΆ (αα·ααααα HPA) ααΎααααΈααααΎααΆαααααααΆαααΆαααΆαααααααΆαα
αα
ααααααααααα»αα½αα ScaledObject
ααααΌαααΆααα»α αα·ααααα HPA αααααααΌαααααΆααααΌαααΆααααααα
αααααΊααΆαα·ααααα ScaledObject
αααααΆααα§ααΆα αααααααααΎα ααΆααααΎα§αααααααΆααααααα Prometheus
:
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: prometheus-scaledobject
namespace: default
labels:
deploymentName: go-prom-app
spec:
scaleTargetRef:
deploymentName: go-prom-app
pollingInterval: 15
cooldownPeriod: 30
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
metadata:
serverAddress:
http://prometheus-service.default.svc.cluster.local:9090
metricName: access_frequency
threshold: '3'
query: sum(rate(http_requests[2m]))
ααΌααα·α αΆαααΆα ααα»α ααΆααααααα
- ααΆααα
ααα’α»ααα
Deployment
ααΆαα½ααααααgo-prom-app
. - ααααααααα -
Prometheus
. α’αΆααααααΆααααΆαααΈααα Prometheus ααααΌαααΆααααααΆαααα½αααΆαα½αααΉαααααααααααα ααααα·α αα·ααααα½α PromQL αααααΉαααααΌαααΆαααααΎα αααα½α PromQL -sum(rate(http_requests[2m]))
. - αααααΎαααααΆα
pollingInterval
KEDA ααααΎαα»αααααα ααΈ Prometheus αααααΆαααααααααΆααα·ααΆααΈα αααΆαα αα ααΆαααα½ααα ααααα (minReplicaCount
) α αΎαα ααα½αα’αα·ααααΆαα pods αα·αααΎαααΈmaxReplicaCount
(αααα»αα§ααΆα αααααα - ααα) α
α’αΆα
ααααΌαααΆαααα‘αΎα minReplicaCount
ααααΎααΉαααΌαααα αααα»αααααΈααα KEDA ααααΎαααΆαααΆαααΆαααααααΆαααΈααΌααααα
αα½α α αΎααααααΆαααααααα αΆα HPA αααααΆααααΆαααααΎααΆαααααααΆααααααααααααααααα·αααααααααα ααααΆαααααα
αααΆαααα’αΆα
ααααΎαα
ααΆαααα αααααΊααααΎααΆαααααααΆαααΈαα½ααα
ααΌαααα αααα»αα§ααΆα ααα ααΎααα·αααΆαααααΎαααΎαααΌααααα ααααααααααΆααααΆ HTTP αα·ααα·ααααααΆααααααααααΆααααααΌαααΆαα
ααααααααα ααΆααααα»α autoscaling
ααααα·αβααααΌαβααΆαβααααΎβααΆβαααααΉαβααΎααααΈβααααΎβααΆαααααααΆαβααΆαβααΆααβαααααΆαα αααα»αα§ααΆα αααααααααΎα αααα½α PromQL sum(rate (http_requests [2m]))
αααα‘ααα’ααααΆααααΎ HTTP ααα»α (ααααΎαααα»ααα½ααα·ααΆααΈ) αααααΆααααααααα»αααααααααΈαααΆααΈα
α»ααααααα
αααααΆααααααααααα·αααΊααΈ ααΆααΆααααααΆααΉαααΆααα½ααα
ααααααααααααααααααα sum(rate (http_requests [2m]))
αα·α
ααΆαααΈα ααααα·αααΎαααααααΎαα‘αΎα α’αα»ααααααααααΌαααΆαααααααααΆααααα sum(rate (http_requests [2m]))
ααΎαα‘αΎαααΈα α§ααΆα αααααααα·αααΎαααααααΊααΈ 12 αα
14 αααα
ααα½αααααΊαα½αα
α₯α‘αΌαβααΎαβααααΆααΆαβαααα αβααΆβ!
ααΆααααααααΆαα»α
α’αααΈαααα’αααααααΌαααΆαααΊα
ααααα Kubernetes αα·αα§αααααααααΎααααΆαααααααΆαααααααα
ααΆαααααααα kubectl
. α§ααΆα ααααααααααΎα
ααααα minikube
ααα»ααααα’αααα’αΆα
αααα½αααααααααα ααΎααααΈααα‘αΎαα
αααααααΆα
ααα‘αΎαααααα α»αααααααααα»ααα ααΎ Macα
curl -Lo minikube
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/
ααααα
ααα‘αΎαααααα α»αααααααααα»ααα ααΎ Macα
curl -LO
"https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version
ααΆαααα‘αΎα KEDA
α’αααα’αΆα
ααΆαααααααΆα KEDA ααΆααα·ααΈααΆα
αααΎα αα½αααααααΌαααΆαααΆααααα»ααααααΈ
kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaScaleController.yaml
KEDA αα·ααααΆαααΆαα»ααααααΆααααΌαααΆαααα‘αΎααα
αααα»α namespace keda
. ααΆααααααααΆααΎααααΈαα·αα·αααα
kubectl get pods -n keda
αααα
αΆαα±ααααααα·ααααα·αα KEDA α
αΆααααααΎα α αΎαα
αΌααα
ααΆαα Running State
. α αΎααααααΆααααΈαααααααα
ααΆαααα‘αΎα Redis αααααααΎ Helm
ααααα·αααΎα’ααααα·αααΆαααα‘αΎα Helm ααΌαααααΎααΆ
brew install kubernetes-helm
helm init --history-max 200
helm init
α
αΆααααααΎαα
ααα»α
αααααΆαααααααΆααααΆααααααααΆααΌαααααΆα α αΎαααα‘αΎααααααα Tiller
αα
ααΆαααααα»α Kubernetes α
kubectl get pods -n kube-system | grep tiller
αααα αΆαα±αα Tiller pod α αΌααα αααα»αααααΆαααΆαααααααα»αααααΎαααΆαα
αααααα αααΆαααααα’αααααααααα α’ααααα·ααααααααΎ Helm@2 ααααααααΌαα±ααααα‘αΎααααΆαααΆααααΆαααΈααα Tiller α α₯α‘αΌαααα Helm@3 ααΊααΆααααααα ααΆαα·ααααααΌαα±ααααΆαααααααααΆαααΈαααααα
αααααΆααααΈααα‘αΎα Helm ααΆααααααααΆαα½αααΊαααααααααΆααααΎααααΈα αΆααααααΎα Redisα
helm install --name redis-server --set cluster.enabled=false --set
usePassword=false stable/redis
αααααααααΆααααΆ Redis ααΆαα αΆααααααΎααααααααααα
kubectl get pods/redis-server-master-0
αααα
αΆα Redis α
αΌααα
αααα»ααααα Running
.
ααΆαααΆααα±ααααααΎααααΆαααααααα·ααΈ
ααΆααααααααΆααΆαααααααΆαα
kubectl apply -f go-app.yaml
//output
deployment.apps/go-prom-app created
service/go-prom-app-service created
αα·αα·αααααΎαααΆα’αααΈαααΆαα αΆααααααΎαα
kubectl get pods -l=app=go-prom-app
αααα
αΆα Redis α
αΌααααα Running
.
ααΆαααΆαααααααΆααααΆαααΈααα Prometheus
Prometheus manifest ααααΎ
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_run]
regex: go-prom-app-service
action: keep
ααΎααααΈααΆαααααααΆαα
kubectl apply -f prometheus.yaml
//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created
αα·αα·αααααΎαααΆα’αααΈαααΆαα αΆααααααΎαα
kubectl get pods -l=app=prometheus-server
αααα
αΆα Prometheus α
αΌααα
αααα»ααααα Running
.
ααααΎ kubectl port-forward
ααΎααααΈα
αΌααα
ααΆααα
ααα»α
αααααΆααα’αααααααΎ Prometheus (α¬αααΆαααΈααα API) αα
kubectl port-forward service/prometheus-service 9090
ααΆαααΆαααααααΆαααΆαααααααα ααΆααααααααααααααααααααα· KEDA
ααΆααααααααΆααΎααααΈαααααΎα ScaledObject
:
kubectl apply -f keda-prometheus-scaledobject.yaml
αα·αα·αααααΎααααααα ααα»ααααα·ααααα·αα KEDAα
KEDA_POD_NAME=$(kubectl get pods -n keda
-o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda
ααααααααΎααα ααΌα αααα
time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:
default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with
namespace default and name keda-hpa-go-prom-app"
αα·αα·ααααα
ααααααααααα·ααΈα α§ααΆα ααααα½αααααΌαααααααΎαααΆααααααΆααα minReplicaCount
ααααΎ α‘α
kubectl get pods -l=app=go-prom-app
αααααααααΆααααΆααααΆα HPA ααααΌαααΆααααααΎααααααααααα
kubectl get hpa
α’ααααα½αααααΎαα’αααΈαα½αααΌα ααΆα
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
keda-hpa-go-prom-app Deployment/go-prom-app 0/3 (avg) 1 10 1 45s
ααΆααα·αα·ααααα»αααΆαα ααΆαα αΌαααααΎαααααα·ααΈ
ααΎααααΈα αΌαααααΎα ααα»α αααα αα REST αααααααα·ααΈααααααΎα ααΌαααααΎαααΆαα
kubectl port-forward service/go-prom-app-service 8080
α₯α‘αΌαααα α’αααα’αΆα
α
αΌαααααΎαααααα·ααΈ Go ααααα’ααααααααααΎα’αΆααααααΆα
curl http://localhost:8080/test
ααααααααΎααα ααΌα αααα
Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC
m=+406004.817901246
Access count 1
αα
α
ααα»α
ααααααα·αα·αααααΎα Redis αααααα α’αααααΉαααΎααααααΉααααα access_count
ααΎαα‘αΎαααα α‘α
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
"1"
ααααΌαααααΆααααΆαααααααααααααΊ http_requests
ααΌα
βααααΆ:
curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1
αααα»αααΆααααααΎα
ααΎαααΉαααααΎ
curl -o hey https://storage.googleapis.com/hey-release/hey_darwin_amd64
&& chmod a+x hey
α’αααααα’αΆα
ααΆαααα§αααααααααΎααααΆαααααααΆαα
ααααΎαααΆαααΆα
./hey http://localhost:8080/test
ααΆαααααΆαααΎα α§αααααααααΎααααΆααααααΎααααΎα ααα½α 200 α α’αααα’αΆα αααααααααΆααααΆαααααααΎαααααΆαα Prometheus ααααΌα ααΆ Redis α
curl http://localhost:8080/metrics | grep http_requests
//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201
kubectl exec -it redis-server-master-0 -- redis-cli get access_count
//output
201
ααααΎα±ααααααααααααααααα·αααααΆαα (αααα‘ααααααααα½α PromQL)α
curl -g
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
//output
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}
αααα»αααααΈαααααααααααΆααααααα 1,686057971014493
αα·αααααΌαααΆααααα αΆααα
αααα»αααΆα value
. ααααα·ααααααααααΆαααααααΆααααΆαααααΎααΆαααααααΆααα αααααΆαααααα·ααααααΎααααααααΊ 3 α
αααα»αααΆααααα αααΎα!
αα αααα»αααααΆααΈαααααΈ ααΌαααΆαααΆαα ααα½ααααααα·ααΈ podsα
kubectl get pods -l=app=go-prom-app -w
ααααααααΎαααααα»ααααααααΎααΆααααααααΆα
./hey -n 2000 http://localhost:8080/test
αα½αααααα»αααααααα α’αααααΉαααΎα HPA ααααΎααΆαααααααΆαααΆαααΆαααααααΆα αα·αααΎαααααΎαααΆα pods ααααΈα αα·αα·ααα HPA ααααα’αααααΎααααΈααααΆααααΆα
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
keda-hpa-go-prom-app Deployment/go-prom-app 1830m/3 (avg) 1 10 6 4m22s
ααααα·αααΎααΆααααα»ααα·ααααΈαααααΆααααααΆ ααΆαααΆαααααααΆαααΉαααααΌαααΆαααΆααααααααααα ααα»α αααααΆααααααα½ααααα»αααααΎαααΆαα ααααα·αααΎα’αααα αααα·αα·αααααΎααααααααα·αααααΆαα (αααα‘ααααααααα½α PromQL) αααααΆααααααααΎααΆααααααααΆα
curl -g
'http://localhost:9090/api/v1/query?query=sum(rate(http_requests[2m]))'
ααΆααααα’αΆα
//Delete KEDA
kubectl delete namespace keda
//Delete the app, Prometheus server and KEDA scaled object
kubectl delete -f .
//Delete Redis
helm del --purge redis-server
ααα ααααΈααααα·ααααΆα
KEDA α’αα»ααααΆαα±ααα’αααααααΎααΆαααααααΆαααΆαααΆαααααααΆα Kubernetes ααααα’ααααααααααααααααααα· (αα /ααΈααΌααα) αααααα’ααααΎαα·ααααααααΈααααααααΆααααα α α§ααΆα ααα ααα’ααααΎαααααΆαα Prometheus αααααααα½ααααα»α Redis ααΆαααΊααααΆαααααα’αααααααΎααααΆαααα αααα»ααααααΆααα Kafkaα
KEDA αα½ααααα αΌαααΆαα½ααααααααΆααααα α αΎααααααΆαααααααααΌααααααΆααααααααΆααΆαααα Metrics Server αα ααΆαα Horizontal Pod Autoscalerα
ααΌαααααΆαααα’!
ααΎααΆαα’αααΈααααααααααΌαα’αΆαα
ααΆαα’αα»ααααααα’αααα»α αα·αααΆαα’αα»ααααααα’αααα»ααααααΆααααΆαααααΎαααΆααα»αααΊααα αα·α Kubernetes αα αααα»αααα·ααΆααΆαααα·ααααα .α§αααααααΆααααααααα 90+ αααααΆαα Kubernetesα ααΆαααΆαααααααΆα ααΆαααααααααα ααΆααααα½ααα·αα·ααα αα»ααααα·ααΆα αα·αα αααΎαααα .ααα»αααα·αααααααΎααα αα»ααα·α Kubernetes αα αααα»α Telegram .
ααααα: www.habr.com