ProHoster > Blog > Pulega > Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

O le CD ua fa'ailoaina ose faiga fa'akomepiuta atina'e ma ose suiga fa'anatura o mataupu fa'avae CI. Ae ui i lea, o CD e seasea lava, masalo ona o le lavelave o le pulega ma le fefe i le le manuia o le faʻapipiʻiina e aʻafia ai le avanoa o le polokalama.

Fuʻa o se fa'amatalaga tatala Kubernetes operator lea e fa'amoemoe e fa'aumatia mafutaga fenumiai. Otometi le fa'alauiloaina o le fa'aogaina o canary e fa'aaoga ai Istio traffic offsets ma Prometheus metrics e iloilo ai amioga fa'aoga i le taimi o le fa'atonuina o le ta'avale.

O loʻo i lalo se taʻiala taʻitasi i le faʻatulagaina ma le faʻaogaina o le Fuʻa ile Google Kubernetes Engine (GKE).

Fa'atulaga se vaega Kubernetes

E te amata i le fatuina o se fuifui GKE ma le Istio add-on (afai e leai sau GCP account, e mafai ona e sainia iinei - ia maua tupe maua fua).

Saini i totonu ile Google Cloud, fai se poloketi, ma fa'agaoioi le pili. Fa'apipi'i le fa'aogaina o le laina fa'atonu gcloud ma fetuutuunai lau poloketi ma gcloud init.

Seti le galuega fa'aletonu, vaega fa'atatau, ma le sone (sui PROJECT_ID mo lau galuega faatino):

gcloud config set project PROJECT_ID
gcloud config set compute/region us-central1
gcloud config set compute/zone us-central1-a

Fa'aaga le auaunaga GKE ma fai se fuifui fa'atasi ma HPA ma Istio fa'aopoopo:

gcloud services enable container.googleapis.com
K8S_VERSION=$(gcloud beta container get-server-config --format=json | jq -r '.validMasterVersions[0]')
gcloud beta container clusters create istio 
--cluster-version=${K8S_VERSION} 
--zone=us-central1-a 
--num-nodes=2 
--machine-type=n1-standard-2 
--disk-size=30 
--enable-autorepair 
--no-enable-cloud-logging 
--no-enable-cloud-monitoring 
--addons=HorizontalPodAutoscaling,Istio 
--istio-config=auth=MTLS_PERMISSIVE

O le faʻatonuga o loʻo i luga o le a fatuina ai se vaitaele le aoga e aofia ai VM e lua n1-standard-2 (vCPU: 2, RAM 7,5 GB, tisiki: 30 GB). O le mea e lelei ai, o vaega Istio e tatau ona vavae ese mai a latou galuega, ae leai se auala faigofie e taʻavale ai Istio pods i luga o se vaitaele faʻapitoa. O fa'aaliga Istio e manatu na'o le faitau, ma o le a toe fa'afo'isia e le GKE so'o se suiga e pei o le fusifusia i se node po'o le to'ese mai se pusa.

Seti fa'ailoga mo kubectl:

gcloud container clusters get-credentials istio

Fausia se vaega o le pulega o lo'o fusifusia:

kubectl create clusterrolebinding "cluster-admin-$(whoami)" 
--clusterrole=cluster-admin 
--user="$(gcloud config get-value core/account)"

Fa'apipi'i le meafaigaluega laina fa'atonu pūlou faʻafao:

brew install kubernetes-helm

Homebrew 2.0 ua avanoa nei mo Linux.

Fausia se tala o auaunaga ma fa'apipi'i matafaioi mo Tiller:

kubectl -n kube-system create sa tiller && 
kubectl create clusterrolebinding tiller-cluster-rule 
--clusterrole=cluster-admin 
--serviceaccount=kube-system:tiller

Fa'alautele le Tiller ile igoa avanoa kube-system:

helm init --service-account tiller

E tatau ona e mafaufau e faʻaaoga SSL i le va o Helm ma Tiller. Mo nisi fa'amatalaga e uiga i le puipuia o lau fa'apipi'i Helm, va'ai docs.helm.sh

Fa'amaonia tulaga:

kubectl -n istio-system get svc

A maeʻa ni nai sekone, e tatau ona tuʻuina atu e le GCP se tuatusi IP fafo i le tautua istio-ingressgateway.

Fa'atuina se Istio Ingress Gateway

Fausia se tuatusi IP tumau ma le igoa istio-gatewayfa'aaoga le tuatusi IP faitotoa Istio:

export GATEWAY_IP=$(kubectl -n istio-system get svc/istio-ingressgateway -ojson | jq -r .status.loadBalancer.ingress[0].ip)
gcloud compute addresses create istio-gateway --addresses ${GATEWAY_IP} --region us-central1

O lea e te manaʻomia se initaneti ma avanoa i lau resitala DNS. Faaopoopo i ai ni faamaumauga A se lua (sui example.com i lau vaega):

istio.example.com   A ${GATEWAY_IP}
*.istio.example.com A ${GATEWAY_IP}

Fa'amaonia o lo'o galue le DNS wildcard:

watch host test.istio.example.com

Fausia se faitotoa lautele Istio e tuʻuina atu ai auaunaga i fafo atu o le mesh auaunaga i luga ole HTTP:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Faasaoina le punaoa o loʻo i luga o le public-gateway.yaml ona faʻaoga lea:

kubectl apply -f ./public-gateway.yaml

Leai se faiga gaosiga e tatau ona tuʻuina atu auaunaga i luga ole Initaneti e aunoa ma le SSL. Ina ia faʻamautu lau faitotoa Istio ingress ma le pule faʻamaonia, CloudDNS ma Let's Encrypt, faʻamolemole faitau fa'amaumauga Fu'a G.K.E.

Fa'apipi'i fu'a

O le GKE Istio faʻaopoopo e le aofia ai le Prometheus faʻataʻitaʻiga e faʻamamāina ai le Istio telemetry auaunaga. Talu ai o le Flagger e faʻaogaina Istio HTTP metrics e faʻatino ai suʻesuʻega canary, e tatau ona e faʻaogaina le faʻatulagaina o Prometheus, e tutusa ma le mea e sau ma le Istio Helm schema.

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/gke/istio-prometheus.yaml

Fa'aopoopo le fa'aputuga o le Fu'a Helm:

helm repo add flagger [https://flagger.app](https://flagger.app/)

Fa'alautele le Fu'a ile igoa avanoa istio-systeme ala i le faʻatagaina o faʻamatalaga Slack:

helm upgrade -i flagger flagger/flagger 
--namespace=istio-system 
--set metricsServer=http://prometheus.istio-system:9090 
--set slack.url=https://hooks.slack.com/services/YOUR-WEBHOOK-ID 
--set slack.channel=general 
--set slack.user=flagger

E mafai ona e faʻapipiʻi le Flagger i soʻo se igoa avanoa pe a mafai ona fesoʻotaʻi ma le Istio Prometheus auaunaga ile taulaga 9090.

O lo'o i ai i le Fu'a le fa'ailoga o le Grafana mo su'esu'ega o canary. Faʻapipiʻi le Grafana ile igoa avanoa istio-system:

helm upgrade -i flagger-grafana flagger/grafana 
--namespace=istio-system 
--set url=http://prometheus.istio-system:9090 
--set user=admin 
--set password=change-me

Fa'aali atu le Grafana i se faitoto'a tatala e ala i le faia o se auaunaga fa'apitoa (sui example.com i lau vaega):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
  namespace: istio-system
spec:
  hosts:
    - "grafana.istio.example.com"
  gateways:
    - public-gateway.istio-system.svc.cluster.local
  http:
    - route:
        - destination:
            host: flagger-grafana

Faasaoina le punaoa o loʻo i luga o le grafana-virtual-service.yaml ona faʻaoga lea:

kubectl apply -f ./grafana-virtual-service.yaml

Pe a alu i http://grafana.istio.example.com O lau su'esu'ega e tatau ona toe fa'afeiloa'i oe i le itulau e saini ai le Grafana.

Fa'aogaina o talosaga i luga ole laiga ile Fu'a

Fua fa'apipi'i Kubernetes ma, pe a mana'omia, fa'ata'ita'i ta'avale (HPA), ona fa'atupuina lea o se fa'asologa o mea faitino (fa'apipi'i Kubernetes, 'au'aunaga ClusterIP ma Istio virtual services). O nei mea faitino e faʻaalia ai le faʻaogaina i le 'auʻaunaga ma faʻatautaia suʻesuʻega ma faʻalauiloa canary.

Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

Fausia se su'ega igoa avanoa fa'atasi ai ma le fa'atinoga o le Istio Sidecar ua mafai ai:

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml

Fausia se fa'apipi'i ma se mea faigaluega fa'alava fa'ata'atia mo le pod:

kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml

Fa'atu se auaunaga su'ega uta e fa'atupu ai feoaiga i le taimi o su'esu'ega o le canary:

helm upgrade -i flagger-loadtester flagger/loadtester 
--namepace=test

Fausia se punaoa masani canary (sui example.com i lau vaega):

apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 60
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: podinfo
  service:
    port: 9898
    gateways:
    - public-gateway.istio-system.svc.cluster.local
    hosts:
    - app.istio.example.com
  canaryAnalysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: istio_requests_total
      threshold: 99
      interval: 30s
    - name: istio_request_duration_seconds_bucket
      threshold: 500
      interval: 30s
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

Faasaoina le punaoa o loʻo i luga e pei o podinfo-canary.yaml ona faʻaoga lea:

kubectl apply -f ./podinfo-canary.yaml

O le suʻesuʻega o loʻo i luga, pe a manuia, o le a tamoe mo le lima minute, siaki HTTP metrics i afa minute. E mafai ona e fuafuaina le taimi laʻititi e manaʻomia e suʻe ai ma faʻalauiloa le faʻapipiʻiina o canary e faʻaaoga ai le fua faʻatatau: interval * (maxWeight / stepWeight). Canary CRD fanua o loʻo faʻamaumau iinei.

A maeʻa ni nai sekone, o le a faia e Flagger ni mea taʻavale:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo
# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
virtualservice.networking.istio.io/podinfo

Tatala lau su'ega ma alu i app.istio.example.com, e tatau ona e vaʻai i le numera numera talosaga demo.

Otometi su'esu'ega canary ma fa'alauiloa

E fa'atino e le Flagger se matasele fa'atonutonu e fa'agasolo malie atu ai le ta'avale i le canary a'o fuaina fa'ailoga autu o fa'atinoga e pei o le HTTP talosaga manuia fua, averesi le umi ole talosaga, ma le soifua maloloina. Faʻavae i luga ole suʻesuʻega KPI, o le canary e faʻalauiloaina pe faʻamutaina, ma o taunuʻuga o suʻesuʻega o loʻo lolomiina i Slack.

Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

O le fa'aogaina o Canary e fa'aosofia pe a suia se tasi o mea nei:

  • Fa'atino le PodSpec (fa'atusa ata, fa'atonuga, ports, env, etc.)
  • ConfigMaps o loʻo faʻapipiʻiina e pei o voluma pe liliu i suiga o le siosiomaga
  • O mea lilo o lo'o fa'apipi'iina e pei o voluma po'o le liua i suiga o le si'osi'omaga

Fa'ata'ita'i le fa'aogaina o canary pe a fa'afou le ata o pusa:

kubectl -n test set image deployment/podinfo 
podinfod=quay.io/stefanprodan/podinfo:1.4.1

Ua iloa e le Fu'a ua suia le fa'ata'ita'iga ma amata ona su'esu'eina:

kubectl -n test describe canary/podinfo

Events:

New revision detected podinfo.test
Scaling up podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test

I le taimi o suʻesuʻega, e mafai ona mataʻituina taunuuga o canary e faʻaaoga ai le Grafana:

Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

Fa'amolemole maitau: afai e fa'aoga ni suiga fou i le fa'apipi'iina i le taimi o su'esu'ega canary, o le a toe amata e le Fu'a le vaega au'ili'ili.

Fai se lisi o canaries uma i lau fuifui:

watch kubectl get canaries --all-namespaces
NAMESPACE   NAME      STATUS        WEIGHT   LASTTRANSITIONTIME
test        podinfo   Progressing   15       2019-01-16T14:05:07Z
prod        frontend  Succeeded     0        2019-01-15T16:15:07Z
prod        backend   Failed        0        2019-01-14T17:05:07Z

Afai na e faʻatagaina faʻamatalaga Slack, o le ae mauaina feʻau nei:

Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

Otometi le solomuli

I le taimi o le su'esu'ega o canary, e mafai ona e fa'atupuina mea sese HTTP 500 ma maualuga le tali atu e siaki pe o le a taofia e le Flagger le fa'apipi'iina.

Fausia se suʻega suʻega ma fai mea nei i totonu:

kubectl -n test run tester 
--image=quay.io/stefanprodan/podinfo:1.2.1 
-- ./podinfo --port=9898
kubectl -n test exec -it tester-xx-xx sh

Fausia HTTP 500 mea sese:

watch curl http://podinfo-canary:9898/status/500

Fa'atuai tupuaga:

watch curl http://podinfo-canary:9898/delay/1

A o'o le numera o siaki ua le taulau i se fa'ailoga, ona toe fa'afo'i atu lea o fe'avea'i i le alalaupapa muamua, fa'atele le canary i le zero, ma fa'ailogaina le fa'agaioiina ua le manuia.

O mea sese a Canary ma fa'agasolo o lo'o fa'amauina e pei o mea Kubernetes ma fa'amauina e le Fu'a ile JSON fa'asologa:

kubectl -n istio-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement success rate 47.00% < 99%
Halt podinfo.test advancement success rate 37.00% < 99%
Halt podinfo.test advancement request duration 1.515s > 500ms
Halt podinfo.test advancement request duration 1.600s > 500ms
Halt podinfo.test advancement request duration 1.915s > 500ms
Halt podinfo.test advancement request duration 2.050s > 500ms
Halt podinfo.test advancement request duration 2.515s > 500ms
Rolling back podinfo.test failed checks threshold reached 10
Canary failed! Scaling down podinfo.test

Afai na e faʻatagaina faʻamatalaga Slack, o le ae mauaina se feʻau pe a sili atu le taimi e faʻamaeʻa ai pe ausia ai le numera maualuga o iloiloga le manuia i se auiliiliga:

Fa'apipi'i otometi canary fa'atasi ma le Fu'a ma le Istio

I le faaiuga

O le fa'atinoina o se 'au'aunaga e pei o Istio i le pito i luga o Kubernetes o le a maua ai fua fa'atatau, ogalaau, ma ogalaau, ae o le fa'aogaina o galuega e fa'alagolago lava i meafaigaluega i fafo. Fu'a fa'amoemoe e sui lenei mea e ala i le fa'aopoopoina o le Istio gafatia fa'agasolo le tu'uina atu.

E fetaui lelei le Flagger ma soʻo se tali CI/CD mo Kubernetes, ma e faigofie ona faʻalauteleina le suʻesuʻega o canary i webhooks e fa'atino su'ega tu'ufa'atasia/ talia, su'ega uta po'o so'o se isi lava su'ega masani. Talu ai ona o le Flagger e faʻaalia ma tali atu i mea Kubernetes, e mafai ona faʻaogaina i laina paipa GitOps faʻatasi ma Lalaga Flux poʻo JenkinsX. Afai o loʻo e faʻaaogaina JenkinsX, e mafai ona e faʻapipiʻi le Flagger faʻatasi ma le jx add-ons.

Lagolagoina le fu'a Lalaga ma tu'uina atu fa'ata'otoga canary i Lalaga Ao. O le poloketi o loʻo faʻataʻitaʻiina ile GKE, EKS ma le uʻamea faʻatasi ma le kubeadm.

Afai e iai ni au fautuaga mo le fa'aleleia atili o le Fu'a, fa'amolemole lafo mai se fa'amatalaga po'o se PR ile GitHub ile stefanprodan/fu'a. E sili atu le talia o sao!

Спасибо Ray Tsang.

puna: www.habr.com

Faaopoopo i ai se faamatalaga