🥇Otomatiki canary deployments neFlager uye Istio

CD inozivikanwa sebhizinesi software tsika uye mhedzisiro yekushanduka kwechisikigo kweyakasimbiswa CI nheyo. Nekudaro, CD ichiri kushoma, pamwe nekuda kwekuoma kwehutungamiriri uye kutya kwekutadza kutumirwa kunokanganisa kuwanikwa kwehurongwa.

Mureza ndeye yakavhurika sosi Kubernetes opareta inovavarira kubvisa hukama hunovhiringidza. Inoita otomatiki kukwidziridzwa kwecanary deployments uchishandisa Istio traffic offset uye Prometheus metrics yekuongorora maitiro ekushandisa panguva yekuburitswa kwakagadziriswa.

Pazasi pane nhanho nhanho gwara rekumisikidza uye kushandisa Flagger paGoogle Kubernetes Engine (GKE).

Kugadzira Kubernetes cluster

Iwe unotanga nekugadzira GKE cluster neIstio add-on (kana iwe usina GCP account, unogona kusaina. pano - kuwana zvikwereti zvemahara).

Pinda muGoogle Cloud, gadzira chirongwa, uye wogonesa kubhadhara kwayo. Isa mutsara wekuraira utility gcloud uye gadzira chirongwa chako ne gcloud init.

Seta default purojekiti, compute nzvimbo, uye zone (tsiva PROJECT_ID kune purojekiti yako):

gcloud config set project PROJECT_ID
gcloud config set compute/region us-central1
gcloud config set compute/zone us-central1-a

Gonesa iyo GKE sevhisi uye gadzira sumbu neHPA uye Istio yekuwedzera:

gcloud services enable container.googleapis.com
K8S_VERSION=$(gcloud beta container get-server-config --format=json | jq -r '.validMasterVersions[0]')
gcloud beta container clusters create istio 
--cluster-version=${K8S_VERSION} 
--zone=us-central1-a 
--num-nodes=2 
--machine-type=n1-standard-2 
--disk-size=30 
--enable-autorepair 
--no-enable-cloud-logging 
--no-enable-cloud-monitoring 
--addons=HorizontalPodAutoscaling,Istio 
--istio-config=auth=MTLS_PERMISSIVE

Murairo wepamusoro uchagadzira dziva renode default kusanganisira maVM maviri n1-standard-2 (vCPU: 2, RAM 7,5 GB, dhisiki: 30 GB). Zvakanaka, iwe unofanirwa kutsaura zvikamu zveIstio kubva pamabasa ako, asi hapana nzira iri nyore yekumhanyisa Istio Pods mudziva rakatsaurirwa remanodhi. Istio inoratidza inoonekwa sekuverenga-chete uye GKE ichagadzirisa chero shanduko, sekubatanidza node kana kubvisa kubva pane pod.

Gadzira zvitupa zve kubectl:

gcloud container clusters get-credentials istio

Gadzira cluster administrator basa rinosunga:

kubectl create clusterrolebinding "cluster-admin-$(whoami)" 
--clusterrole=cluster-admin 
--user="$(gcloud config get-value core/account)"

Isa chirairo chekuraira Helmamu:

brew install kubernetes-helm

Homebrew 2.0 yave kuwanikwa zvakare Linux.

Gadzira account yesevhisi uye basa remasumbu rinosunga reTiller:

kubectl -n kube-system create sa tiller && 
kubectl create clusterrolebinding tiller-cluster-rule 
--clusterrole=cluster-admin 
--serviceaccount=kube-system:tiller

Wedzera Tiller munzvimbo yemazita kube-system:

helm init --service-account tiller

Iwe unofanirwa kufunga nezve kushandisa SSL pakati peHelm neTiller. Kuti uwane rumwe ruzivo nezve kuchengetedza yako Helm kuisirwa, ona docs.helm.sh

Simbisa marongero:

kubectl -n istio-system get svc

Mushure memasekondi mashoma, GCP inofanirwa kugovera yekunze IP kero yebasa racho istio-ingressgateway.

Kugadzirisa iyo Istio Ingress Gateway

Gadzira static IP kero ine zita istio-gatewayuchishandisa IP kero yeIstio gedhi:

export GATEWAY_IP=$(kubectl -n istio-system get svc/istio-ingressgateway -ojson | jq -r .status.loadBalancer.ingress[0].ip)
gcloud compute addresses create istio-gateway --addresses ${GATEWAY_IP} --region us-central1

Iye zvino iwe unoda internet domain uye kuwana kune yako DNS registrar. Wedzera maviri A marekodhi (tsiva example.com kune yako domain):

istio.example.com   A ${GATEWAY_IP}
*.istio.example.com A ${GATEWAY_IP}

Tarisa kuti DNS wildcard iri kushanda:

watch host test.istio.example.com

Gadzira gedhi reIstio gedhi rekupa masevhisi kunze kwesevhisi mesh pamusoro peHTTP:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Sevha chiri pamusoro apa sosi se public-gateway.yaml wobva waishandisa:

kubectl apply -f ./public-gateway.yaml

Hapana hurongwa hwekugadzira hunofanirwa kupa masevhisi paInternet pasina SSL. Kuti uchengetedze Istio ingress gedhi necert-maneja, CloudDNS uye Let's Encrypt, ndapota verenga zvinyorwa Mureza G.K.E.

Flagger Installation

Iyo GKE Istio yekuwedzera haisanganisi chiitiko chePrometheus chinochenesa Istio telemetry service. Nekuti Mureza anoshandisa Istio HTTP metrics kuita canary ongororo, iwe unofanirwa kuendesa inotevera Prometheus kumisikidzwa, yakafanana neiyo inouya neiyo official Istio Helm schema.

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/gke/istio-prometheus.yaml

Wedzera iyo Flagger Helm repository:

helm repo add flagger [https://flagger.app](https://flagger.app/)

Wedzera Mureza kusvika panzvimbo yemazita istio-systemnekugonesa Slack zviziviso:

helm upgrade -i flagger flagger/flagger 
--namespace=istio-system 
--set metricsServer=http://prometheus.istio-system:9090 
--set slack.url=https://hooks.slack.com/services/YOUR-WEBHOOK-ID 
--set slack.channel=general 
--set slack.user=flagger

Iwe unogona kuisa Flagger mune chero nzvimbo yezita chero bedzi ichikwanisa kutaurirana neIstio Prometheus sevhisi pachiteshi 9090.

Flagger ine Grafana dashboard yekuongorora canary. Isa Grafana munzvimbo yezita istio-system:

helm upgrade -i flagger-grafana flagger/grafana 
--namespace=istio-system 
--set url=http://prometheus.istio-system:9090 
--set user=admin 
--set password=change-me

Ratidza Grafana kuburikidza negedhi rakavhurika nekugadzira chaiyo sevhisi (tsiva example.com kune yako domain):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
  namespace: istio-system
spec:
  hosts:
    - "grafana.istio.example.com"
  gateways:
    - public-gateway.istio-system.svc.cluster.local
  http:
    - route:
        - destination:
            host: flagger-grafana

Sevha chiri pamusoro apa sosi se grafana-virtual-service.yaml wobva waishandisa:

kubectl apply -f ./grafana-virtual-service.yaml

Pakufamba kuenda http://grafana.istio.example.com mubrowser, iwe unofanirwa kutungamirwa kune iyo Grafana login peji.

Kutumira mawebhu application neFlager

Mureza anotumira Kubernetes uye nesarudzo anoyera otomatiki (HPA), obva agadzira nhevedzano yezvinhu (Kubernetes deployments, ClusterIP services, uye Istio virtual services). Izvi zvinhu zvinofumura application kune mesh sevhisi uye kutonga canary ongororo uye kufambira mberi.

Gadzira nzvimbo yekuyedza neIstio Sidecar jekiseni inogoneswa:

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml

Gadzira kutumirwa uye pod otomatiki chiyero-kunze chishandiso:

kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml

Tumira sevhisi yemutoro wekuyedza kugadzira traffic panguva yekuongorora canary:

helm upgrade -i flagger-loadtester flagger/loadtester 
--namepace=test

Gadzira tsika canary sosi (chinja example.com kune yako domain):

apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 60
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: podinfo
  service:
    port: 9898
    gateways:
    - public-gateway.istio-system.svc.cluster.local
    hosts:
    - app.istio.example.com
  canaryAnalysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: istio_requests_total
      threshold: 99
      interval: 30s
    - name: istio_request_duration_seconds_bucket
      threshold: 500
      interval: 30s
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

Sevha iyo iri pamusoro sosi se podinfo-canary.yaml wobva waishandisa:

kubectl apply -f ./podinfo-canary.yaml

Ongororo iri pamusoro, kana ikabudirira, inomhanya kwemaminetsi mashanu, ichitarisa HTTP metrics hafu yeminiti. Iwe unogona kuona iyo shoma nguva inodiwa kusimbisa uye kusimudzira kuendesa canary uchishandisa inotevera fomula: interval * (maxWeight / stepWeight). Canary CRD minda yakanyorwa pano.

Mushure memasekondi mashoma, Mureza achagadzira zvinhu zvecanary:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo
# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
virtualservice.networking.istio.io/podinfo

Vhura browser uye enda ku app.istio.example.com, unofanira kuona nhamba yeshanduro demo apps.

Otomatiki canary kuongorora uye kukwidziridzwa

Mureza inoshandisa dhizaini yekudzora iyo zvishoma nezvishoma inofambisa traffic kune canary uku ichiyera makiyi ekuita metrics akadai seHTTP chikumbiro chekubudirira, avhareji yenguva yekukumbira, uye pod hutano. Zvichienderana nekuongorora kweKPI, canary inosimudzirwa kana kukanganiswa, uye mhedzisiro yekuongorora inoburitswa kuna Slack.

Canary deployment inokonzereswa kana chimwe chezvinhu zvinotevera chichichinja:

Deploy PodSpec (container image, command, ports, env, etc.)
ConfigMaps akaiswa semavhoriyamu kana mepu kune zvakatipoteredza zvinosiyana
Zvakavanzika zvinoiswa semavhoriyamu kana kuchinjirwa kune zvakatipoteredza zvinosiyana

Mhanya canary deploy paunenge uchivandudza mufananidzo wemudziyo:

kubectl -n test set image deployment/podinfo 
podinfod=quay.io/stefanprodan/podinfo:1.4.1

Mureza anoona kuti vhezheni yekutumirwa yachinja uye otanga kuiburitsa:

kubectl -n test describe canary/podinfo

Events:

New revision detected podinfo.test
Scaling up podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test

Panguva yekuongorora, canary mhedzisiro inogona kuteverwa uchishandisa Grafana:

Ndokumbira utarise kuti kana shanduko nyowani dzikaiswa pakutumirwa panguva yekuongorora canary, ipapo Flagger ichatangazve chikamu chekuongorora.

Ita runyoro rwese canaries musumbu rako:

watch kubectl get canaries --all-namespaces
NAMESPACE   NAME      STATUS        WEIGHT   LASTTRANSITIONTIME
test        podinfo   Progressing   15       2019-01-16T14:05:07Z
prod        frontend  Succeeded     0        2019-01-15T16:15:07Z
prod        backend   Failed        0        2019-01-14T17:05:07Z

Kana iwe wakagonesa Slack zviziviso, iwe unogashira anotevera mameseji:

Otomatiki rollback

Panguva yekuongorora canary, unogona kugadzira zvikanganiso zveHTTP 500 uye yakakwira mhinduro latency kuti uone kana Mureza achamisa kutumirwa.

Gadzira bvunzo pod uye ita zvinotevera mairi:

kubectl -n test run tester 
--image=quay.io/stefanprodan/podinfo:1.2.1 
-- ./podinfo --port=9898
kubectl -n test exec -it tester-xx-xx sh

Kugadzira HTTP 500 kukanganisa:

watch curl http://podinfo-canary:9898/status/500

Kunonoka kugadzira:

watch curl http://podinfo-canary:9898/delay/1

Kana nhamba yecheki yakakundikana inosvika pachikumbaridzo, traffic inodzoserwa kunzira yekutanga, canary inoyerwa kusvika zero, uye kutumirwa kunoiswa chiratidzo sekukundikana.

Zvikanganiso zveCanary uye latency spikes zvakarogwa sezviitiko zveKubernetes uye zvakadhindwa neMureza muJSON fomati:

kubectl -n istio-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement success rate 47.00% < 99%
Halt podinfo.test advancement success rate 37.00% < 99%
Halt podinfo.test advancement request duration 1.515s > 500ms
Halt podinfo.test advancement request duration 1.600s > 500ms
Halt podinfo.test advancement request duration 1.915s > 500ms
Halt podinfo.test advancement request duration 2.050s > 500ms
Halt podinfo.test advancement request duration 2.515s > 500ms
Rolling back podinfo.test failed checks threshold reached 10
Canary failed! Scaling down podinfo.test

Kana iwe wakagonesa Slack zviziviso, iwe unogashira meseji kana nguva yekupedzisira yadarika kana kuwanda kwehuwandu hweakatadza cheki mukuongororwa kwasvika:

Mukupedzisa

Kumhanyisa mesh sevhisi seIstio mukuwedzera kuKubernetes kunopa otomatiki metrics, matanda, uye mapuroteni, asi basa rekutumira rinoramba richienderana nezvishandiso zvekunze. Mureza unovavarira kushandura izvi nekuwedzera Istio kugona kupa kunofambira mberi.

Flagger inoenderana nechero Kubernetes CI/CD mhinduro, uye canary ongororo inogona kuwedzerwa nyore ne webhooks kuita system yekubatanidza / yekubvuma bvunzo, mutoro bvunzo, kana chero imwe tsika cheki. Sezvo Flagger ichizivisa uye ichipindura Kubernetes zviitiko, inogona kushandiswa muGitOps mapaipi pamwe chete. Weave Flux kana JenkinsX. Kana uri kushandisa JenkinsX unogona kuisa Flagger nejx addons.

Flagger inotsigirwa Weaveworks uye inopa canary deployments mukati Weave Cloud. Iyo purojekiti iri kuedzwa paGKE, EKS, uye isina simbi ine kubeadm.

Kana uine mazano ekuvandudza Mureza, ndapota tumira nyaya kana PR paGitHub pa stefanprodan/flagger. Mipiro yakawanda kupfuura inogamuchirwa!

Спасибо Ray Tsang.

Source: www.habr.com

Otomatiki canary deployments neFlager uye Istio