ProHoster > Блог > Gudanarwa > Aiwatar da canary ta atomatik tare da Flagger da Istio

Aiwatar da canary ta atomatik tare da Flagger da Istio

Aiwatar da canary ta atomatik tare da Flagger da Istio

CD ana gane shi azaman aikin software na kasuwanci kuma juyin halitta ne na ka'idojin CI da aka kafa. Koyaya, CD ɗin har yanzu ba kasafai ba ne, watakila saboda sarƙaƙƙiyar gudanarwa da fargabar gazawar tura kayan aiki da ke shafar samuwar tsarin.

Alamar wani buɗaɗɗen tushen ma'aikacin Kubernetes ne wanda ke nufin kawar da alaƙar ruɗani. Yana sarrafa haɓaka ayyukan canary ta amfani da abubuwan kashe zirga-zirgar Istio da ma'auni na Prometheus don nazarin halayen aikace-aikacen yayin aiwatar da aiwatarwa.

Da ke ƙasa akwai jagorar mataki-mataki don kafawa da amfani da Flagger akan Injin Kubernetes na Google (GKE).

Kafa gungu na Kubernetes

Kuna farawa da ƙirƙirar gungu na GKE tare da ƙarawar Istio (idan ba ku da asusun GCP, kuna iya yin rajista. a nan - don karɓar kyauta kyauta).

Shiga zuwa Google Cloud, ƙirƙiri aiki, kuma ba da damar yin lissafin kuɗi don shi. Shigar da amfanin layin umarni gcloud kuma saita aikin ku tare da gcloud init.

Saita tsoho aikin, lissafin yanki, da yanki (maye gurbin PROJECT_ID don aikinku):

gcloud config set project PROJECT_ID
gcloud config set compute/region us-central1
gcloud config set compute/zone us-central1-a

Kunna sabis ɗin GKE kuma ƙirƙirar tari tare da ƙari na HPA da Istio:

gcloud services enable container.googleapis.com
K8S_VERSION=$(gcloud beta container get-server-config --format=json | jq -r '.validMasterVersions[0]')
gcloud beta container clusters create istio 
--cluster-version=${K8S_VERSION} 
--zone=us-central1-a 
--num-nodes=2 
--machine-type=n1-standard-2 
--disk-size=30 
--enable-autorepair 
--no-enable-cloud-logging 
--no-enable-cloud-monitoring 
--addons=HorizontalPodAutoscaling,Istio 
--istio-config=auth=MTLS_PERMISSIVE

Umurnin da ke sama zai ƙirƙiri wani wurin tsoho na node wanda ya ƙunshi VM guda biyu n1-standard-2 (vCPU: 2, RAM 7,5 GB, faifai: 30 GB). Da kyau, abubuwan Istio yakamata a ware su daga nauyin aikinsu, amma babu wata hanya mai sauƙi don gudanar da kwas ɗin Istio akan tafkin kumburin da aka keɓe. Ana ɗaukar bayyanar Istio a karanta kawai, kuma GKE zai dawo da kowane canje-canje kamar ɗaure zuwa kumburi ko cirewa daga kwasfa.

Saita takaddun shaida don kubectl:

gcloud container clusters get-credentials istio

Ƙirƙiri ɗaurin rawar gudanarwa ta gungu:

kubectl create clusterrolebinding "cluster-admin-$(whoami)" 
--clusterrole=cluster-admin 
--user="$(gcloud config get-value core/account)"

Shigar da kayan aikin layin umarni Hanya:

brew install kubernetes-helm

Homebrew 2.0 yanzu kuma yana samuwa don Linux.

Ƙirƙiri asusun sabis da ɗaurin rawar tari don Tiller:

kubectl -n kube-system create sa tiller && 
kubectl create clusterrolebinding tiller-cluster-rule 
--clusterrole=cluster-admin 
--serviceaccount=kube-system:tiller

Fadada Tiller a cikin sararin suna kube-system:

helm init --service-account tiller

Ya kamata ku yi la'akari da amfani da SSL tsakanin Helm da Tiller. Don ƙarin bayani game da kare shigarwa Helm, duba docs.helm.sh

Tabbatar da saituna:

kubectl -n istio-system get svc

Bayan ƴan daƙiƙa, GCP yakamata ya sanya adireshin IP na waje zuwa sabis ɗin istio-ingressgateway.

Kafa Ƙofar Istio Ingress

Ƙirƙiri adreshin IP na tsaye tare da sunan istio-gatewayta amfani da adireshin IP na Gateway Istio:

export GATEWAY_IP=$(kubectl -n istio-system get svc/istio-ingressgateway -ojson | jq -r .status.loadBalancer.ingress[0].ip)
gcloud compute addresses create istio-gateway --addresses ${GATEWAY_IP} --region us-central1

Yanzu kuna buƙatar yankin intanet da samun dama ga mai rejista na DNS. Ƙara bayanan A guda biyu (maye gurbin example.com zuwa yankinku):

istio.example.com   A ${GATEWAY_IP}
*.istio.example.com A ${GATEWAY_IP}

Tabbatar da cewa DNS wildcard yana aiki:

watch host test.istio.example.com

Ƙirƙirar ƙofa ta Istio gabaɗaya don samar da ayyuka a wajen layin sabis akan HTTP:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Ajiye albarkatun da ke sama azaman public-gateway.yaml sannan a yi amfani da shi:

kubectl apply -f ./public-gateway.yaml

Babu tsarin samarwa da yakamata ya samar da sabis akan Intanet ba tare da SSL ba. Don tabbatar da ƙofar shiga Istio tare da cert-manager, CloudDNS da Let's Encrypt, da fatan za a karanta takardun shaida Flagger G.K.E.

Shigar da tuta

GKE Istio add-on baya haɗa da misalin Prometheus wanda ke tsaftace sabis na telemetry na Istio. Tunda Flagger yana amfani da ma'aunin Istio HTTP don yin nazarin canary, kuna buƙatar tura tsarin Prometheus mai zuwa, kama da wanda ya zo tare da tsarin Istio Helm na hukuma.

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/gke/istio-prometheus.yaml

Ƙara ma'ajiyar Tutar Helm:

helm repo add flagger [https://flagger.app](https://flagger.app/)

Fadada Tuta zuwa sararin suna istio-systemta hanyar kunna sanarwar Slack:

helm upgrade -i flagger flagger/flagger 
--namespace=istio-system 
--set metricsServer=http://prometheus.istio-system:9090 
--set slack.url=https://hooks.slack.com/services/YOUR-WEBHOOK-ID 
--set slack.channel=general 
--set slack.user=flagger

Kuna iya shigar da Flagger a kowane sarari suna muddin yana iya sadarwa tare da sabis na Istio Prometheus akan tashar jiragen ruwa 9090.

Flagger yana da dashboard ɗin Grafana don nazarin canary. Sanya Grafana a cikin sararin suna istio-system:

helm upgrade -i flagger-grafana flagger/grafana 
--namespace=istio-system 
--set url=http://prometheus.istio-system:9090 
--set user=admin 
--set password=change-me

Nuna Grafana ta hanyar bude kofa ta hanyar ƙirƙirar sabis na kama-da-wane (maye gurbin example.com zuwa yankinku):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
  namespace: istio-system
spec:
  hosts:
    - "grafana.istio.example.com"
  gateways:
    - public-gateway.istio-system.svc.cluster.local
  http:
    - route:
        - destination:
            host: flagger-grafana

Ajiye albarkatun da ke sama azaman grafana-virtual-service.yaml sannan a yi amfani da su:

kubectl apply -f ./grafana-virtual-service.yaml

Lokacin zuwa http://grafana.istio.example.com Mai binciken ku yakamata ya tura ku zuwa shafin shiga Grafana.

Ana tura aikace-aikacen yanar gizo tare da Flagger

Flagger yana tura Kubernetes kuma, idan ya cancanta, autoscaling autoscaling (HPA), sannan ya ƙirƙiri jerin abubuwa (aikin Kubernetes, sabis na ClusterIP da sabis na kama-da-wane na Istio). Waɗannan abubuwa suna fallasa aikace-aikacen zuwa ragar sabis kuma suna sarrafa bincike da haɓakawa.

Aiwatar da canary ta atomatik tare da Flagger da Istio

Ƙirƙirar filin suna na gwaji tare da kunna aikin Istio Sidecar:

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml

Ƙirƙirar turawa da kayan aiki a kwance ta atomatik don kwafsa:

kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml

Ƙaddamar da sabis na gwajin lodi don samar da zirga-zirga yayin nazarin canary:

helm upgrade -i flagger-loadtester flagger/loadtester 
--namepace=test

Ƙirƙiri albarkatun canary na al'ada (maye gurbin example.com zuwa yankinku):

apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 60
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: podinfo
  service:
    port: 9898
    gateways:
    - public-gateway.istio-system.svc.cluster.local
    hosts:
    - app.istio.example.com
  canaryAnalysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: istio_requests_total
      threshold: 99
      interval: 30s
    - name: istio_request_duration_seconds_bucket
      threshold: 500
      interval: 30s
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

Ajiye albarkatun da ke sama azaman podinfo-canary.yaml sannan a yi amfani da su:

kubectl apply -f ./podinfo-canary.yaml

Binciken da ke sama, idan ya yi nasara, zai yi aiki na mintuna biyar, yana duba ma'aunin HTTP kowane rabin minti. Kuna iya ƙayyade mafi ƙarancin lokacin da ake buƙata don gwadawa da haɓaka jigilar canary ta amfani da dabara mai zuwa: interval * (maxWeight / stepWeight). Ana rubuta filayen Canary CRD a nan.

Bayan daƙiƙa biyu, Flagger zai ƙirƙiri abubuwan canary:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo
# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
virtualservice.networking.istio.io/podinfo

Bude burauzar ku kuma je zuwa app.istio.example.com, ya kamata ku ga lambar sigar demo aikace-aikace.

Binciken Canary ta atomatik da haɓakawa

Flagger yana aiwatar da madauki na sarrafawa wanda sannu a hankali ke motsa zirga-zirga zuwa canary yayin da ake auna mahimman alamun aiki kamar ƙimar nasarar buƙatar HTTP, matsakaicin lokacin buƙatar buƙata, da lafiyar kwasfa. Dangane da binciken KPI, ana haɓaka ko ƙare canary, kuma ana buga sakamakon binciken a cikin Slack.

Aiwatar da canary ta atomatik tare da Flagger da Istio

Ana haifar da jigilar Canary lokacin da ɗayan abubuwa masu zuwa ya canza:

  • Sanya PodSpec (hoton kwantena, umarni, tashar jiragen ruwa, env, da sauransu)
  • An ɗora ConfigMaps azaman juzu'i ko an canza su zuwa masu canjin yanayi
  • Ana ɗora asirin a matsayin juzu'i ko canzawa zuwa masu canjin yanayi

Gudanar da tura canary lokacin da ake sabunta hoton akwati:

kubectl -n test set image deployment/podinfo 
podinfod=quay.io/stefanprodan/podinfo:1.4.1

Flagger ya gano cewa sigar turawa ta canza kuma ya fara bincikar ta:

kubectl -n test describe canary/podinfo

Events:

New revision detected podinfo.test
Scaling up podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test

Yayin bincike, ana iya lura da sakamakon canary ta amfani da Grafana:

Aiwatar da canary ta atomatik tare da Flagger da Istio

Lura: idan an yi amfani da sabbin canje-canje ga turawa yayin binciken canary, Flagger zai sake farawa lokacin bincike.

Yi lissafin duk kanari a cikin tarin ku:

watch kubectl get canaries --all-namespaces
NAMESPACE   NAME      STATUS        WEIGHT   LASTTRANSITIONTIME
test        podinfo   Progressing   15       2019-01-16T14:05:07Z
prod        frontend  Succeeded     0        2019-01-15T16:15:07Z
prod        backend   Failed        0        2019-01-14T17:05:07Z

Idan kun kunna sanarwar Slack, zaku karɓi saƙonni masu zuwa:

Aiwatar da canary ta atomatik tare da Flagger da Istio

Juyawa ta atomatik

A yayin binciken canary, zaku iya haifar da kurakuran HTTP 500 na roba da kuma babban jinkirin amsawa don bincika ko Flagger zai dakatar da turawa.

Ƙirƙiri fasfo ɗin gwaji kuma yi waɗannan abubuwa a ciki:

kubectl -n test run tester 
--image=quay.io/stefanprodan/podinfo:1.2.1 
-- ./podinfo --port=9898
kubectl -n test exec -it tester-xx-xx sh

Samar da kurakurai HTTP 500:

watch curl http://podinfo-canary:9898/status/500

Ƙirƙirar jinkiri:

watch curl http://podinfo-canary:9898/delay/1

Lokacin da adadin cak ɗin da ya gaza ya kai bakin kofa, ana korar zirga-zirgar ababen hawa zuwa tashar farko, ana daidaita canary zuwa sifili, kuma an yi alamar aikin da ya gaza.

Ana shigar da kurakurai Canary da latency spikes azaman abubuwan Kubernetes kuma Flagger ya rubuta su a tsarin JSON:

kubectl -n istio-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement success rate 47.00% < 99%
Halt podinfo.test advancement success rate 37.00% < 99%
Halt podinfo.test advancement request duration 1.515s > 500ms
Halt podinfo.test advancement request duration 1.600s > 500ms
Halt podinfo.test advancement request duration 1.915s > 500ms
Halt podinfo.test advancement request duration 2.050s > 500ms
Halt podinfo.test advancement request duration 2.515s > 500ms
Rolling back podinfo.test failed checks threshold reached 10
Canary failed! Scaling down podinfo.test

Idan kun kunna sanarwar Slack, zaku karɓi saƙo lokacin da ƙarshen ƙarshe don kammalawa ko cimma matsakaicin adadin faɗuwar bita a cikin bincike ya wuce:

Aiwatar da canary ta atomatik tare da Flagger da Istio

A ƙarshe

Gudanar da ragamar sabis kamar Istio a saman Kubernetes zai samar da ma'auni na atomatik, rajistan ayyukan, da rajistan ayyukan, amma ƙaddamar da kayan aiki har yanzu ya dogara da kayan aikin waje. Flagger yana nufin canza wannan ta ƙara iyawar Istio isar da ci gaba.

Flagger ya dace da kowane bayani na CI / CD don Kubernetes, kuma ana iya fadada binciken canary cikin sauƙi tare da gidan yanar gizo don yin gwaje-gwajen haɗakarwa/karɓar tsarin, gwaje-gwajen kaya ko duk wani gwaje-gwaje na al'ada. Saboda Flagger yana bayyanawa kuma yana amsa abubuwan da suka faru na Kubernetes, ana iya amfani da shi a cikin bututun GitOps tare da Saƙa Flux ko JenkinsX. Idan kana amfani da JenkinsX, za ka iya shigar da Flagger tare da jx add-ons.

Ana goyan bayan tuta Saƙa kuma yana ba da jigilar canary a cikin Saƙa Cloud. An gwada aikin akan GKE, EKS da kuma ƙarfe mara ƙarfi tare da kubeadm.

Idan kuna da shawarwari don inganta Flagger, da fatan za a ƙaddamar da batun ko PR akan GitHub a stefanprodan/flagger. Gudunmawa sun fi maraba!

Спасибо Ray Tsang.

source: www.habr.com

Add a comment