ProHoster > Блог > Whakahaerenga > Ko nga whakangao kanari aunoa me te Flagger me te Istio

Ko nga whakangao kanari aunoa me te Flagger me te Istio

Ko nga whakangao kanari aunoa me te Flagger me te Istio

E mohiotia ana te CD hei mahinga rorohiko hinonga, a, ko te hua o te kukuwhatanga maori o nga maataapono CI kua whakapumautia. Heoi, he onge tonu te CD, na te uaua o te whakahaere me te mataku kei rahua te tukunga ka pa ki te waatea o te punaha.

Haki he puna tuwhera Kubernetes kaiwhakahaere e whai ana ki te whakakore i nga hononga rangirua. Ka whakahaere aunoa i te whakatairanga o nga tukunga canary ma te whakamahi Istio traffic offset me Prometheus metrics ki te tātari i te whanonga tono i te wa e whakahaeretia ana te whakaputanga.

Kei raro nei he aratohu taahiraa-i-taahiraa ki te whakarite me te whakamahi i te Kara i runga i te Google Kubernetes Engine (GKE).

Te whakatu i te roopu Kubernetes

Ka timata koe ma te hanga i te roopu GKE me te taapiri Istio (mehemea karekau he putea GCP, ka taea e koe te haina konei - ki te whiwhi nama kore utu).

Waitohu ki a Google Cloud, hanga he kaupapa, ka taea te utu nama. Tāutahia te whaipainga raina whakahau gcloud me te whakarite i to kaupapa me gcloud init.

Tautuhia te kaupapa taunoa, te horahanga tatau me te rohe (whakakapi PROJECT_ID mo to kaupapa):

gcloud config set project PROJECT_ID
gcloud config set compute/region us-central1
gcloud config set compute/zone us-central1-a

Whakahohehia te ratonga GKE ka waihangahia he huinga me nga taapiri HPA me Istio:

gcloud services enable container.googleapis.com
K8S_VERSION=$(gcloud beta container get-server-config --format=json | jq -r '.validMasterVersions[0]')
gcloud beta container clusters create istio 
--cluster-version=${K8S_VERSION} 
--zone=us-central1-a 
--num-nodes=2 
--machine-type=n1-standard-2 
--disk-size=30 
--enable-autorepair 
--no-enable-cloud-logging 
--no-enable-cloud-monitoring 
--addons=HorizontalPodAutoscaling,Istio 
--istio-config=auth=MTLS_PERMISSIVE

Ko te whakahau i runga ake nei ka waihangahia he poka wai node taunoa tae atu ki nga VM e rua n1-standard-2 (vCPU: 2, RAM 7,5 GB, kōpae: 30 GB). Ko te mea pai, me wehe koe i nga waahanga Istio mai i o taumahatanga mahi, engari kaore he huarahi ngawari ki te whakahaere Istio Pods i roto i te puna kaukau kua whakatapua. Ko nga whakaaturanga Istio e kiia ana he panui-anake ka whakakorehia e te GKE nga huringa, penei i te hono ki te node, te wehe mai ranei i te putunga.

Whakaritehia nga tohu mo kubectl:

gcloud container clusters get-credentials istio

Waihangahia he roopu kaiwhakahaere roopu here:

kubectl create clusterrolebinding "cluster-admin-$(whoami)" 
--clusterrole=cluster-admin 
--user="$(gcloud config get-value core/account)"

Tāutahia te taputapu raina whakahau Helm:

brew install kubernetes-helm

Kei te waatea ano te Homebrew 2.0 mo Linux.

Waihangahia he kaute ratonga me te roopu roopu here mo Tiller:

kubectl -n kube-system create sa tiller && 
kubectl create clusterrolebinding tiller-cluster-rule 
--clusterrole=cluster-admin 
--serviceaccount=kube-system:tiller

Roha Tiller ki te mokowāingoa kube-system:

helm init --service-account tiller

Me whakaaro koe ki te whakamahi SSL i waenga i te Helm me te Tiller. Mo etahi atu korero mo te tiaki i to whakaurunga Helm, tirohia docs.helm.sh

Whakaū tautuhinga:

kubectl -n istio-system get svc

Whai muri i etahi hēkona, me tautapa e GCP he wāhitau IP waho mo te ratonga istio-ingressgateway.

Te whirihora i te Istio Ingress Gateway

Waihangatia he wahitau IP pateko me te ingoa istio-gatewayte whakamahi i te wahitau IP o te kuaha Istio:

export GATEWAY_IP=$(kubectl -n istio-system get svc/istio-ingressgateway -ojson | jq -r .status.loadBalancer.ingress[0].ip)
gcloud compute addresses create istio-gateway --addresses ${GATEWAY_IP} --region us-central1

Inaianei kei te hiahia koe ki tetahi rohe ipurangi me te uru ki to rehita DNS. Tāpirihia kia rua nga rekoata A (whakakapi example.com ki to rohe):

istio.example.com   A ${GATEWAY_IP}
*.istio.example.com A ${GATEWAY_IP}

Manatokohia kei te mahi te kaari mohoao DNS:

watch host test.istio.example.com

Waihangahia he kuaha Istio hei whakarato ratonga i waho o te mata ratonga i runga HTTP:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Tiakina te rauemi i runga ake hei public-gateway.yaml katahi ka whakamahi:

kubectl apply -f ./public-gateway.yaml

Kaore he punaha whakaputa e whakarato ratonga i runga i te Ipurangi me te kore SSL. Hei whakapumau i te kuaha Istio ingress me te kaiwhakahaere-tohu, CloudDNS me Let's Encrypt, panui koa tuhinga Kara G.K.E.

Tāutanga Kara

Ko te taapiri GKE Istio kaore he tauira Prometheus e horoi ana i te ratonga waea a Istio. Na te mea ka whakamahi a Flagger i nga inenga HTTP Istio ki te mahi tātaritanga canary, me tuku e koe te whirihoranga Prometheus e whai ake nei, he rite ki te mea ka tae mai me te kaupapa Istio Helm whaimana.

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/gke/istio-prometheus.yaml

Tāpirihia te putunga Helm Flagger:

helm repo add flagger [https://flagger.app](https://flagger.app/)

Whakaroha Kara ki te mokowāingoa istio-systemmā te whakahohe i ngā whakamōhiotanga Slack:

helm upgrade -i flagger flagger/flagger 
--namespace=istio-system 
--set metricsServer=http://prometheus.istio-system:9090 
--set slack.url=https://hooks.slack.com/services/YOUR-WEBHOOK-ID 
--set slack.channel=general 
--set slack.user=flagger

Ka taea e koe te whakauru i te Flagger ki tetahi waahi ingoa mena ka taea e ia te korero ki te ratonga Istio Prometheus i runga i te tauranga 9090.

Kei a Flagger he papatohu Grafana mo te tātari kanari. Tāutahia a Grafana ki te mokowāingoa istio-system:

helm upgrade -i flagger-grafana flagger/grafana 
--namespace=istio-system 
--set url=http://prometheus.istio-system:9090 
--set user=admin 
--set password=change-me

Whakaatuhia a Grafana ma te kuaha tuwhera ma te hanga i tetahi ratonga mariko (whakakapi example.com ki to rohe):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
  namespace: istio-system
spec:
  hosts:
    - "grafana.istio.example.com"
  gateways:
    - public-gateway.istio-system.svc.cluster.local
  http:
    - route:
        - destination:
            host: flagger-grafana

Tiakina te rauemi i runga ake nei hei grafana-virtual-service.yaml ka whakamahi:

kubectl apply -f ./grafana-virtual-service.yaml

Ina neke ki http://grafana.istio.example.com i roto i te kaitirotiro, me tuku koe ki te wharangi takiuru a Grafana.

Te tuku tono tukutuku me te Flagger

Ka horahia e Flagger nga Kubernetes me te kowhiringa ka whakatauhia aunoatia (HPA), katahi ka hanga i te raupapa o nga mea (nga tukunga Kubernetes, ratonga ClusterIP, me nga ratonga mariko Istio). Ko enei taonga e whakaatu ana i te tono ki te mata ratonga me te whakahaere i te tātaritanga me te ahunga whakamua.

Ko nga whakangao kanari aunoa me te Flagger me te Istio

Waihangahia he mokowāingoa whakamātautau me te werohanga a Istio Sidecar kua whakahohea:

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml

Waihangahia he whakangao me tetahi taputapu tauine-aunoa pod:

kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml

Tukuna he ratonga uta whakamatautau ki te whakaputa waka i te wa o te tātaritanga kanary:

helm upgrade -i flagger-loadtester flagger/loadtester 
--namepace=test

Waihangahia he rauemi canary ritenga (whakakapi example.com ki to rohe):

apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 60
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: podinfo
  service:
    port: 9898
    gateways:
    - public-gateway.istio-system.svc.cluster.local
    hosts:
    - app.istio.example.com
  canaryAnalysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: istio_requests_total
      threshold: 99
      interval: 30s
    - name: istio_request_duration_seconds_bucket
      threshold: 500
      interval: 30s
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

Tiakina te rauemi i runga ake nei hei podinfo-canary.yaml ka whakamahi:

kubectl apply -f ./podinfo-canary.yaml

Ko te tātaritanga i runga ake nei, ki te angitu, ka haere mo te rima meneti, ka tirohia nga inenga HTTP ia hawhe meneti. Ka taea e koe te whakatau i te wa iti e hiahiatia ana ki te whakamana me te whakatairanga i te tukunga o te canary ma te whakamahi i te tauira e whai ake nei: interval * (maxWeight / stepWeight). Kua tuhia nga mara CRD Canary konei.

Whai muri i nga hēkona e rua, ka waihangahia e Flagger nga ahanoa canary:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo
# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
virtualservice.networking.istio.io/podinfo

Whakatuwheratia he kaitirotiro ka haere ki app.istio.example.com, me kite koe i te tau putanga taupānga demo.

Te tātari kanary aunoa me te whakatairanga

Ka whakatinanahia e Flagger he kohanga mana e neke haere ana nga waka ki te kanari i te ine i nga inenga mahi matua penei i te reeti angitu o te tono HTTP, te roanga roa o te tono, me te hauora pod. I runga i te tātaritanga KPI, ka whakatairangahia, ka haukotia ranei te canary, ka whakaputaina nga hua o te tātaritanga ki a Slack.

Ko nga whakangao kanari aunoa me te Flagger me te Istio

Ka puta te tuku Canary ina huri tetahi o nga mea e whai ake nei:

  • Hoatuhia te PodSpec (whakaahua ipu, whakahau, tauranga, env, etc.)
  • Ko nga ConfigMaps ka whakairihia hei pukapuka, ka mapi ranei ki nga taurangi taiao
  • Ko nga mea ngaro ka whakairihia hei pukapuka, ka huri ranei ki nga taurangi taiao

Whakahaerehia te whakangao canary ina whakahōu ana i te atahanga ipu:

kubectl -n test set image deployment/podinfo 
podinfod=quay.io/stefanprodan/podinfo:1.4.1

Ka kitea e Flagger kua huri te putanga tukunga ka timata te poroporoaki:

kubectl -n test describe canary/podinfo

Events:

New revision detected podinfo.test
Scaling up podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test

I te wa o te tātaritanga, ka taea te whai i nga hua canary ma te whakamahi i te Grafana:

Ko nga whakangao kanari aunoa me te Flagger me te Istio

Kia mahara mai mena ka tukuna nga huringa hou ki te whakatakotoranga i te wa o te tātaritanga kanari, katahi ka timata ano a Flagger i te wahanga tātari.

Hangaia he rarangi o nga canaries katoa i roto i to huinga:

watch kubectl get canaries --all-namespaces
NAMESPACE   NAME      STATUS        WEIGHT   LASTTRANSITIONTIME
test        podinfo   Progressing   15       2019-01-16T14:05:07Z
prod        frontend  Succeeded     0        2019-01-15T16:15:07Z
prod        backend   Failed        0        2019-01-14T17:05:07Z

Mena kua whakahohea e koe nga whakamohiotanga Slack, ka whiwhi koe i nga panui e whai ake nei:

Ko nga whakangao kanari aunoa me te Flagger me te Istio

Whakahoki Aunoa

I te wa o te tātaritanga canary, ka taea e koe te whakaputa i nga hapa HTTP 500 waihanga me te roanga whakautu nui kia kite mena ka aukati a Flagger i te tuku.

Waihangahia he putea whakamatautau ka mahi i nga mea e whai ake nei:

kubectl -n test run tester 
--image=quay.io/stefanprodan/podinfo:1.2.1 
-- ./podinfo --port=9898
kubectl -n test exec -it tester-xx-xx sh

Te whakaputa hapa HTTP 500:

watch curl http://podinfo-canary:9898/status/500

Whakaputa whakaroa:

watch curl http://podinfo-canary:9898/delay/1

Ka tae te maha o nga arowhai i rahua ki te paepae, ka hoki te waka ki te hongere tuatahi, ka whakahekehia te kanari ki te kore, ka tohua te tukunga kua rahua.

Ko nga hapa o te Canary me nga koikoi torohe ka tuhia hei huihuinga Kubernetes ka tuhia e Flagger ki te whakatakotoranga JSON:

kubectl -n istio-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement success rate 47.00% < 99%
Halt podinfo.test advancement success rate 37.00% < 99%
Halt podinfo.test advancement request duration 1.515s > 500ms
Halt podinfo.test advancement request duration 1.600s > 500ms
Halt podinfo.test advancement request duration 1.915s > 500ms
Halt podinfo.test advancement request duration 2.050s > 500ms
Halt podinfo.test advancement request duration 2.515s > 500ms
Rolling back podinfo.test failed checks threshold reached 10
Canary failed! Scaling down podinfo.test

Mena kua whakahohea e koe nga whakamohiotanga Slack, ka whiwhi koe i te panui ka eke ki te wa mutunga ka eke ranei te maha o nga arowhai rahua i roto i te tātaritanga:

Ko nga whakangao kanari aunoa me te Flagger me te Istio

I te mutunga

Ko te whakahaere i tetahi mata ratonga penei i a Istio hei taapiri atu ki a Kubernetes ka whakarato i nga inenga aunoa, nga raarangi me nga kawa, engari kei te whakawhirinaki tonu te tuku mahi ki nga taputapu o waho. Ko te whai a Flagger ki te whakarereke i tenei ma te taapiri i nga kaha o Istio tuku whakamua.

He hototahi a Flagger ki tetahi otinga Kubernetes CI/CD, a ka ngawari te whakaroa ki te tātari canary matau tukutuku ki te mahi i nga whakamatautau whakauru/whakaaetanga punaha, nga whakamatautau uta, etahi atu arowhai ritenga ranei. I te mea he korero a Flagger me te whakautu ki nga huihuinga Kubernetes, ka taea te whakamahi ki nga raina paipa GitOps me Weave Flux ranei JenkinsX. Mena kei te whakamahi koe i a JenkinsX ka taea e koe te whakauru i te Flagger me nga taapiri jx.

Ka tautokohia te kara Nga mahi raranga me te whakarato i nga whakatakotoranga kanari ki roto Raranga Kapua. Kei te whakamatautauhia te kaupapa i runga i te GKE, EKS, me te whakarewa ma te kubeadm.

Mena kei a koe etahi whakaaro hei whakapai ake i a Flagger, tukuna mai he take, PR ranei ki GitHub i stefanprodan / haki. He nui noa atu nga takoha!

Tuhinga Ray Tsang.

Source: will.com

Tāpiri i te kōrero