Automated canary deployments with Flagger and Istio

CD is recognized as a practice of enterprise software; it is the result of the natural evolution of established CI principles. However, CD remains a relatively rare phenomenon, possibly due to the complexity of management and fear of failed deployments affecting system availability.

Flagger is an open-source Kubernetes operator aimed at eliminating complicated relationships. It automates the promotion of canary deployments using Istio traffic shifting and Prometheus metrics to analyze application behavior during a controlled rollout.

Below is a step-by-step guide on setting up and using Flagger in Google Kubernetes Engine (GKE).

Setting Up the Kubernetes Cluster

You start by creating a GKE cluster with Istio add-ons (if you don't have a GCP account, you can sign up here for free credits).

Log in to Google Cloud, create a project, and enable billing for it. Install the command-line utility gcloud and set up your project using gcloud init.

Set your default project, compute region, and zone (replace PROJECT_ID with your project):

gcloud config set project PROJECT_ID
gcloud config set compute/region us-central1
gcloud config set compute/zone us-central1-a

Enable the GKE service and create a cluster with HPA and Istio add-ons:

gcloud services enable container.googleapis.com
K8S_VERSION=$(gcloud beta container get-server-config --format=json | jq -r '.validMasterVersions[0]')
gcloud beta container clusters create istio 
--cluster-version=${K8S_VERSION} 
--zone=us-central1-a 
--num-nodes=2 
--machine-type=n1-standard-2 
--disk-size=30 
--enable-autorepair 
--no-enable-cloud-logging 
--no-enable-cloud-monitoring 
--addons=HorizontalPodAutoscaling,Istio 
--istio-config=auth=MTLS_PERMISSIVE

The command above will create a default node pool consisting of two VMs n1-standard-2 (vCPU: 2, RAM 7.5 GB, Disk: 30 GB). Ideally, you should isolate Istio components from your workloads, but there is no straightforward way to run Istio pods in a dedicated node pool. Istio manifests are considered read-only, and GKE will revert any changes, such as node binding or pod detachment.

Configure credentials for kubectl:

gcloud container clusters get-credentials istio

Create a cluster admin role binding:

kubectl create clusterrolebinding "cluster-admin-$(whoami)" 
--clusterrole=cluster-admin 
--user="$(gcloud config get-value core/account)"

Install the command-line tool Helm:

brew install kubernetes-helm

Homebrew 2.0 is now also available for Linux.

Create a service account and cluster role binding for Tiller:

kubectl -n kube-system create sa tiller && 
kubectl create clusterrolebinding tiller-cluster-rule 
--clusterrole=cluster-admin 
--serviceaccount=kube-system:tiller

Deploy Tiller in the namespace kube-system:

helm init --service-account tiller

You should consider using SSL between Helm and Tiller. For more information on securing your Helm installation, see docs.helm.sh

Confirm the settings:

kubectl -n istio-system get svc

Within a few seconds, GCP should assign an external IP address to the service istio-ingressgateway.

Setting up Istio Ingress Gateway

Create a static IP address named istio-gateway, using the Istio gateway IP address:

export GATEWAY_IP=$(kubectl -n istio-system get svc/istio-ingressgateway -ojson | jq -r .status.loadBalancer.ingress[0].ip)
gcloud compute addresses create istio-gateway --addresses ${GATEWAY_IP} --region us-central1

Now you need an internet domain and access to your DNS registrar. Add two A records (replace example.com with your domain):

istio.example.com   A ${GATEWAY_IP}
*.istio.example.com A ${GATEWAY_IP}

Ensure that the wildcard DNS is working:

watch host test.istio.example.com

Create a public Istio gateway to provide services outside the service mesh over HTTP:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Save the above resource as public-gateway.yaml and then apply it:

kubectl apply -f ./public-gateway.yaml

No production system should expose services to the internet without SSL. To secure the Istio ingress gateway with cert-manager, CloudDNS, and Let’s Encrypt, please read documentation Flagger GKE.

Installing Flagger

The GKE Istio add-on does not include a Prometheus instance, which handles the telemetry service cleanup for Istio. Since Flagger uses Istio HTTP metrics for performing canary analysis, you need to deploy the following Prometheus configuration, similar to the one provided with the official Istio Helm chart.

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/gke/istio-prometheus.yaml

Add the Flagger Helm repository:

helm repo add flagger [https://flagger.app](https://flagger.app/)

Deploy Flagger in the namespace istio-system, including Slack notifications:

helm upgrade -i flagger flagger/flagger 
--namespace=istio-system 
--set metricsServer=http://prometheus.istio-system:9090 
--set slack.url=https://hooks.slack.com/services/YOUR-WEBHOOK-ID 
--set slack.channel=general 
--set slack.user=flagger

You can install Flagger in any namespace as long as it can communicate with the Istio Prometheus service over port 9090.

Flagger has a Grafana dashboard for canary analysis. Install Grafana in the namespace istio-system:

helm upgrade -i flagger-grafana flagger/grafana 
--namespace=istio-system 
--set url=http://prometheus.istio-system:9090 
--set user=admin 
--set password=change-me

Expose Grafana through the open gateway by creating a virtual service (replace example.com with your domain):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
  namespace: istio-system
spec:
  hosts:
    - "grafana.istio.example.com"
  gateways:
    - public-gateway.istio-system.svc.cluster.local
  http:
    - route:
        - destination:
            host: flagger-grafana

Save the above resource as grafana-virtual-service.yaml and then apply it:

kubectl apply -f ./grafana-virtual-service.yaml

Navigating to http://grafana.istio.example.com in your browser should direct you to the Grafana login page.

Deploy Web Applications with Flagger

Flagger deploys Kubernetes and, if necessary, horizontal pod autoscaling (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services, and Istio virtual services). These objects expose the application in the service mesh and manage canary analysis and promotion.

Create a test namespace with Istio sidecar injection enabled:

REPO=https://raw.githubusercontent.com/stefanprodan/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml

Create a deployment and a horizontal pod autoscaler resource:

kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml

Deploy a load test service to generate traffic during the canary analysis:

helm upgrade -i flagger-loadtester flagger/loadtester 
--namespace=test

Create a custom canary resource (replace example.com with your domain):

apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 60
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name: podinfo
  service:
    port: 9898
    gateways:
    - public-gateway.istio-system.svc.cluster.local
    hosts:
    - app.istio.example.com
  canaryAnalysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: istio_requests_total
      threshold: 99
      interval: 30s
    - name: istio_request_duration_seconds_bucket
      threshold: 500
      interval: 30s
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

The above analysis, if successful, will run for five minutes, checking HTTP metrics every half a minute. You can determine the minimum time required to verify and promote the canary deployment using the following formula: interval * (maxWeight / stepWeight). The fields of the Canary CRD are documented here.

In a few seconds, Flagger will create canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo
# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
virtualservice.networking.istio.io/podinfo

Open your browser and navigate to app.istio.example.com, you should see the version number of the demo application.

Automated canary analysis and promotion

Flagger implements a management cycle that gradually shifts traffic to the canary while measuring key performance indicators such as HTTP request success rate, average request duration, and pod availability. Based on the KPI analysis, the canary is promoted or stopped, and the analysis results are published to Slack.

Canary deployment is triggered when any of the following objects are updated:

• Deployment PodSpec (container image, command, ports, env, etc.)
• ConfigMaps are mounted as volumes or transformed into environment variables
• Secrets are mounted as volumes or transformed into environment variables

Triggering canary deployment when the container image is updated:

kubectl -n test set image deployment/podinfo 
podinfod=quay.io/stefanprodan/podinfo:1.4.1

Flagger detects that the deployment version has changed and starts analyzing it:

kubectl -n test describe canary/podinfo

Events:

New revision detected podinfo.test
Scaling up podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test

During the analysis, the canary results can be monitored using Grafana:

Note: if new changes are applied to the deployment during canary analysis, Flagger will restart the analysis phase.

List all the ‘canaries’ in your cluster:

watch kubectl get canaries --all-namespaces
NAMESPACE   NAME      STATUS        WEIGHT   LASTTRANSITIONTIME
test        podinfo   Progressing   15       2019-01-16T14:05:07Z
prod        frontend  Succeeded     0        2019-01-15T16:15:07Z
prod        backend   Failed        0        2019-01-14T17:05:07Z

If you have enabled Slack notifications, you will receive the following messages:

Automatic rollback

During canary analysis, you can generate synthetic HTTP 500 errors and high response latency to check if Flagger will halt the deployment.

Create a test pod and run the following command in it:

kubectl -n test run tester 
--image=quay.io/stefanprodan/podinfo:1.2.1 
-- ./podinfo --port=9898
kubectl -n test exec -it tester-xx-xx sh

Generating HTTP 500 errors:

watch curl http://podinfo-canary:9898/status/500

Generating latency:

watch curl http://podinfo-canary:9898/delay/1

When the number of failed checks reaches the threshold, traffic is redirected back to the primary channel, the canary scales down to zero, and the deployment is marked as failed.

Canary errors and latency spikes are logged as Kubernetes events and recorded by Flagger in JSON format:

kubectl -n istio-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement success rate 47.00% < 99%
Halt podinfo.test advancement success rate 37.00%  500ms
Halt podinfo.test advancement request duration 1.600s > 500ms
Halt podinfo.test advancement request duration 1.915s > 500ms
Halt podinfo.test advancement request duration 2.050s > 500ms
Halt podinfo.test advancement request duration 2.515s > 500ms
Rolling back podinfo.test failed checks threshold reached 10
Canary failed! Scaling down podinfo.test

If you have Slack notifications enabled, you will receive a message when the execution deadline is exceeded or the maximum number of failed checks is reached during analysis:

In conclusion

Launching a service mesh, such as Istio, alongside Kubernetes provides automatic metrics, logs, and traces, but deployment of workloads still relies on external tools. Flagger aims to change this situation by adding Istio capabilities for progressive delivery.

Flagger is compatible with any CI/CD solution for Kubernetes, and canary analysis can easily be extended with webhooks to execute integration/acceptance tests, load tests, or any other custom checks. Since Flagger is declarative and responds to Kubernetes events, it can be used in GitOps pipelines along with Weave Flux or JenkinsX. If you are using JenkinsX, you can install Flagger with jx addons.

Flagger is supported by Weaveworks and enables canary deployments in Weave Cloud. The project is tested on GKE, EKS, and bare metal with kubeadm.

If you have suggestions for improving Flagger, please submit an issue or PR on GitHub at stefanprodan/flagger. Contributions are more than welcome!

Thank you Ray Tsan.

Source: habr.com