🥇Kubernetes deployment strategies: rolling, recreate, blue/green, canary, dark (A/B testing)

Note. translation: This walkthrough from Weaveworks introduces the most popular application rollout strategies and how you can implement the most advanced ones with the Flagger Kubernetes operator. It is written in simple language and contains visual diagrams that allow even novice engineers to understand the issue.

Diagram taken from another review rollout strategies made in Container Solutions

One of the biggest challenges in developing cloud native applications today is deployment speed. With a microservice approach, developers are already working with and designing fully modular applications, allowing different teams to write code and make changes to the application at the same time.

Shorter and more frequent deployments have the following benefits:

Reduced time to market.
New features get to users faster.
User feedback reaches the development team faster. This means the team can add features and fix issues more quickly.
Improves developer morale: More features in development are more fun to work with.

But as release frequency increases, the chances of negatively impacting app reliability or user experience also increase. That is why it is important for operations and DevOps teams to build processes and manage deployment strategies in a way that minimizes risk to the product and users. (To learn more about automating the CI/CD pipeline, see here.)

In this post, we will discuss various Kubernetes deployment strategies, including rolling deployments and more advanced methods such as canary rollouts and their variations.

Deployment strategies

There are several different types of deployment strategies that can be used depending on the purpose. For example, you may need to make changes to some environment for further testing, or to a subset of users/clients, or you may need to do limited testing on users before making a feature publicly available.

Rolling (gradual, rolling deployment)

This is the standard deployment strategy for Kubernetes. It gradually, one by one, replaces the pods with the old version of the application with pods with the new version - without downtime of the cluster.

Kubernetes waits for new pods to be ready to go (checking them with readiness tests) before proceeding to roll up the old ones. If a problem occurs, such a rolling update can be aborted without stopping the entire cluster. In the deployment type YAML file, the new image replaces the old image:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: awesomeapp
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: awesomeapp
    spec:
      containers:
        - name: awesomeapp
          image: imagerepo-user/awesomeapp:new
          ports:
            - containerPort: 8080

The parameters of the rolling update can be specified in the manifest file:

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
       maxSurge: 25%
       maxUnavailable: 25%  
  template:
  ...

Recreate (re-creation)

In this simplest type of deployment, old pods are killed all at once and replaced with new ones:

The corresponding manifest looks something like this:

spec:
  replicas: 3
  strategy:
    type: Recreate
  template:
  ...

Blue/Green (blue-green deployments)

The blue-green deployment strategy (sometimes also referred to as red / black, i.e. red-black) involves the simultaneous deployment of the old (green) and new (blue) versions of the application. Once both versions are hosted, the green one is available to the general user, while the blue one is available for the QA team to automate tests via a separate service or direct port forwarding:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: awesomeapp-02
spec:
  template:
    metadata:
      labels:
        app: awesomeapp
        version: "02"

After the blue (new) version has been tested and approved for release, the service is switched to it, and the green (old) is minimized:

apiVersion: v1
kind: Service
metadata:
  name: awesomeapp
spec:
  selector:
    app: awesomeapp
    version: "02"
...

Canary (canary deployments)

Canary rollouts are similar to blue-green, but better controlled and used progressive step by step approach. Several different strategies fall into this category, including covert launches and A/B testing.

This strategy is used when some new functionality needs to be tested, usually in the application's backend. The essence of the approach is to create two almost identical servers: one serves almost all users, and the other, with new features, serves only a small subgroup of users, after which the results of their work are compared. If everything goes without errors, the new version is gradually rolled out to the entire infrastructure.

Although this strategy can be implemented exclusively using Kubernetes, replacing old pods with new ones, it is much more convenient and easier to use a service mesh like Istio.

For example, you might have two different Git manifests: a regular one with a 0.1.0 tag and a canary one with a 0.2.0 tag. By changing the weights in the Istio Virtual Gateway manifest, you can control the distribution of traffic between these two deployments:

A step-by-step guide to implementing canary deployments with Istio can be found in GitOps Workflows with Istio. (Note. transl.: We also translated material about canary rolls in Istio here.)

Canary deployments with Weaveworks Flagger

Weaveworks Flagger allows easy and efficient control of canary rolls.

Flagger automates working with them. It uses Istio or AWS App Mesh to route and switch traffic, and Prometheus metrics to analyze the results. In addition, the analysis of canary deployments can be supplemented with webhooks for conducting acceptance tests, load tests, and any other types of checks.

Based on the deployment of Kubernetes and, if necessary, scale-out pods (HPA), Flagger creates sets of objects (Kubernetes deployments, ClusterIP services and Istio or App Mesh virtual services) to analyze and implement canary deployments:

Implementing a control loop (control loop)Flagger gradually switches traffic to the canary server while measuring key performance metrics such as HTTP request success rate, average request duration, and pod health. Based on the analysis of KPIs (Key Performance Indicators), the canary part either grows or shrinks, and the results of the analysis are published to Slack. A description and demonstration of this process can be found in the material Progressive Delivery for App Mesh.

Dark (hidden) or A/B deployments

Stealth deployment is another variation of the canary strategy (which, by the way, Flagger can also work with). The difference between stealth and canary deployments is that stealth deployments deal with the front end and not the backend like canary deployments do.

Another name for these deployments is A/B testing. Instead of opening access to a new feature to all users, it is offered only to a limited part of them. Typically, these users are unaware that they are pioneer testers (hence the term "silent deployment").

With function switches (feature toggle) and other tools, you can monitor how users interact with a new feature, whether they are addicted to it or find the new user interface confusing, and other types of metrics.

Flagger and A/B Deployments

In addition to weighted routing, Flagger can also route traffic to the canary server based on HTTP parameters. A/B testing can use HTTP headers or cookies to redirect a specific segment of users. This is especially effective in the case of frontend applications that require the session to be bound to the server. (session affinity). More information can be found in the Flagger documentation.

The author is grateful Stefan Prodan, engineer of Weaveworks (and creator of Flagger), for all these amazing deployment schemes.

Kubernetes deployment strategies: rolling, recreate, blue/green, canary, dark (A/B testing)