🥇Deploying Applications Across Multiple Kubernetes Clusters with Helm

How Dailymotion Uses Kubernetes: Deploying Applications

At Dailymotion, we started using Kubernetes in production three years ago. However, deploying applications across multiple clusters has been quite a challenge, so over the past few years, we have been working to improve our tools and workflows.

How It Started

Here, we will explain how we deploy our applications across multiple Kubernetes clusters worldwide.

To deploy multiple Kubernetes objects at once, we use Helm, and all our charts are stored in a single git repository. To deploy a full stack of applications consisting of multiple services, we utilize a so-called umbrella chart. Essentially, this is a chart that declares dependencies and allows for the initialization of the API and its services with a single command.

Additionally, we wrote a small Python script on top of Helm to handle checks, create charts, add secrets, and deploy applications. All of these tasks are executed on a central CI platform using a Docker image.

Let's get to the point.

Note: By the time you read this, the first release candidate for Helm 3 has already been announced. The main version includes a whole set of enhancements aimed at resolving some issues we faced in the past.

Chart Development Workflow

For applications, we use branching, and we decided to apply the same approach to charts.

The branch dev is used to create charts that will be tested on development clusters.
When a pull request is submitted to master, they are validated in staging.
Finally, we create a pull request to merge changes into the branch prod and apply them in production.

Each environment has its own private repository that stores our charts, and we use Chartmuseum with very useful APIs. This ensures strict isolation between environments and the validation of charts in real conditions before using them in production.

Chart Repositories in Different Environments

It is worth noting that when developers push to the dev branch, their chart version is automatically sent to the dev Chartmuseum. This way, all developers use a single dev repository, and it is crucial to specify your chart version carefully to avoid unintentionally using someone else's changes.

Moreover, our small Python script checks Kubernetes objects against the Kubernetes OpenAPI specifications using Kubeval, before publishing them to Chartmuseum.

Overview of the chart development workflow

Configuring pipeline tasks according to specifications gazr.io for quality control (lint, unit-test).
Sending the Docker image with Python tools that deploy our applications.
Setting up the environment by branch name.
Validating Kubernetes yaml files with Kubeval.
Automatically incrementing the version of the chart and its parent charts (charts that depend on the changing chart).
Sending the chart to Chartmuseum corresponding to its environment

Managing differences in clusters

Cluster federation

There was a time when we used Kubernetes cluster federation, where Kubernetes objects could be declared from a single API endpoint. But issues arose. For example, some Kubernetes objects could not be created at the federation endpoint, making it difficult to manage unioned objects and others for individual clusters.

To solve the issue, we began managing clusters independently, significantly simplifying the process (we used the first version of federation; something might have changed in the second).

Geo-distributed platform

Currently, our platform is distributed across 6 regions — 3 locally and 3 in the cloud.

Distributed deployment

Global Helm values

4 global Helm values allow defining differences between clusters. For all our charts, there are minimum default values.

global:
  cloud: True
  env: staging
  region: us-central1
  clusterName: staging-us-central1

Global values

These values help define the context for our applications and are used for different tasks: monitoring, tracing, logging, making external calls, scaling, etc.

"cloud": we have a hybrid Kubernetes platform. For example, our API is deployed in GCP zones and in our data centers.
"env": some values may change for non-production environments. For instance, resource definitions and autoscaling configurations.
"region": this information helps determine the location of the cluster and can be used to identify the nearest endpoints for external services.
"clusterName": if and when we want to define a value for an individual cluster.

Here is a specific example:

{{
/* Returns Horizontal Pod Autoscaler replicas for GraphQL*/}}
{{- define "graphql.hpaReplicas" -}}
{{- if eq .Values.global.env "prod" }}
{{- if eq .Values.global.region "europe-west1" }}
minReplicas: 40
{{- else }}
minReplicas: 150
{{- end }}
maxReplicas: 1400
{{- else }}
minReplicas: 4
maxReplicas: 20
{{- end }}
{{- end -}}

Example of a Helm template

This logic is defined in a helper template to avoid cluttering Kubernetes YAML.

Application declaration

Our deployment tools are based on several YAML files. Below is an example of how we declare a service and its scaling topology (number of replicas) in the cluster.

releases:
  - foo.world

foo.world:                # Release name
  services:               # List of dailymotion's apps/projects
    foobar:
      chart_name: foo-foobar
      repo: git@github.com:dailymotion/foobar
      contexts:
        prod-europe-west1:
          deployments:
            - name: foo-bar-baz
              replicas: 18
            - name: another-deployment
              replicas: 3

Service definition

This is a diagram of all the steps that define our deployment workflow. The last step deploys the application simultaneously across multiple working clusters.

Deployment steps in Jenkins

And what about secrets?

Regarding security, we track all secrets from various sources and store them in a unique repository Vault in Paris.

Our deployment tools extract secret values from Vault, and when it’s time to deploy, they insert them into Helm.

For this, we defined a mapping between secrets in Vault and the secrets required by our applications:

secrets:                                                                                                                                                                                                        
     - secret_id: "stack1-app1-password"                                                                                                                                                                                  
       contexts:                                                                                                                                                                                                   
         - name: "default"                                                                                                                                                                                         
           vaultPath: "\/kv\/dev\/stack1\/app1\/test"                                                                                                                                                               
           vaultKey: "password"                                                                                                                                                                                    
         - name: "cluster1"                                                                                                                                                                           
           vaultPath: "\/kv\/dev\/stack1\/app1\/test"                                                                                                                                                               
           vaultKey: "password"

We have established general rules that must be followed when recording secrets in the Vault.
If the secret pertains to a specific context or cluster, a specific entry needs to be added. (In this case, the context cluster1 has a distinct value for the secret stack-app1-password).
Otherwise, the default.
value is used. For each item in this list, a Kubernetes secret

is inserted as a key-value pair. Therefore, the secret template in our charts is very straightforward.

Issues and Limitations

Working with Multiple Repositories

Currently, we separate the development of charts and applications. This means that developers have to work in two git repositories: one for the application and another for defining its deployment in Kubernetes. Two git repositories mean two workflows, and it can be easy for newcomers to get confused.

Managing generic charts is cumbersome.

As mentioned, generic charts are very convenient for defining dependencies and quickly deploying multiple applications. However, we use --reuse-values, to avoid passing all values each time we deploy an application that is part of this generic chart.

In the continuous delivery workflow, we have only two values that change regularly: the number of replicas and the image tag (version). Other, more stable values are changed manually, which is quite complex. Moreover, a single error in deploying a generalized chart can lead to serious failures, as we have seen from our own experience.

Updating several configuration files

When a developer adds a new application, they have to modify several files: the application declaration, the secrets list, and add the application to dependencies if it is included in the generalized chart.

Jenkins permissions are overly broad in Vault

Currently, we have one AppRole, which reads all secrets from Vault.

The rollback process is not automated

To roll back, a command needs to be executed on several clusters, which is prone to errors. We perform this operation manually to ensure the correct version identifier is specified.

We are moving towards GitOps

Our goal

We want to return the chart to the repository of the application it deploys.

The workflow will be the same as for development. For example, when a branch is pushed to master, deployment will start automatically. The main difference between this approach and the current workflow will be that everything will be managed in git (the application itself and the method of its deployment in Kubernetes).

There are several advantages:

Much clearer for the developer. It's easier to learn how to apply changes in the local chart.
The service deployment definition can be specified right where the code of the service is.
Managing the deletion of generalized charts. The service will have its own Helm release. This will allow for managing the application lifecycle (rollback, upgrade) at the finest level, without affecting other services.
The advantages of git for managing charts: undo changes, audit log, etc. If you need to revert a chart change, this can be done using git. The deployment starts automatically.
One can think about enhancing the development workflow using tools like Skaffold, which allows developers to test changes in an environment similar to production.

Two-stage migration

Our developers have been using this workflow for 2 years, so we need a seamless migration. That’s why we decided to add an intermediate step on the way to our goal.
The first step is simple:

We maintain a similar structure for application deployment configuration, but in a single object named DailymotionRelease.

apiVersion: "v1"
kind: "DailymotionRelease"
metadata:
  name: "app1.ns1"
  environment: "dev"
  branch: "mybranch"
spec:
  slack_channel: "#admin"
  chart_name: "app1"
  scaling:
    - context: "dev-us-central1-0"
      replicas:
        - name: "hermes"
          count: 2
    - context: "dev-europe-west1-0"
      replicas:
        - name: "app1-deploy"
          count: 2
  secrets:
    - secret_id: "app1"
      contexts:
        - name: "default"
          vaultPath: "\/kv\/dev\/ns1\/app1\/test"
          vaultKey: "password"
        - name: "dev-europe-west1-0"
          vaultPath: "\/kv\/dev\/ns1\/app1\/test"
          vaultKey: "password"

1 release per application (no generic charts).
Charts in the application's git repository.

We spoke with all developers, so the migration process has already begun. The first stage is still monitored using the CI platform. Soon I will write another post about the second stage: how we transitioned to the GitOps workflow with Flux. I will share how we set everything up and what challenges we faced (multiple repositories, secrets, etc.). Stay tuned for updates.

Here we tried to outline our progress in the application deployment workflow over the last few years, which has led to thoughts about the GitOps approach. We have not yet reached our goal and will report on the results, but we are now convinced that we made the right decision to simplify everything and bring it closer to developers' habits.

Source: habr.com