🥇Is it easy and convenient to prepare a Kubernetes cluster? Announcing addon-operator

After shell-operator we represent his older brother - addon operator. This is an Open Source project that is used to install system components in a Kubernetes cluster, which can be called in a common word - add-ons.

Why add any add-ons at all?

It's no secret that Kubernetes is not a ready-made all-in-one product, and various add-ons will be needed to build an "adult" cluster. The addon-operator will help you install, configure and keep these addons up to date.

The need for additional components in a cluster is covered in report Colleagues driusha. In short, the situation with Kubernetes at the moment is such that for a simple “play around” installation, you can get by with the components out of the box, for developers and testing, you can add Ingress, but for a full-fledged installation, which you can say “your production is ready”, you need to add with a dozen different additions: something for monitoring, something for logs, don't forget ingress and cert-manager, select groups of nodes, add network policies, spice up sysctl and pod autoscaler settings...

What are the specifics of working with them?

As practice shows, the matter is not limited to one installation. For comfortable work with the cluster, add-ons will need to be updated, disabled (removed from the cluster), and you want to test something before installing it in the production cluster.

So, maybe Ansible is enough here? Maybe. But full-fledged add-ons in the general case do not live without settings. These settings may differ depending on the cluster variant (aws, gce, azure, bare-metal, do, ...). Some settings cannot be set in advance - they must be obtained from the cluster. And the cluster is not static: for some settings, you will have to follow the changes. And here Ansible is not enough: you need a program that lives in a cluster, i.e. Kubernetes operator.

Those who have tried shell-operator, they will say that the tasks of installing and updating add-ons and monitoring settings can be completely solved using hooks for shell operator. You can write a script that will do a conditional kubectl apply and follow, for example, ConfigMap, where the settings will be stored. This is approximately what is implemented in the addon-operator.

How is it organized in addon-operator?

Creating a new solution, we proceeded from the following principles:

The add-on installer must support templating and declarative configuration. We do not make magic scripts that install add-ons. The addon-operator uses Helm to install addons. To install, you need to create a chart and select the values that will be used for configuration.
Settings can generate on install, they can get from clusteror receive updateswhile monitoring cluster resources. These operations can be implemented using hooks.
Settings can store in a cluster. To store settings in the cluster, a ConfigMap/addon-operator is created and the Addon-operator monitors changes to this ConfigMap. The addon-operator gives hooks access to settings through simple conventions.
Addition depends on settings. If the settings have changed, then Addon-operator rolls out a Helm-chart with new values. We called the combination of the Helm chart, values for it, and hooks a module (see below for more details).
Staging. There are no magic release scripts. The update mechanism is similar to a regular application - collect add-ons and addon-operator into an image, tag and roll out.
Result control. Addon-operator can return metrics for Prometheus.

What is the addition in addon-operator?

Anything that adds new features to the cluster can be considered an add-on. For example, the Ingress installation is a great example of an add-on. This can be any operator or controller with its own CRD: prometheus-operator, cert-manager, kube-controller-manager, etc. Or something small but easier to use, such as a secret copier that copies registry secrets to new namespaces, or a sysctl tuner that tunes sysctl parameters on new nodes.

To implement additions, Addon-operator provides several concepts:

Helm chart used to install various software into the cluster - for example, Prometheus, Grafana, nginx-ingress. If the desired component has a Helm chart, then installing it using the Addon-operator will be very simple.
values storage. Helm charts usually have many different settings that can change over time. The addon-operator supports storing these settings and keeps track of their changes in order to reset the Helm chart with the new values.
Hooks are executable files that the Addon-operator runs on events and that access the values store. The hook can watch for changes in the cluster and update the values in the values store. Those. using hooks, you can do discovery to collect values from the cluster at startup or on a schedule, or you can do continuous discovery, collecting values from the cluster based on changes in the cluster.
Module is an amalgamation of a Helm chart, values storage, and hooks. Modules can be enabled and disabled. Disabling a module is deleting all Helm-chart releases. Modules can include themselves dynamically, for example, if all the modules it needs are enabled or if discovery found the necessary parameters in hooks - this is done using an auxiliary enabled script.
Global Hooks. These are hooks "on their own", they are not included in modules and have access to the global values store, the values from which are available to all hooks in modules.

How do these parts work together? Consider the picture from the documentation:

There are two work scenarios:

The global hook is triggered by an event - for example, when a resource in the cluster changes. This hook handles the changes and writes the new values to the global values store. The addon-operator notices that the global store has changed and runs all modules. Each module uses its hooks to determine if it needs to be enabled and updates its values store. If the module is enabled, then the Addon-operator starts the installation of the Helm-chart. At the same time, values from the module storage and from the global storage are available to the Helm chart.
The second scenario is simpler: the module hook is triggered by an event, changes the values in the module's values store. The addon-operator notices this and launches a Helm chart with updated values.

An add-on can be implemented as a single hook, or as a single Helm chart, or even as multiple dependent modules - it depends on the complexity of the component installed in the cluster and on the desired level of configuration flexibility. For example, in the repository (/examples) there is an add-on sysctl-tuner, which is implemented both as a simple module with a hook and a Helm-chart, and using the values store, which makes it possible to add settings by editing the ConfigMap.

Delivery of updates

A few words about the organization of component updates that the Addon-operator installs.

To run Addon-operator on a cluster, you need assemble an image with additions as hook files and Helm charts, add a binary file addon-operator and everything you need for hooks: bash, kubectl, jq, python etc. Further, this image can be rolled out to the cluster as a normal application, and most likely you will want to organize one or another tagging scheme. If there are few clusters, the same approach as with applications may be suitable: new release, new version, go through all the clusters and fix the image of the Pods. However, in the case of rolling out to a significant number of clusters, the concept of self-updating from the channel was more suitable for us.

We have it set up like this:

A channel is essentially an identifier that can be set to anything (for example, dev/stage/ea/stable).
The channel name is an image tag. When you need to roll out updates to a channel, a new image is built and tagged with the channel name.
When a new image appears in the registry, Addon-operator restarts and starts with the new image.

This is not best practice, as it is written in Kubernetes documentation. It is not recommended to do this, but we are talking about a normal application that lives in one cluster. In the case of Addon-operator, the application is a lot of Deployments scattered across clusters, and self-updating helps a lot and makes life easier.

Channels help and in testing: if there is an auxiliary cluster, you can set it to a channel stage and roll updates into it before rolling out to channels ea и stable. If with a cluster on the channel ea an error has occurred, you can change it to stablewhile the issue with this cluster is being investigated. If the cluster is taken out of active support, it switches to its "frozen" channel - for example, freeze-2019-03-20.

In addition to updating hooks and Helm charts, you may need to update and third party component. For example, you noticed a bug in the conditional node-exporter and even figured out how to patch it. Then we opened PR and are waiting for a new release to go through all the clusters and increase the version of the image. In order not to wait indefinitely, you can build your own node-exporter and switch to it before accepting the PR.

In general, this can be done without the Addon-operator, but with the Addon-operator, the module for installing node-exporter will be visible in one repository, you can keep the Dockerfile to build your image right there, it becomes easier for all participants in the process to understand that happens… And if there are several clusters, it becomes easier both to test your PR and roll out a new version!

This component update organization works successfully for us, but any other suitable scheme can be implemented - after all in this case addon-operator is a simple binary file.

Conclusion

The principles implemented in the Addon-operator allow you to build a transparent process for creating, testing, installing and updating add-ons in a cluster, similar to the development of ordinary applications.

Addons for Addon-operator in the format of modules (Helm-chart + hooks) can be made publicly available. We, the Flant company, plan to publish our developments in the form of such additions during the summer. Join the development on GitHub (shell-operator, addon operator), try to make your addition based on examples и documentation, wait for news on Habré and on our YouTube channel!

Is it easy and convenient to prepare a Kubernetes cluster? Announcing addon-operator