🥇Go? Bash! Meet the shell-operator (review and video talk from KubeCon EU'2020)

This year, the main European Kubernetes conference - KubeCon + CloudNativeCon Europe 2020 - was virtual. However, such a change in format did not prevent us from delivering a long-planned report “Go? Bash! Meet the Shell-operator" dedicated to our Open Source project shell-operator.

This article, inspired by the talk, presents an approach to simplifying the process of creating operators for Kubernetes and shows how you can make your own with minimal effort using the shell operator.

Introducing video with the report (~23 minutes in English, noticeably more informative than the article) and the main extract from it in text form. Go!

We at Flant are constantly optimizing and automating everything. Today we will talk about another fascinating concept. Meet: cloud-native shell scripting!

However, let's start with the context in which all this happens - with Kubernetes.

Kubernetes APIs and Controllers

API in Kubernetes can be represented as a kind of file server with directories for each type of object. Objects (resources) on this server are represented by YAML files. In addition, the server has a basic API that allows you to do three things:

get resource by its kind and name;
change resource (at the same time, the server stores only "correct" objects - all incorrectly formed or intended for other directories are discarded);
follow behind the resource (in this case, the user immediately receives its current/updated version).

Thus, Kubernetes acts as a kind of file server (for YAML manifests) with three basic methods (yes, there are actually others, but we will omit them for now).

The problem is that the server can only store information. To make it work, you need controller is the second most important and fundamental concept in the world of Kubernetes.

There are two main types of controllers. The first one takes information from Kubernetes, processes it according to nested logic, and returns it to K8s. The second one takes information from Kubernetes, but, unlike the first type, it changes the state of some external resources.

Let's take a closer look at the process of creating a Deployment in Kubernetes:

Deployment Controller (included in kube-controller-manager) gets information about the Deployment and creates a ReplicaSet.
The ReplicaSet creates two replicas (two pods) based on this information, but these pods are not scheduled yet.
The scheduler schedules pods and adds node information to their YAMLs.
Kubelets make changes to an external resource (say, Docker).

Then this whole sequence is repeated in reverse order: kubelet checks the containers, calculates the status of the pod, and sends it back. The ReplicaSet controller receives the status and updates the state of the replica set. The same happens with the Deployment Controller and the user finally gets the updated (current) status.

Shell operator

It turns out that Kubernetes is based on the collaboration of various controllers (Kubernetes operators are also controllers). The question arises, how to create your own operator with minimal effort? And here comes to the aid developed by us shell-operator. It allows system administrators to create their own statements using familiar methods.

Simple Example: Copying Secrets

Let's look at a simple example.

Let's say we have a Kubernetes cluster. It has a namespace default with some secret mysecret. In addition, there are other namespaces in the cluster. Some of them have a specific label attached to them. Our goal is to copy Secret into namespaces with a label.

The task is complicated by the fact that new namespaces may appear in the cluster, and some of them may have this label. On the other hand, when the label is removed, the Secret must also be removed. In addition to everything, the Secret itself can also change: in this case, the new Secret must be copied to all namespaces with labels. If a Secret is accidentally deleted in any namespace, our operator must restore it immediately.

Now that the task has been formulated, it's time to start implementing it using the shell-operator. But first it is worth saying a few words about the shell-operator's itself.

How the shell operator works

Like other workloads in Kubernetes, the shell-operator runs in its own pod. In this pod in the directory /hooks executable files are stored. These can be scripts in Bash, Python, Ruby, etc. We call such executable files hooks (hooks).

The shell-operator subscribes to Kubernetes events and fires these hooks in response to the events we want.

How does the shell-operator know which hook to run and when? The point is that each hook has two stages. During startup, the shell-operator runs all hooks with an argument --config is the configuration stage. And after it, hooks are launched in a normal way - in response to the events to which they are attached. In the latter case, the hook receives the binding context (binding context) is JSON data, which we will discuss in more detail below.

Making a statement in Bash

Now we are ready for implementation. To do this, we need to write two functions (by the way, we recommend library shell_lib, which greatly simplifies writing hooks in Bash):

the first is needed for the configuration stage - it displays the binding context;
the second contains the main logic of the hook.

#!/bin/bash

source /shell_lib.sh

function __config__() {
  cat << EOF
    configVersion: v1
    # BINDING CONFIGURATION
EOF
}

function __main__() {
  # THE LOGIC
}

hook::run "$@"

The next step is to decide what objects we need. In our case, we need to track:

source secret for changes;
all namespaces in the cluster to know which of them the label is attached to;
target secrets to make sure they are all in sync with the source secret.

Subscribe to secret source

Binding configuration for it is quite simple. We indicate that we are interested in Secret with the name mysecret in namespace default:

function __config__() {
  cat << EOF
    configVersion: v1
    kubernetes:
    - name: src_secret
      apiVersion: v1
      kind: Secret
      nameSelector:
        matchNames:
        - mysecret
      namespace:
        nameSelector:
          matchNames: ["default"]
      group: main
EOF

As a result, the hook will run when the source secret changes (src_secret) and get the following binding context:

As you can see, it contains the name and the entire object.

Keeping track of namespaces

Now we need to subscribe to namespaces. To do this, specify the following binding configuration:

- name: namespaces
  group: main
  apiVersion: v1
  kind: Namespace
  jqFilter: |
    {
      namespace: .metadata.name,
      hasLabel: (
       .metadata.labels // {} |  
         contains({"secret": "yes"})
      )
    }
  group: main
  keepFullObjectsInMemory: false

As you can see, a new field has appeared in the configuration with the name jqFilter. As its name suggests, jqFilter filters out all unnecessary information and creates a new JSON object with fields that are of interest to us. A hook with this configuration will receive the following binding context:

It contains an array filterResults for each namespace in the cluster. Boolean hasLabel indicates whether the label is attached to the given namespace. Selector keepFullObjectsInMemory: false says that there is no need to keep complete objects in memory.

Keeping track of secrets

We subscribe to all Secrets that have an annotation managed-secret: "yes" (these are our targets dst_secrets):

- name: dst_secrets
  apiVersion: v1
  kind: Secret
  labelSelector:
    matchLabels:
      managed-secret: "yes"
  jqFilter: |
    {
      "namespace":
        .metadata.namespace,
      "resourceVersion":
        .metadata.annotations.resourceVersion
    }
  group: main
  keepFullObjectsInMemory: false

In this case jqFilter filters out all information except namespace and parameter resourceVersion. The last parameter was passed to the annotation when the secret was created: it allows you to compare versions of secrets and keep them up to date.

A hook configured in this way will receive the three binding contexts described above when executed. They can be thought of as a kind of snapshot (snapshot) cluster.

Based on all this information, a basic algorithm can be developed. It iterates over all namespaces and:

if hasLabel has the meaning true for the current namespace:
- compares the global secret with the local one:
  - if they are the same, it does nothing;
  - if they differ, execute kubectl replace or create;
if hasLabel has the meaning false for the current namespace:
- makes sure Secret is not in the given namespace:
  - if local Secret is present, remove it with kubectl delete;
  - if the local Secret is not found, it does nothing.

Implementation of the algorithm in Bash you can download in our repositories with examples.

This is how we were able to create a simple Kubernetes controller using 35 lines of YAML configs and about the same amount of Bash code! The job of the shell-operator is to link them together.

However, copying secrets is not the only scope of the utility. Here are a few more examples to show what he can do.

Example 1: Making Changes to a ConfigMap

Let's consider a Deployment consisting of three pods. Pods use ConfigMap to store some configuration. During the launch of the pods, ConfigMap was in some state (let's call it v.1). Accordingly, all pods use this version of ConfigMap.

Now suppose ConfigMap has changed (v.2). However, pods will use the old version of ConfigMap (v.1):

How to make them switch to the new ConfigMap (v.2)? The answer is simple: use a template. Let's add a checksum annotation to the section template Deployment configurations:

As a result, this checksum will be written in all pods, and it will be the same as that of Deployment. Now you just need to update the annotation when the ConfigMap changes. And shell-operator comes in handy in this case. All you need is to program a hook that will subscribe to the ConfigMap and update the checksum.

If the user makes changes to the ConfigMap, the shell-operator will notice them and recalculate the checksum. After that, the magic of Kubernetes will come into play: the orchestrator will kill the pod, create a new one, wait until it becomes Ready, and move on to the next one. As a result, Deployment will be synchronized and will switch to the new version of ConfigMap.

Example 2: Working with Custom Resource Definitions

As you know, Kubernetes allows you to create custom types (kinds) of objects. For example, you can create kind MysqlDatabase. Let's say this type has two metadata parameters: name и namespace.

apiVersion: example.com/v1alpha1
kind: MysqlDatabase
metadata:
  name: foo
  namespace: bar

We have a Kubernetes cluster with different namespaces where we can create MySQL databases. In this case shell-operator can be used to track resources MysqlDatabase, connecting them to the MySQL server, and synchronizing the desired and observed cluster states.

Example 3: Monitoring a Cluster Network

As you know, using ping is the simplest way to monitor the network. In this example, we will show how to implement such monitoring using the shell-operator.

First of all, you need to subscribe to nodes. The shell operator needs the name and IP address of each node. With their help, he will ping these nodes.

configVersion: v1
kubernetes:
- name: nodes
  apiVersion: v1
  kind: Node
  jqFilter: |
    {
      name: .metadata.name,
      ip: (
       .status.addresses[] |  
        select(.type == "InternalIP") |
        .address
      )
    }
  group: main
  keepFullObjectsInMemory: false
  executeHookOnEvent: []
schedule:
- name: every_minute
  group: main
  crontab: "* * * * *"

Parameter executeHookOnEvent: [] prevents the hook from running in response to any event (i.e. in response to changing, adding, deleting nodes). However, he will run (and update the list of nodes) Scheduled - every minute, as required by the field schedule.

Now the question is, how exactly do we know about problems like packet loss? Let's take a look at the code:

function __main__() {
  for i in $(seq 0 "$(context::jq -r '(.snapshots.nodes | length) - 1')"); do
    node_name="$(context::jq -r '.snapshots.nodes['"$i"'].filterResult.name')"
    node_ip="$(context::jq -r '.snapshots.nodes['"$i"'].filterResult.ip')"
    packets_lost=0
    if ! ping -c 1 "$node_ip" -t 1 ; then
      packets_lost=1
    fi
    cat >> "$METRICS_PATH" <<END
      {
        "name": "node_packets_lost",
        "add": $packets_lost,
        "labels": {
          "node": "$node_name"
        }
      }
END
  done
}

We loop through the list of hosts, get their names and IP addresses, ping and send the results to Prometheus. Shell-operator can export metrics to Prometheus, saving them to a file located according to the path specified in the environment variable $METRICS_PATH.

Like this you can make an operator for simple network monitoring in a cluster.

Queue mechanism

This article would be incomplete without describing another important mechanism built into the shell-operator. Imagine that it executes some kind of hook in response to an event in the cluster.

What happens if at the same time in the cluster happens one more event?
Will shell-operator run another instance of the hook?
But what if, say, five events happen in the cluster at once?
Will the shell-operator process them in parallel?
What about consumed resources like memory and CPU?

Fortunately, the shell-operator has a built-in queuing mechanism. All events are queued and processed sequentially.

Let's illustrate this with examples. Let's say we have two hooks. The first event goes to the first hook. After it has been processed, the queue moves forward. The next three events are redirected to the second hook - they are removed from the queue and come into it in a “batch”. That is hook receives an array of events — or, more accurately, an array of binding contexts.

Also these events can be combined into one big. The parameter is responsible for this. group in the binding configuration.

You can create any number of queues / hooks and their various combinations. For example, one queue can work with two hooks, or vice versa.

All you have to do is set up the field accordingly. queue in the binding configuration. If no queue name is specified, the hook runs on the default queue (default). Such a queuing mechanism allows you to completely solve all the resource management problems when working with hooks.

Conclusion

We explained what the shell-operator is, showed how it can be used to quickly and effortlessly create Kubernetes operators, and gave some examples of its use.

Detailed information about the shell-operator, as well as a brief guide to its use, are available in the corresponding repositories on GitHub. Feel free to contact us with questions: you can discuss them in a special Telegram group (in Russian) or in this forum (In English).

And if you like it, we are always happy to see new issues/PR/stars on GitHub, where, by the way, you can find others interesting projects. Among them, it is worth highlighting addon operator, which is the older brother of shell-operator. This utility uses Helm charts to install add-ons, can deliver updates and monitor various chart parameters/values, controls the chart installation process, and can also modify them in response to events in the cluster.

Videos and slides

Video from the performance (~23 minutes):

Report presentation:

go? Bash! Meet the shell-operator (review and video talk from KubeCon EU'2020)