Kubernetes Operator in Python without frameworks and SDK.

Go is currently the dominant language among programming languages that people choose for writing operators for Kubernetes. There are objective reasons for this, such as:

1. There is a powerful framework for developing operators in Go — Operator SDK.
2. Notable applications such as Docker and Kubernetes have been written in Go. Writing your operator in Go means communicating with the ecosystem in a common language.
3. High performance of applications in Go and simple tools for working with concurrency "out of the box."

NB: By the way, how to write your operator in Go, we have already described in one of our translations of foreign authors.

But what if a lack of time or, simply, motivation, is preventing you from studying Go? The article provides an example of how to write a decent operator using one of the most popular languages known to almost every DevOps engineer — Python.

Introducing: Copy Operator — the copying operator!

As an example, let's consider the development of a simple operator designed for copying a ConfigMap either when a new namespace appears or when one of two entities is changed: ConfigMap and Secret. From a practical standpoint, the operator can be useful for bulk updating application configurations (by updating the ConfigMap) or for updating secret data — for example, keys for working with Docker Registry (when adding a Secret to a namespace).

So, what should be in a good operator:

1. Interaction with the operator is carried out using Custom Resource Definitions (hereinafter referred to as CRD).
2. The operator can be configured. For this, we will use command-line flags and environment variables.
3. The Docker container and Helm chart build processes are designed so that users can easily (with just one command) install the operator in their Kubernetes cluster.

CRD

To ensure the operator knows what resources to look for and where, we need to set a rule for it. Each rule will be represented as a single CRD object. What fields should this CRD have?

1. Resource type, which we will be looking for (ConfigMap or Secret).
2. List of namespaces, in which the resources should reside.
3. Selector, with which we will search for resources in the namespace.

Let's describe the CRD:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: copyrator.flant.com
spec:
  group: flant.com
  versions:
  - name: v1
    served: true
    storage: true
  scope: Namespaced
  names:
    plural: copyrators
    singular: copyrator
    kind: CopyratorRule
    shortNames:
    - copyr
  validation:
    openAPIV3Schema:
      type: object
      properties:
        ruleType:
          type: string
        namespaces:
          type: array
          items:
            type: string
        selector:
          type: string

And let's create a simple rule — to search in the namespace named default for all ConfigMaps with labels of the form copyrator: "true":

apiVersion: flant.com/v1
kind: CopyratorRule
metadata:
  name: main-rule
  labels:
    module: copyrator
ruleType: configmap
selector:
  copyrator: "true"
namespace: default

Done! Now we need to get information about our rule. I should clarify right away that we won't be writing requests to the API Server of the cluster ourselves. Instead, we'll use the ready-made Python library kubernetes-client:

import kubernetes
from contextlib import suppress


CRD_GROUP = 'flant.com'
CRD_VERSION = 'v1'
CRD_PLURAL = 'copyrators'


def load_crd(namespace, name):
    client = kubernetes.client.ApiClient()
    custom_api = kubernetes.client.CustomObjectsApi(client)

    with suppress(kubernetes.client.api_client.ApiException):
        crd = custom_api.get_namespaced_custom_object(
            CRD_GROUP,
            CRD_VERSION,
            namespace,
            CRD_PLURAL,
            name,
        )
    return {x: crd[x] for x in ('ruleType', 'selector', 'namespace')}

As a result of running this code, we will get the following:

{'ruleType': 'configmap', 'selector': {'copyrator': 'true'}, 'namespace': ['default']}

Great: we managed to get the rule for the operator. And most importantly — we did it in the Kubernetes way.

Environment variables or flags? We take everything!

Now we move on to the main configuration of the operator. There are two basic approaches to configuring applications:

1. using command-line parameters;
2. using environment variables.

Command-line parameters allow for more flexible reading of settings, with support and validation of data types. The Python standard library has a module argparser, which we will use. Details and examples of its capabilities are available at the official documentation..

Here’s how an example of setting up command-line flag reading would look for our case:

   parser = ArgumentParser(
        description='Copyrator - copy operator.',
        prog='copyrator'
    )
    parser.add_argument(
        '--namespace',
        type=str,
        default=getenv('NAMESPACE', 'default'),
        help='Operator Namespace'
    )
    parser.add_argument(
        '--rule-name',
        type=str,
        default=getenv('RULE_NAME', 'main-rule'),
        help='CRD Name'
    )
    args = parser.parse_args()

On the other hand, using environment variables in Kubernetes, we can easily transfer service information about a pod into the container. For example, we can obtain the information about the namespace in which the pod is running with the following construct:

env:
- name: NAMESPACE
  valueFrom:
     fieldRef:
         fieldPath: metadata.namespace 

The operator's logic

To understand how to separate methods for working with ConfigMap and Secret, let's use special maps. This will help us understand which methods we need for tracking and creating the object:

LIST_TYPES_MAP = {
    'configmap': 'list_namespaced_config_map',
    'secret': 'list_namespaced_secret',
}

CREATE_TYPES_MAP = {
    'configmap': 'create_namespaced_config_map',
    'secret': 'create_namespaced_secret',
}

Next, we need to receive events from the API server. We will implement it as follows:

def handle(specs):
    kubernetes.config.load_incluster_config()
    v1 = kubernetes.client.CoreV1Api()

    # Get the method for tracking objects
    method = getattr(v1, LIST_TYPES_MAP[specs['ruleType']])
    func = partial(method, specs['namespace'])

    w = kubernetes.watch.Watch()
    for event in w.stream(func, _request_timeout=60):
        handle_event(v1, specs, event)

After receiving the event, we move on to its main processing logic:

# Типы событий, на которые будем реагировать
ALLOWED_EVENT_TYPES = {'ADDED', 'UPDATED'}


def handle_event(v1, specs, event):
    if event['type'] not in ALLOWED_EVENT_TYPES:
        return

    object_ = event['object']
    labels = object_['metadata'].get('labels', {})

    # Ищем совпадения по selector'у
    for key, value in specs['selector'].items():
        if labels.get(key) != value:
            return
    # Получаем активные namespace'ы
    namespaces = map(
        lambda x: x.metadata.name,
        filter(
            lambda x: x.status.phase == 'Active',
            v1.list_namespace().items
        )
    )
    for namespace in namespaces:
        # Очищаем метаданные, устанавливаем namespace
        object_['metadata'] = {
            'labels': object_['metadata']['labels'],
            'namespace': namespace,
            'name': object_['metadata']['name'],
        }
        # Вызываем метод создания/обновления объекта
        methodcaller(
            CREATE_TYPES_MAP[specs['ruleType']],
            namespace,
            object_
        )(v1)

The main logic is ready! Now we need to package all of this into one Python package. We format the file setup.py, writing the meta-information about the project there:

from sys import version_info

from setuptools import find_packages, setup

if version_info[:2] < (3, 5):
    raise RuntimeError(
        'Unsupported python version %s.' % '.'.join(version_info)
    )


_NAME = 'copyrator'
setup(
    name=_NAME,
    version='0.0.1',
    packages=find_packages(),
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
        'Programming Language :: Python :: 3.7',
    ],
    author='Flant',
    author_email='maksim.nabokikh@flant.com',
    include_package_data=True,
    install_requires=[
        'kubernetes==9.0.0',
    ],
    entry_points={
        'console_scripts': [
            '{0} = {0}.cli:main'.format(_NAME),
        ]
    }
)

NB: The Kubernetes client for Python has its own versioning schema. More information about the compatibility of client versions and Kubernetes versions can be found in the compatibility matrix.

Currently, our project looks like this:

copyrator
├── copyrator
│   ├── cli.py # Command line logic
│   ├── constant.py # Constants we specified above
│   ├── load_crd.py # CRD loading logic
│   └── operator.py # Main logic of the operator
└── setup.py # Package setup

Docker and Helm

The Dockerfile will be absurdly simple: we will take the base image python-alpine and install our package. We will postpone its optimization for better times:

FROM python:3.7.3-alpine3.9

ADD . /app

RUN pip3 install /app

ENTRYPOINT ["copyrator"]

Deployment for the operator is also very simple:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Chart.Name }}
spec:
  selector:
    matchLabels:
      name: {{ .Chart.Name }}
  template:
    metadata:
      labels:
        name: {{ .Chart.Name }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: privaterepo.yourcompany.com/copyrator:latest
        imagePullPolicy: Always
        args: ["--rule-type", "main-rule"]
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      serviceAccountName: {{ .Chart.Name }}-acc

Finally, it is necessary to create the corresponding role for the operator with the required permissions:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Chart.Name }}-acc

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: {{ .Chart.Name }}
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "watch", "list"]
  - apiGroups: [""]
    resources: ["secrets", "configmaps"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: {{ .Chart.Name }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {{ .Chart.Name }}
subjects:
- kind: ServiceAccount
  name: {{ .Chart.Name }}

Summary

This way, without fear, reproach, or learning Go, we were able to create our own operator for Kubernetes in Python. Certainly, there is still room for growth: in the future it will be able to handle multiple rules, work in parallel, and autonomously monitor changes to its CRD…

To get a closer look at the code, we have compiled it in a public repository. If you are interested in more serious operators implemented using Python, you can check out two operators for deploying mongodb (the first and the second).

P.S. And if you find dealing with Kubernetes events cumbersome or simply prefer using Bash — our colleagues have prepared a ready-made solution in the form of shell-operator (we announced it in April).

P.P.S.

Also read in our blog:

• «Is setting up a Kubernetes cluster easy and convenient? Announcing the addon-operator»;
• «Introducing shell-operator: creating operators for Kubernetes has become even easier»;
• «Expanding and complementing Kubernetes (overview and presentation video)»;
• «Writing an operator for Kubernetes in Golang»;
• «Operators for Kubernetes: how to run stateful applications».

Source: habr.com