🥇Writing an Operator for Kubernetes in Golang

Note: translation.: Operators are auxiliary software for Kubernetes designed to automate routine actions on cluster objects in response to certain events. We have previously written about operators in from one of the authors., where we discussed the fundamental ideas and principles of their operation. However, while that material was more of a perspective on operating ready-made components for Kubernetes, the current translation of this new article represents the viewpoint of a developer/DevOps engineer tasked with implementing a new operator.

I decided to write this post with a real-life example after my attempts to find documentation on creating an operator for Kubernetes, which went through code study.

The example to be described is as follows: in our Kubernetes cluster, each Namespace represents a sandbox environment for some team, and we wanted to restrict access to them so that teams could only play in their own sandboxes.

This can be achieved by assigning the user a group that has RoleBinding specific Namespace and ClusterRole with editing rights. The YAML representation would look like this:

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kubernetes-team-1
  namespace: team-1
subjects:
- kind: Group
  name: kubernetes-team-1
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
apiGroup: rbac.authorization.k8s.io

(rolebinding.yaml, in .raw)

Creating such RoleBinding can be done manually, but after surpassing the hundred namespace mark, it becomes a tedious task. This is exactly where Kubernetes operators help—they allow for the automation of resource creation in Kubernetes based on changes in resources. In our case, we want to create RoleBinding upon creation Namespace.

First, we will define the function main, which performs the necessary configuration to start the operator and then calls the operator's action:

(Note: translation.: here and afterwards, comments in the code are translated into Russian. Additionally, the indents have been corrected to spaces instead of the [recommended in Go] tabs solely for better readability in the context of Habr's layout. After each listing, links to the original on GitHub are provided, where the English comments and tabs are preserved.)

func main() {
  // Setting log output to console STDOUT
  log.SetOutput(os.Stdout)

  sigs := make(chan os.Signal, 1) // Creating a channel to receive OS signals
  stop := make(chan struct{})     // Creating a channel for stop signal

  // Registering to receive SIGTERM in the sigs channel
  signal.Notify(sigs, os.Interrupt, syscall.SIGTERM, syscall.SIGINT) 

  // Goroutines can add themselves to WaitGroup,
  // to wait for their completion
  wg := &sync.WaitGroup{} 

  runOutsideCluster := flag.Bool("run-outside-cluster", false, "Set this flag when running outside of the cluster.")
  flag.Parse()
  // Creating clientset for interacting with the Kubernetes cluster
  clientset, err := newClientSet(*runOutsideCluster)

  if err != nil {
    panic(err.Error())
  }

  controller.NewNamespaceController(clientset).Run(stop, wg)

  <-sigs // Waiting for signals (nothing happens until a signal is received)
  log.Printf("Shutting down...")

  close(stop) // Telling goroutines to stop
  wg.Wait()   // Waiting for all to stop
}

(main.go, in .raw)

We do the following:

We set up handlers for specific operating system signals to trigger a graceful shutdown of the operator.
We use WaitGroup, to properly stop all goroutines before the application shuts down.
We provide access to the cluster by creating clientset.
Launch NamespaceController, where all our logic will reside.

Now we need a foundation for logic, and in our case, this is the mentioned NamespaceController:

// NamespaceController следит через Kubernetes API за изменениями
// в пространствах имен и создает RoleBinding для конкретного namespace.
type NamespaceController struct {
  namespaceInformer cache.SharedIndexInformer
  kclient           *kubernetes.Clientset
}

// NewNamespaceController создает новый NewNamespaceController
func NewNamespaceController(kclient *kubernetes.Clientset) *NamespaceController {
  namespaceWatcher := &NamespaceController{}

  // Создаем информер для слежения за Namespaces
  namespaceInformer := cache.NewSharedIndexInformer(
    &cache.ListWatch{
      ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
        return kclient.Core().Namespaces().List(options)
      },
      WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
        return kclient.Core().Namespaces().Watch(options)
      },
    },
    &v1.Namespace{},
    3*time.Minute,
    cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
  )

  namespaceInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: namespaceWatcher.createRoleBinding,
  })

  namespaceWatcher.kclient = kclient
  namespaceWatcher.namespaceInformer = namespaceInformer

  return namespaceWatcher
}

(controller.go, in .raw)

Here we set up SharedIndexInformer, which will effectively (using a cache) wait for changes in namespaces (for more on informers, see the article "How does the Kubernetes scheduler really work?» — translator's note.). After that, we connect EventHandler to the informer, so when a namespace is added (Namespace), the function createRoleBinding.

The next step is to define this function createRoleBinding:

func (c *NamespaceController) createRoleBinding(obj interface{}) {
  namespaceObj := obj.(*v1.Namespace)
  namespaceName := namespaceObj.Name

  roleBinding := &v1beta1.RoleBinding{
    TypeMeta: metav1.TypeMeta{
      Kind:       "RoleBinding",
      APIVersion: "rbac.authorization.k8s.io/v1beta1",
    },
    ObjectMeta: metav1.ObjectMeta{
      Name:      fmt.Sprintf("ad-kubernetes-%s", namespaceName),
      Namespace: namespaceName,
    },
    Subjects: []v1beta1.Subject{
      v1beta1.Subject{
        Kind: "Group",
        Name: fmt.Sprintf("ad-kubernetes-%s", namespaceName),
      },
    },
    RoleRef: v1beta1.RoleRef{
      APIGroup: "rbac.authorization.k8s.io",
        Kind:     "ClusterRole",
        Name:     "edit",
    },
  }

  _, err := c.kclient.Rbac().RoleBindings(namespaceName).Create(roleBinding)

  if err != nil {
    log.Println(fmt.Sprintf("Failed to create Role Binding: %s", err.Error()))
  } else {
    log.Println(fmt.Sprintf("Created AD RoleBinding for Namespace: %s", roleBinding.Name))
  }
}

(controller.go, in .raw)

We receive the namespace as obj and convert it into an object Namespace. We then define RoleBinding, based on what is mentioned in the initial YAML file, using the provided object Namespace and creating RoleBinding. Finally, we log whether the creation was successful.

The last function to define is Run:

// Run запускает процесс ожидания изменений в пространствах имён
// и действия в соответствии с этими изменениями.
func (c *NamespaceController) Run(stopCh <-chan struct{}, wg *sync.WaitGroup) {
  // Когда эта функция завершена, пометим как выполненную
  defer wg.Done()

  // Инкрементируем wait group, т.к. собираемся вызвать goroutine
  wg.Add(1)

  // Вызываем goroutine
  go c.namespaceInformer.Run(stopCh)

  // Ожидаем получения стоп-сигнала
  <-stopCh
}

(controller.go, in .raw)

Here we state WaitGroup, that we will launch a goroutine and then call namespaceInformer, which was defined earlier. When a stop signal is received, it will complete the function, reporting WaitGroup, that it is no longer running, and this function will finish its execution.

Information about deploying and running this operator in a Kubernetes cluster can be found in GitHub repository.

At this point, the operator that creates RoleBinding upon appearance Namespace in the Kubernetes cluster is ready.

Source: habr.com

Writing an operator for Kubernetes in Golang