🥇Introducing Kubernetes CCM (Cloud Controller Manager) for Yandex.Cloud

In continuation to the recent CSI driver release for Yandex.Cloud, we are publishing another Open Source project for this cloud — Cloud Controller Manager. CCM is required not only for the cluster as a whole, but also for the CSI driver itself. Details about its purpose and some implementation features are under the cut.

Introduction

Why is that?

The motives that prompted us to develop CCM for Yandex.Cloud completely coincide with those already described in announcement CSI drivers. We serve many Kubernetes clusters from different cloud providers, for which we use a single tool. It implements numerous conveniences “bypassing” the managed solutions of these providers. Yes, we have a rather specific case and needs, but the developments created because of them may be useful to other users.

What is CCM anyway?

As a rule, we prepare our environment for the cluster outside - for example, using Terraform. But sometimes there is a need to manage the cloud environment around us. from the cluster. Such a possibility is provided, and it implements it CCM.

In particular, Cloud Controller Manager provides five main types of interaction:

Instances – implements a 1:1 relationship between a node object in Kubernetes (Node) and a virtual machine in the cloud provider. For this we:
- fill in the field spec.providerID in object Node. For example, for OpenStack CCM this field has the following format: openstack:///d58a78bf-21b0-4682-9dc6-2132406d2bb0. You can see the name of the cloud provider and the unique UUID of the server (virtual machine in OpenStack) of the object;
- supplement nodeInfo in object Node information about the virtual machine. For example, we specify instance type in AWS;
- check the presence of a virtual machine in the cloud. For example, if the object Node moved to state NotReady, you can check if a virtual machine even exists in the cloud provider by providerID. If it doesn't exist, delete the object. Node, which would otherwise remain in the cluster forever;
Zones – sets the failure domain for the object Nodeso that the scheduler can select a node for the Pod according to regions and zones in the cloud provider;
LoadBalancer - when creating an object Service with type LoadBalancer creates a balancer that will direct traffic from the outside to the cluster nodes. For example, in Yandex.Cloud you can use NetworkLoadBalancer и TargetGroup for these purposes;
Route - builds a network between nodes, because Kubernetes requires each pod to have its own IP address and be able to reach any other pod. For these purposes, you can use an overlay network (VXLAN, GENEVE) or set the routing table directly in the virtual network of the cloud provider:
Volume – allows you to dynamically order PV using PVC and SC. Initially, this functionality was part of the CCM, but due to its great complexity, it was moved to a separate Container Storage Interface (CSI) project. We talk about CSI more than once wrote and, as already mentioned, even released CSI driver.

Previously, all code interacting with the cloud was in the main Git repository of the Kubernetes project at k8s.io/kubernetes/pkg/cloudprovider/providers, but they decided to abandon this due to the inconvenience of working with a large code base. All old implementations have been moved to separate repository. For the convenience of further support and development, all common components were also moved to separate repository.

As with CSIs, many major cloud service providers have already developed their CCMs to run clouds on Kubernetes. If the supplier does not have CCM, but all the necessary functions are available through the API, then you can implement the CCM yourself.

To write your own implementation of CCM, it is enough to implement required Go interfaces.

И here is what we got.

implementation

How did you come to this

Development (or rather, even use) we started with finished(!) CCM for Yandex.Cloud a year ago.

However, in this implementation, we lacked:

authentication via JWT IAM token;
Service controller support.

By agreement with the author (dlyin) in Telegram, we forked yandex-cloud-controller-manager and added the missing functions.

Main Features

CCM currently supports the following interfaces:

Instances;
Zones;
LoadBalancer.

In the future, when Yandex.Cloud starts working with advanced VPC features, we will add an interface Routes.

LoadBalanacer as main call

Initially, we tried, like other CCM implementations, to create a pair of LoadBalancer и TargetGroup for each Service with type LoadBalancer. However, Yandex.Cloud has one interesting limitation: you cannot use TargetGroups with intersecting Targets (pair SubnetID — IpAddress).

Therefore, a controller is launched inside the created CCM, which, when objects are changed Node collects information about all interfaces on each virtual machine, groups them according to their belonging to certain NetworkID, creates by TargetGroup on NetworkIDand keeps up to date. Subsequently, when creating an object Service with type LoadBalanacer we simply attach a pre-created TargetGroup to new NetworkLoadBalanacer'am.

How to start using?

CCM supports Kubernetes version 1.15 and above. In a cluster, for it to work, it is required that the flag --cloud-provider=external has been set to true for kube-apiserver, kube-controller-manager, kube-scheduler and all kubelets.

All the necessary steps for the installation itself are described in README. Installation comes down to creating objects in Kubernetes from manifests.

To use CCM you will also need:

specify in the manifest directory ID (folder-id) Yandex.Cloud;
a service account for interacting with the Yandex.Cloud API. In the manifest Secret necessary transfer authorized keys from a service account. Documentation describedhow to create a service account and get the keys.

We welcome feedback and new issuesif you run into any problems!

Results

We have been using the implemented CCM in five Kubernetes clusters over the past two weeks and plan to expand their number to 20 in the next month. We do not currently recommend using CCM for large and critical K8s installations.

As in the case of CSI, we will be happy if Yandex developers take over the development and support of this project - we are ready to transfer the repository at their request in order to deal with more specialized tasks for us.

Introducing Kubernetes CCM (Cloud Controller Manager) for Yandex.Cloud