🥇Our experience in developing a CSI driver in Kubernetes for Yandex.Cloud

We are pleased to announce that Flant has expanded its contribution to Open Source tools for Kubernetes by releasing the alpha version of the CSI driver (Container Storage Interface) for Yandex.Cloud.

But before diving into the implementation details, let's address the question of why this is needed at all, given that Yandex already offers Managed Service for Kubernetes.

Introduction

Why is this needed?

Within our company, since the very beginning of running Kubernetes in production (i.e., for several years now), we have been developing our own tool (deckhouse), which we also plan to make available as an Open Source project soon. With it, we configure and manage all our clusters uniformly, currently numbering over 100, deployed on various hardware configurations and across all available cloud services.

Clusters using deckhouse incorporate all the components necessary for operation: load balancers, monitoring with convenient graphs, metrics, and alerts, user authentication via external providers for access to all dashboards, and so on. Such an 'enhanced' cluster makes little sense to implement in a managed solution, as often this either isn't possible or would require disabling half of its components.

NB: This is our experience, and it is quite specific. We do not claim that everyone should independently deploy Kubernetes clusters instead of using ready-made solutions. By the way, we do not have any real-world experience with Yandex's Kubernetes service, and we will not make any assessments of this service in the present article.

What is this and for whom?

So, we have already discussed the modern approach to storage in Kubernetes: how CSI works and how the community arrived at such an approach.

Currently, many large cloud service providers have developed drivers to use their 'cloud' disks as Persistent Volumes in Kubernetes. However, if a provider does not have such a driver but provides all necessary functions through an API, there is nothing preventing us from implementing a driver ourselves. That’s how we came to create one for Yandex.Cloud.

We based our development on the CSI driver for DigitalOcean and a few ideas from the driver for GCP, as the interaction with the APIs of these clouds (Google and Yandex) has several similarities. In particular, both APIs and GCP, and Yandex return an object Operation for tracking the status of long operations (for example, creating a new disk). To interact with the Yandex.Cloud API, the Yandex.Cloud Go SDK.

The result of the work done is published on GitHub and may be useful for those who for some reason use their own Kubernetes installation on Yandex.Cloud virtual machines (but not a ready-made managed cluster) and would like to use (order) disks via CSI.

Implementation

Key Features

Currently, the driver supports the following features:

Ordering disks in all cluster zones according to the topology of the available nodes in the cluster;
Deleting previously ordered disks;
Offline resize for disks (Yandex.Cloud does not support increases the disks that are mounted to the virtual machine). For information on how the driver had to be modified to perform the resize as painlessly as possible, see below.

In the future, support for creating and deleting snapshots of disks is planned.

The main challenge and overcoming it

The absence of the ability to increase disks in real-time in the Yandex.Cloud API is a limitation that complicates the resize operation for PV (Persistent Volume): in this case, it is necessary for the application pod using the disk to be stopped, which can cause application downtime.

According to CSI specification, if the CSI controller reports that it can only perform the disk resize ‘offline’ (VolumeExpansion.OFFLINE), then the process of increasing the disk must proceed as follows:

If the plugin has only VolumeExpansion.OFFLINE expansion capability and the volume is currently published or available on a node then ControllerExpandVolume MUST be called ONLY after either:

The plugin has controller PUBLISH_UNPUBLISH_VOLUME capability and ControllerUnpublishVolume has been invoked successfully.
OR ELSE

The plugin does NOT have controller PUBLISH_UNPUBLISH_VOLUME capability, the plugin has node STAGE_UNSTAGE_VOLUME capability, and NodeUnstageVolume has been completed successfully.
OR ELSE

The plugin does NOT have controller PUBLISH_UNPUBLISH_VOLUME capability, nor node STAGE_UNSTAGE_VOLUME capability, and NodeUnpublishVolume has completed successfully.

Essentially, this means that the disk must be detached from the virtual machine before it can be increased.

However, unfortunately, implementation the CSI specifications through sidecars do not meet these requirements:

In the sidecar container csi-attacher, which should be responsible for ensuring the necessary interval between mounts, this functionality has simply not been implemented during offline resizing. This discussion was initiated here.
What is a sidecar container in this context? The CSI plugin itself does not interact with the Kubernetes API; it only responds to gRPC calls sent to it by sidecar containers. The latter are developed is managed by the Kubernetes community.

In our case (CSI plugin), the disk expansion operation looks like this:

We receive a gRPC call ControllerExpandVolume;
We attempt to expand the disk in the API but receive an error about the inability to perform the operation since the disk is mounted;
We save the disk ID in a map containing disks for which the expansion operation needs to be performed. For brevity, we will refer to this map as volumeResizeRequired;
We manually delete the pod that uses the disk. Kubernetes will restart it. To prevent the disk from being mounted (ControllerPublishVolume) before the expansion operation is complete during the mount attempt, we check that this disk is still in volumeResizeRequired and return an error;
The CSI driver attempts to retry the resize operation. If the operation is successful, we remove the disk from volumeResizeRequired;
Since the disk ID is not present in volumeResizeRequired, ControllerPublishVolume is successful, the disk is mounted, and the pod starts running.

Everything seems quite simple, but as always, there are pitfalls. Disk resizing is handled by external-resizer, which, in case of an error during the operation, uses a queue with an exponential increase in timeout up to 1000 seconds:

func DefaultControllerRateLimiter() RateLimiter {
  return NewMaxOfRateLimiter(
  NewItemExponentialFailureRateLimiter(5*time.Millisecond, 1000*time.Second),
  \/\/ 10 qps, 100 bucket size. This is only for retry speed and its only the overall factor (not per item)
  &BucketRateLimiter{Limiter: rate.NewLimiter(rate.Limit(10), 100)},
  )
}

This can periodically lead to the disk expansion operation stretching over 15+ minutes, thus causing the corresponding pod to be unavailable.

The only option that allowed us to reduce potential downtime relatively easily was to use our version of external-resizer with a maximum timeout limit of 5 seconds:

workqueue.NewItemExponentialFailureRateLimiter(5*time.Millisecond, 5*time.Second)

We didn’t find it necessary to urgently initiate a discussion and patch external-resizer because offline disk resizing is an artifact that will soon disappear with all cloud providers.

How to get started?

The driver is supported in Kubernetes version 1.15 and above. The following requirements must be met for the driver to function:

Flag --allow-privileged set to true for the API server and kubelet;
Enabled --feature-gates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true for the API server and kubelet;
Mount propagation (mount propagation) must be enabled in the cluster. When using Docker, the daemon must be configured to allow shared mounts.

All necessary installation steps are described in README. The installation involves creating objects in Kubernetes from manifests.

To use the driver, you'll need the following:

Specify the folder ID in the manifest (folder-id) of Yandex.Cloud (see documentation);
. To interact with Yandex.Cloud API in the CSI driver, a service account is used. In the Secret manifest, you need to provide authorized keys from the service account. The documentation explains is describedhow to create a service account and obtain the keys.

In general, please try, and we would appreciate your feedback and new issues, if you encounter any problems!

Further support

In conclusion, we would like to note that we developed this CSI driver not out of a strong desire to play with writing applications in Go, but due to a pressing need within the company. We don't see maintaining our own implementation as feasible, so if Yandex shows interest and decides to continue supporting the driver, we would be happy to transfer the repository to their disposal.

Additionally, Yandex probably has its own implementation of a CSI driver in the managed Kubernetes cluster that could be released as Open Source. Such a development option also seems favorable to us — the community will be able to use a proven driver from the service provider rather than a third-party company.