There are several ways to set up data storage for applications running on a Kubernetes cluster. Some of them are already outdated, others have appeared recently. In this article, we will consider the concept of three options for connecting storage systems, including the most recent one - connecting through the Container Storage Interface.
Method 1: Specifying the PV in the Pod Manifest
A typical manifest describing a Pod in a Kubernetes cluster:
Parts of the manifest are highlighted in color, where it is described which volume is connected and where.
In section volumeMounts indicate the mount points (mountPath) - to which directory inside the container the persistent volume will be mounted, as well as the name of the volume.
In section x list all the volumes that are used in the pod. Specify the name of each volume, as well as the type (in our case: awsElasticBlockStore) and connection options. Which parameters are listed in the manifest depends on the volume type.
The same volume can be mounted to multiple Pod containers at the same time. This way different application processes can access the same data.
This connection method was invented at the very beginning, when Kubernetes was in its infancy, and today the method is outdated.
There are several problems when using it:
- all volumes must be created manually, Kubernetes will not be able to create anything for us;
- access parameters for each of the volumes are unique, and they must be specified in the manifests of all pods that use the volume;
- to change the storage system (for example, move from AWS to Google Cloud), you need to change the settings and the type of mounted volumes in all manifests.
All this is very inconvenient, therefore, in reality, only some special types of volumes are used in this way to connect: configMap, secret, emptyDir, hostPath:
-
configMap and secret - service volumes, allow you to create a volume in the container with files from Kubernetes manifests.
-
emptyDir - temporary volume, created only for the lifetime of the pod. Convenient to use for testing or storing temporary data. When a pod is deleted, the emptyDir volume is also deleted and all data is lost.
-
hostPath - allows you to mount inside the container with the application any directory on the local disk of the server on which the application is running - including /etc/kubernetes. This is not a secure feature, so security policies usually prohibit the use of this type of volume. Otherwise, an attacker's application will be able to mount the HTC Kubernetes directory inside its container and steal all the cluster certificates. Typically, hostPath volumes are only allowed to be used by system applications that run in the kube-system namespace.
are given in the documentation.
Method 2: Connecting to SC/PVC/PV pods
An alternative connection method is the concept of Storage class, PersistentVolumeClaim, PersistentVolume.
storage class stores connection parameters to the storage system.
PersistentVolumeClaim describes the requirements for the one that the application needs.
PersistentVolume stores access settings and volume status.
The essence of the idea: in the pod manifest, specify the volume of the PersistentVolumeClaim type and indicate the name of this entity in the claimName parameter.
The PersistentVolumeClaim manifest describes the requirements for a data volume that an application needs. Including:
- disk size;
- access method: ReadWriteOnce or ReadWriteMany;
- reference to the Storage class - in which data storage system we want to create a volume.
The Storage class manifest stores the type and settings for connecting to a storage system. The kublet needs them to mount the volume on its node.
PersistentVolume manifests specify the Storage class and access parameters for a particular volume (volume ID, path, etc.).
When creating a PVC, Kubernetes looks at the volume of what size and from which Storage class it will need, and selects a free PersistentVolume.
If such PVs are not available, Kubernetes can run a special program - Provisioner (its name is indicated in the Storage class). This program connects to the storage system, creates a volume of the required size, obtains an identifier, and creates a PersistentVolume manifest in the Kubernetes cluster, which is associated with the PersistentVolumeClaim.
All this set of abstractions allows you to remove information about which storage system the application works with from the application manifest level to the administration level.
All storage connection settings are in the Storage class, which is the responsibility of cluster administrators. All that needs to be done when moving from AWS to Google Cloud is to change the name of the Storage class to PVC in the application manifests. Persistance Volume for data storage will be created in the cluster automatically using the Provisioner program.
Method 3. Container Storage Interface
All code that interacts with various storage systems is part of the Kubernetes core. The release of bug fixes or new functionality is tied to new releases, the code has to be changed for all supported versions of Kubernetes. All this is hard to maintain and add new functionality.
To solve the problem, developers from Cloud Foundry, Kubernetes, Mesos and Docker created the Container Storage Interface (CSI) - a simple unified interface that describes the interaction of a container management system and a special driver (CSI Driver) that works with a specific storage system. All code for interacting with the storage system was moved from the Kubernetes core to a separate system.
.
As a rule, CSI Driver consists of two components: Node Plugin and Controller plugin.
The Node Plugin runs on each node and is responsible for volume mounting and operations. The Controller plugin interacts with the storage system: creates or deletes volumes, assigns access rights, etc.
While the old drivers remain in the Kubernetes core, they are no longer recommended to use and everyone is advised to install the CSI Driver specifically for the system with which they will work.
The innovation may scare those who are already accustomed to setting up data storage through the Storage class, but in fact nothing terrible has happened. For programmers, nothing changes for sure - they both worked only with the name of the Storage class, and will continue. For administrators, the helm chart installation has been added and the settings structure has changed. If earlier the settings were entered directly into the Storage class, now they must first be set in the helm chart, and then in the Storage class. If you figure it out, nothing bad happened.
Let's look at an example of what benefits can be obtained by switching to connecting Ceph storage using the CSI driver.
When working with Ceph, the CSI plugin provides more options for working with storage systems than the built-in drivers.
- Dynamic disk creation. Typically, RBDs are only used in RWO mode, while CSI for Ceph allows them to be used in RWX mode. Several pods on different nodes can mount the same RDB disk to their nodes and work with them in parallel. In fairness, not everything is so radiant - this disk can only be connected as a block device, that is, you will have to adapt the application to work with it in multiple access mode.
- Snapshot creation. On a Kubernetes cluster, you can create a manifest that requires you to create a snapshot. The CSI plugin will see it and take a snapshot from the disk. Based on it, it will be possible to make either a backup or a copy of PersistentVolume.
- Increasing disk size on storage and PersistentVolume in a Kubernetes cluster.
- Quotas. CephFS drivers built into Kubernetes do not support quotas, and fresh CSI plugins with fresh Ceph Nautilus can enable quotas on CephFS partitions.
- Metrics. The CSI plugin can give Prometheus a lot of metrics about which volumes are mounted, which interactions are taking place, etc.
- topology aware. Allows you to specify in manifests how the cluster is geographically distributed and avoid connecting to pods running in London on a storage system located in Amsterdam.
How to connect Ceph to a Kubernetes cluster via CSI, see . You can also subscribe to which will be launched on October 15th.
Author of the article: Sergey Bondarev, practicing Southbridge architect, Certified Kubernetes Administrator, one of the developers of kubespray.
A little Post Scriptum is not for advertising, but for the sake of ...
PS Sergey Bondarev leads two intensives: updated September 28-30 and advanced October 14β16.
Source: habr.com