A practical example of connecting a Ceph-based storage to a Kubernetes cluster

The Container Storage Interface (CSI) is a unified interface for interaction between Kubernetes and storage systems. We've briefly covered it already, talkedand today we will take a closer look at the connection between CSI and Ceph: we'll demonstrate how to connect Ceph storage to a Kubernetes cluster.
This article provides real, albeit slightly simplified examples for easier understanding. We will not discuss the installation and configuration of Ceph and Kubernetes clusters.

Are you interested in how this works?

So, you have a Kubernetes cluster set up, for example, using kubespray.Nearby, there is a Ceph cluster — it can also be deployed using, for example, this set of playbooks.I hope it’s unnecessary to mention that for production, there should be a network between them with a bandwidth of at least 10 Gbps.

If you have all this, let's get started!

First, let's log in to one of the Ceph cluster nodes and check that everything is fine:

ceph health
ceph -s

Next, we will create a pool for RBD disks:

ceph osd pool create kube 32
ceph osd pool application enable kube rbd

Now, let's move to the Kubernetes cluster. The first thing we'll do is install the Ceph CSI driver for RBD. We will install it as we should, using Helm.
We add the repository containing the chart and get a set of chart variables for ceph-csi-rbd:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml

Now we need to fill in the cephrbd.yml file. To do this, we will find out the cluster ID and the IP addresses of the monitors in Ceph:

ceph fsid  # this will show us the clusterID
ceph mon dump  # this will show the IP addresses of the monitors

We will insert the obtained values into the cephrbd.yml file. Simultaneously, we will enable the creation of PSP (Pod Security Policies). The options in the sections nodeplugin, and provisioner, are already in the file, and can be modified as shown below:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true

Next, all that remains is to install the chart in the Kubernetes cluster.

helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace

Great, the RBD driver is working!
Let's create a new StorageClass in Kubernetes. This will require a little more work with Ceph.

We will create a new user in Ceph and grant it write access to the pool kube.:

ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'

Now let's view the access key right there:

ceph auth get-key client.rbdkube

The command will output something like:

AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==

We will insert this value into a Secret in the Kubernetes cluster — where it's needed userKey.:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  # The key values correspond to the username and its key, as specified in
  # the Ceph cluster. The user ID must have access to the pool,
  # specified in the storage class
  userID: rbdkube
  userKey:

And we create our secret:

kubectl apply -f secret.yaml

Next, we need a manifest similar to this StorageClass:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 
   pool: kube

   imageFeatures: layering

   # These secrets must contain data for authentication
   # to your pool.
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

You need to fill in clusterID, which we already learned with the command ceph fsid, and apply this manifest in the Kubernetes cluster:

kubectl apply -f storageclass.yaml

To check the operation of clusters in tandem, let's create a PVC (Persistent Volume Claim) like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

Let's immediately see how Kubernetes created the requested volume in Ceph:

kubectl get pvc
kubectl get pv

Everything seems great! But how does it look on the Ceph side?
We get a list of volumes in the pool and view information about our volume:

rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # of course, there will be a different volume ID provided by the previous command

Now, let's see how RBD volume resizing works.
We change the volume size in the pvc.yaml manifest to 2Gi and apply it:

kubectl apply -f pvc.yaml

Let's wait for the changes to take effect and check the volume size again.

rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc

We see that the size of the PVC has not changed. To find out why, we can request the YAML description of the PVC from Kubernetes:

kubectl get pvc rbd-pvc -o yaml

And here is the problem:

message: Waiting for user to (re-)start a pod to finish file system resize of volume on node. type: FileSystemResizePending

This means the disk has increased, but the filesystem on it has not.
To enlarge the filesystem, the volume must be mounted. Our created PVC/PV is not being used in any way right now.

We can create a test Pod, for example like this:

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

And now let's take a look at the PVC:

kubectl get pvc

The size has changed, everything is fine.

In the first part, we worked with the RBD block device (which stands for Rados Block Device), but this approach isn't suitable when multiple microservices need simultaneous access to the disk. For file operations rather than disk images, CephFS is a much better fit.
Using the example of Ceph and Kubernetes clusters, we will configure CSI and the other necessary entities to work with CephFS.

We will extract values from the required new Helm chart:

helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml

We need to fill in the cephfs.yml file again. As before, Ceph commands will help:

ceph fsid
ceph mon dump

We fill in the values file like this:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true

Note that the addresses of the monitors are specified in the straightforward address:port format. For mounting cephfs on the node, these addresses are passed to the kernel module, which does not yet support v2 monitor protocols.
We change the port for httpMetrics (where Prometheus will request metrics for monitoring) to avoid conflicts with the nginx-proxy installed by Kubespray. You might not need to do this.

We install the Helm chart in the Kubernetes cluster:

helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace

Now, let's move to the Ceph data storage to create a separate user there. The documentation states that the CephFS provisioner requires cluster administrator access. However, we will create a separate user fs with limited permissions:

ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'

And let's immediately check its access key, which we will need later:

ceph auth get-key client.fs

We will create separate Secret and StorageClass.
Nothing new; we have seen this before with RBD:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  # Necessary for dynamically created volumes
  adminID: fs
  adminKey:

We apply the manifest:

kubectl apply -f secret.yaml

And now – a separate StorageClass:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: 

  # Name of the CephFS file system where the volume will be created
  fsName: cephfs

  # (optional) Pool in Ceph where the volume data will be stored
  # pool: cephfs_data

  # (optional) Comma-separated mount options for Ceph-fuse
  # for example:
  # fuseMountOptions: debug

  # (optional) Comma-separated mount options for CephFS for kernel
  # See man mount.ceph for a list of these options. For example:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # Secrets must contain access tokens for admin and/or Ceph user.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # (optional) The driver can use either ceph-fuse (fuse), 
  # or ceph kernelclient (kernel).
  # If not specified, the default volume mounting will be used,
  # determined by searching for ceph-fuse and mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

Let's fill this in clusterID and apply it in Kubernetes:

kubectl apply -f storageclass.yaml

Check

To verify, as in the previous example, let's create a PVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

And check for the presence of PVC/PV:

kubectl get pvc
kubectl get pv

If you want to look at the files and directories in CephFS, you can mount this file system somewhere. For example, as shown below.

Let's go to one of the nodes in the Ceph cluster and perform the following actions:

# Точка монтирования
mkdir -p /mnt/cephfs

# Создаём файл с ключом администратора
ceph auth get-key client.admin >/etc/ceph/secret.key

# Добавляем запись в /etc/fstab
# !! Изменяем ip адрес на адрес нашего узла
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs

Of course, this kind of FS mounting on a Ceph node is only suitable for training purposes, which is what we're doing in our Slurm courses. I don't think anyone would do this in production due to the high risk of accidentally deleting important files.

Lastly, let's check how resizing a volume works in CephFS. We return to Kubernetes and edit our PVC manifest — let's increase the size, for example, to 7Gi.

We will apply the edited file:

kubectl apply -f pvc.yaml

Let's look in the mounted directory to see how the quota has changed:

getfattr -n ceph.quota.max_bytes

You might need to install the package system for this command to work attr.

Fear makes the eyes wide, but hands get to work

At first glance, all these spells and lengthy YAML manifests may seem complicated, but in practice, Slayer students grasp them quite quickly.
In this article, we did not delve into the intricacies — for that, there is official documentation. If you're interested in details on configuring Ceph storage with a Kubernetes cluster, these links will help:

General Principles of Kubernetes with Volumes
RBD Documentation
Integration of RBD and Kubernetes from the Perspective of Ceph
Integration of RBD and Kubernetes from the Perspective of CSI
General Documentation on CephFS
Integration of CephFS and Kubernetes from the Perspective of CSI

In the Slayer course Kubernetes Base , you can go a bit further and deploy a real application in Kubernetes that will use CephFS as storage for files. Through GET/POST requests, you will be able to transfer files and retrieve them from Ceph.

If you're more interested in data storage, then sign up for the new course on Ceph. While beta testing is ongoing, the course can be obtained at a discount, and you can influence its content.

Author of the article: Alexander Shvalov, practicing engineer Southbridge, Certified Kubernetes Administrator, author and developer of Slayer courses.

Source: habr.com