🥇Praktičan primer povezivanja skladišta zasnovanog na Ceph-u u Kubernetes klaster

Container Storage Interface (CSI) je enotna vmesna plast za interakcijo med Kubernetesom in sistemi za shranjevanje podatkov. Kratko o njem smo že su pričali, danes pa bomo podrobneje obravnavali povezavo CSI in Ceph: prikazali bomo, kako povezati shrambo Ceph s Kubernetes klastrom.
V članku so predstavljeni resnični, čeprav nekoliko poenostavljeni primeri za lažje razumevanje. Namestitve in konfiguracije Ceph in Kubernetes klastrov ne obravnavamo.

Vas zanima, kako to deluje?

Torej, imate na voljo Kubernetes klaster, razporejen, na primer, kubespray. Ob njem deluje klaster Ceph — ta se lahko prav tako namesti, na primer, s tem naborom igralk. Upam, da ni treba omenjati, da mora biti za produkcijo med njima omrežje s pasovno širino vsaj 10 Gb/s.

Če imate vse to, gremo naprej!

Najprej se prijavimo na eno od vozlišč klasterja Ceph in preverimo, ali je vse v redu:

ceph health
ceph -s

Nato takoj ustvarimo bazen za RBD diske:

ceph osd pool create kube 32
ceph osd pool application enable kube rbd

Preidemo v Kubernetes klster. Tam bomo najprej namestili Ceph CSI gonilnik za RBD. Namestitev bomo izvedli, kot je potrebno, preko Helma.
Dodajamo repozitorij s chartom in pridobimo niz spremenljivk charta ceph-csi-rbd:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml

Zdaj je treba izpolniti datoteko cephrbd.yml. Za to izvedemo ID klasterja in IP-naslove monitorjev v Ceph:

ceph fsid  # tako ugotovimo clusterID
ceph mon dump  # in tako bomo videli IP-naslove monitorjev

Pridobljene vrednosti vnesemo v datoteko cephrbd.yml. Hkrati vklopimo ustvarjanje politik PSP (Pod Security Policies). Možnosti v razdelkih nodeplugin in provisioner so že v datoteki, jih lahko popravimo tako, kot je razloženo spodaj:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true

Nato je vse, kar nam ostane, namestiti chart v Kubernetes kluster.

helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace

Odlično, RBD gonilnik deluje!
Ustvarimo v Kubernetes nov StorageClass. Za to bo treba spet malo delati s Ceph.

Ustvarjamo novega uporabnika v Ceph in mu dodelimo pravice za pisanje v bazen kube:

ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'

Zdaj pa si oglejmo ključe dostopa vse tam:

ceph auth get-key client.rbdkube

Ukaz bo prikazal nekaj takega:

AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==

Zabeležimo to vrednost v Secret v klastru Kubernetes — tja, kjer jo potrebujemo userKey:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  # Ključne vrijednosti odgovaraju korisničkom imenu i njegovom ključu, kako je navedeno u
  # Ceph klasteru. ID korisnika mora imati pristup bazenu,
  # navedenom u storage class
  userID: rbdkube
  userKey:

I stvaramo naš tajni ključ:

kubectl apply -f secret.yaml

Zatim nam treba otprilike ovakav manifest StorageClass:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 
   pool: kube

   imageFeatures: layering

   # Ovi tajni podaci trebaju sadržavati informacije za autentifikaciju
   # u vaš bazen.
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

Treba popuniti clusterID, kojeg smo već saznali naredbom ceph fsid, i primijeniti ovaj manifest u Kubernetes klasteru:

kubectl apply -f storageclass.yaml

Da bismo provjerili rad klastera u suradnji, stvorimo ovakav PVC (Persistent Volume Claim):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

Odmah ćemo provjeriti kako je Kubernetes stvorio traženi volumen u Ceph-u:

kubectl get pvc
kubectl get pv

Izgleda odlično! Kako to izgleda s Ceph strane?
Dobivamo popis volumena u bazenu i pregledavamo informacije o našem volumenu:

rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # ovdje će, naravno, biti drugačiji ID volumena koji je prethodna naredba izdala

Sada pogledajmo kako funkcionira promjena veličine RBD volumena.
Promijenimo veličinu volumena u manifestu pvc.yaml na 2Gi i primijenimo ga:

kubectl apply -f pvc.yaml

Sačekajmo da promjene stupe na snagu i još jednom provjerimo veličinu volumena.

rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc

Vidimo da veličina PVC nije promijenjena. Da bismo saznali razlog, možemo zatražiti od Kubernetesa opisati PVC u YAML formatu:

kubectl get pvc rbd-pvc -o yaml

A evo i problema:

message: Čekanje da korisnik (re-)pokrene pod za završetak promjene veličine datotečnog sustava volumena na čvoru. type: FileSystemResizePending

Dakle, disk se povećao, ali datotečni sustav na njemu — nije.
Da bismo povećali datotečni sustav, moramo montirati volumen. Naš PVC/PV trenutno nije u upotrebi.

Možemo stvoriti testni Pod, na primjer ovako:

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

In now let's take a look at the PVC:

kubectl get pvc

The size has changed, everything is fine.

In the first part we worked with the RBD block device (which stands for Rados Block Device), but this approach is not suitable if different microservices need to access this disk concurrently. For working with files rather than a disk image, CephFS is a much better choice.
We will set up CSI and the other necessary entities for working with CephFS using Ceph and Kubernetes clusters as an example.

We will obtain values from the relevant new Helm chart:

helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml

Once again, we need to fill in the cephfs.yml file. As before, Ceph commands will assist us:

ceph fsid
ceph mon dump

We fill in the values file approximately like this:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true

Note that the addresses of the monitors are specified in a simple form address:port. To mount cephfs on the node, these addresses are passed to the kernel module, which does not yet support the v2 monitor protocol.
We change the port for httpMetrics (where Prometheus will fetch metrics for monitoring) so that it does not conflict with the nginx-proxy set up by Kubespray. You might not need this.

We install the Helm chart in the Kubernetes cluster:

helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace

We go to the Ceph data storage to create a separate user there. The documentation states that the CephFS provisioner needs admin privileges to the cluster. However, we will create a separate user fs with limited permissions:

ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'

And now let's check its access key, which will be useful later:

ceph auth get-key client.fs

We will create separate Secret and StorageClass.
Nothing new, we've already seen this with RBD:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  # Required for dynamically created volumes
  adminID: fs
  adminKey:

We apply the manifest:

kubectl apply -f secret.yaml

And now – a separate StorageClass:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: 

  # The name of the CephFS file system where the volume will be created
  fsName: cephfs

  # (optional) Ceph pool where the volume data will be stored
  # pool: cephfs_data

  # (optional) Comma-separated mount options for Ceph-fuse
  # e.g.:
  # fuseMountOptions: debug

  # (optional) Comma-separated mount options for CephFS for kernel
  # See man mount.ceph for a list of these options. For example:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # Secrets must contain access for admin and/or user of Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # (optional) The driver can use either ceph-fuse (fuse), 
  # or ceph kernelclient (kernel).
  # If not specified, the default volume mounting will be used,
  # determined by searching for ceph-fuse and mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

Let's fill this in here clusterID and apply it in Kubernetes:

kubectl apply -f storageclass.yaml

Проверка

To check, as in the previous example, we will create a PVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

And check for the presence of PVC/PV:

kubectl get pvc
kubectl get pv

If you want to look at the files and directories in CephFS, you can mount this file system somewhere. For example, as shown below.

Let's go to one of the nodes in the Ceph cluster and perform the following actions:

# Точка монтирования
mkdir -p /mnt/cephfs

# Создаём файл с ключом администратора
ceph auth get-key client.admin >/etc/ceph/secret.key

# Добавляем запись в /etc/fstab
# !! Изменяем ip адрес на адрес нашего узла
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs

Of course, this kind of FS mounting on a Ceph node is suitable only for educational purposes, which we are engaged in on our Slurm courses. I don't think anyone would do this in production, as there is a high risk of accidentally overwriting important files.

And lastly, let's check how the volume resizing works with CephFS. We return to Kubernetes and edit our PVC manifest — let's increase the size, for example, to 7Gi.

We will apply the edited file:

kubectl apply -f pvc.yaml

Let's check the mounted directory to see how the quota has changed:

getfattr -n ceph.quota.max_bytes

For this command to work, you may need to install the package in your system attr.

The eyes are afraid, but the hands are doing

Na prvi pogled, svi ovi alati i dugi YAML manifesti izgledaju složeno, ali u praksi studenti Slërma se brzo snalaze sa njima.
U ovom članku se nećemo zateći u dubinama — za to postoji zvanična dokumentacija. Ako vas zanimaju detalji konfiguracije Ceph skladišta u vezi sa Kubernetes klasterom, pomoći će vam sljedeće veze:

Opšti principi rada Kubernetes-a sa volumenima
Dokumentacija o RBD-u
Integracija RBD-a i Kubernetes-a iz perspektive Ceph-a
Integracija RBD-a i Kubernetes-a iz perspektive CSI-a
Opšta dokumentacija o CephFS-u
Integracija CephFS-a i Kubernetes-a iz perspektive CSI-a

Na kursu Slërm Kubernetes Osnova možete da odete još malo dalje i implementirate stvarnu aplikaciju u Kubernetes-u koja će koristiti CephFS kao skladište za datoteke. Putem GET/POST zahteva moći ćete da šaljete datoteke i preuzimate ih iz Ceph-a.

A ako vas više zanimaju podaci za skladištenje, prijavite se na novi kurs o Ceph-u. Dok traje beta test, kurs možete dobiti sa popustom i uticati na njegov sadržaj.

Autor članka: Aleksandar Švalov, praktični inženjer Southbridge, Certified Kubernetes Administrator, autor i developer kurseva Slërm.

Izvor: habr.com

Praktičen primer povezovanja shranjevanja na osnovi Ceph v Kubernetes klaster

Проверка

The eyes are afraid, but the hands are doing