Chitsanzo chothandiza cholumikiza kusungirako kwa Ceph ku gulu la Kubernetes

Container Storage Interface (CSI) ndi mawonekedwe ogwirizana pakati pa Kubernetes ndi makina osungira. Takambirana kale mwachidule anauza, ndipo lero tiwona mwatsatanetsatane kuphatikiza kwa CSI ndi Ceph: tiwonetsa momwe kulumikiza Ceph yosungirako ku gulu la Kubernetes.
Nkhaniyi ili ndi zitsanzo zenizeni, ngakhale zophweka pang'ono kuti mumvetsetse bwino. Sitiganizira kukhazikitsa ndi kukonza magulu a Ceph ndi Kubernetes.

Kodi mukudabwa momwe zimagwirira ntchito?

Chifukwa chake, muli ndi gulu la Kubernetes m'manja mwanu, loyikidwa, mwachitsanzo, kukhalaspray. Pali gulu la Ceph lomwe likugwira ntchito pafupi - mutha kuyiyikanso, mwachitsanzo, ndi izi gulu la playbooks. Ndikuyembekeza kuti palibe chifukwa chonena kuti kupanga pakati pawo payenera kukhala maukonde okhala ndi bandwidth osachepera 10 Gbit / s.

Ngati muli ndi zonsezi, tiyeni tipite!

Choyamba, tiyeni tipite ku imodzi mwa magulu a Ceph ndikuwonetsetsa kuti zonse zili bwino:

ceph health
ceph -s

Kenako, nthawi yomweyo tipanga dziwe la ma disks a RBD:

ceph osd pool create kube 32
ceph osd pool application enable kube rbd

Tiyeni tipitirire ku gulu la Kubernetes. Kumeneko, choyamba, tidzakhazikitsa dalaivala wa Ceph CSI wa RBD. Tidzakhazikitsa, monga tikuyembekezeredwa, kudzera mu Helm.
Timawonjezera malo okhala ndi tchati, timapeza zosinthika za tchati cha ceph-csi-rbd:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml

Tsopano muyenera kudzaza fayilo ya cephrbd.yml. Kuti muchite izi, pezani ma ID a cluster ndi ma adilesi a IP a oyang'anira mu Ceph:

ceph fsid  # так мы узнаем clusterID
ceph mon dump  # а так увидим IP-адреса мониторов

Timalowetsa zomwe tapeza mu fayilo ya cephrbd.yml. Panthawi imodzimodziyo, timathandiza kupanga ndondomeko za PSP (Pod Security Policies). Zosankha m'magawo nodeplugin и wopereka zomwe zili mufayilo, zitha kuwongoleredwa monga zikuwonetsedwa pansipa:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true

Chotsatira, chomwe chatsalira kwa ife ndikuyika tchati mu gulu la Kubernetes.

helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace

Zabwino, dalaivala wa RBD amagwira ntchito!
Tiyeni tipange StorageClass yatsopano ku Kubernetes. Izi zimafunanso kuwongolera pang'ono ndi Ceph.

Timapanga wogwiritsa ntchito watsopano ku Ceph ndikumupatsa ufulu wolembera dziwe kyubu:

ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'

Tsopano tiyeni tiwone kiyi yolowera ikadalipo:

ceph auth get-key client.rbdkube

Lamulo lidzatulutsa chonchi:

AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==

Tiyeni tiwonjezere mtengo uwu Chinsinsi mu gulu la Kubernetes - komwe timafunikira userKey:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  # Значения ключей соответствуют имени пользователя и его ключу, как указано в
  # кластере Ceph. ID юзера должен иметь доступ к пулу,
  # указанному в storage class
  userID: rbdkube
  userKey: <user-key>

Ndipo timapanga chinsinsi chathu:

kubectl apply -f secret.yaml

Kenako, tifunika StorageClass chiwonetsero chonga ichi:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   # Эти секреты должны содержать данные для авторизации
   # в ваш пул.
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

Iyenera kudzazidwa guluID, zomwe taphunzira kale ndi timu ceph fsid, ndikuyika chiwonetserochi ku gulu la Kubernetes:

kubectl apply -f storageclass.yaml

Kuti muwone momwe maguluwa amagwirira ntchito limodzi, tiyeni tipange PVC yotsatirayi (Persistent Volume Claim):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

Tiyeni tiwone nthawi yomweyo momwe Kubernetes adapangira voliyumu yomwe adafunsidwa ku Ceph:

kubectl get pvc
kubectl get pv

Chilichonse chikuwoneka bwino! Kodi izi zikuwoneka bwanji kumbali ya Ceph?
Timapeza mndandanda wa mavoliyumu mu dziwe ndikuwona zambiri za voliyumu yathu:

rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # тут, конечно же, будет другой ID тома, который выдала предыдущая команда

Tsopano tiyeni tiwone momwe kusinthira kukula kwa voliyumu ya RBD kumagwirira ntchito.
Sinthani kukula kwa voliyumu mu pvc.yaml manifest kukhala 2Gi ndikuyiyika:

kubectl apply -f pvc.yaml

Tiyeni tidikire kuti kusintha kuchitike ndikuyang'ananso kukula kwa voliyumu.

rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc

Tikuwona kuti kukula kwa PVC sikunasinthe. Kuti mudziwe chifukwa chake, mutha kufunsa Kubernetes kufotokozera kwa YAML pa PVC:

kubectl get pvc rbd-pvc -o yaml

Nali vuto:

uthenga: Kudikirira kuti wogwiritsa ntchito (re-) ayambitsenso pod kuti amalize kukula kwa fayilo pa node. mtundu: FileSystemResizePending

Ndiko kuti, disk yakula, koma mawonekedwe a fayilo pa iyo sanatero.
Kuti mukulitse fayilo ya fayilo, muyenera kukweza voliyumu. M'dziko lathu, PVC / PV yopangidwa sikugwiritsidwa ntchito mwanjira iliyonse.

Titha kupanga Pod yoyeserera, mwachitsanzo motere:

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

Tsopano tiyeni tiwone PVC:

kubectl get pvc

Kukula kwasintha, zonse zili bwino.

Mu gawo loyamba, tinagwira ntchito ndi chipangizo chotchinga cha RBD (chimayimira Rados Block Device), koma izi sizingachitike ngati ma microservices osiyana ayenera kugwira ntchito ndi disk iyi panthawi imodzi. CephFS ndiyoyenera kugwira ntchito ndi mafayilo osati zithunzi za disk.
Pogwiritsa ntchito chitsanzo cha magulu a Ceph ndi Kubernetes, tidzakonza CSI ndi mabungwe ena ofunikira kuti agwire ntchito ndi CephFS.

Tiyeni titenge zamtengo wapatali kuchokera ku Helm chart yatsopano yomwe tikufuna:

helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml

Apanso muyenera kudzaza fayilo ya cephfs.yml. Monga kale, malamulo a Ceph athandiza:

ceph fsid
ceph mon dump

Lembani fayiloyo ndi mfundo monga izi:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true

Chonde dziwani kuti ma adilesi owunika amatchulidwa mu adilesi yosavuta: port. Kuyika ma cephf pa node, ma adilesi awa amaperekedwa ku gawo la kernel, lomwe silikudziwa momwe angagwiritsire ntchito ndi v2 monitor protocol.
Timasintha doko la httpMetrics (Prometheus apita kumeneko kukayang'anira ma metrics) kuti asasemphane ndi nginx-proxy, yomwe imayikidwa ndi Kubespray. Mwina simungafune izi.

Ikani tchati cha Helm mu gulu la Kubernetes:

helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace

Tiyeni tipite ku sitolo ya data ya Ceph kuti tipange wogwiritsa ntchito wina kumeneko. Zolembazo zimati wopereka za CephFS amafunikira ufulu wofikira wowongolera magulu. Koma tidzapanga wogwiritsa ntchito wina fs ndi maufulu ochepa:

ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'

Ndipo tiyeni tiwone nthawi yomweyo kiyi yake yolowera, tidzayifuna pambuyo pake:

ceph auth get-key client.fs

Tiyeni tipange Secret and StorageClass yosiyana.
Palibe chatsopano, taziwona kale mu chitsanzo cha RBD:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  # Необходимо для динамически создаваемых томов
  adminID: fs
  adminKey: <вывод предыдущей команды>

Kugwiritsa ntchito chiwonetserochi:

kubectl apply -f secret.yaml

Ndipo tsopano - StorageClass yosiyana:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  # Имя файловой системы CephFS, в которой будет создан том
  fsName: cephfs

  # (необязательно) Пул Ceph, в котором будут храниться данные тома
  # pool: cephfs_data

  # (необязательно) Разделенные запятыми опции монтирования для Ceph-fuse
  # например:
  # fuseMountOptions: debug

  # (необязательно) Разделенные запятыми опции монтирования CephFS для ядра
  # См. man mount.ceph чтобы узнать список этих опций. Например:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # Секреты должны содержать доступы для админа и/или юзера Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # (необязательно) Драйвер может использовать либо ceph-fuse (fuse), 
  # либо ceph kernelclient (kernel).
  # Если не указано, будет использоваться монтирование томов по умолчанию,
  # это определяется поиском ceph-fuse и mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

Tiyeni tilembe apa guluID ndikugwira ntchito ku Kubernetes:

kubectl apply -f storageclass.yaml

kuyendera

Kuti muwone, monga momwe tawonera, tiyeni tipange PVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

Ndipo onani kukhalapo kwa PVC/PV:

kubectl get pvc
kubectl get pv

Ngati mukufuna kuyang'ana mafayilo ndi zolemba mu CephFS, mutha kuyika fayiloyi kwinakwake. Mwachitsanzo monga momwe zilili pansipa.

Tiyeni tipite ku imodzi mwa magulu a Ceph ndikuchita izi:

# Точка монтирования
mkdir -p /mnt/cephfs

# Создаём файл с ключом администратора
ceph auth get-key client.admin >/etc/ceph/secret.key

# Добавляем запись в /etc/fstab
# !! Изменяем ip адрес на адрес нашего узла
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs

Zachidziwikire, kuyika FS pa Ceph node ngati iyi ndi koyenera pazolinga zophunzitsira, zomwe ndizomwe timachita pazathu. Maphunziro a Slurm. Sindikuganiza kuti aliyense angachite izi popanga; pali chiopsezo chachikulu chochotsa mwangozi mafayilo ofunikira.

Ndipo pomaliza, tiyeni tiwone momwe zinthu zimagwirira ntchito ndikusinthira ma voliyumu pankhani ya CephFS. Tiyeni tibwerere ku Kubernetes ndikusintha chiwonetsero chathu cha PVC - onjezani kukula kumeneko, mwachitsanzo, mpaka 7Gi.

Tiyeni tigwiritse ntchito fayilo yosinthidwa:

kubectl apply -f pvc.yaml

Tiyeni tiwone chikwatu chokwera kuti tiwone momwe gawo lasinthira:

getfattr -n ceph.quota.max_bytes <каталог-с-данными>

Kuti lamuloli ligwire ntchito, mungafunike kuyika phukusi pakompyuta yanu attr.

Maso amachita mantha, koma manja amachita

Matchulidwe onsewa ndi mawonekedwe aatali a YAML amawoneka ovuta pamwamba, koma m'malo mwake, ophunzira a Slurm amawapeza mwachangu.
M'nkhaniyi sitinalowe m'nkhalango - pali zolemba zovomerezeka za izo. Ngati mukufuna tsatanetsatane wa kukhazikitsa Ceph yosungirako ndi gulu la Kubernetes, maulalo awa adzakuthandizani:

Mfundo zambiri za Kubernetes zikugwira ntchito ndi mavoliyumu
Zolemba za RBD
Kuphatikiza RBD ndi Kubernetes kuchokera ku Ceph
Kuphatikiza RBD ndi Kubernetes kuchokera ku CSI
General CephFS Documentation
Kuphatikiza CephFS ndi Kubernetes kuchokera ku CSI

Pa maphunziro a Slurm Kubernetes Base mutha kupita patsogolo pang'ono ndikuyika pulogalamu yeniyeni ku Kubernetes yomwe idzagwiritse ntchito CephFS ngati kusungira mafayilo. Kupyolera mu zopempha za GET/POST mudzatha kusamutsa mafayilo ndikuwalandira kuchokera ku Ceph.

Ndipo ngati mumakonda kwambiri kusungirako deta, lembani maphunziro atsopano pa Ceph. Ngakhale kuyesa kwa beta kukupitilira, maphunzirowa atha kugulidwa pamtengo wotsika ndipo mutha kukhudza zomwe zili.

Wolemba nkhaniyo: Alexander Shvalov, katswiri wodziwa ntchito Southbridge, Certified Kubernetes Administrator, wolemba ndi wopanga maphunziro a Slurm.

Source: www.habr.com