🥇Isibonelo esisebenzayo sokuxhuma isitoreji esisekelwe ku-Ceph kuqoqo le-Kubernetes

I-Container Storage Interface (CSI) iwukuxhumana okuhlanganisiwe phakathi kwe-Kubernetes nezinhlelo zokugcina. Sesike sakhuluma ngakho kafushane utshele, futhi namuhla sizobhekisisa inhlanganisela ye-CSI ne-Ceph: sizobonisa ukuthi kanjani xhuma i-Ceph storage kwiqoqo leKubernetes.
I-athikili ihlinzeka ngezibonelo zangempela, nakuba zenziwe lula ukuze kube lula ukuqonda. Asicabangi ukufaka nokumisa amaqoqo e-Ceph ne-Kubernetes.

Uyazibuza ukuthi kusebenza kanjani?

Ngakho-ke, uneqoqo le-Kubernetes ezandleni zakho, elisetshenzisiwe, ngokwesibonelo, kubespray. Kukhona iqoqo le-Ceph elisebenza eduze - ungalifaka futhi, isibonelo, nalokhu iqoqo lezincwadi zokudlala. Ngithemba ukuthi asikho isidingo sokusho ukuthi ukukhiqiza phakathi kwabo kufanele kube nenethiwekhi enomkhawulokudonsa okungenani we-10 Gbit / s.

Uma unakho konke lokhu, asihambe!

Okokuqala, ake siye kwenye ye-Ceph cluster node futhi sihlole ukuthi yonke into ihlelekile:

ceph health
ceph -s

Okulandelayo, sizokwakha ngokushesha ichibi lamadiski e-RBD:

ceph osd pool create kube 32
ceph osd pool application enable kube rbd

Asiqhubekele kwiqoqo le-Kubernetes. Lapho, okokuqala, sizofaka umshayeli we-Ceph CSI we-RBD. Sizofaka, njengoba kulindelekile, nge-Helm.
Sengeza inqolobane eneshadi, sithola isethi yezinto eziguquguqukayo zeshadi le-ceph-csi-rbd:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml

Manje udinga ukugcwalisa ifayela elithi cephrbd.yml. Ukuze wenze lokhu, thola i-ID yeqoqo namakheli e-IP wabaqaphi ku-Ceph:

ceph fsid  # так мы узнаем clusterID
ceph mon dump  # а так увидим IP-адреса мониторов

Sifaka amanani atholiwe kufayela elithi cephrbd.yml. Ngesikhathi esifanayo, sinika amandla ukudalwa kwezinqubomgomo ze-PSP (Izinqubomgomo Zokuphepha Ze-Pod). Izinketho ezigabeni i-nodeplugin и umnikezeli asevele efayelini, angalungiswa njengoba kukhonjisiwe ngezansi:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true

Okulandelayo, okusele kithi ukufaka ishadi kuqoqo le-Kubernetes.

helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace

Kuhle, umshayeli we-RBD uyasebenza!
Masidale i-StorageClass entsha ku-Kubernetes. Lokhu futhi kudinga ukuthintana kancane noCeph.

Sidala umsebenzisi omusha ku-Ceph futhi simnike amalungelo okubhalela ichibi ikhiyubhu:

ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'

Manje ake sibone ukhiye wokufinyelela usekhona:

ceph auth get-key client.rbdkube

Umyalo uzokhipha okufana nalokhu:

AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==

Ake sengeze leli nani kokuthi Imfihlo kuqoqo le-Kubernetes - lapho silidinga khona umsebenzisiKey:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  # Значения ключей соответствуют имени пользователя и его ключу, как указано в
  # кластере Ceph. ID юзера должен иметь доступ к пулу,
  # указанному в storage class
  userID: rbdkube
  userKey: <user-key>

Futhi sakha imfihlo yethu:

kubectl apply -f secret.yaml

Okulandelayo, sidinga i-StorageClass manifest into efana nale:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   # Эти секреты должны содержать данные для авторизации
   # в ваш пул.
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

Idinga ukugcwaliswa iqoqoID, esesivele siyifundile yiqembu ceph fsid, futhi usebenzise le-manifest kuqoqo le-Kubernetes:

kubectl apply -f storageclass.yaml

Ukuze uhlole ukuthi amaqoqo asebenzisana kanjani, ake sakhe i-PVC elandelayo (Isimangalo Somthamo Oqhubekayo):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

Ake sibone ngokushesha ukuthi uKubernetes udale kanjani ivolumu eceliwe ku-Ceph:

kubectl get pvc
kubectl get pv

Konke kubonakala kukuhle! Kubukeka kanjani lokhu ohlangothini lweCeph?
Sithola uhlu lwamavolumu echibini futhi sibuka ulwazi mayelana nevolumu yethu:

rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # тут, конечно же, будет другой ID тома, который выдала предыдущая команда

Manje ake sibone ukuthi ukukhulisa usayizi wevolumu ye-RBD kusebenza kanjani.
Shintsha usayizi wevolumu ku-manifest ye-pvc.yaml uye ku-2Gi futhi uyisebenzise:

kubectl apply -f pvc.yaml

Asilinde ukuthi izinguquko ziqale ukusebenza bese sibheka usayizi wevolumu futhi.

rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc

Siyabona ukuthi ubukhulu be-PVC abukashintshi. Ukuze uthole ukuthi kungani, ungabuza u-Kubernetes ngencazelo ye-YAML ye-PVC:

kubectl get pvc rbd-pvc -o yaml

Nansi inkinga:

umyalezo: Ilinde umsebenzisi ukuthi (kabusha) aqale i-pod ukuze aqedele usayizi wohlelo lwefayela wevolumu endaweni. uhlobo: FileSystemResizePending

Okusho ukuthi, i-disk ikhulile, kodwa uhlelo lwefayela olukulo aluzange.
Ukuze ukhulise isistimu yefayela, udinga ukukhweza ivolumu. Ezweni lakithi, i-PVC/PV edaliwe okwamanje ayisetshenziswa nganoma iyiphi indlela.

Singakha i-Pod yokuhlola, isibonelo kanje:

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

Futhi manje ake sibheke i-PVC:

kubectl get pvc

Usayizi ushintshile, konke kuhamba kahle.

Engxenyeni yokuqala, sisebenze nedivayisi ye-block ye-RBD (imele i-Rados Block Device), kodwa lokhu akunakwenziwa uma ama-microservices ahlukene adinga ukusebenza nale disk ngesikhathi esisodwa. I-CephFS ifaneleka kangcono ukusebenza ngamafayela kunezithombe zediski.
Sisebenzisa isibonelo samaqoqo e-Ceph ne-Kubernetes, sizomisa i-CSI nezinye izinhlangano ezidingekayo ukuze zisebenze ne-CephFS.

Ake sithole amanani eshadini elisha le-Helm esilidingayo:

helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml

Futhi udinga ukugcwalisa ifayela elithi cephfs.yml. Njengangaphambili, imiyalo kaCeph izosiza:

ceph fsid
ceph mon dump

Gcwalisa ifayela ngamavelu afana nalawa:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true

Sicela uqaphele ukuthi amakheli okuqapha acaciswe ekhelini elilula elithi:port. Ukuze ukhweze ama-cephf endaweni, lawa makheli adluliselwa kumojula ye-kernel, engakazi okwamanje ukuthi isebenza kanjani ne-v2 monitor protocol.
Sishintsha imbobo ye-httpMetrics (i-Prometheus izoya lapho ukuze ihlole amamethrikhi) ukuze ingangqubuzani ne-nginx-proxy, efakwe yi-Kubespray. Ungase ungakudingi lokhu.

Faka ishadi le-Helm kuqoqo le-Kubernetes:

helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace

Ake siye esitolo sedatha ye-Ceph ukuze sakhe umsebenzisi ohlukile lapho. Imibhalo ithi umhlinzeki we-CephFS udinga amalungelo okufinyelela omlawuli weqoqo. Kodwa sizodala umsebenzisi ohlukile fs namalungelo anomkhawulo:

ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'

Futhi masibheke ngokushesha ukhiye wakhe wokufinyelela, sizowudinga kamuva:

ceph auth get-key client.fs

Masidale i- Secret and StorageClass ehlukene.
Akukho okusha, sesivele sikubonile lokhu esibonelweni se-RBD:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  # Необходимо для динамически создаваемых томов
  adminID: fs
  adminKey: <вывод предыдущей команды>

Ukusebenzisa i-manifest:

kubectl apply -f secret.yaml

Futhi manje - i-StorageClass ehlukile:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  # Имя файловой системы CephFS, в которой будет создан том
  fsName: cephfs

  # (необязательно) Пул Ceph, в котором будут храниться данные тома
  # pool: cephfs_data

  # (необязательно) Разделенные запятыми опции монтирования для Ceph-fuse
  # например:
  # fuseMountOptions: debug

  # (необязательно) Разделенные запятыми опции монтирования CephFS для ядра
  # См. man mount.ceph чтобы узнать список этих опций. Например:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # Секреты должны содержать доступы для админа и/или юзера Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # (необязательно) Драйвер может использовать либо ceph-fuse (fuse), 
  # либо ceph kernelclient (kernel).
  # Если не указано, будет использоваться монтирование томов по умолчанию,
  # это определяется поиском ceph-fuse и mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

Asigcwalise lapha iqoqoID futhi kusebenza ku-Kubernetes:

kubectl apply -f storageclass.yaml

wokuhlola

Ukuze uhlole, njengasesibonelweni sangaphambilini, ake sakhe i-PVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

Futhi hlola ubukhona be-PVC/PV:

kubectl get pvc
kubectl get pv

Uma ufuna ukubuka amafayela nezinkomba ku-CephFS, ungafaka lolu hlelo lwefayela endaweni ethile. Isibonelo njengoba kukhonjisiwe ngezansi.

Ake siye kwenye ye-Ceph cluster node futhi senze lezi zenzo ezilandelayo:

# Точка монтирования
mkdir -p /mnt/cephfs

# Создаём файл с ключом администратора
ceph auth get-key client.admin >/etc/ceph/secret.key

# Добавляем запись в /etc/fstab
# !! Изменяем ip адрес на адрес нашего узла
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs

Vele, ukukhweza i-FS ku-Ceph node efana nalena kulungele izinjongo zokuqeqesha kuphela, okuyinto esiyenzayo kuhlelo lwethu lokusebenza. Izifundo ze-slurm. Angicabangi ukuthi kukhona ongenza lokhu ekukhiqizeni; kunengozi enkulu yokusula amafayela abalulekile ngephutha.

Futhi ekugcineni, ake sibheke ukuthi izinto zisebenza kanjani ngokushintsha usayizi wamavolumu esimweni se-CephFS. Masibuyele ku-Kubernetes futhi sihlele i-manifest yethu ye-PVC - sandise usayizi lapho, ngokwesibonelo, ukuya ku-7Gi.

Masisebenzise ifayela elihleliwe:

kubectl apply -f pvc.yaml

Ake sibheke uhla lwemibhalo ukuze sibone ukuthi i-quota ishintshe kanjani:

getfattr -n ceph.quota.max_bytes <каталог-с-данными>

Ukuze lo myalo usebenze, kungase kudingeke ukuthi ufake iphakheji kusistimu yakho attr.

Amehlo ayesaba, kodwa izandla ziyesaba

Zonke lezi ziphonso kanye nokubonakaliswa okude kwe-YAML kubonakala kuyinkimbinkimbi ngaphezulu, kodwa empeleni, abafundi be-Slurm bathola ukulengiswa kwakho ngokushesha okukhulu.
Kulesi sihloko asingenanga kakhulu ehlathini - kukhona imibhalo esemthethweni yalokho. Uma unentshisekelo emininingwaneni yokusetha isitoreji se-Ceph ngeqoqo le-Kubernetes, lezi zixhumanisi zizosiza:

Izimiso ezijwayelekile ze-Kubernetes esebenza ngamavolumu
Imibhalo ye-RBD
Ukuhlanganisa i-RBD ne-Kubernetes ngokombono we-Ceph
Ukuhlanganisa i-RBD ne-Kubernetes ngokombono we-CSI
Imibhalo Ye-CephFS Ejwayelekile
Ukuhlanganisa i-CephFS ne-Kubernetes ngokombono we-CSI

Esifundweni se-Slurm Kubernetes Base ungaqhubeka kancane futhi usebenzise uhlelo lokusebenza lwangempela ku-Kubernetes oluzosebenzisa i-CephFS njengokugcina ifayela. Ngezicelo ze-GET/POST uzokwazi ukudlulisela amafayela futhi uwathole ku-Ceph.

Futhi uma unentshisekelo enkulu yokugcina idatha, bhalisela isifundo esisha ku-Ceph. Ngenkathi ukuhlolwa kwe-beta kusaqhubeka, isifundo singatholwa ngesaphulelo futhi ungathonya okuqukethwe kwayo.

Umbhali walesi sihloko: Alexander Shvalov, unjiniyela I-Southbridge, Umqondisi Oqinisekisiwe we-Kubernetes, umbhali kanye nomthuthukisi wezifundo ze-Slurm.

Source: www.habr.com

Isibonelo esisebenzayo sokuxhuma isitoreji esisekelwe ku-Ceph kuqoqo le-Kubernetes

wokuhlola

Amehlo ayesaba, kodwa izandla ziyesaba

Yuri gagarin