🥇Umzekelo osebenzayo wokudibanisa indawo yokugcina esekwe kwiCeph kwiqela leKubernetes

IsiBonelelo sokuGcinwa kweSikhongozeli (CSI) lunxibelelwano olumanyeneyo phakathi kweKubernetes kunye neenkqubo zokugcina. Sele sithethile ngayo ngokufutshane uxelelwe, kwaye namhlanje siza kujonga ngakumbi ukudibanisa kweCSI kunye neCeph: siya kubonisa indlela qhagamshela iCeph yokugcina ukuya kwiqela leKubernetes.
Inqaku libonelela ngemizekelo yokwenyani, nangona yenziwe lula kancinane ukuze kube lula ukuqonda. Asicingi ngokufaka kunye nokuqwalasela amaqela e-Ceph kunye ne-Kubernetes.

Ngaba uyazibuza ukuba isebenza njani?

Ke, uneqela le-Kubernetes ezandleni zakho, ibekwe, umzekelo, kubespray. Kukho iqela leCeph elisebenza kufutshane - ungayifaka kwakhona, umzekelo, ngale nto iseti yeencwadi zokudlala. Ndiyathemba ukuba akukho mfuneko yokukhankanya ukuba ukuveliswa phakathi kwabo kufuneka kubekho inethiwekhi ene-bandwidth ubuncinane ye-10 Gbit / s.

Ukuba unayo yonke le nto, masihambe!

Okokuqala, masiye kwenye yeenodi zeqela leCeph kwaye sijonge ukuba yonke into ilungile na:

ceph health
ceph -s

Okulandelayo, ngokukhawuleza siza kudala ichibi leediski ze-RBD:

ceph osd pool create kube 32
ceph osd pool application enable kube rbd

Masiqhubele phambili kwiqela leKubernetes. Apho, okokuqala, siya kufaka umqhubi weCeph CSI weRBD. Siza kufaka, njengoko kulindelekile, ngeHelm.
Songeza indawo yokugcina kunye netshati, sifumana uluhlu lwezinto eziguquguqukayo zetshathi ye-ceph-csi-rbd:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml

Ngoku kufuneka ugcwalise ifayile ye-cephrbd.yml. Ukwenza oku, fumana i-ID yeqela kunye needilesi ze-IP zabahloli kwi-Ceph:

ceph fsid  # так мы узнаем clusterID
ceph mon dump  # а так увидим IP-адреса мониторов

Sifaka amaxabiso afunyenweyo kwifayile ye-cephrbd.yml. Ngexesha elifanayo, senza ukuba kuqulunqwe imigaqo-nkqubo ye-PSP (iPolisi zoKhuseleko lwePod). Iinketho kumacandelo nodeplugin и umboneleli sele ikwifayile, inokulungiswa njengoko kubonisiwe ngezantsi:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true

Okulandelayo, konke okuseleyo kuthi kukufakela itshati kwiqela leKubernetes.

helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace

Kakhulu, umqhubi weRBD uyasebenza!
Masenze iClass entsha yokuGcina kwiKubernetes. Oku kwakhona kufuna kancinci ukukhenketha noCeph.

Senza umsebenzisi omtsha kwiCeph kwaye simnike amalungelo okubhala echibini Ityhubhu:

ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'

Ngoku makhe sibone ukuba isitshixo sokufikelela sisekho:

ceph auth get-key client.rbdkube

Umyalelo uya kukhupha into enje:

AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==

Masiyongeze eli xabiso kwiMfihlo kwiqela le-Kubernetes - apho silifuna khona owona msbenzisi:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  # Значения ключей соответствуют имени пользователя и его ключу, как указано в
  # кластере Ceph. ID юзера должен иметь доступ к пулу,
  # указанному в storage class
  userID: rbdkube
  userKey: <user-key>

Kwaye sidala imfihlo yethu:

kubectl apply -f secret.yaml

Okulandelayo, sifuna iStoreClass ibonakalise into enje:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   # Эти секреты должны содержать данные для авторизации
   # в ваш пул.
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

Kufuneka izaliswe clusterID, esele siyifundile liqela ceph fsid, kwaye usebenzise le manifesto kwiqela leKubernetes:

kubectl apply -f storageclass.yaml

Ukujonga indlela amaqela asebenza ngayo kunye, makhe senze le PVC ilandelayo (iBango loMqulu oZingisayo):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

Makhe sibone ngokukhawuleza ukuba uKubernetes udale njani umthamo oceliweyo kwiCeph:

kubectl get pvc
kubectl get pv

Yonke into ibonakala ilungile! Ijongeka njani lento kwicala likaCeph?
Sifumana uluhlu lwemithamo equleni kwaye sijonge ulwazi malunga nomthamo wethu:

rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # тут, конечно же, будет другой ID тома, который выдала предыдущая команда

Ngoku makhe sibone ukuba ukusebenza ngokutsha kwevolumu ye-RBD kusebenza njani.
Guqula ubungakanani bevolyum kwi-manifest ye-pvc.yaml ukuya ku-2Gi kwaye uyisebenzise:

kubectl apply -f pvc.yaml

Masilinde ukuba utshintsho lusebenze kwaye sijonge ubungakanani bevolumu kwakhona.

rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc

Siyabona ukuba ubukhulu bePVC abukatshintshi. Ukufumanisa ukuba kutheni, ungabuza uKubernetes ngenkcazo ye-YAML yePVC:

kubectl get pvc rbd-pvc -o yaml

Nantsi ingxaki:

umyalezo: Ilindele umsebenzisi ukuba (aphinde) aqale i-pod ukugqiba isixokelelwano sefayile yobungakanani bobungakanani bomthamo kwindawo. uhlobo: FileSystemResizePending

Oko kukuthi, idiski ikhulile, kodwa inkqubo yefayile kuyo ayikhange.
Ukukhulisa inkqubo yefayile, kufuneka unyuse umthamo. Kwilizwe lethu, i-PVC / PV eyenziwe ngoku ayisetyenziswanga nangayiphi na indlela.

Sinokwenza iPod yovavanyo, umzekelo onje:

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

Kwaye ngoku makhe sijonge iPVC:

kubectl get pvc

Ubungakanani butshintshile, yonke into ilungile.

Kwinxalenye yokuqala, sisebenze kunye nesixhobo sebhloko se-RBD (imele i-Rados Block Device), kodwa oku akunakwenziwa ukuba ii-microservices ezahlukeneyo kufuneka zisebenze nale diski ngaxeshanye. I-CephFS ilunge ngakumbi ekusebenzeni ngeefayile kunemifanekiso yedisk.
Ukusebenzisa umzekelo wamaqela e-Ceph kunye ne-Kubernetes, siya kumisela i-CSI kunye namanye amaziko ayimfuneko ukuze asebenze ne-CephFS.

Masifumane amaxabiso kwitshathi yeHelm entsha esiyidingayo:

helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml

Kwakhona kufuneka ugcwalise ifayile ye-cephfs.yml. Njengangaphambili, imiyalelo kaCeph iya kunceda:

ceph fsid
ceph mon dump

Gcwalisa ifayile ngamaxabiso anje:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true

Nceda uqaphele ukuba iidilesi zokubeka iliso zichazwe kwidilesi yefom elula: port. Ukunyusa i-cephfs kwi-node, ezi dilesi zigqithiselwa kwimodyuli ye-kernel, engekayazi indlela yokusebenza kunye ne-v2 monitor protocol.
Sitshintsha i-port ye-http Metrics (i-Prometheus iya kuya apho i-metrics esweni) ukuze ingangqubani ne-nginx-proxy, efakwe yi-Kubespray. Usenokungayifuni le nto.

Faka itshathi yeHelm kwiqela leKubernetes:

helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace

Makhe siye kwivenkile yedatha yeCeph ukwenza umsebenzisi owahlukileyo apho. Amaxwebhu achaza ukuba umboneleli weCephFS ufuna amalungelo okufikelela kumlawuli weqela. Kodwa siya kudala umsebenzisi owahlukileyo fs ngamalungelo alinganiselweyo:

ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'

Kwaye masijonge kwangoko isitshixo sakhe sokufikelela, siya kusidinga kamva:

ceph auth get-key client.fs

Masidale iMfihlo kunye neClass yokuGcina eyahlukileyo.
Akukho nto intsha, sele siyibonile oku kumzekelo we-RBD:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  # Необходимо для динамически создаваемых томов
  adminID: fs
  adminKey: <вывод предыдущей команды>

Ukusebenzisa i-manifest:

kubectl apply -f secret.yaml

Kwaye ngoku-Igumbi lokuGcina elahlukileyo:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  # Имя файловой системы CephFS, в которой будет создан том
  fsName: cephfs

  # (необязательно) Пул Ceph, в котором будут храниться данные тома
  # pool: cephfs_data

  # (необязательно) Разделенные запятыми опции монтирования для Ceph-fuse
  # например:
  # fuseMountOptions: debug

  # (необязательно) Разделенные запятыми опции монтирования CephFS для ядра
  # См. man mount.ceph чтобы узнать список этих опций. Например:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # Секреты должны содержать доступы для админа и/или юзера Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # (необязательно) Драйвер может использовать либо ceph-fuse (fuse), 
  # либо ceph kernelclient (kernel).
  # Если не указано, будет использоваться монтирование томов по умолчанию,
  # это определяется поиском ceph-fuse и mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

Masiyigcwalise apha clusterID kwaye iyasebenza eKubernetes:

kubectl apply -f storageclass.yaml

ukuhlola

Ukujonga, njengakumzekelo wangaphambili, masenze iPVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

Kwaye jonga ubukho bePVC/PV:

kubectl get pvc
kubectl get pv

Ukuba ufuna ukujonga iifayile kunye nabalawuli kwi-CephFS, ungayinyusa le nkqubo yefayile kwenye indawo. Umzekelo njengoko kubonisiwe ngezantsi.

Makhe siye kwenye yeenodi zeqela leCeph kwaye senze ezi ntshukumo zilandelayo:

# Точка монтирования
mkdir -p /mnt/cephfs

# Создаём файл с ключом администратора
ceph auth get-key client.admin >/etc/ceph/secret.key

# Добавляем запись в /etc/fstab
# !! Изменяем ip адрес на адрес нашего узла
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs

Ewe kunjalo, ukufaka iFS kwindawo yeCeph efana nale ilungele kuphela iinjongo zoqeqesho, yile nto siyenzayo yethu. Izifundo ze-slurm. Andiqondi ukuba nabani na onokwenza oku kwimveliso; kukho umngcipheko omkhulu wokucima iifayile ezibalulekileyo ngengozi.

Kwaye okokugqibela, makhe sijonge ukuba izinto zisebenza njani ngohlengahlengiso lweevolumu kwimeko yeCephFS. Masibuyele kwi-Kubernetes kwaye sihlele i-manifest yethu ye-PVC-yandisa ubungakanani apho, umzekelo, ukuya kwi-7Gi.

Masisebenzise ifayile ehleliweyo:

kubectl apply -f pvc.yaml

Makhe sijonge kuluhlu olunyusiweyo ukuze sibone ukuba i-quota itshintshe njani:

getfattr -n ceph.quota.max_bytes <каталог-с-данными>

Ukuze lo myalelo usebenze, kungafuneka ufakele ipakethe kwindlela yakho attr.

Amehlo ayoyika, kodwa izandla ziyayoyika

Zonke ezi zipela kunye nokubonakaliswa okude kwe-YAML kubonakala kuntsonkothile kumphezulu, kodwa ngokuziqhelanisa, abafundi be-Slurm bafumana ukuxhoma kubo ngokukhawuleza.
Kweli nqaku asizange singene nzulu ehlathini - kukho amaxwebhu asemthethweni aloo nto. Ukuba unomdla kwiinkcukacha zokuseta i-Ceph yokugcina kunye neqela le-Kubernetes, la makhonkco aya kunceda:

Imigaqo ngokubanzi yeKubernetes esebenza ngemiqulu
Uxwebhu lweRBD
Ukudibanisa i-RBD kunye ne-Kubernetes ukusuka kumbono weCeph
Ukudibanisa i-RBD kunye ne-Kubernetes ukusuka kumbono we-CSI
Uxwebhu lweCephFS ngokubanzi
Ukudibanisa i-CephFS kunye ne-Kubernetes ukusuka kumbono we-CSI

Kwikhosi yeSlurm Kubernetes Base ungaya phambili kancinci kwaye usebenzise usetyenziso lokwenyani kwi-Kubernetes eya kusebenzisa i-CephFS njengogcino lwefayile. Ngezicelo ze-GET/POST uya kuba nako ukuthumela iifayile kwaye uzifumane kuCeph.

Kwaye ukuba unomdla ngakumbi kugcino lwedatha, ke ubhalisele ikhosi entsha kuCeph. Ngelixa uvavanyo lwe-beta luqhubeka, ikhosi inokufumaneka ngesaphulelo kwaye unokuphembelela umxholo wayo.

Umbhali wenqaku: Alexander Shvalov, injineli eqhubayo I-Southbridge, Umlawuli oqinisekisiweyo we-Kubernetes, umbhali kunye nomphuhlisi wezifundo ze-Slurm.

umthombo: www.habr.com

Umzekelo osebenzayo wokudibanisa i-Ceph-based storage ukuya kwiqela le-Kubernetes

ukuhlola

Amehlo ayoyika, kodwa izandla ziyayoyika