ProHoster > Blog > Pulega > O se faʻataʻitaʻiga faʻapitoa o le faʻafesoʻotaʻi o le teuina o le Ceph i le Kubernetes cluster

O se faʻataʻitaʻiga faʻapitoa o le faʻafesoʻotaʻi o le teuina o le Ceph i le Kubernetes cluster

Container Storage Interface (CSI) ose feso'ota'iga tu'ufa'atasi i le va o Kubernetes ma faiga teu. Ua uma ona tatou talanoa puupuu i ai taʻu, ma i aso nei o le a tatou vaʻavaʻai totoʻa i le tuʻufaʻatasiga o CSI ma Ceph: o le a tatou faʻaalia pe faʻafefea faʻafesoʻotaʻi le teuina o Ceph i le fuifui Kubernetes.
O le tusiga o loʻo tuʻuina mai ai faʻataʻitaʻiga moni, e ui lava ina faʻafaigofie ona faʻafaigofie ona iloa. Matou te le mafaufau e faʻapipiʻi ma faʻapipiʻi fuifui Ceph ma Kubernetes.

O e mafaufau pe fa'apefea ona galue?

O se faʻataʻitaʻiga faʻapitoa o le faʻafesoʻotaʻi o le teuina o le Ceph i le Kubernetes cluster

O lea, o loʻo i ai sau faʻapipiʻi Kubernetes i ou tamatamai lima, faʻapipiʻi, mo se faʻataʻitaʻiga, kubespray. O loʻo i ai se fuifui Ceph o loʻo galue lata ane - e mafai foi ona e faʻapipiʻiina, mo se faʻataʻitaʻiga, ma lenei se seti o tusi taalo. Ou te faʻamoemoe e leai se manaʻoga e taʻu atu mo le gaosiga i le va oi latou e tatau ona i ai se fesoʻotaʻiga ma se bandwidth e le itiiti ifo i le 10 Gbit / s.

Afai o ia te oe nei mea uma, ta o!

Muamua, tatou o atu i se tasi o le Ceph cluster nodes ma siaki o loʻo lelei mea uma:

ceph health
ceph -s

Ma le isi, matou te vave faia se vaitaele mo RBD disks:

ceph osd pool create kube 32
ceph osd pool application enable kube rbd

Se'i o tatou aga'i atu i le vaega Kubernetes. O iina, muamua lava, matou te faʻapipiʻi le avetaʻavale Ceph CSI mo RBD. O le a matou faʻapipiʻi, e pei ona faʻamoemoeina, e ala i le Helm.
Matou te faʻaopoopoina se fale teu oloa ma se siata, matou te maua se seti o fesuiaiga mo le siata ceph-csi-rbd:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml

Ole taimi nei e te manaʻomia e faʻatumu le faila cephrbd.yml. Ina ia faia lenei mea, suʻe le ID faʻapipiʻi ma tuatusi IP o siaki i Ceph:

ceph fsid  # так мы узнаем clusterID
ceph mon dump  # а так увидим IP-адреса мониторов

Matou te tuʻuina faʻatauga na maua i le faila cephrbd.yml. I le taimi lava e tasi, matou te faʻatagaina le fausiaina o faiga faʻavae PSP (Pod Security Policies). Filifiliga i vaega nodeplugin и foa'i ua i ai i le faila, e mafai ona faʻasaʻo e pei ona faʻaalia i lalo:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true

O le isi, pau lava le mea e totoe mo i matou o le faʻapipiʻiina lea o le siata i le vaega Kubernetes.

helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace

Lelei, galue le avetaavale RBD!
Se'i tatou faia se StorageClass fou ile Kubernetes. O lenei mea e manaʻomia ai sina faʻataʻitaʻiga ma Ceph.

Matou te fatuina se tagata fou i Ceph ma tuʻuina atu ia te ia aia tatau e tusi ai i le vaitaele pusa:

ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'

Se'i tatou va'ai o lo'o i ai pea le ki avanoa:

ceph auth get-key client.rbdkube

O le poloaiga o le a faʻaalia se mea e pei o lenei:

AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==

Se'i o tatou fa'aopoopo le taulia lea ile Secret i le fuifui Kubernetes - o fea tatou te mana'omia ai userKey:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  # Значения ключей соответствуют имени пользователя и его ключу, как указано в
  # кластере Ceph. ID юзера должен иметь доступ к пулу,
  # указанному в storage class
  userID: rbdkube
  userKey: <user-key>

Ma matou faia la matou mealilo:

kubectl apply -f secret.yaml

Le isi, matou te manaʻomia se StorageClass faʻaalia se mea e pei o lenei:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   # Эти секреты должны содержать данные для авторизации
   # в ваш пул.
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

E mana'omia ona fa'atumu clusterID, lea ua uma ona matou aoaoina e le au ceph fsid, ma faʻaoga lenei faʻaaliga i le Kubernetes cluster:

kubectl apply -f storageclass.yaml

Ina ia siaki pe faʻapefea ona galulue faʻatasi ia fuifui, seʻi o tatou fatuina le PVC (Persistent Volume Claim):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

Sei o tatou vaʻavaʻai pe faʻapefea ona faia e Kubernetes le voluma talosagaina i Ceph:

kubectl get pvc
kubectl get pv

E foliga mai e lelei mea uma! O le a le foliga o lenei mea i le itu Ceph?
Matou te maua se lisi o voluma i totonu o le vaitaele ma matamata i faʻamatalaga e uiga i la matou voluma:

rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # тут, конечно же, будет другой ID тома, который выдала предыдущая команда

Se'i o tatou va'ai pe fa'afefea ona fai le suiga o le volumes RBD.
Suia le tele o le voluma i le pvc.yaml manifest i le 2Gi ma faʻaoga:

kubectl apply -f pvc.yaml

Se'i fa'atali se'i o'o mai suiga ma toe va'ai i le tele o le leo.

rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc

Matou te vaʻaia o le tele o le PVC e leʻi suia. Ina ia iloa pe aisea, e mafai ona e fesiligia Kubernetes mo se faʻamatalaga YAML o le PVC:

kubectl get pvc rbd-pvc -o yaml

O le fa'afitauli lea:

fe'au: Fa'atali mo le tagata fa'aoga e (toe-) amata se pod e fa'auma ai faila faila toe fa'aopoopo le leo ile node. ituaiga: FileSystemResizePending

O lona uiga, ua tupu le disk, ae o le faila faila i luga e leai.
Ina ia faʻalauteleina le faila faila, e tatau ona e faʻapipiʻi le leo. I totonu o lo tatou atunuu, o le PVC / PV na faia e le o faʻaaogaina i le taimi nei i soo se auala.

E mafai ona matou faia se suʻega Pod, mo se faʻataʻitaʻiga pei o lenei:

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

Ma o lenei sei o tatou tilotilo i le PVC:

kubectl get pvc

Ua suia le tele, ua lelei mea uma.

I le vaega muamua, matou te galulue ma le masini poloka poloka RBD (e tu mo Rados Block Device), ae e le mafai ona faia lenei mea pe a manaʻomia e microservices eseese ona galulue faʻatasi ma lenei disk i le taimi e tasi. O le CephFS e sili atu ona fetaui mo le galue i faila nai lo ata faʻapipiʻi.
I le faʻaaogaina o le faʻataʻitaʻiga o fuifui Ceph ma Kubernetes, o le a matou faʻatulagaina CSI ma isi faʻalapotopotoga talafeagai e galulue ai ma CephFS.

Sei o tatou maua tau mai le siata Helm fou tatou te manaʻomia:

helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml

Toe e manaʻomia le faʻatumu o le faila cephfs.yml. E pei o le taimi muamua, o le a fesoasoani le poloaiga a Ceph:

ceph fsid
ceph mon dump

Faatumu le faila i tulaga taua e pei o lenei:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true

Fa'amolemole maitau o tuatusi mata'itu o lo'o fa'amaoti mai ile fa'ailoga faigofie tuatusi:port. Ina ia faʻapipiʻi cephfs i luga o se node, o nei tuatusi e tuʻuina atu i le kernel module, lea e leʻi iloa pe faʻafefea ona galue ma le v2 monitor protocol.
Matou te suia le taulaga mo httpMetrics (Prometheus o le a alu iina mo le mataʻituina o fua) ina ia le feteʻenaʻi ma le nginx-proxy, lea e faʻapipiʻi e Kubespray. Atonu e te le mana'omia lenei mea.

Fa'apipi'i le siata Helm ile fa'aputuga Kubernetes:

helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace

Sei o tatou o i le Ceph data store e fatu ai se isi tagata faʻaoga iina. O fa'amaumauga o lo'o ta'u mai ai e mana'omia e le kamupani tu'uina atu le CephFS aia tatau e fa'aoga ai le pule fa'aputu. Ae o le a matou fatuina se tagata faʻaoga ese fs fa'atasi ai ma aia tatau:

ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'

Ma seʻi o tatou vaʻavaʻai i lana ki avanoa, tatou te manaʻomia mulimuli ane:

ceph auth get-key client.fs

Sei o tatou faia se mea lilo ma le StorageClass.
E leai se mea fou, ua uma ona matou vaʻaia lenei mea i le faʻataʻitaʻiga a RBD:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  # Необходимо для динамически создаваемых томов
  adminID: fs
  adminKey: <вывод предыдущей команды>

Faʻaaogaina le faʻaaliga:

kubectl apply -f secret.yaml

Ma o lenei - seisi StorageClass:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  # Имя файловой системы CephFS, в которой будет создан том
  fsName: cephfs

  # (необязательно) Пул Ceph, в котором будут храниться данные тома
  # pool: cephfs_data

  # (необязательно) Разделенные запятыми опции монтирования для Ceph-fuse
  # например:
  # fuseMountOptions: debug

  # (необязательно) Разделенные запятыми опции монтирования CephFS для ядра
  # См. man mount.ceph чтобы узнать список этих опций. Например:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # Секреты должны содержать доступы для админа и/или юзера Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # (необязательно) Драйвер может использовать либо ceph-fuse (fuse), 
  # либо ceph kernelclient (kernel).
  # Если не указано, будет использоваться монтирование томов по умолчанию,
  # это определяется поиском ceph-fuse и mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

Tatou faatumu iinei clusterID ma talafeagai i Kubernetes:

kubectl apply -f storageclass.yaml

asiasiga

Ina ia siaki, pei o le faʻataʻitaʻiga muamua, seʻi o tatou faia se PVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

Ma siaki le i ai o le PVC/PV:

kubectl get pvc
kubectl get pv

Afai e te manaʻo e vaʻai i faila ma faʻatonuga i le CephFS, e mafai ona e faʻapipiʻi le faila faila i se mea. Mo se faʻataʻitaʻiga e pei ona faʻaalia i lalo.

Se'i o tatou o i se tasi o node fuifui Ceph ma fai taga nei:

# Точка монтирования
mkdir -p /mnt/cephfs

# Создаём файл с ключом администратора
ceph auth get-key client.admin >/etc/ceph/secret.key

# Добавляем запись в /etc/fstab
# !! Изменяем ip адрес на адрес нашего узла
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs

O le mea moni, o le faʻapipiʻiina o le FS i luga o le Ceph node e pei o lenei e talafeagai mo faʻamoemoega aʻoaʻoga, o le mea lea matou te faia i luga o matou. Kosi Slurm. Ou te le manatu e faia e se tasi lenei mea i le gaosiga o loʻo i ai se tulaga maualuga o le tapeina faʻafuaseʻi o faila taua.

Ma le mea mulimuli, seʻi o tatou siaki pe faʻafefea ona galue mea i le toe faʻaleleia o voluma i le tulaga o CephFS. Sei o tatou toe foʻi i Kubernetes ma faʻasaʻo la tatou faʻaaliga mo PVC - faʻateleina le tele iina, mo se faʻataʻitaʻiga, i le 7Gi.

Se'i o tatou fa'aoga le faila fa'atonu:

kubectl apply -f pvc.yaml

Se'i o tatou va'ai ile fa'atonuga fa'apipi'i e va'ai pe fa'apefea ona suia le quota:

getfattr -n ceph.quota.max_bytes <каталог-с-данными>

Mo lenei faʻatonuga e galue, atonu e te manaʻomia le faʻapipiʻiina o le afifi i lau masini attr.

E fefefe mata, ae fefe lima

O nei sipela uma ma le umi o le YAML faʻaalia e foliga faigata i luga, ae o le faʻataʻitaʻiga, o tamaiti aʻoga Slurm e vave ona latou tautau.
I lenei tusiga matou te leʻi oʻo i totonu o le vaomatua - o loʻo i ai faʻamaumauga aloaia mo lena mea. Afai e te fiafia i faʻamatalaga o le faʻatulagaina o le teuina o Ceph ma le Kubernetes cluster, o fesoʻotaʻiga nei o le a fesoasoani:

O mataupu faavae lautele o Kubernetes o loʻo galue ma voluma
RBD Fa'amaumauga
Tuʻufaʻatasia RBD ma Kubernetes mai se vaaiga Ceph
Tu'ufa'atasia RBD ma Kubernetes mai se va'aiga CSI
Fa'amaumauga Lautele CephFS
Tuʻufaʻatasia CephFS ma Kubernetes mai se vaaiga CSI

I luga ole vasega Slurm Kubernetes Base e mafai ona e alu i luma ma faʻapipiʻi se talosaga moni i Kubernetes o le a faʻaogaina ai le CephFS e teu ai faila. E ala i talosaga GET/POST o le a mafai ona e fesiitai faila i ma maua mai ia Ceph.

Ma afai e sili atu lou fiafia i le teuina o faʻamatalaga, ona saini lea mo ala fou i Ceph. A'o fa'aauau pea le su'ega o le beta, e mafai ona maua le kosi i se fa'aitiitiga ma e mafai ona e fa'aaafia ana mea.

Tusitala o le tusiga: Alexander Shvalov, enisinia faataitai Southbridge, Pule Fa'amaonia Kubernetes, tusitala ma le fa'atupuina o kosi a Slurm.

puna: www.habr.com