ProHoster > Blog > Utongi > Muenzaniso unoshanda wekubatanidza Ceph-based storage kune Kubernetes cluster

Muenzaniso unoshanda wekubatanidza Ceph-based storage kune Kubernetes cluster

Container Storage Interface (CSI) ishamwari yakabatana pakati peKubernetes uye masisitimu ekuchengetedza. Takatotaura nezvazvo muchidimbu kuudzwa, uye nhasi tichanyatsotarisa kusanganiswa kweCSI neCeph: ticharatidza sei batanidza Ceph kuchengetedza kune Kubernetes cluster.
Chinyorwa chinopa mienzaniso chaiyo, kunyangwe yakarerutswa zvishoma kuitira nyore kuona. Isu hatifunge kuisa nekugadzirisa Ceph uye Kubernetes masumbu.

Uri kushamisika kuti zvinoshanda sei?

Muenzaniso unoshanda wekubatanidza Ceph-based storage kune Kubernetes cluster

Saka, iwe une Kubernetes cluster pamunwe wako, akaiswa, semuenzaniso, kubespray. Kune ceph cluster inoshanda padyo - iwe unogona zvakare kuimisa, semuenzaniso, neizvi seti yemabhuku ekutamba. Ndinotarisira kuti hapana chikonzero chekutaura kuti pakugadzirwa pakati pavo panofanira kunge kune network ine bandwidth yeinenge 10 Gbit / s.

Kana uine zvese izvi, handei!

Kutanga, ngatiende kune imwe yeCeph cluster node uye tarisa kuti zvese zvakarongeka:

ceph health
ceph -s

Tevere, isu tichakurumidza kugadzira dziva reRBD disks:

ceph osd pool create kube 32
ceph osd pool application enable kube rbd

Ngatienderei kuKubernetes cluster. Ikoko, chekutanga pane zvese, tichaisa iyo Ceph CSI mutyairi weRBD. Tichaisa, sezvinotarisirwa, kuburikidza neHelm.
Isu tinowedzera repository ine chati, tinowana seti yezvakasiyana zveiyo ceph-csi-rbd chati:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml

Iye zvino unofanirwa kuzadza iyo cephrbd.yml faira. Kuti uite izvi, tsvaga iyo cluster ID uye IP kero yevatariri muCeph:

ceph fsid  # Ρ‚Π°ΠΊ ΠΌΡ‹ ΡƒΠ·Π½Π°Π΅ΠΌ clusterID
ceph mon dump  # Π° Ρ‚Π°ΠΊ ΡƒΠ²ΠΈΠ΄ΠΈΠΌ IP-адрСса ΠΌΠΎΠ½ΠΈΡ‚ΠΎΡ€ΠΎΠ²

Isu tinoisa iyo yakawana kukosha muiyo cephrbd.yml faira. Panguva imwecheteyo, tinogonesa kugadzirwa kwePSP mitemo (Pod Security Policies). Sarudzo muzvikamu nodeplugin ΠΈ provider dzatova mufaira, dzinogona kugadziriswa sezvinoratidzwa pazasi:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true

Tevere, chasara kwatiri kuisa chati muKubernetes cluster.

helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace

Zvakanaka, mutyairi weRBD anoshanda!
Ngatigadzire itsva StorageClass muKubernetes. Izvi zvakare zvinoda kumboti tarisei naCeph.

Isu tinogadzira mushandisi mutsva muCeph uye tinomupa kodzero yekunyorera dziva Cube:

ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'

Zvino ngationei kiyi yekuwana ichiripo:

ceph auth get-key client.rbdkube

Murairo uchaburitsa chinhu chakadai:

AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==

Ngatiwedzerei kukosha uku kuChakavanzika muKubernetes cluster - kwatinoida userKey:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  # ЗначСния ΠΊΠ»ΡŽΡ‡Π΅ΠΉ ΡΠΎΠΎΡ‚Π²Π΅Ρ‚ΡΡ‚Π²ΡƒΡŽΡ‚ ΠΈΠΌΠ΅Π½ΠΈ ΠΏΠΎΠ»ΡŒΠ·ΠΎΠ²Π°Ρ‚Π΅Π»Ρ ΠΈ Π΅Π³ΠΎ ΠΊΠ»ΡŽΡ‡Ρƒ, ΠΊΠ°ΠΊ ΡƒΠΊΠ°Π·Π°Π½ΠΎ Π²
  # кластСрС Ceph. ID ΡŽΠ·Π΅Ρ€Π° Π΄ΠΎΠ»ΠΆΠ΅Π½ ΠΈΠΌΠ΅Ρ‚ΡŒ доступ ΠΊ ΠΏΡƒΠ»Ρƒ,
  # ΡƒΠΊΠ°Π·Π°Π½Π½ΠΎΠΌΡƒ Π² storage class
  userID: rbdkube
  userKey: <user-key>

Uye isu tinogadzira chakavanzika chedu:

kubectl apply -f secret.yaml

Tevere, isu tinoda StorageClass kuratidza chimwe chinhu chakadai:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   # Π­Ρ‚ΠΈ сСкрСты Π΄ΠΎΠ»ΠΆΠ½Ρ‹ ΡΠΎΠ΄Π΅Ρ€ΠΆΠ°Ρ‚ΡŒ Π΄Π°Π½Π½Ρ‹Π΅ для Π°Π²Ρ‚ΠΎΡ€ΠΈΠ·Π°Ρ†ΠΈΠΈ
   # Π² ваш ΠΏΡƒΠ».
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

Inoda kuzadzwa clusterID, izvo zvatakatodzidza nechikwata ceph fsid, uye shandisa iyi manifest kune Kubernetes cluster:

kubectl apply -f storageclass.yaml

Kuti utarise kuti masumbu anoshanda pamwe chete sei, ngatigadzirei inotevera PVC (Inoenderera Volume Claim):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

Ngationei nekukurumidza kuti Kubernetes akagadzira sei vhoriyamu yakakumbirwa muCeph:

kubectl get pvc
kubectl get pv

Zvese zvinoita kunge zvakanaka! Izvi zvinotaridzika sei kudivi reCeph?
Isu tinowana runyorwa rwemavhoriyamu mudziva uye tinoona ruzivo nezve vhoriyamu yedu:

rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # Ρ‚ΡƒΡ‚, ΠΊΠΎΠ½Π΅Ρ‡Π½ΠΎ ΠΆΠ΅, Π±ΡƒΠ΄Π΅Ρ‚ Π΄Ρ€ΡƒΠ³ΠΎΠΉ ID Ρ‚ΠΎΠΌΠ°, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹ΠΉ Π²Ρ‹Π΄Π°Π»Π° прСдыдущая ΠΊΠΎΠΌΠ°Π½Π΄Π°

Zvino ngationei kuti kudzoreredza vhoriyamu yeRBD kunoshanda sei.
Chinja saizi yevhoriyamu mu pvc.yaml manifest kuita 2Gi woishandisa:

kubectl apply -f pvc.yaml

Ngatimirirei kuti shanduko dziite totarisa size yevhoriyamu zvakare.

rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc

Isu tinoona kuti saizi yePVC haina kuchinja. Kuti uzive kuti sei, unogona kubvunza Kubernetes kune YAML tsananguro yePVC:

kubectl get pvc rbd-pvc -o yaml

Heino dambudziko:

meseji: Kumirira kuti mushandisi (re-) atange pod kupedzisa faira system resize yevhoriyamu pane node. mhando: FileSystemResizePending

Ndiko kuti, dhisiki yakakura, asi iyo faira system pairi haina.
Kuti ukure iyo faira system, unofanirwa kukwidza vhoriyamu. Munyika yedu, iyo yakagadzirwa PVC/PV haisati yashandiswa chero nzira.

Tinogona kugadzira bvunzo Pod, semuenzaniso seizvi:

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

Uye zvino ngatitarisei PVC:

kubectl get pvc

Saizi yachinja, zvese zvakanaka.

Muchikamu chekutanga, takashanda neRBD block device (inomirira Rados Block Device), asi izvi hazvigone kuitwa kana microservices yakasiyana inoda kushanda ne disk iyi panguva imwe chete. CephFS inonyanya kukodzera kushanda nemafaira pane dhisiki mifananidzo.
Tichishandisa muenzaniso wemasumbu eCeph neKubernetes, isu tichagadzirisa CSI nemamwe masangano anodiwa kuti ashande neCephFS.

Ngatitorei kukosha kubva kune itsva Helm chati yatinoda:

helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml

Zvakare iwe unofanirwa kuzadza iyo cephfs.yml faira. Sekare, Ceph mirairo ichabatsira:

ceph fsid
ceph mon dump

Zadza faira nemaitiro akadai:

csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true

Ndokumbira utarise kuti kero yekutarisa inotsanangurwa mune yakapusa fomu kero: port. Kuti uise cephs pane node, kero idzi dzinoendeswa kune kernel module, iyo isati yaziva kushanda ne v2 monitor protocol.
Isu tinoshandura chiteshi che httpMetrics (Prometheus ichaenda ikoko yekutarisa metrics) kuitira kuti isapesane nenginx-proxy, iyo yakaiswa neKubespray. Unogona kunge usingade izvi.

Isa iyo Helm chati muKubernetes cluster:

helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace

Handei kuCeph data chitoro kuti tigadzire akaparadzana mushandisi ipapo. Zvinyorwa zvinoti mupi weCephFS anoda kodzero dzekuwana maneja wemasumbu. Asi isu tichagadzira mushandisi akasiyana fs vane kodzero shoma:

ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'

Uye ngatitarisei kiyi yake yekuwana, tichaida gare gare:

ceph auth get-key client.fs

Ngatigadzire yakaparadzana Chakavanzika uye StorageClass.
Hapana chitsva, takatoona izvi mumuenzaniso weRBD:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  # НСобходимо для динамичСски создаваСмых Ρ‚ΠΎΠΌΠΎΠ²
  adminID: fs
  adminKey: <Π²Ρ‹Π²ΠΎΠ΄ ΠΏΡ€Π΅Π΄Ρ‹Π΄ΡƒΡ‰Π΅ΠΉ ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹>

Kushandisa manifestation:

kubectl apply -f secret.yaml

Uye ikozvino - yakaparadzana StorageClass:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  # Имя Ρ„Π°ΠΉΠ»ΠΎΠ²ΠΎΠΉ систСмы CephFS, Π² ΠΊΠΎΡ‚ΠΎΡ€ΠΎΠΉ Π±ΡƒΠ΄Π΅Ρ‚ создан Ρ‚ΠΎΠΌ
  fsName: cephfs

  # (Π½Π΅ΠΎΠ±ΡΠ·Π°Ρ‚Π΅Π»ΡŒΠ½ΠΎ) ΠŸΡƒΠ» Ceph, Π² ΠΊΠΎΡ‚ΠΎΡ€ΠΎΠΌ Π±ΡƒΠ΄ΡƒΡ‚ Ρ…Ρ€Π°Π½ΠΈΡ‚ΡŒΡΡ Π΄Π°Π½Π½Ρ‹Π΅ Ρ‚ΠΎΠΌΠ°
  # pool: cephfs_data

  # (Π½Π΅ΠΎΠ±ΡΠ·Π°Ρ‚Π΅Π»ΡŒΠ½ΠΎ) Π Π°Π·Π΄Π΅Π»Π΅Π½Π½Ρ‹Π΅ запятыми ΠΎΠΏΡ†ΠΈΠΈ монтирования для Ceph-fuse
  # Π½Π°ΠΏΡ€ΠΈΠΌΠ΅Ρ€:
  # fuseMountOptions: debug

  # (Π½Π΅ΠΎΠ±ΡΠ·Π°Ρ‚Π΅Π»ΡŒΠ½ΠΎ) Π Π°Π·Π΄Π΅Π»Π΅Π½Π½Ρ‹Π΅ запятыми ΠΎΠΏΡ†ΠΈΠΈ монтирования CephFS для ядра
  # Π‘ΠΌ. man mount.ceph Ρ‡Ρ‚ΠΎΠ±Ρ‹ ΡƒΠ·Π½Π°Ρ‚ΡŒ список этих ΠΎΠΏΡ†ΠΈΠΉ. НапримСр:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # Π‘Π΅ΠΊΡ€Π΅Ρ‚Ρ‹ Π΄ΠΎΠ»ΠΆΠ½Ρ‹ ΡΠΎΠ΄Π΅Ρ€ΠΆΠ°Ρ‚ΡŒ доступы для Π°Π΄ΠΌΠΈΠ½Π° ΠΈ/ΠΈΠ»ΠΈ ΡŽΠ·Π΅Ρ€Π° Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # (Π½Π΅ΠΎΠ±ΡΠ·Π°Ρ‚Π΅Π»ΡŒΠ½ΠΎ) Π”Ρ€Π°ΠΉΠ²Π΅Ρ€ ΠΌΠΎΠΆΠ΅Ρ‚ ΠΈΡΠΏΠΎΠ»ΡŒΠ·ΠΎΠ²Π°Ρ‚ΡŒ Π»ΠΈΠ±ΠΎ ceph-fuse (fuse), 
  # Π»ΠΈΠ±ΠΎ ceph kernelclient (kernel).
  # Если Π½Π΅ ΡƒΠΊΠ°Π·Π°Π½ΠΎ, Π±ΡƒΠ΄Π΅Ρ‚ ΠΈΡΠΏΠΎΠ»ΡŒΠ·ΠΎΠ²Π°Ρ‚ΡŒΡΡ ΠΌΠΎΠ½Ρ‚ΠΈΡ€ΠΎΠ²Π°Π½ΠΈΠ΅ Ρ‚ΠΎΠΌΠΎΠ² ΠΏΠΎ ΡƒΠΌΠΎΠ»Ρ‡Π°Π½ΠΈΡŽ,
  # это опрСдСляСтся поиском ceph-fuse ΠΈ mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

Ngatizadzei pano clusterID uye inoshanda muKubernetes:

kubectl apply -f storageclass.yaml

kuonorora

Kuti utarise, semumuenzaniso wapfuura, ngatigadzire PVC:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

Uye tarisa kuvapo kwePVC/PV:

kubectl get pvc
kubectl get pv

Kana iwe uchida kutarisa mafaera uye madhairekitori muCephFS, unogona kukwira iyi faira system kumwe kunhu. Somuenzaniso sezvinoratidzwa pasi apa.

Ngatiende kune imwe yeCeph cluster node uye tiite zvinotevera zviito:

# Π’ΠΎΡ‡ΠΊΠ° монтирования
mkdir -p /mnt/cephfs

# Π‘ΠΎΠ·Π΄Π°Ρ‘ΠΌ Ρ„Π°ΠΉΠ» с ΠΊΠ»ΡŽΡ‡ΠΎΠΌ администратора
ceph auth get-key client.admin >/etc/ceph/secret.key

# ДобавляСм запись Π² /etc/fstab
# !! ИзмСняСм ip адрСс Π½Π° адрСс нашСго ΡƒΠ·Π»Π°
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs

Ehe, kukwira FS pane yeCeph node seizvi inokodzera chete zvinangwa zvekudzidzisa, ndizvo zvatinoita pane yedu. Slurm courses. Ini handifunge kuti chero munhu angaita izvi mukugadzira; kune njodzi yakakura yekudzima netsaona mafaera akakosha.

Uye pakupedzisira, ngatitarisei kuti zvinhu zvinoshanda sei nekugadzirisa mavhoriyamu munyaya yeCephFS. Ngatidzokerei kuKubernetes uye tigadzirise manifesto yedu yePVC - wedzera saizi ipapo, semuenzaniso, ku7Gi.

Ngatishandise iyo yakagadziridzwa faira:

kubectl apply -f pvc.yaml

Ngatitarisei dhairekitori rakaiswa kuti tione kuti quota yachinja sei:

getfattr -n ceph.quota.max_bytes <ΠΊΠ°Ρ‚Π°Π»ΠΎΠ³-с-Π΄Π°Π½Π½Ρ‹ΠΌΠΈ>

Kuti uyu murairo ushande, ungangoda kuisa pasuru pane yako system attr.

Meso anotya, asi maoko anotya

Ese aya zviperengo uye akareba YAML anoratidza anoita seakaomarara pamusoro, asi mukuita, vadzidzi veSlurm vanowana kurembera kwavo nekukurumidza.
Muchikamu chino hatina kupinda mukati mesango - pane zvinyorwa zvepamutemo zveizvi. Kana iwe uchifarira ruzivo rwekumisikidza Ceph kuchengetedza neKubernetes cluster, aya malink anozobatsira:

General misimboti yeKubernetes inoshanda nemavhoriyamu
RBD Zvinyorwa
Kubatanidza RBD uye Kubernetes kubva kuCeph maonero
Kubatanidza RBD uye Kubernetes kubva kune CSI maonero
General CephFS Documentation
Kubatanidza CephFS uye Kubernetes kubva pane CSI maonero

PaSlurm course Kubernetes Base iwe unogona kuenda mberi zvishoma uye kuendesa chaiyo application muKubernetes iyo inoshandisa CephFS sekuchengetedza faira. Kuburikidza neGET/POST zvikumbiro iwe unozokwanisa kuendesa mafaera uye nekuagamuchira kubva kuCeph.

Uye kana iwe uchifarira zvakanyanya kuchengetedza data, wobva wanyoresa kosi itsva paCeph. Ipo bvunzo yebeta ichienderera mberi, kosi yacho inogona kuwanikwa nemutengo wakaderera uye unogona kupesvedzera zvirimo.

Munyori wenyaya: Alexander Shvalov, kudzidzira injiniya Southbridge, Certified Kubernetes Administrator, munyori uye mugadziri weSlurm makosi.

Source: www.habr.com