Container Storage Interface (CSI) yog ib qho kev sib koom ua ke ntawm Kubernetes thiab cov tshuab khaws cia. Peb twb tau tham txog nws luv luv
Cov kab lus muab qhov tseeb, txawm tias me ntsis yooj yim piv txwv rau kev nkag siab yooj yim. Peb tsis xav txog kev txhim kho thiab teeb tsa Ceph thiab Kubernetes pawg.
Koj puas xav tias nws ua haujlwm li cas?
Yog li, koj muaj Kubernetes pawg ntawm koj lub ntsis ntiv tes, xa mus, piv txwv li,
Yog tias koj muaj tag nrho cov no, cia peb mus!
Ua ntej, cia peb mus rau ib qho ntawm Ceph pawg nodes thiab xyuas tias txhua yam nyob rau hauv kev txiav txim:
ceph health
ceph -s
Tom ntej no, peb mam li tsim ib lub pas dej ua ke rau RBD disks:
ceph osd pool create kube 32
ceph osd pool application enable kube rbd
Cia peb mus rau Kubernetes pawg. Nyob ntawd, ua ntej ntawm tag nrho cov, peb yuav nruab Ceph CSI tsav tsheb rau RBD. Peb yuav nruab, raws li xav tau, los ntawm Helm.
Peb ntxiv ib lub chaw cia khoom nrog ib daim ntawv qhia, peb tau txais ib txheej ntawm qhov sib txawv rau daim ntawv ceph-csi-rbd:
helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml
Tam sim no koj yuav tsum sau cov ntaub ntawv cephrbd.yml. Txhawm rau ua qhov no, tshawb xyuas pawg ID thiab IP chaw nyob ntawm cov saib hauv Ceph:
ceph fsid # ΡΠ°ΠΊ ΠΌΡ ΡΠ·Π½Π°Π΅ΠΌ clusterID
ceph mon dump # Π° ΡΠ°ΠΊ ΡΠ²ΠΈΠ΄ΠΈΠΌ IP-Π°Π΄ΡΠ΅ΡΠ° ΠΌΠΎΠ½ΠΈΡΠΎΡΠΎΠ²
Peb nkag mus rau qhov tau txais txiaj ntsig rau hauv cov ntaub ntawv cephrbd.yml. Nyob rau tib lub sijhawm, peb pab kom tsim PSP cov cai (Pod Security Policy). Kev xaiv hauv ntu nodeplugin ΠΈ tus muab twb nyob rau hauv cov ntaub ntawv, lawv tuaj yeem raug kho raws li qhia hauv qab no:
csiConfig:
- clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
monitors:
- "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
- "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
- "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"
nodeplugin:
podSecurityPolicy:
enabled: true
provisioner:
podSecurityPolicy:
enabled: true
Tom ntej no, txhua yam uas tseem tshuav rau peb yog txhawm rau txhim kho daim ntawv qhia hauv Kubernetes pawg.
helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace
Zoo heev, tus tsav tsheb RBD ua haujlwm!
Cia peb tsim StorageClass tshiab hauv Kubernetes. Qhov no dua yuav tsum tau me ntsis tinkering nrog Ceph.
Peb tsim ib tus neeg siv tshiab hauv Ceph thiab muab nws txoj cai sau rau hauv pas dej kub:
ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'
Tam sim no cia peb pom tus yuam sij nkag tseem muaj:
ceph auth get-key client.rbdkube
Cov lus txib yuav tso tawm ib yam dab tsi zoo li no:
AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==
Cia peb ntxiv tus nqi no rau Secret hauv Kubernetes pawg - qhov twg peb xav tau userKey:
---
apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
namespace: ceph-csi-rbd
stringData:
# ΠΠ½Π°ΡΠ΅Π½ΠΈΡ ΠΊΠ»ΡΡΠ΅ΠΉ ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²ΡΡΡ ΠΈΠΌΠ΅Π½ΠΈ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ ΠΈ Π΅Π³ΠΎ ΠΊΠ»ΡΡΡ, ΠΊΠ°ΠΊ ΡΠΊΠ°Π·Π°Π½ΠΎ Π²
# ΠΊΠ»Π°ΡΡΠ΅ΡΠ΅ Ceph. ID ΡΠ·Π΅ΡΠ° Π΄ΠΎΠ»ΠΆΠ΅Π½ ΠΈΠΌΠ΅ΡΡ Π΄ΠΎΡΡΡΠΏ ΠΊ ΠΏΡΠ»Ρ,
# ΡΠΊΠ°Π·Π°Π½Π½ΠΎΠΌΡ Π² storage class
userID: rbdkube
userKey: <user-key>
Thiab peb tsim peb qhov zais cia:
kubectl apply -f secret.yaml
Tom ntej no, peb xav tau StorageClass manifest ib yam dab tsi zoo li no:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
clusterID: <cluster-id>
pool: kube
imageFeatures: layering
# ΠΡΠΈ ΡΠ΅ΠΊΡΠ΅ΡΡ Π΄ΠΎΠ»ΠΆΠ½Ρ ΡΠΎΠ΄Π΅ΡΠΆΠ°ΡΡ Π΄Π°Π½Π½ΡΠ΅ Π΄Π»Ρ Π°Π²ΡΠΎΡΠΈΠ·Π°ΡΠΈΠΈ
# Π² Π²Π°Ρ ΠΏΡΠ».
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd
csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
Yuav tsum tau ua kom tiav pawgID, uas peb twb tau kawm los ntawm pab neeg seb fsd, thiab siv qhov manifest no rau Kubernetes pawg:
kubectl apply -f storageclass.yaml
Txhawm rau xyuas seb cov pawg ua haujlwm li cas, cia peb tsim cov PVC hauv qab no (Persistent Volume Claim):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: csi-rbd-sc
Cia peb pom tam sim ntawd Kubernetes tsim qhov ntim thov hauv Ceph:
kubectl get pvc
kubectl get pv
Txhua yam zoo li zoo heev! Qhov no zoo li cas ntawm Ceph sab?
Peb tau txais cov npe ntawm cov ntim hauv pas dej thiab saib cov ntaub ntawv hais txog peb lub ntim:
rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653 # ΡΡΡ, ΠΊΠΎΠ½Π΅ΡΠ½ΠΎ ΠΆΠ΅, Π±ΡΠ΄Π΅Ρ Π΄ΡΡΠ³ΠΎΠΉ ID ΡΠΎΠΌΠ°, ΠΊΠΎΡΠΎΡΡΠΉ Π²ΡΠ΄Π°Π»Π° ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ ΠΊΠΎΠΌΠ°Π½Π΄Π°
Tam sim no cia saib yuav ua li cas resizing ib RBD ntim ua haujlwm.
Hloov qhov ntim loj hauv pvc.yaml manifest rau 2Gi thiab siv nws:
kubectl apply -f pvc.yaml
Cia peb tos rau qhov kev hloov pauv kom muaj txiaj ntsig thiab saib lub ntim loj dua.
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653
kubectl get pv
kubectl get pvc
Peb pom tias qhov loj ntawm PVC tsis tau hloov. Yog xav paub vim li cas, koj tuaj yeem nug Kubernetes rau YAML cov lus piav qhia ntawm PVC:
kubectl get pvc rbd-pvc -o yaml
Nov yog qhov teeb meem:
lus: Tos rau tus neeg siv kom (re-) pib lub pod kom tiav cov ntaub ntawv kaw lus resize ntawm ntim ntawm node. hom: FileSystemResizePending
Ntawd yog, lub disk tau loj hlob, tab sis cov ntaub ntawv kaw lus ntawm nws tsis tau.
Yuav kom loj hlob cov ntaub ntawv system, koj yuav tsum tau mount lub ntim. Nyob rau hauv peb lub teb chaws, cov tsim PVC / PV tam sim no tsis siv nyob rau hauv txhua txoj kev.
Peb tuaj yeem tsim Pod xeem, piv txwv li no:
---
apiVersion: v1
kind: Pod
metadata:
name: csi-rbd-demo-pod
spec:
containers:
- name: web-server
image: nginx:1.17.6
volumeMounts:
- name: mypvc
mountPath: /data
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: rbd-pvc
readOnly: false
Thiab tam sim no cia saib PVC:
kubectl get pvc
Qhov loj me tau hloov, txhua yam zoo.
Hauv thawj ntu, peb tau ua haujlwm nrog RBD thaiv cov cuab yeej (nws sawv cev rau Rados Block Device), tab sis qhov no tsis tuaj yeem ua tiav yog tias cov microservices sib txawv yuav tsum ua haujlwm nrog cov disk no ib txhij. CephFS yog qhov zoo dua rau kev ua haujlwm nrog cov ntaub ntawv ntau dua li cov duab disk.
Siv cov piv txwv ntawm Ceph thiab Kubernetes pawg, peb yuav teeb tsa CSI thiab lwm qhov chaw tsim nyog los ua haujlwm nrog CephFS.
Cia peb tau txais cov txiaj ntsig los ntawm daim ntawv qhia Helm tshiab peb xav tau:
helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml
Ib zaug ntxiv koj yuav tsum sau cov ntaub ntawv cephfs.yml. Raws li ua ntej, Ceph cov lus txib yuav pab:
ceph fsid
ceph mon dump
Sau cov ntaub ntawv nrog cov txiaj ntsig zoo li no:
csiConfig:
- clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
monitors:
- "172.18.8.5:6789"
- "172.18.8.6:6789"
- "172.18.8.7:6789"
nodeplugin:
httpMetrics:
enabled: true
containerPort: 8091
podSecurityPolicy:
enabled: true
provisioner:
replicaCount: 1
podSecurityPolicy:
enabled: true
Thov nco ntsoov tias cov chaw nyob saib xyuas tau teev nyob rau hauv daim ntawv yooj yim chaw nyob: chaw nres nkoj. Txhawm rau mount cephfs ntawm ib lub ntawm, cov chaw nyob no raug xa mus rau lub ntsiav module, uas tseem tsis tau paub ua haujlwm nrog v2 saib raws tu qauv.
Peb hloov qhov chaw nres nkoj rau httpMetrics (Prometheus yuav mus rau qhov ntsuas ntsuas) kom nws tsis cuam tshuam nrog nginx-proxy, uas tau teeb tsa los ntawm Kubespray. Tej zaum koj yuav tsis xav tau qhov no.
Nruab daim ntawv Helm hauv Kubernetes pawg:
helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace
Cia peb mus rau Ceph cov ntaub ntawv khaws cia los tsim ib tus neeg siv cais nyob ntawd. Cov ntaub ntawv hais tias CephFS tus neeg muab kev pabcuam yuav tsum muaj cai nkag mus rau pawg thawj tswj hwm. Tab sis peb yuav tsim ib tus neeg siv cais fs nrog txwv txoj cai:
ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'
Thiab tam sim ntawd saib nws tus yuam sij nkag, peb yuav xav tau nws tom qab:
ceph auth get-key client.fs
Cia peb tsim cais Secret thiab StorageClass.
Tsis muaj dab tsi tshiab, peb twb pom qhov no hauv qhov piv txwv ntawm RBD:
---
apiVersion: v1
kind: Secret
metadata:
name: csi-cephfs-secret
namespace: ceph-csi-cephfs
stringData:
# ΠΠ΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ Π΄Π»Ρ Π΄ΠΈΠ½Π°ΠΌΠΈΡΠ΅ΡΠΊΠΈ ΡΠΎΠ·Π΄Π°Π²Π°Π΅ΠΌΡΡ
ΡΠΎΠΌΠΎΠ²
adminID: fs
adminKey: <Π²ΡΠ²ΠΎΠ΄ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ ΠΊΠΎΠΌΠ°Π½Π΄Ρ>
Siv lub manifest:
kubectl apply -f secret.yaml
Thiab tam sim no - cais StorageClass:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
clusterID: <cluster-id>
# ΠΠΌΡ ΡΠ°ΠΉΠ»ΠΎΠ²ΠΎΠΉ ΡΠΈΡΡΠ΅ΠΌΡ CephFS, Π² ΠΊΠΎΡΠΎΡΠΎΠΉ Π±ΡΠ΄Π΅Ρ ΡΠΎΠ·Π΄Π°Π½ ΡΠΎΠΌ
fsName: cephfs
# (Π½Π΅ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎ) ΠΡΠ» Ceph, Π² ΠΊΠΎΡΠΎΡΠΎΠΌ Π±ΡΠ΄ΡΡ Ρ
ΡΠ°Π½ΠΈΡΡΡΡ Π΄Π°Π½Π½ΡΠ΅ ΡΠΎΠΌΠ°
# pool: cephfs_data
# (Π½Π΅ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎ) Π Π°Π·Π΄Π΅Π»Π΅Π½Π½ΡΠ΅ Π·Π°ΠΏΡΡΡΠΌΠΈ ΠΎΠΏΡΠΈΠΈ ΠΌΠΎΠ½ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π΄Π»Ρ Ceph-fuse
# Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ:
# fuseMountOptions: debug
# (Π½Π΅ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎ) Π Π°Π·Π΄Π΅Π»Π΅Π½Π½ΡΠ΅ Π·Π°ΠΏΡΡΡΠΌΠΈ ΠΎΠΏΡΠΈΠΈ ΠΌΠΎΠ½ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ CephFS Π΄Π»Ρ ΡΠ΄ΡΠ°
# Π‘ΠΌ. man mount.ceph ΡΡΠΎΠ±Ρ ΡΠ·Π½Π°ΡΡ ΡΠΏΠΈΡΠΎΠΊ ΡΡΠΈΡ
ΠΎΠΏΡΠΈΠΉ. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ:
# kernelMountOptions: readdir_max_bytes=1048576,norbytes
# Π‘Π΅ΠΊΡΠ΅ΡΡ Π΄ΠΎΠ»ΠΆΠ½Ρ ΡΠΎΠ΄Π΅ΡΠΆΠ°ΡΡ Π΄ΠΎΡΡΡΠΏΡ Π΄Π»Ρ Π°Π΄ΠΌΠΈΠ½Π° ΠΈ/ΠΈΠ»ΠΈ ΡΠ·Π΅ΡΠ° Ceph.
csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs
# (Π½Π΅ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎ) ΠΡΠ°ΠΉΠ²Π΅Ρ ΠΌΠΎΠΆΠ΅Ρ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ Π»ΠΈΠ±ΠΎ ceph-fuse (fuse),
# Π»ΠΈΠ±ΠΎ ceph kernelclient (kernel).
# ΠΡΠ»ΠΈ Π½Π΅ ΡΠΊΠ°Π·Π°Π½ΠΎ, Π±ΡΠ΄Π΅Ρ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡΡΡ ΠΌΠΎΠ½ΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΡΠΎΠΌΠΎΠ² ΠΏΠΎ ΡΠΌΠΎΠ»ΡΠ°Π½ΠΈΡ,
# ΡΡΠΎ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅ΡΡΡ ΠΏΠΎΠΈΡΠΊΠΎΠΌ ceph-fuse ΠΈ mount.ceph
# mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- debug
Wb sau rau ntawm no pawgID thiab siv tau hauv Kubernetes:
kubectl apply -f storageclass.yaml
soj ntsuam
Txhawm rau txheeb xyuas, zoo li hauv qhov piv txwv dhau los, cia peb tsim PVC:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-cephfs-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: csi-cephfs-sc
Thiab xyuas qhov muaj ntawm PVC / PV:
kubectl get pvc
kubectl get pv
Yog tias koj xav saib cov ntaub ntawv thiab cov npe hauv CephFS, koj tuaj yeem txuas cov ntaub ntawv no rau qhov chaw. Piv txwv li qhia hauv qab no.
Cia wb mus rau ib qho ntawm Ceph pawg nodes thiab ua cov haujlwm hauv qab no:
# Π’ΠΎΡΠΊΠ° ΠΌΠΎΠ½ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ
mkdir -p /mnt/cephfs
# Π‘ΠΎΠ·Π΄Π°ΡΠΌ ΡΠ°ΠΉΠ» Ρ ΠΊΠ»ΡΡΠΎΠΌ Π°Π΄ΠΌΠΈΠ½ΠΈΡΡΡΠ°ΡΠΎΡΠ°
ceph auth get-key client.admin >/etc/ceph/secret.key
# ΠΠΎΠ±Π°Π²Π»ΡΠ΅ΠΌ Π·Π°ΠΏΠΈΡΡ Π² /etc/fstab
# !! ΠΠ·ΠΌΠ΅Π½ΡΠ΅ΠΌ ip Π°Π΄ΡΠ΅Ρ Π½Π° Π°Π΄ΡΠ΅Ρ Π½Π°ΡΠ΅Π³ΠΎ ΡΠ·Π»Π°
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev 0 2" >> /etc/fstab
mount /mnt/cephfs
Tau kawg, mounting FS ntawm Ceph node zoo li qhov no tsuas yog tsim nyog rau kev cob qhia, uas yog qhov peb ua ntawm peb.
Thiab thaum kawg, cia peb xyuas seb cov khoom ua haujlwm li cas nrog kev hloov pauv ntim hauv rooj plaub ntawm CephFS. Cia peb rov qab mus rau Kubernetes thiab kho peb cov manifest rau PVC - nce qhov loj nyob ntawd, piv txwv li, rau 7Gi.
Cia peb siv cov ntaub ntawv edited:
kubectl apply -f pvc.yaml
Cia peb saib ntawm cov npe mounted kom pom tias cov quota tau hloov pauv li cas:
getfattr -n ceph.quota.max_bytes <ΠΊΠ°ΡΠ°Π»ΠΎΠ³-Ρ-Π΄Π°Π½Π½ΡΠΌΠΈ>
Rau cov lus txib no ua haujlwm, koj yuav tsum tau nruab lub pob ntawm koj lub cev attr.
Qhov muag ntshai, tab sis txhais tes ua
Tag nrho cov kev sau ntawv no thiab ntev YAML manifests zoo li nyuaj rau saum npoo, tab sis hauv kev xyaum, Slurm cov tub ntxhais kawm tau txais qhov dai ntawm lawv sai sai.
Hauv tsab xov xwm no peb tsis tau mus tob rau hauv hav zoov - muaj cov ntaub ntawv raug cai rau qhov ntawd. Yog tias koj txaus siab rau cov ntsiab lus ntawm kev teeb tsa Ceph cia nrog Kubernetes pawg, cov kev txuas no yuav pab tau:
Hauv chav kawm Slurm
Thiab yog tias koj xav paub ntau ntxiv hauv cov ntaub ntawv khaws cia, ces sau npe rau
Tus sau ntawm tsab xov xwm: Alexander Shvalov, xyaum engineer
Tau qhov twg los: www.hab.com