Lati Rook tabi kii ṣe si Rook - iyẹn ni ibeere naa

Ni ibẹrẹ oṣu yii, ni Oṣu Karun ọjọ 3, itusilẹ pataki ti “eto iṣakoso fun ibi ipamọ data pinpin ni Kubernetes” ti kede - Rook 1.0.0. Die e sii ju odun kan seyin a tẹlẹ atejade gbogboogbo Akopọ ti Rook. Lẹhinna a beere lọwọ wa lati sọrọ nipa iriri rẹ lo ninu iwa - ati ni bayi, ni akoko fun iru iṣẹlẹ pataki kan ninu itan-akọọlẹ iṣẹ akanṣe naa, a ni idunnu lati pin awọn iwunilori ikojọpọ wa.

Ni kukuru, Rook jẹ eto kan awọn oniṣẹ fun Kubernetes, eyiti o gba iṣakoso ni kikun ti imuṣiṣẹ, iṣakoso, imularada laifọwọyi ti awọn solusan ipamọ data gẹgẹbi Ceph, EdgeFS, Minio, Cassandra, CockroachDB.

Ni akoko julọ ni idagbasoke (ati nikan ni ọkan в idurosinsin ipele) ojutu ni rook-ceph-onišẹ.

Daakọ: Lara awọn iyipada pataki ninu igbasilẹ Rook 1.0.0 ti o ni ibatan si Ceph, a le ṣe akiyesi atilẹyin fun Ceph Nautilus ati agbara lati lo NFS fun CephFS tabi awọn buckets RGW. Ohun ti o ṣe pataki laarin awọn miiran ni idagbasoke ti atilẹyin EdgeFS si ipele beta.

Nitorinaa, ninu nkan yii a:

• Jẹ ki a dahun ibeere naa nipa awọn anfani wo ni a rii ni lilo Rook lati fi Ceph ranṣẹ ni iṣupọ Kubernetes;
• A yoo pin iriri wa ati awọn iwunilori ti lilo Rook ni iṣelọpọ;
• Jẹ ki a sọ idi ti a fi sọ “Bẹẹni!” fun Rook, ati nipa awọn eto wa fun u.

Jẹ ki a bẹrẹ pẹlu awọn imọran gbogbogbo ati imọran.

"Mo ni anfani ti Rook kan!" (Ẹrọ chess ti a ko mọ)

Ọkan ninu awọn anfani akọkọ ti Rook ni pe ibaraenisepo pẹlu awọn ile itaja data ni a ṣe nipasẹ awọn ilana Kubernetes. Eyi tumọ si pe o ko nilo lati daakọ awọn aṣẹ lati tunto Ceph lati inu iwe sinu console.

— Ṣe o fẹ lati ran CephFS sinu iṣupọ kan? Kan kọ faili YAML kan!
- Kini? Ṣe o tun fẹ lati ran ile itaja ohun kan lọ pẹlu S3 API? Kan kọ faili YAML keji!

A ṣẹda Rook ni ibamu si gbogbo awọn ofin ti oniṣẹ aṣoju kan. Ibaraṣepọ pẹlu rẹ waye nipa lilo CRD (Awọn itumọ orisun orisun Aṣa), ninu eyiti a ṣe apejuwe awọn abuda ti awọn nkan Ceph ti a nilo (niwọn igba ti eyi jẹ imuse iduroṣinṣin nikan, nipa aiyipada nkan yii yoo sọrọ nipa Ceph, ayafi ti o ba sọ ni gbangba bibẹẹkọ). Gẹgẹbi awọn paramita ti a ti sọ tẹlẹ, oniṣẹ yoo ṣiṣẹ laifọwọyi awọn aṣẹ pataki fun iṣeto ni.

Jẹ ki a wo awọn pato ni lilo apẹẹrẹ ṣiṣẹda Ile-itaja Ohun kan, tabi dipo - CephObjectStoreUser.

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  gateway:
    type: s3
    sslCertificateRef:
    port: 80
    securePort:
    instances: 1
    allNodes: false
---
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  store: {{ .Values.s3.crdName }}
  displayName: {{ .Values.s3.username }}

Awọn paramita ti a tọka si ninu atokọ jẹ boṣewa ati pe ko nilo awọn asọye, ṣugbọn o tọ lati san ifojusi pataki si awọn ti a pin si awọn oniyipada awoṣe.

Eto gbogbogbo ti iṣẹ wa si otitọ pe a “paṣẹ” awọn orisun nipasẹ faili YAML kan, eyiti oniṣẹ ṣiṣẹ awọn aṣẹ to wulo ati da aṣiri “kii ṣe-gidi” pada wa pẹlu eyiti a le ṣiṣẹ siwaju sii. (wo isalẹ). Ati lati awọn oniyipada ti a ṣe akojọ loke, aṣẹ ati orukọ aṣiri yoo ṣajọ.

Iru egbe wo ni eyi? Nigbati o ba ṣẹda olumulo kan fun ibi ipamọ ohun, oniṣẹ Rook inu adarọ-ese yoo ṣe atẹle naa:

radosgw-admin user create --uid="rook-user" --display-name="{{ .Values.s3.username }}"

Abajade ti pipaṣẹ aṣẹ yii yoo jẹ eto JSON kan:

{
    "user_id": "rook-user",
    "display_name": "{{ .Values.s3.username }}",
    "keys": [
        {
           "user": "rook-user",
           "access_key": "NRWGT19TWMYOB1YDBV1Y",
           "secret_key": "gr1VEGIV7rxcP3xvXDFCo4UDwwl2YoNrmtRlIAty"
        }
    ],
    ...
}

Keys - kini awọn ohun elo iwaju yoo nilo lati wọle si ibi ipamọ ohun nipasẹ S3 API. Oṣiṣẹ Rook fi inurere yan wọn o si fi wọn si aaye orukọ rẹ ni irisi aṣiri pẹlu orukọ naa rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}.

Lati lo data lati inu aṣiri yii, kan ṣafikun si apoti bi awọn oniyipada ayika. Gẹgẹbi apẹẹrẹ, Emi yoo fun awoṣe fun Job, ninu eyiti a ṣẹda awọn buckets laifọwọyi fun agbegbe olumulo kọọkan:

{{- range $bucket := $.Values.s3.bucketNames }}
apiVersion: batch/v1
kind: Job
metadata:
  name: create-{{ $bucket }}-bucket-job
  annotations:
    "helm.sh/hook": post-install
    "helm.sh/hook-weight": "2"
spec:
  template:
    metadata:
      name: create-{{ $bucket }}-bucket-job
    spec:
      restartPolicy: Never
      initContainers:
      - name: waitdns
        image: alpine:3.6
        command: ["/bin/sh", "-c", "while ! getent ahostsv4 rook-ceph-rgw-{{ $.Values.s3.crdName }}; do sleep 1; done" ]
      - name: config
        image: rook/ceph:v1.0.0
        command: ["/bin/sh", "-c"]
        args: ["s3cmd --configure --access_key=$(ACCESS-KEY) --secret_key=$(SECRET-KEY) -s --no-ssl --dump-config | tee /config/.s3cfg"]
        volumeMounts:
        - name: config
          mountPath: /config
        env:
        - name: ACCESS-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: AccessKey
        - name: SECRET-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: SecretKey
      containers:
      - name: create-bucket
        image: rook/ceph:v1.0.0
        command: 
        - "s3cmd"
        - "mb"
        - "--host=rook-ceph-rgw-{{ $.Values.s3.crdName }}"
        - "--host-bucket= "
        - "s3://{{ $bucket }}"
        ports:
        - name: s3-no-sll
          containerPort: 80
        volumeMounts:
        - name: config
          mountPath: /root
      volumes:
      - name: config
        emptyDir: {}
---
{{- end }}

Gbogbo awọn iṣe ti a ṣe akojọ si ni iṣẹ yii ni a ṣe laarin ilana ti Kubernetes. Awọn ẹya ti a ṣalaye ninu awọn faili YAML ti wa ni ipamọ sinu ibi ipamọ Git kan ati tun lo ni ọpọlọpọ igba. A rii eyi bi afikun nla fun awọn onimọ-ẹrọ DevOps ati ilana CI/CD lapapọ.

Dun pẹlu Rook ati Rados

Lilo apapọ Ceph + RBD fa awọn ihamọ kan lori awọn ipele iṣagbesori si awọn adarọ-ese.

Ni pataki, aaye orukọ gbọdọ ni aṣiri kan fun iraye si Ceph ni ibere fun awọn ohun elo ipinlẹ lati ṣiṣẹ. O dara ti o ba ni awọn agbegbe 2-3 ni awọn aaye orukọ wọn: o le lọ daakọ aṣiri pẹlu ọwọ. Ṣugbọn kini ti ẹya kọọkan ba ṣẹda agbegbe lọtọ pẹlu aaye orukọ tirẹ fun awọn olupilẹṣẹ?

A yanju iṣoro yii funrararẹ nipa lilo ikarahun-onišẹ, eyiti o daakọ awọn aṣiri laifọwọyi si awọn aaye orukọ titun (apẹẹrẹ iru kio kan jẹ apejuwe ninu Arokọ yi).

#! /bin/bash

if [[ $1 == “--config” ]]; then
   cat <<EOF
{"onKubernetesEvent":[
 {"name": "OnNewNamespace",
  "kind": "namespace",
  "event": ["add"]
  }
]}
EOF
else
    NAMESPACE=$(kubectl get namespace -o json | jq '.items | max_by( .metadata.creationTimestamp ) | .metadata.name')
    kubectl -n ${CEPH_SECRET_NAMESPACE} get secret ${CEPH_SECRET_NAME} -o json | jq ".metadata.namespace="${NAMESPACE}"" | kubectl apply -f -
fi

Sibẹsibẹ, nigba lilo Rook isoro yi nìkan ko ni tẹlẹ. Ilana iṣagbesori waye nipa lilo awọn awakọ ti ara rẹ ti o da lori Flexvolume tabi CSI (si tun wa ni ipele beta) ati nitorinaa ko nilo awọn aṣiri.

Rook laifọwọyi yanju ọpọlọpọ awọn iṣoro, eyiti o gba wa niyanju lati lo ninu awọn iṣẹ akanṣe tuntun.

idoti ti Rook

Jẹ ki a pari apakan ilowo nipa gbigbe Rook ati Ceph ṣiṣẹ ki a le ṣe awọn idanwo tiwa. Lati jẹ ki o rọrun lati ji ile-iṣọ impregnable yii, awọn olupilẹṣẹ ti pese idii Helm kan. Jẹ ki a ṣe igbasilẹ rẹ:

$ helm fetch rook-master/rook-ceph --untar --version 1.0.0

Ninu faili rook-ceph/values.yaml o le wa ọpọlọpọ awọn eto oriṣiriṣi. Ohun pataki julọ ni lati pato awọn ifarada fun awọn aṣoju ati wiwa. A ṣe apejuwe ni apejuwe ohun ti ẹrọ taints / tolerations le ṣee lo fun ninu Arokọ yi.

Ni kukuru, a ko fẹ ki awọn adarọ-ese ohun elo alabara wa lori awọn apa kanna bi awọn disiki ipamọ data. Idi naa rọrun: ni ọna yii iṣẹ awọn aṣoju Rook kii yoo ni ipa lori ohun elo funrararẹ.

Nitorinaa, ṣii faili naa rook-ceph/values.yaml pẹlu olootu ayanfẹ rẹ ki o ṣafikun bulọọki atẹle ni ipari:

discover:
  toleration: NoExecute
  tolerationKey: node-role/storage
agent:
  toleration: NoExecute
  tolerationKey: node-role/storage
  mountSecurityMode: Any

Fun ipade kọọkan ti o wa ni ipamọ fun ibi ipamọ data, ṣafikun taint ti o baamu:

$ kubectl taint node ${NODE_NAME} node-role/storage="":NoExecute

Lẹhinna fi sori ẹrọ iwe apẹrẹ Helm pẹlu aṣẹ naa:

$ helm install --namespace ${ROOK_NAMESPACE} ./rook-ceph

Bayi o nilo lati ṣẹda iṣupọ kan ki o pato ipo naa OSD:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  clusterName: "ceph"
  finalizers:
  - cephcluster.ceph.rook.io
  generation: 1
  name: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v13
  dashboard:
    enabled: true
  dataDirHostPath: /var/lib/rook/osd
  mon:
    allowMultiplePerNode: false
    count: 3
  network:
    hostNetwork: true
  rbdMirroring:
    workers: 1
  placement:
    all:
      tolerations:
      - key: node-role/storage
        operator: Exists
  storage:
    useAllNodes: false
    useAllDevices: false
    config:
      osdsPerDevice: "1"
      storeType: filestore
    resources:
      limits:
        memory: "1024Mi"
      requests:
        memory: "1024Mi"
    nodes:
    - name: host-1
      directories:
      - path: "/mnt/osd"
    - name: host-2
      directories:
      - path: "/mnt/osd"
    - name: host-3
      directories:
      - path: "/mnt/osd"

Ṣiṣayẹwo ipo Ceph - nireti lati rii HEALTH_OK:

$ kubectl -n ${ROOK_NAMESPACE} exec $(kubectl -n ${ROOK_NAMESPACE} get pod -l app=rook-ceph-operator -o name -o jsonpath='{.items[0].metadata.name}') -- ceph -s

Ni akoko kanna, jẹ ki a ṣayẹwo pe awọn adarọ-ese pẹlu ohun elo alabara ko pari lori awọn apa ti o wa ni ipamọ fun Ceph:

$ kubectl -n ${APPLICATION_NAMESPACE} get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Siwaju sii, awọn paati afikun le tunto bi o ṣe fẹ. Awọn alaye diẹ sii nipa wọn ni itọkasi ni iwe. Fun iṣakoso, a ṣeduro ni iyanju fifi dasibodu ati apoti irinṣẹ sori ẹrọ.

Rook ati kio: ni Rook to fun ohun gbogbo?

Bi o ti le ri, idagbasoke ti Rook wa ni kikun. Ṣugbọn awọn iṣoro tun wa ti ko gba wa laaye lati kọ iṣeto ni afọwọṣe ti Ceph patapata:

• Ko si Rook Driver ko le okeere metiriki lori awọn lilo ti agesin ohun amorindun, eyi ti o deprives wa ti monitoring.
• Flexvolume ati CSI ko mọ bi yi awọn iwọn ti awọn iwọn didun (bi o lodi si kanna RBD), ki Rook ti wa ni finnufindo ti a wulo (ati ki o ma farabale nilo!) ọpa.
• Rook ko tun rọ bi Ceph deede. Ti a ba fẹ tunto adagun-omi fun metadata CephFS lati wa ni fipamọ sori SSD, ati pe data funrararẹ lati wa ni fipamọ sori HDD, a yoo nilo lati forukọsilẹ awọn ẹgbẹ lọtọ ti awọn ẹrọ ni awọn maapu CRUSH pẹlu ọwọ.
• Bíótilẹ o daju wipe rook-ceph-operator ti wa ni ka idurosinsin, nibẹ ni o wa Lọwọlọwọ diẹ ninu awọn isoro nigba ti igbegasoke Ceph lati version 13 to 14.

awari

"Ni bayi Rook ti wa ni pipade lati ita ita nipasẹ awọn pawn, ṣugbọn a gbagbọ pe ni ọjọ kan o yoo ṣe ipa pataki ninu ere naa!" (asọ ti a ṣe pataki fun nkan yii)

Ise agbese Rook ti gba ọkan wa laiseaniani - a gbagbọ pe [pẹlu gbogbo awọn anfani ati alailanfani rẹ] dajudaju o yẹ akiyesi rẹ.

Awọn ero iwaju wa ṣan silẹ lati ṣe rook-ceph module kan fun addoni-onišẹ, eyiti yoo jẹ ki lilo rẹ ni ọpọlọpọ awọn iṣupọ Kubernetes paapaa rọrun ati irọrun diẹ sii.

PS

Ka tun lori bulọọgi wa:

• «Rook - ile-ipamọ data “iṣẹ-ara ẹni” fun Kubernetes»;
• «Ṣiṣẹda ibi ipamọ itẹramọṣẹ pẹlu ipese ni Kubernetes ti o da lori Ceph»;
• «Awọn aaye data ati Kubernetes (atunyẹwo ati ijabọ fidio)»;
• «Iṣafihan onišẹ ikarahun: ṣiṣẹda awọn oniṣẹ fun Kubernetes kan ni irọrun»;
• «Awọn oniṣẹ fun Kubernetes: bii o ṣe le ṣiṣẹ awọn ohun elo ipinlẹ».

orisun: www.habr.com