To Rook or not to Rook - lowo ngumbuzo

Ekuqaleni kwale nyanga, ngoMeyi 3, kwamenyezelwa ukukhululwa okukhulu “kohlelo lokuphatha lokusatshalaliswa kwedatha eKubernetes” - I-Rook 1.0.0. Ngaphezu konyaka odlule sesivele eshicilelwe Uhlolojikelele olujwayelekile lweRook. Sabe sesicelwa ukuba sikhulume ngokuhlangenwe nakho kwakhe ukusetshenziswa ekusebenzeni - futhi manje, ngesikhathi esifanele sengqophamlando ebaluleke kangaka emlandweni wephrojekthi, siyajabula ukwabelana ngemibono yethu enqwabelene.

Ngamafuphi, i-Rook iyisethi opharetha ye-Kubernetes, ethatha ukulawula okugcwele kokuthunyelwa, ukuphathwa, ukutholwa okuzenzakalelayo kwezixazululo zokugcina idatha ezifana ne-Ceph, EdgeFS, Minio, Cassandra, CockroachDB.

Okwamanje ethuthuke kakhulu (kanye yedwa в ezinzile stage) ikhambi i-rook-ceph-opharetha.

Ukubhala: Phakathi kwezinguquko ezibalulekile ekukhishweni kwe-Rook 1.0.0 okuhlobene ne-Ceph, singaphawula ukusekelwa kwe-Ceph Nautilus kanye nekhono lokusebenzisa i-NFS kumabhakede e-CephFS noma e-RGW. Okugqamayo phakathi kokunye ukuvuthwa kokusekelwa kwe-EdgeFS kuya ezingeni le-beta.

Ngakho, kulesi sihloko:

• Ake siphendule umbuzo mayelana nokuthi yiziphi izinzuzo esizibonayo ekusebenziseni i-Rook ukuze kusetshenziswe i-Ceph kuqoqo le-Kubernetes;
• Sizokwabelana ngolwazi lwethu kanye nemibono yokusebenzisa i-Rook ekukhiqizeni;
• Ake sikutshele ukuthi kungani sithi “Yebo!” kuRook, nangezinhlelo zethu ngaye.

Ake siqale ngemiqondo evamile kanye nethiyori.

"Nginenzuzo ye-Rook eyodwa!" (umdlali we-chess ongaziwa)

Enye yezinzuzo eziyinhloko ze-Rook ukuthi ukusebenzisana nezitolo zedatha kwenziwa ngokusebenzisa izindlela ze-Kubernetes. Lokhu kusho ukuthi awusadingi ukukopisha imiyalo ukuze ulungiselele i-Ceph kusuka eshidini ukuya kukhonsoli.

- Uyafuna ukufaka i-CephFS kuqoqo? Vele ubhale ifayela le-YAML!
- Ini? Uyafuna futhi ukuphakela isitolo sezinto nge-S3 API? Vele ubhale ifayela lesibili le-YAML!

I-Rook idalwe ngokuvumelana nayo yonke imithetho yomqhubi ojwayelekile. Ukusebenzisana naye kwenzeka ngokusebenzisa I-CRD (Izincazelo Zensiza Yangokwezifiso), lapho sichaza khona izici ze-Ceph esizidingayo (njengoba lokhu kuwukuphela kokuqaliswa okuzinzile, ngokuzenzakalelayo le ndatshana izokhuluma nge-Ceph, ngaphandle uma kuchazwe ngokucacile ngenye indlela). Ngokusho kwemingcele ecacisiwe, opharetha uzokhipha ngokuzenzakalelayo imiyalo edingekayo yokucushwa.

Ake sibheke okucacisiwe sisebenzisa isibonelo sokwenza Isitolo Sezinto, noma kunalokho - CephObjectStoreUser.

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  gateway:
    type: s3
    sslCertificateRef:
    port: 80
    securePort:
    instances: 1
    allNodes: false
---
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  store: {{ .Values.s3.crdName }}
  displayName: {{ .Values.s3.username }}

Amapharamitha akhonjiswe ohlwini ajwayelekile futhi awadingi ukuphawula, kodwa kufanelekile ukunaka ngokukhethekile lawo abelwe okuguquguqukayo kwezifanekiso.

Uhlelo olujwayelekile lomsebenzi lwehlela eqinisweni lokuthi “siyala” izinsiza ngefayela le-YAML, lapho u-opharetha enza khona imiyalo edingekayo futhi asibuyisele imfihlo “okungeyona eyangempela” esingaqhubeka ngayo sisebenze. (bona ngezansi). Futhi kusukela kokuguquguqukayo okubalwe ngenhla, umyalo negama eliyimfihlo lizohlanganiswa.

Iqembu elinjani leli? Lapho udala umsebenzisi wokugcina into, u-opharetha we-Rook ngaphakathi kwe-pod uzokwenza lokhu okulandelayo:

radosgw-admin user create --uid="rook-user" --display-name="{{ .Values.s3.username }}"

Umphumela wokwenza lo myalo uzoba isakhiwo se-JSON:

{
    "user_id": "rook-user",
    "display_name": "{{ .Values.s3.username }}",
    "keys": [
        {
           "user": "rook-user",
           "access_key": "NRWGT19TWMYOB1YDBV1Y",
           "secret_key": "gr1VEGIV7rxcP3xvXDFCo4UDwwl2YoNrmtRlIAty"
        }
    ],
    ...
}

Keys - yiziphi izinhlelo zokusebenza zesikhathi esizayo ezizodinga ukufinyelela isitoreji sento nge-S3 API. Umsebenzisi we-Rook uwakhetha ngomusa futhi awabeke endaweni yakhe yamagama ngendlela eyimfihlo enegama rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}.

Ukuze usebenzise idatha evela kule mfihlo, vele uyengeze esitsheni njengokuguquguquka kwemvelo. Njengesibonelo, ngizonikeza isifanekiso somsebenzi, lapho sakha khona amabhakede ngokuzenzakalelayo endaweni ngayinye yomsebenzisi:

{{- range $bucket := $.Values.s3.bucketNames }}
apiVersion: batch/v1
kind: Job
metadata:
  name: create-{{ $bucket }}-bucket-job
  annotations:
    "helm.sh/hook": post-install
    "helm.sh/hook-weight": "2"
spec:
  template:
    metadata:
      name: create-{{ $bucket }}-bucket-job
    spec:
      restartPolicy: Never
      initContainers:
      - name: waitdns
        image: alpine:3.6
        command: ["/bin/sh", "-c", "while ! getent ahostsv4 rook-ceph-rgw-{{ $.Values.s3.crdName }}; do sleep 1; done" ]
      - name: config
        image: rook/ceph:v1.0.0
        command: ["/bin/sh", "-c"]
        args: ["s3cmd --configure --access_key=$(ACCESS-KEY) --secret_key=$(SECRET-KEY) -s --no-ssl --dump-config | tee /config/.s3cfg"]
        volumeMounts:
        - name: config
          mountPath: /config
        env:
        - name: ACCESS-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: AccessKey
        - name: SECRET-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: SecretKey
      containers:
      - name: create-bucket
        image: rook/ceph:v1.0.0
        command: 
        - "s3cmd"
        - "mb"
        - "--host=rook-ceph-rgw-{{ $.Values.s3.crdName }}"
        - "--host-bucket= "
        - "s3://{{ $bucket }}"
        ports:
        - name: s3-no-sll
          containerPort: 80
        volumeMounts:
        - name: config
          mountPath: /root
      volumes:
      - name: config
        emptyDir: {}
---
{{- end }}

Zonke izenzo ezibalwe kulo Jobe zenziwa ngaphansi kohlaka lukaKubernetes. Izakhiwo ezichazwe kumafayela e-YAML zigcinwa endaweni yokugcina ye-Git futhi ziphinde zisetshenziswe izikhathi eziningi. Lokhu sikubona njengokuhlanganisa okukhulu konjiniyela be-DevOps kanye nenqubo ye-CI/CD iyonke.

Ngijabule ngoRook noRados

Ukusebenzisa inhlanganisela ye-Ceph + RBD kubeka imingcele ethile ekukhuphuleni amavolumu kuma-pods.

Ikakhulukazi, indawo yamagama kumele iqukathe imfihlo yokufinyelela i-Ceph ukuze izinhlelo zokusebenza ezisezingeni eliphezulu zisebenze. Kulungile uma unezindawo ezingu-2-3 ezindaweni zazo zamagama: ungahamba futhi ukopishe imfihlo ngokwenza. Kodwa kuthiwani uma isici ngasinye kwakheka indawo ehlukile enegama layo lonjiniyela?

Le nkinga siyixazulule ngokwethu sisebenzisa igobolondo-opharetha, ekopishe ngokuzenzakalelayo izimfihlo ezindaweni zamagama ezintsha (isibonelo sehuku elinjalo sichazwe ku lesi sihloko).

#! /bin/bash

if [[ $1 == “--config” ]]; then
   cat <<EOF
{"onKubernetesEvent":[
 {"name": "OnNewNamespace",
  "kind": "namespace",
  "event": ["add"]
  }
]}
EOF
else
    NAMESPACE=$(kubectl get namespace -o json | jq '.items | max_by( .metadata.creationTimestamp ) | .metadata.name')
    kubectl -n ${CEPH_SECRET_NAMESPACE} get secret ${CEPH_SECRET_NAME} -o json | jq ".metadata.namespace="${NAMESPACE}"" | kubectl apply -f -
fi

Kodwa-ke, uma usebenzisa i-Rook le nkinga ayikho. Inqubo ukukhweza kwenzeka usebenzisa abashayeli bayo esekelwe Ivolumu ye-Flex noma CSI (usesesigabeni se-beta) ngakho-ke ayidingi izimfihlo.

I-Rook ixazulula izinkinga eziningi ngokuzenzakalelayo, okusikhuthaza ukuthi siyisebenzise kumaphrojekthi amasha.

Ukuvinjezelwa kweRook

Masiqedele ingxenye esebenzayo ngokuthumela u-Rook no-Ceph ukuze sikwazi ukwenza okwethu ukuhlola. Ukwenza kube lula ukuhlasela lo mbhoshongo ongangeneki, abathuthukisi balungiselele iphakethe le-Helm. Masiyilande:

$ helm fetch rook-master/rook-ceph --untar --version 1.0.0

Kufayela rook-ceph/values.yaml ungathola izilungiselelo eziningi ezahlukene. Into ebaluleke kakhulu ukucacisa ukubekezelelwa kwama-ejenti nokusesha. Sichaze ngokuningiliziwe ukuthi indlela yokubekezelela ingasetshenziselwa ini lesi sihloko.

Ngamafuphi, asifuni ukuthi ama-pods ohlelo lweklayenti abekwe ezindaweni ezifanayo namadiski okugcina idatha. Isizathu silula: ngale ndlela umsebenzi we-Rook agents ngeke uthinte uhlelo lokusebenza ngokwalo.

Ngakho, vula ifayela rook-ceph/values.yaml nomhleli wakho owuthandayo bese wengeza ibhulokhi elandelayo ekugcineni:

discover:
  toleration: NoExecute
  tolerationKey: node-role/storage
agent:
  toleration: NoExecute
  tolerationKey: node-role/storage
  mountSecurityMode: Any

Kunodi ngayinye ibekelwe ukugcinwa kwedatha, engeza ukugcotshwa okuhambisanayo:

$ kubectl taint node ${NODE_NAME} node-role/storage="":NoExecute

Bese ufaka ishadi le-Helm ngomyalo:

$ helm install --namespace ${ROOK_NAMESPACE} ./rook-ceph

Manje udinga ukudala iqoqo futhi ucacise indawo I-OSD:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  clusterName: "ceph"
  finalizers:
  - cephcluster.ceph.rook.io
  generation: 1
  name: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v13
  dashboard:
    enabled: true
  dataDirHostPath: /var/lib/rook/osd
  mon:
    allowMultiplePerNode: false
    count: 3
  network:
    hostNetwork: true
  rbdMirroring:
    workers: 1
  placement:
    all:
      tolerations:
      - key: node-role/storage
        operator: Exists
  storage:
    useAllNodes: false
    useAllDevices: false
    config:
      osdsPerDevice: "1"
      storeType: filestore
    resources:
      limits:
        memory: "1024Mi"
      requests:
        memory: "1024Mi"
    nodes:
    - name: host-1
      directories:
      - path: "/mnt/osd"
    - name: host-2
      directories:
      - path: "/mnt/osd"
    - name: host-3
      directories:
      - path: "/mnt/osd"

Ihlola isimo se-Ceph - lindela ukubona HEALTH_OK:

$ kubectl -n ${ROOK_NAMESPACE} exec $(kubectl -n ${ROOK_NAMESPACE} get pod -l app=rook-ceph-operator -o name -o jsonpath='{.items[0].metadata.name}') -- ceph -s

Ngasikhathi sinye, ake sihlole ukuthi ama-pods anohlelo lweklayenti awapheli kumanodi abekelwe i-Ceph:

$ kubectl -n ${APPLICATION_NAMESPACE} get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Ngaphezu kwalokho, izingxenye ezengeziwe zingahlelwa ngendlela othanda ngayo. Imininingwane eminingi ngabo ikhonjisiwe ku imibhalo. Ngokuphatha, sincoma kakhulu ukuthi ufake ideshibhodi nebhokisi lamathuluzi.

I-Rook namahhuku: ingabe i-Rook yanele yonke into?

Njengoba ubona, ukuthuthukiswa kwe-Rook kuhamba ngokugcwele. Kepha kusenezinkinga ezingasivumeli ukuthi silahle ngokuphelele ukucushwa kwe-Ceph:

• Akekho umshayeli we-Rook ngeke ukuthekelisa amamethrikhi ekusetshenzisweni kwamabhulokhi agxunyekiwe, okusincisha ukuqapha.
• I-Flexvolume ne-CSI angazi kanjani shintsha usayizi wamavolumu (ngokungafani ne-RBD efanayo), ngakho u-Rook uncishwa ithuluzi eliwusizo (futhi ngezinye izikhathi elidingeka kakhulu!)
• U-Rook akakaguquki njengeCeph evamile. Uma sifuna ukulungisa ichibi ukuze imethadatha ye-CephFS igcinwe ku-SSD, futhi idatha ngokwayo izogcinwa ku-HDD, sizodinga ukubhalisa amaqembu ahlukene wamadivayisi kumamephu we-CRUSH mathupha.
• Ngaphandle kweqiniso lokuthi i-rook-ceph-opharetha ibhekwa njengezinzile, okwamanje kunezinkinga ezithile lapho kuthuthukiswa i-Ceph kusuka kunguqulo 13 kuya ku-14.

okutholakele

“Njengamanje uRook uvalelwe ngaphandle yizimbangi, kodwa sikholwa ukuthi ngolunye usuku uzodlala indima ebalulekile kulo mdlalo!” (Ingcaphuno esungulwe ngokukhethekile lesi sihloko)

Iphrojekthi ye-Rook ngokungangabazeki izizuzile izinhliziyo zethu - sikholelwa ukuthi [nabo bonke ubuhle nobubi bayo] ikufanele nakanjani ukunakwa.

Izinhlelo zethu zesikhathi esizayo zigxila ekwenzeni i-rook-ceph ibe imojuli yayo i-addon-opharetha, okuzokwenza ukusetshenziswa kwayo kumaqoqo ethu amaningi e-Kubernetes kube lula futhi kube lula nakakhulu.

PS

Funda futhi kubhulogi yethu:

• «I-Rook - indawo yokugcina idatha "yokuzisiza" ye-Kubernetes";
• «Ukudala isitoreji esiqhubekayo ngokuhlinzekwa ku-Kubernetes okusekelwe ku-Ceph";
• «Imininingo egciniwe kanye ne-Kubernetes (isibuyekezo kanye nombiko wevidiyo)";
• «Sethula i-shell-opharetha: ukudala ama-opharetha e-Kubernetes sekulula";
• «Ama-Opharetha e-Kubernetes: ukuthi zisebenza kanjani izinhlelo zokusebenza ezisezingeni eliphakeme".

Source: www.habr.com