I Rook pe leai i Rook - o le fesili lena

I le amataga o lenei masina, i le aso 3 o Me, na faʻasalalau ai se faʻasalalauga tele o se "faiga faʻatautaia mo le tufatufaina atu o faʻamaumauga i Kubernetes" na faʻasalalau - Rook 1.0.0. Ua silia ma le tausaga talu ai ua uma lomia faasalalau aotelega lautele o Rook. Ona talosagaina lea o i matou e talanoa e uiga i lona aafiaga fa'aoga i le fa'atinoga - ma o lenei, i le taimi tonu mo se faʻailoga taua tele i le talafaasolopito o le poloketi, matou te fiafia e faʻasoa a matou faʻaputuga lagona.

I se faapuupuuga, o Rook o se seti tagata e faafoeina mo Kubernetes, lea e pulea atoatoa le faʻapipiʻiina, pulega, toe faʻaleleia otometi o fofo faʻamaumauga e pei o Ceph, EdgeFS, Minio, Cassandra, CockroachDB.

I le taimi nei o le sili ona atinaʻe (ma na o le tasi в fale mautu tulaga) o le fofo o le rook-ceph-operator.

mataʻi: Faatasi ai ma suiga taua i le Rook 1.0.0 faʻasalalauga e fesoʻotaʻi ma Ceph, e mafai ona matou maitauina le lagolago mo Ceph Nautilus ma le mafai ona faʻaoga NFS mo pakete CephFS poʻo RGW. O le mea e tu i fafo i isi o le matua o le EdgeFS lagolago i le tulaga beta.

O lea la, i lenei tusiga tatou:

• Sei o tatou taliina le fesili e uiga i mea lelei tatou te vaʻaia i le faʻaaogaina o Rook e faʻapipiʻi ai Ceph i totonu o le Kubernetes cluster;
• Matou te faʻasoa atu o matou poto masani ma lagona o le faʻaaogaina o Rook i le gaosiga;
• Se'i ta'u atu ia te oe le mafuaaga tatou te fai atu ai "Ioe!" ia Rook, ma e uiga ia tatou fuafuaga mo ia.

Sei o tatou amata i manatu lautele ma talitonuga.

“E iai laʻu avanoa i le Rook e tasi!” (tagata ta'alo e le iloa)

O se tasi o faʻamanuiaga autu o Rook o le fegalegaleai ma faleoloa faʻamaumauga e faia e ala i faiga Kubernetes. O lona uiga e te le toe manaʻomia le kopiina o poloaiga e faʻapipiʻi ai Ceph mai le laupepa i totonu o le faʻamafanafanaga.

- E te manaʻo e faʻapipiʻi le CephFS i se fuifui? Na'o le tusi o se faila YAML!
- O le ā? E te manaʻo foʻi e faʻapipiʻi se faleoloa mea faitino ma S3 API? Na'o le tusi o se faila YAML lona lua!

Ua faia Rook e tusa ai ma tulafono uma a se tagata faʻapitoa masani. Fegalegaleai ma ia e tupu i le faaaogaina CRD (Custom Resource Definition), lea tatou te faamatalaina ai uiga o Ceph entities tatou te manaomia (Talu ai e na o le pau lea o le faʻatinoga mautu, e le mafai e lenei tusiga ona talanoa e uiga ia Ceph, seʻi vagana ua faʻaalia manino). E tusa ai ma taʻiala faʻapitoa, o le a otometi lava ona faʻatinoina e le tagata faʻatonu poloaiga e manaʻomia mo le faʻatulagaina.

Seʻi o tatou vaʻavaʻai i mea faʻapitoa e faʻaaoga ai le faʻataʻitaʻiga o le fatuina o se Faleoloa Mea, pe sili atu - CephObjectStoreUser.

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  gateway:
    type: s3
    sslCertificateRef:
    port: 80
    securePort:
    instances: 1
    allNodes: false
---
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  store: {{ .Values.s3.crdName }}
  displayName: {{ .Values.s3.username }}

O faʻamaufaʻailoga o loʻo faʻaalia i le lisi e fai si tulaga masani ma e tau le manaʻomia ni faʻamatalaga, ae e aoga le faʻalogo faʻapitoa ia i latou e tuʻuina atu i fesuiaiga o faʻataʻitaʻiga.

O le fuafuaga lautele o le galuega e oʻo mai i le mea moni tatou te "faʻatonu" punaoa e ala i se faila YAML, lea e faʻatino ai e le tagata faʻatonu tulafono talafeagai ma toe faʻafoʻi mai ia i tatou se mea lilo "e le o se mea moni" lea e mafai ai ona tatou galulue atili. (silasila i lalo). Ma mai fesuiaiga o loʻo lisiina i luga, o le poloaiga ma le igoa faalilolilo o le a tuʻufaʻatasia.

O le a le ituaiga 'au lea? Pe a fatuina se tagata faʻaoga mo le teuina o mea, o le Rook operator i totonu o le pod o le a faia mea nei:

radosgw-admin user create --uid="rook-user" --display-name="{{ .Values.s3.username }}"

O le taunuuga o le faʻatinoina o lenei poloaiga o le a avea ma se fausaga JSON:

{
    "user_id": "rook-user",
    "display_name": "{{ .Values.s3.username }}",
    "keys": [
        {
           "user": "rook-user",
           "access_key": "NRWGT19TWMYOB1YDBV1Y",
           "secret_key": "gr1VEGIV7rxcP3xvXDFCo4UDwwl2YoNrmtRlIAty"
        }
    ],
    ...
}

Keys - o a talosaga i le lumana'i e mana'omia e maua ai mea e teu ai mea e ala ile S3 API. E filifilia ma le agalelei e le Rook operator ma tuʻu i totonu o lona igoa avanoa i foliga o se mealilo ma le igoa rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}.

Mo le faʻaogaina o faʻamatalaga mai lenei mealilo, naʻo le faʻapipiʻi i le koneteina e fai ma suiga o le siosiomaga. Mo se faʻataʻitaʻiga, o le a ou tuʻuina atu se faʻataʻitaʻiga mo Iopu, lea tatou te otometi ai ona fatuina pakete mo tagata taʻitasi siosiomaga:

{{- range $bucket := $.Values.s3.bucketNames }}
apiVersion: batch/v1
kind: Job
metadata:
  name: create-{{ $bucket }}-bucket-job
  annotations:
    "helm.sh/hook": post-install
    "helm.sh/hook-weight": "2"
spec:
  template:
    metadata:
      name: create-{{ $bucket }}-bucket-job
    spec:
      restartPolicy: Never
      initContainers:
      - name: waitdns
        image: alpine:3.6
        command: ["/bin/sh", "-c", "while ! getent ahostsv4 rook-ceph-rgw-{{ $.Values.s3.crdName }}; do sleep 1; done" ]
      - name: config
        image: rook/ceph:v1.0.0
        command: ["/bin/sh", "-c"]
        args: ["s3cmd --configure --access_key=$(ACCESS-KEY) --secret_key=$(SECRET-KEY) -s --no-ssl --dump-config | tee /config/.s3cfg"]
        volumeMounts:
        - name: config
          mountPath: /config
        env:
        - name: ACCESS-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: AccessKey
        - name: SECRET-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: SecretKey
      containers:
      - name: create-bucket
        image: rook/ceph:v1.0.0
        command: 
        - "s3cmd"
        - "mb"
        - "--host=rook-ceph-rgw-{{ $.Values.s3.crdName }}"
        - "--host-bucket= "
        - "s3://{{ $bucket }}"
        ports:
        - name: s3-no-sll
          containerPort: 80
        volumeMounts:
        - name: config
          mountPath: /root
      volumes:
      - name: config
        emptyDir: {}
---
{{- end }}

O gaioiga uma o loʻo lisiina i lenei Iopu na faia i totonu o le faʻavae a Kubernetes. O fausaga o loʻo faʻamatalaina i faila YAML o loʻo teuina i totonu o le Git repository ma toe faʻaaogaina i le tele o taimi. Matou te vaʻaia lenei mea o se faʻaopoopoga tele mo DevOps inisinia ma le CI / CD faʻasologa atoa.

Fiafia ma Rook ma Rados

O le faʻaaogaina o le Ceph + RBD tuʻufaʻatasiga e tuʻuina atu ai ni tapulaʻa i luga o le faʻapipiʻiina o voluma i pods.

Aemaise lava, o le igoa avanoa e tatau ona i ai se mealilo mo le mauaina o Ceph ina ia mafai ai ona faʻaogaina talosaga faʻapitoa. E lelei pe a iai sau siosiomaga 2-3 i o latou igoa: e mafai ona e alu e kopi le mealilo ma le lima. Ae faʻafefea pe a fai mo faʻaaliga taʻitasi se siosiomaga eseʻese ma lona lava igoa igoa e faia mo tagata atiaʻe?

Na matou foia lenei faafitauli ia i matou lava e faʻaaogaina tagata fai atigi, lea e otometi ona kopiina mea lilo i igoa fou (o se faʻataʻitaʻiga o se matau o loʻo faʻamatalaina i lenei tusiga).

#! /bin/bash

if [[ $1 == “--config” ]]; then
   cat <<EOF
{"onKubernetesEvent":[
 {"name": "OnNewNamespace",
  "kind": "namespace",
  "event": ["add"]
  }
]}
EOF
else
    NAMESPACE=$(kubectl get namespace -o json | jq '.items | max_by( .metadata.creationTimestamp ) | .metadata.name')
    kubectl -n ${CEPH_SECRET_NAMESPACE} get secret ${CEPH_SECRET_NAME} -o json | jq ".metadata.namespace="${NAMESPACE}"" | kubectl apply -f -
fi

Ae ui i lea, pe a faʻaaogaina Rook e le o iai lenei faʻafitauli. O le faʻapipiʻiina o faʻagasologa e tupu i le faʻaaogaina o ona lava avetaavale e faʻavae i luga Flexvolume poʻo CSI (o loʻo i ai pea i le beta stage) ma o lea e le manaʻomia ai mea lilo.

E otometi lava ona foia e Rook le tele o faafitauli, lea e faʻamalosia ai i tatou e faʻaaogaina i galuega fou.

Siege of Rook

Se'i fa'amae'a le vaega fa'atino e ala i le fa'aogaina o Rook ma Ceph ina ia mafai ai ona fa'atino a tatou lava su'ega. Ina ia faʻafaigofie le faʻafefe o lenei olo e le mafai ona faʻafefe, ua saunia e le au atiaʻe se pusa Helm. Se'i tatou la'uina mai:

$ helm fetch rook-master/rook-ceph --untar --version 1.0.0

I le faila rook-ceph/values.yaml e mafai ona e mauaina le tele o faatulagaga eseese. O le mea pito sili ona taua o le faʻamaonia o toleniga mo sui ma suʻesuʻega. Na matou fa'amatala auiliili pe o le a le mea e mafai ona fa'aoga ai le tants/toleration mechanism lenei tusiga.

I se faapuupuuga, matou te le manaʻo e tuʻu i luga o pona tutusa ma tisiki e teu ai faʻamatalaga. E faigofie le mafuaʻaga: o le auala lea e le afaina ai le talosaga a le Rook agents.

O lea, tatala le faila rook-ceph/values.yaml ma lau faatonu e sili ona e fiafia i ai ma faaopoopo le poloka lea i le faaiuga:

discover:
  toleration: NoExecute
  tolerationKey: node-role/storage
agent:
  toleration: NoExecute
  tolerationKey: node-role/storage
  mountSecurityMode: Any

Mo node ta'itasi ua fa'aagaga mo le teuina o fa'amaumauga, fa'aopoopo le taint talafeagai:

$ kubectl taint node ${NODE_NAME} node-role/storage="":NoExecute

Ona faʻapipiʻi lea o le siata Helm ma le faʻatonuga:

$ helm install --namespace ${ROOK_NAMESPACE} ./rook-ceph

Ole taimi nei e tatau ona e fatuina se fuifui ma faʻamaonia le nofoaga OSD:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  clusterName: "ceph"
  finalizers:
  - cephcluster.ceph.rook.io
  generation: 1
  name: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v13
  dashboard:
    enabled: true
  dataDirHostPath: /var/lib/rook/osd
  mon:
    allowMultiplePerNode: false
    count: 3
  network:
    hostNetwork: true
  rbdMirroring:
    workers: 1
  placement:
    all:
      tolerations:
      - key: node-role/storage
        operator: Exists
  storage:
    useAllNodes: false
    useAllDevices: false
    config:
      osdsPerDevice: "1"
      storeType: filestore
    resources:
      limits:
        memory: "1024Mi"
      requests:
        memory: "1024Mi"
    nodes:
    - name: host-1
      directories:
      - path: "/mnt/osd"
    - name: host-2
      directories:
      - path: "/mnt/osd"
    - name: host-3
      directories:
      - path: "/mnt/osd"

Siaki le tulaga Ceph - faʻamoemoe e vaʻai HEALTH_OK:

$ kubectl -n ${ROOK_NAMESPACE} exec $(kubectl -n ${ROOK_NAMESPACE} get pod -l app=rook-ceph-operator -o name -o jsonpath='{.items[0].metadata.name}') -- ceph -s

I le taimi lava e tasi, seʻi o tatou siaki pe o le pods ma le talosaga a le kalani e le faʻamutaina i nodes ua faʻaagaga mo Ceph:

$ kubectl -n ${APPLICATION_NAMESPACE} get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

E le gata i lea, e mafai ona faʻapipiʻi vaega faaopoopo pe a manaʻo ai. O nisi fa'amatalaga e uiga ia i latou o lo'o fa'aalia i totonu fa'amaumauga. Mo le pulega, matou te fautuaina malosi le faʻapipiʻiina o le dashboard ma meafaigaluega.

Rook ma matau: ua lava Rook mo mea uma?

E pei ona mafai ona e vaʻai, o le atinaʻeina o Rook o loʻo faʻasolosolo. Ae o loʻo i ai pea faʻafitauli e le mafai ai ona tatou lafoai atoa le faʻatulagaina o le tusi lesona a Ceph:

• Leai se Rook Driver le mafai fua fa'atau atu i luga o le fa'aogaina o poloka fa'apipi'i, lea e fa'ate'aina ai i tatou mai le mata'ituina.
• Flexvolume ma CSI le iloa pe faapefea suia le tele o voluma (e ese mai i le RBD tutusa), o lea ua le maua ai e Rook se mea aoga (ma o nisi taimi e matua manaʻomia ai!) Meafaigaluega.
• Rook e leʻo faʻafefeteina e pei o Ceph masani. Afai matou te mananaʻo e faʻapipiʻi le vaitaele mo CephFS metadata e teuina i SSD, ma faʻamaumauga lava ia e teuina i luga o le HDD, matou te manaʻomia le resitalaina o vaega eseese o masini i faʻafanua CRUSH ma le lima.
• E ui lava i le mea moni o le rook-ceph-operator ua manatu e mautu, o loʻo i ai nei faʻafitauli pe a faʻaleleia Ceph mai le version 13 i le 14.

sailiiliga

"I le taimi nei ua tapunia Rook mai le lalolagi i fafo e pawns, ae matou te talitonu e i ai se aso o le a ia faia ai se sao taua i le taaloga!" (upusii na faia faapitoa mo lenei tusiga)

O le poloketi Rook e mautinoa lava na manumalo ai o matou loto - matou te talitonu o [ma ona lelei uma ma mea leaga] e mautinoa lava e tatau ona e gauai atu i ai.

O a tatou fuafuaga i le lumanaʻi e faʻapipiʻi i lalo i le faia o rook-ceph o se module mo addon-operator, lea o le a fa'aoga ai i le tele o fuifui Kubernetes e sili atu ona faigofie ma sili atu ona faigofie.

SALA

Faitau foi i la matou blog:

• «Rook - "auaunaga a le tagata lava ia" fale teu oloa mo Kubernetes";
• «Fausia le teuina faifai pea ma le tuʻuina atu i Kubernetes faʻavae i luga ole Ceph";
• «Fa'amaumauga ma Kubernetes (toe iloilo ma lipoti vitio)";
• «Fa'ailoaina o le atigi-operator: fa'atupuina fa'agaioiga mo Kubernetes ua fa'afaigofie";
• «Fa'atonu mo Kubernetes: fa'afefea ona fa'agaoioia talosaga fa'apitoa".

puna: www.habr.com