ืจื•ืง ืึธื“ืขืจ ื ื™ืฉื˜ ืจื•ืง, ื“ืึธืก ืื™ื– ื“ื™ ืงืฉื™ื

ืจื•ืง ืึธื“ืขืจ ื ื™ืฉื˜ ืจื•ืง, ื“ืึธืก ืื™ื– ื“ื™ ืงืฉื™ื

ืื™ืŸ ื“ื™ ืึธื ื”ื™ื™ื‘ ืคื•ืŸ ื“ืขื ื—ื•ื“ืฉ, ืื•ื™ืฃ ืžืื™ 3, ืึท ื”ื•ื™ืคึผื˜ ืžืขืœื“ื•ื ื’ ืคื•ืŸ ืึท "ืคืึทืจื•ื•ืึทืœื˜ื•ื ื’ ืกื™ืกื˜ืขื ืคึฟืึทืจ ืคื•ื ืื ื“ืขืจื’ืขื˜ื™ื™ืœื˜ ื“ืึทื˜ืŸ ืกื˜ืึธืจื™ื“ื–ืฉ ืื™ืŸ Kubernetes" ืื™ื– ืžื•ื“ื™ืข - ืจื•ืงืŸ 1.0.0. ืžืขืจ ื•ื•ื™ ืึท ื™ืึธืจ ืฆื•ืจื™ืง ืžื™ืจ ืฉื•ื™ืŸ ืืจื•ื™ืก ืึทืœื’ืขืžื™ื™ื ืข ืื™ื‘ืขืจื‘ืœื™ืง ืคื•ืŸ Rook. ื“ืขืจื ืึธืš ื”ืึธื˜ ืžืขืŸ ืื•ื ื“ื– ื’ืขื‘ืขื˜ืŸ ืฆื• ืจืขื“ืŸ ื•ื•ืขื’ืŸ ื–ืฒึทืŸ ื“ืขืจืคืึทืจื•ื ื’ ื ื•ืฆืŸ ืื™ืŸ ืคื™ืจ - ืื•ืŸ ืื™ืฆื˜, ืคึผื•ื ืงื˜ ืื™ืŸ ืฆื™ื™ื˜ ืคึฟืึทืจ ืึทื–ืึท ืึท ื‘ืึทื˜ื™ื™ื˜ื™ืง ืžื™ื™ืœืกื˜ืึธื•ืŸ ืื™ืŸ ื“ืขืจ ื’ืขืฉื™ื›ื˜ืข ืคื•ืŸ โ€‹โ€‹ื“ื™ ืคึผืจื•ื™ืขืงื˜, ืžื™ืจ ื–ืขื ืขืŸ ืฆื•ืคืจื™ื“ืŸ ืฆื• ื˜ื™ื™ืœืŸ ืื•ื ื“ื–ืขืจ ืึทืงื™ื•ืžื™ืึทืœื™ื™ื˜ื™ื“ ื™ืžืคึผืจืขืกืกื™ืึธื ืก.

ืื™ืŸ ืงื•ืจืฅ, ืจื•ืง ืื™ื– ืึท ื’ืึทื ื’ ืึธืคึผืขืจื™ื™ื˜ืขืจื– ืคึฟืึทืจ Kubernetes, ื•ื•ืึธืก ื ืขืžืขืŸ ืคื•ืœ ืงืึธื ื˜ืจืึธืœ ืคื•ืŸ ื“ื™ ื“ื™ืคึผืœื•ื™ืžืึทื ื˜, ืคืึทืจื•ื•ืึทืœื˜ื•ื ื’, ืึธื˜ืึทืžืึทื˜ื™ืง ืึธืคึผื–ื•ืš ืคื•ืŸ ื“ืึทื˜ืŸ ืกื˜ืึธืจื™ื“ื–ืฉ ืกืึทืœื•ืฉืึทื ื– ืึทื–ืึท ื•ื•ื™ Ceph, EdgeFS, Minio, Cassandra, CockroachDB.

ืื™ืŸ ื“ืขืจ ืžืึธืžืขื ื˜ ื“ื™ ืžืขืจืกื˜ ื“ืขื•ื•ืขืœืึธืคึผืขื“ (ืื•ืŸ ื“ื™ ืื™ื™ื ืฆื™ื’ืกื˜ืข ะฒ ืกื˜ืึทื‘ื™ืœ ื‘ื™ื ืข) ื“ื™ ืœื™ื™ื–ื•ื ื’ ืื™ื– ืจื•ืง-ืกืขืฃ-ืึธืคึผืขืจืึทื˜ืึธืจ.

ื˜ืึธืŸ: ืฆื•ื•ื™ืฉืŸ ื“ื™ ื‘ืึทื˜ื™ื™ึทื˜ื™ืง ืขื ื“ืขืจื•ื ื’ืขืŸ ืื™ืŸ ื“ื™ Rook 1.0.0 ืžืขืœื“ื•ื ื’ ืฉื™ื™ึทื›ื•ืช ืฆื• Ceph, ืžื™ืจ ืงืขื ืขืŸ ื˜ืึธืŸ ืฉื˜ื™ืฆืŸ ืคึฟืึทืจ Ceph Nautilus ืื•ืŸ ื“ื™ ืคื™ื™ื™ืงื™ื™ื˜ ืฆื• ื ื•ืฆืŸ NFS ืคึฟืึทืจ CephFS ืึธื“ืขืจ RGW ื‘ืึทืงืึทืฅ. ื•ื•ืึธืก ืฉื˜ื™ื™ื˜ ืื•ื™ืก ืฆื•ื•ื™ืฉืŸ ืื ื“ืขืจืข ืื™ื– ื“ื™ ืžืึทื˜ืฉื•ืจื™ื™ืฉืึทืŸ ืคื•ืŸ EdgeFS ืฉื˜ื™ืฆืŸ ืฆื• ื“ื™ ื‘ื™ืชื ืžื“ืจื’ื”.

ืึทื–ื•ื™, ืื™ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ ืžื™ืจ:

  • ืœืึธืžื™ืจ ืขื ื˜ืคึฟืขืจืŸ ื“ื™ ืงืฉื™ื ื•ื•ืขื’ืŸ ื•ื•ืึธืก ืึทื“ื•ื•ืึทื ื˜ื™ื“ื–ืฉื™ื– ืžื™ืจ ื–ืขืŸ ืื™ืŸ ื ื™ืฆืŸ Rook ืฆื• ืฆืขื•ื•ื™ืงืœืขืŸ Ceph ืื™ืŸ ืึท ืงื•ื‘ืขืจื ืขื˜ืขืก ืงื ื•ื™ืœ;
  • ืžื™ืจ ื•ื•ืขืœืŸ ื˜ื™ื™ืœืŸ ืื•ื ื“ื–ืขืจ ื“ืขืจืคืึทืจื•ื ื’ ืื•ืŸ ื™ืžืคึผืจืขืกืกื™ืึธื ืก ืคื•ืŸ ื ื™ืฆืŸ ืจื•ืง ืื™ืŸ ืคึผืจืึธื“ื•ืงืฆื™ืข;
  • ืœืืžื™ืจ ืื™ื™ืš ื–ืื’ืŸ ืคืืจื•ื•ืืก ืžื™ืจ ื–ืื’ืŸ "ื™ื!" ืฆื• ืจื•ืง, ืื•ืŸ ื•ื•ืขื’ืŸ ืื•ื ื–ืขืจืข ืคืœืขื ืขืจ ืคืืจ ืื™ื.

ืœืืžื™ืจ ืื ื”ื™ื™ื‘ืŸ ืžื™ื˜ ืืœื’ืขืžื™ื™ื ืข ื‘ืื’ืจื™ืคืŸ ืื•ืŸ ื˜ืขืืจื™ืข.

"ืื™ืš ื”ืึธื‘ืŸ ืึท ืžื™ื™ึทืœืข ืคื•ืŸ โ€‹โ€‹ืื™ื™ืŸ ืจื•ืง!" (ืื•ืžื‘ืึทืงืึทื ื˜ ืฉืึธืš ืฉืคึผื™ืœืขืจ)

ืจื•ืง ืึธื“ืขืจ ื ื™ืฉื˜ ืจื•ืง, ื“ืึธืก ืื™ื– ื“ื™ ืงืฉื™ื

ืื™ื™ื ืขืจ ืคื•ืŸ ื“ื™ ื”ื•ื™ืคึผื˜ ืึทื“ื•ื•ืึทื ื˜ื™ื“ื–ืฉื™ื– ืคื•ืŸ Rook ืื™ื– ืึทื– ื™ื ื˜ืขืจืึทืงืฉืึทืŸ ืžื™ื˜ ื“ืึทื˜ืŸ ืกื˜ืึธืจื– ืื™ื– ื“ื•ืจื›ื’ืขืงืึธื›ื˜ ื“ื•ืจืš ืงื•ื‘ืขืจื ืขื˜ืขืก ืžืขืงืึทื ื™ื–ืึทืžื–. ื“ืขื ืžื™ื˜ืœ ืึทื– ืื™ืจ ื ื™ื˜ ืžืขืจ ื“ืึทืจืคึฟืŸ ืฆื• ื ืึธื›ืžืึทื›ืŸ ื“ื™ ืงืึทืžืึทื ื“ื– ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ Ceph ืคึฟื•ืŸ ื“ื™ ื‘ืœืึทื˜ ืื™ืŸ ื“ื™ ืงืึทื ืกืึธื•ืœ.

โ€” ืฆื™ ืื™ืจ ื•ื•ื™ืœืŸ ืฆื• ืฆืขื•ื•ื™ืงืœืขืŸ CephFS ืื™ืŸ ืึท ืงื ื•ื™ืœ? ื ืึธืจ ืฉืจื™ื™ึทื‘ืŸ ืึท YAML ื˜ืขืงืข!
- ื•ื•ืืก? ืฆื™ ืื™ืจ ืื•ื™ืš ื•ื•ื™ืœืŸ ืฆื• ืฆืขื•ื•ื™ืงืœืขืŸ ืึท ื›ื™ื™ืคืขืฅ ืงืจืึธื ืžื™ื˜ S3 API? ืฉืจื™ื™ื‘ ื ืืจ ื ืฆื•ื•ื™ื™ื˜ืข ื™ืืžืœ ืคื™ื™ืœ!

ืจื•ืง ืื™ื– ื‘ืืฉืืคืŸ ืœื•ื™ื˜ ืึทืœืข ื“ื™ ื›ึผืœืœื™ื ืคื•ืŸ ืึท ื˜ื™ืคึผื™ืฉ ืึธืคึผืขืจืึทื˜ืึธืจ. ื™ื ื˜ืขืจืึทืงืฉืึทืŸ ืžื™ื˜ ืื™ื ืึทืงืขืจื– ื ื™ืฆืŸ CRD (Custom Resource Definitions), ืื™ืŸ ื•ื•ืึธืก ืžื™ืจ ื‘ืึทืฉืจื™ื™ึทื‘ืŸ ื“ื™ ืงืขืจืึทืงื˜ืขืจื™ืกื˜ื™ืงืก ืคื•ืŸ ืกืขืฃ ืขื ื˜ื™ื˜ื™ื– ืžื™ืจ ื“ืึทืจืคึฟืŸ (ื–ื™ื ื˜ ื“ืึธืก ืื™ื– ื“ืขืจ ื‘ืœื•ื™ื– ืกื˜ืึทื‘ื™ืœ ื™ืžืคึผืœืึทืžืขื ื˜ื™ื™ืฉืึทืŸ, ื“ื•ืจืš ืคืขืœื™ืงื™ื™ึทื˜ ื“ืขื ืึทืจื˜ื™ืงืœ ื•ื•ืขื˜ ืจืขื“ืŸ ื•ื•ืขื’ืŸ Ceph, ืกื™ื™ึทื“ืŸ ื‘ืคื™ืจื•ืฉ ืกื˜ื™ื™ื˜ื™ื“ ืึทื ื“ืขืจืฉ). ืœื•ื™ื˜ ื“ื™ ืกืคึผืขืกื™ืคื™ืขื“ ืคึผืึทืจืึทืžืขื˜ืขืจืก, ื“ืขืจ ืึธืคึผืขืจืึทื˜ืึธืจ ื•ื•ืขื˜ ืื•ื™ื˜ืึธืžืึทื˜ื™ืฉ ื•ื™ืกืคื™ืจืŸ ื“ื™ ืงืึทืžืึทื ื“ื– ื ื™ื™ื˜ื™ืง ืคึฟืึทืจ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ.

ืœืึธืžื™ืจ ืงื•ืงืŸ ืื™ืŸ ื“ื™ ืกืคึผืขืกื™ืคื™ืงืก ื ื™ืฆืŸ ื“ืขื ื‘ื™ื™ืฉืคึผื™ืœ ืคื•ืŸ ืงืจื™ื™ื™ื˜ื™ื ื’ ืึท ืึธื‘ื“ื–ืฉืขืงื˜ ืกื˜ืึธืจ, ืึธื“ืขืจ ื’ืึทื ืฅ - CephObjectStoreUser.

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  gateway:
    type: s3
    sslCertificateRef:
    port: 80
    securePort:
    instances: 1
    allNodes: false
---
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: {{ .Values.s3.crdName }}
  namespace: kube-rook
spec:
  store: {{ .Values.s3.crdName }}
  displayName: {{ .Values.s3.username }}

ื“ื™ ืคึผืึทืจืึทืžืขื˜ืขืจืก ืื ื’ืขื•ื•ื™ื–ืŸ ืื™ืŸ ื“ื™ ืœื™ืกื˜ื™ื ื’ ื–ืขื ืขืŸ ื’ืึทื ืฅ ื ืึธืจืžืึทืœ ืื•ืŸ ืงื•ื™ื ื“ืึทืจืคึฟืŸ ื‘ืึทืžืขืจืงื•ื ื’ืขืŸ, ืึธื‘ืขืจ ืขืก ืื™ื– ื•ื•ืขืจื˜ ืฆื• ื‘ืึทืฆืึธืœืŸ ืกืคึผืขืฆื™ืขืœ ื•ืคืžืขืจืงื–ืึทืžืงื™ื™ื˜ ืฆื• ื“ื™ ืึทืœืึทืงื™ื™ื˜ื™ื“ ืฆื• ืžื•ืกื˜ืขืจ ื•ื•ืขืจื™ืึทื‘ืึทืœื–.

ื“ืขืจ ืึทืœื’ืขืžื™ื™ืŸ ืึทืจื‘ืขื˜ ืกื›ืขืžืข ืงื•ืžื˜ ืฆื• ื“ืขื ืคืึทืงื˜ ืึทื– ืžื™ืจ "ื‘ืึทืฉื˜ืขืœืŸ" ืจืขืกื•ืจืกืŸ ื“ื•ืจืš ืึท ื™ืžืœ ื˜ืขืงืข, ืคึฟืึทืจ ื•ื•ืึธืก ื“ืขืจ ืึธืคึผืขืจืึทื˜ืึธืจ ืขืงืกืึทืงื™ื•ืฅ ื“ื™ ื ื•ื™ื˜ื™ืง ืงืึทืžืึทื ื“ื– ืื•ืŸ ืงืขืจื˜ ืื•ื ื“ื– ืึท "ื ื™ื˜-ืึทื–ื•ื™-ืคืึทืงื˜ื™ืฉ" ืกื•ื“ ืžื™ื˜ ื•ื•ืึธืก ืžื™ืจ ืงืขื ืขืŸ ื•ื•ื™ื™ึทื˜ืขืจ ืึทืจื‘ืขื˜ืŸ (ื–ืข ืื•ื ื˜ืŸ). ืื•ืŸ ืคึฟื•ืŸ ื“ื™ ื•ื•ืขืจื™ืึทื‘ืึทืœื– ืœื™ืกื˜ืขื“ ืื•ื™ื‘ืŸ, ื“ื™ ื‘ืึทืคึฟืขืœ ืื•ืŸ ืกื•ื“ ื ืึธืžืขืŸ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืงืึทืžืคึผื™ื™ืœื“.

ื•ื•ืึธืก ืžื™ืŸ ืคื•ืŸ ืžืึทื ืฉืึทืคึฟื˜ ืื™ื– ื“ืึธืก? ื•ื•ืขืŸ ืงืจื™ื™ื™ื˜ื™ื ื’ ืึท ื‘ืึทื ื™ืฆืขืจ ืคึฟืึทืจ ืึทื‘ื“ื–ืฉืขืงืฅ ืกื˜ืึธืจื™ื“ื–ืฉ, ื“ืขืจ ืจื•ืง ืึธืคึผืขืจืึทื˜ืึธืจ ืื™ืŸ ื“ื™ ืคึผืึธื“ ื•ื•ืขื˜ ื˜ืึธืŸ ื“ื™ ืคืืœื’ืขื ื“ืข:

radosgw-admin user create --uid="rook-user" --display-name="{{ .Values.s3.username }}"

ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜ ืคื•ืŸ ืขืงืกืึทืงื™ื•ื˜ื™ื ื’ ื“ืขื ื‘ืึทืคึฟืขืœ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืึท JSON ืกื˜ืจื•ืงื˜ื•ืจ:

{
    "user_id": "rook-user",
    "display_name": "{{ .Values.s3.username }}",
    "keys": [
        {
           "user": "rook-user",
           "access_key": "NRWGT19TWMYOB1YDBV1Y",
           "secret_key": "gr1VEGIV7rxcP3xvXDFCo4UDwwl2YoNrmtRlIAty"
        }
    ],
    ...
}

Keys - ื•ื•ืึธืก ืฆื•ืงื•ื ืคึฟื˜ ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ื•ื•ืขื˜ ื“ืึทืจืคึฟืŸ ืฆื• ืึทืงืกืขืก ืึทื‘ื“ื–ืฉืขืงืฅ ืกื˜ืึธืจื™ื“ื–ืฉ ื“ื•ืจืš ื“ื™ S3 API. ื“ืขืจ ืจื•ืง ืึธืคึผืขืจืึทื˜ืึธืจ ืกืึทืœืขืงืฅ ื–ื™ื™ ืœื™ื‘ ืื•ืŸ ืœื™ื™ื’ื˜ ื–ื™ื™ ืื™ืŸ ื–ื™ื™ืŸ ื ืึธืžืขืŸ ืื™ืŸ ื“ื™ ืคืึธืจืขื ืคื•ืŸ ืึท ืกื•ื“ ืžื™ื˜ ื“ืขื ื ืึธืžืขืŸ rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}.

ืฆื• ื ื•ืฆืŸ ื“ื™ ื“ืึทื˜ืŸ ืคื•ืŸ ื“ืขื ืกื•ื“, ื ืึธืจ ืœื™ื™ื’ืŸ ืขืก ืฆื• ื“ืขื ืงืึทื ื˜ื™ื™ื ืขืจ ื•ื•ื™ ื™ื ื•ื•ื™ื™ืจืึทื ืžืขื ืึทืœ ื•ื•ืขืจื™ืึทื‘ืึทืœื–. ื•ื•ื™ ืึท ื‘ื™ื™ึทืฉืคึผื™ืœ, ืื™ืš ื•ื•ืขื˜ ื’ืขื‘ืŸ ืึท ืžื•ืกื˜ืขืจ ืคึฟืึทืจ ื“ื–ืฉืึธื‘, ืื™ืŸ ื•ื•ืึธืก ืžื™ืจ ืื•ื™ื˜ืึธืžืึทื˜ื™ืฉ ืฉืึทืคึฟืŸ ื‘ืึทืงืึทืฅ ืคึฟืึทืจ ื™ืขื“ืขืจ ื‘ืึทื ื™ืฆืขืจ ืกื•ื•ื™ื•ื•ืข:

{{- range $bucket := $.Values.s3.bucketNames }}
apiVersion: batch/v1
kind: Job
metadata:
  name: create-{{ $bucket }}-bucket-job
  annotations:
    "helm.sh/hook": post-install
    "helm.sh/hook-weight": "2"
spec:
  template:
    metadata:
      name: create-{{ $bucket }}-bucket-job
    spec:
      restartPolicy: Never
      initContainers:
      - name: waitdns
        image: alpine:3.6
        command: ["/bin/sh", "-c", "while ! getent ahostsv4 rook-ceph-rgw-{{ $.Values.s3.crdName }}; do sleep 1; done" ]
      - name: config
        image: rook/ceph:v1.0.0
        command: ["/bin/sh", "-c"]
        args: ["s3cmd --configure --access_key=$(ACCESS-KEY) --secret_key=$(SECRET-KEY) -s --no-ssl --dump-config | tee /config/.s3cfg"]
        volumeMounts:
        - name: config
          mountPath: /config
        env:
        - name: ACCESS-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: AccessKey
        - name: SECRET-KEY
          valueFrom:
            secretKeyRef:
              name: rook-ceph-object-user-{{ $.Values.s3.crdName }}-{{ $.Values.s3.username }}
              key: SecretKey
      containers:
      - name: create-bucket
        image: rook/ceph:v1.0.0
        command: 
        - "s3cmd"
        - "mb"
        - "--host=rook-ceph-rgw-{{ $.Values.s3.crdName }}"
        - "--host-bucket= "
        - "s3://{{ $bucket }}"
        ports:
        - name: s3-no-sll
          containerPort: 80
        volumeMounts:
        - name: config
          mountPath: /root
      volumes:
      - name: config
        emptyDir: {}
---
{{- end }}

ืึทืœืข ืึทืงืฉืึทื ื– ืœื™ืกื˜ืขื“ ืื™ืŸ ื“ืขื ืึทืจื‘ืขื˜ ื–ืขื ืขืŸ ื“ื•ืจื›ื’ืขืงืึธื›ื˜ ืื™ืŸ ื“ื™ ืคืจื™ื™ืžื•ื•ืขืจืง ืคื•ืŸ Kubernetes. ื“ื™ ืกื˜ืจืึทืงื˜ืฉืขืจื– ื“ื™ืกืงืจื™ื™ื‘ื“ ืื™ืŸ YAML ื˜ืขืงืขืก ื–ืขื ืขืŸ ืกื˜ืึธืจื“ ืื™ืŸ ืึท Git ืจื™ืคึผืึทื–ืึทื˜ืึธืจื™ ืื•ืŸ ืจื™ื•ื–ื“ ืคื™ืœืข ืžืึธืœ. ืžื™ืจ ื–ืขืŸ ื“ืขื ื•ื•ื™ ืึท ืจื™ื–ื™ืง ืคึผืœื•ืก ืคึฟืึทืจ DevOps ืขื ื“ื–ืฉืึทื ื™ืจื– ืื•ืŸ ื“ื™ CI / CD ืคึผืจืึธืฆืขืก ื•ื•ื™ ืึท ื’ืึทื ืฅ.

ืฆื•ืคืจื™ื“ืŸ ืžื™ื˜ ืจื•ืง ืื•ืŸ ืจืึทื“ืึธืก

ื ื™ืฆืŸ ื“ื™ Ceph + RBD ืงืึธืžื‘ื™ื ืึทืฆื™ืข ื™ืžืคึผืึธื•ื–ืึทื– ื–ื™ื›ืขืจ ืจื™ืกื˜ืจื™ืงืฉืึทื ื– ืื•ื™ืฃ ืžืึทื•ื ื˜ื™ื ื’ ื•ื•ืึทืœื™ื•ืžื– ืฆื• ืคึผืึธื“ืก.

ืื™ืŸ ื‘ืึทื–ื•ื ื“ืขืจ, ื“ื™ ื ืึทืžืขืกืคึผืึทืกืข ืžื•ื–ืŸ ืึทื ื˜ื”ืึทืœื˜ืŸ ืึท ืกื•ื“ ืคึฟืึทืจ ืึทืงืกืขืก Ceph ืื™ืŸ ืกื“ืจ ืคึฟืึทืจ ืกื˜ื™ื™ื˜ืคื•ืœ ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ืฆื• ืคื•ื ืงืฆื™ืึธื ื™ืจืŸ. ืขืก ืื™ื– ื’ื•ื˜ ืื•ื™ื‘ ืื™ืจ ื”ืึธื‘ืŸ 2-3 ื™ื ื•ื•ื™ื™ืจืึทื ืžืึทื ืฅ ืื™ืŸ ื–ื™ื™ืขืจ ื ืึธืžืขืŸ ืกืคึผื™ื™ืกืึทื–: ืื™ืจ ืงืขื ืขืŸ ื ืึธื›ืžืึทื›ืŸ ื“ืขื ืกื•ื“ ืžืึทื ื™ื•ืึทืœื™. ืึธื‘ืขืจ ื•ื•ืึธืก ืื•ื™ื‘ ืคึฟืึทืจ ื™ืขื“ืขืจ ืฉื˜ืจื™ืš ืึท ื‘ืึทื–ื•ื ื“ืขืจ ืกื•ื•ื™ื•ื•ืข ืžื™ื˜ ื–ื™ื™ืŸ ืื™ื™ื’ืขื ืข ื ืึธืžืขืŸ ืื™ื– ื‘ืืฉืืคืŸ ืคึฟืึทืจ ื“ืขื•ื•ืขืœืึธืคึผืขืจืก?

ืžื™ืจ ืกืึทืœื•ื•ื“ ื“ืขื ืคึผืจืึธื‘ืœืขื ื–ื™ืš ื ื™ืฆืŸ ืฉืึธืœ-ืึธืคึผืขืจืึทื˜ืึธืจ, ื•ื•ืึธืก ืื•ื™ื˜ืึธืžืึทื˜ื™ืฉ ืงืึทืคึผื™ื“ ืกื™ืงืจื™ืฅ ืฆื• ื ื™ื™ึทืข ื ืึธืžืขืŸ ืกืคึผื™ื™ืกืึทื– (ืึท ื‘ื™ื™ืฉืคึผื™ืœ ืคื•ืŸ ืึทื–ืึท ืึท ืงืจื•ืง ืื™ื– ื“ื™ืกืงืจื™ื™ื‘ื“ ืื™ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ).

#! /bin/bash

if [[ $1 == โ€œ--configโ€ ]]; then
   cat <<EOF
{"onKubernetesEvent":[
 {"name": "OnNewNamespace",
  "kind": "namespace",
  "event": ["add"]
  }
]}
EOF
else
    NAMESPACE=$(kubectl get namespace -o json | jq '.items | max_by( .metadata.creationTimestamp ) | .metadata.name')
    kubectl -n ${CEPH_SECRET_NAMESPACE} get secret ${CEPH_SECRET_NAME} -o json | jq ".metadata.namespace="${NAMESPACE}"" | kubectl apply -f -
fi

ืึธื‘ืขืจ, ื•ื•ืขืŸ ื ื™ืฆืŸ Rook ื“ืขื ืคึผืจืึธื‘ืœืขื ืคืฉื•ื˜ ื˜ื•ื˜ ื ื™ืฉื˜ ืขืงืกื™ืกื˜ื™ืจืŸ. ื“ื™ ืžืึทื•ื ื˜ื™ื ื’ ืคึผืจืึธืฆืขืก ืึทืงืขืจื– ื ื™ืฆืŸ ื–ื™ื™ึทืŸ ืื™ื™ื’ืขื ืข ื“ืจื™ื•ื•ืขืจืก ื‘ืื–ื™ืจื˜ ืื•ื™ืฃ Flexvolum ืึธื“ืขืจ ืงืกื™ (ื ืึธืš ืื™ืŸ ื‘ื™ืชื ื‘ื™ื ืข) ืื•ืŸ ื“ืขืจื™ื‘ืขืจ ื˜ื•ื˜ ื ื™ืฉื˜ ื“ืึทืจืคืŸ ืกื™ืงืจื™ืฅ.

ืจื•ืง ืื•ื™ื˜ืึธืžืึทื˜ื™ืฉ ืกืึทืœื•ื•ื– ืคื™ืœืข ืคึผืจืึธื‘ืœืขืžืก, ื•ื•ืึธืก ื™ื ืงืขืจืึทื“ื–ืฉืึทื– ืื•ื ื“ื– ืฆื• ื ื•ืฆืŸ ืขืก ืื™ืŸ ื ื™ื™ึทืข ืคึผืจืึทื“ื–ืฉืขืงืก.

ืกื™ื“ื–ืฉ ืคื•ืŸ ืจื•ืง

ืœืึธืžื™ืจ ืคืึทืจืขื ื“ื™ืงืŸ ื“ืขื ืคึผืจืึทืงื˜ื™ืฉ ื˜ื™ื™ืœ ื“ื•ืจืš ื“ื™ืคึผืœื™ื™ื™ื ื’ Rook ืื•ืŸ Ceph ืึทื–ื•ื™ ืžื™ืจ ืงืขื ืขืŸ ื“ื•ืจื›ืคื™ืจืŸ ืื•ื ื“ื–ืขืจ ืื™ื™ื’ืขื ืข ื™ืงืกืคึผืขืจืึทืžืึทื ืฅ. ืฆื• ืžืึทื›ืŸ ืขืก ื’ืจื™ื ื’ืขืจ ืฆื• ืฉื˜ื•ืจืขื ื“ืขื ื™ืžืคึผืจืขื’ื ืึทื‘ืึทืœ ื˜ื•ืจืขื, ื“ื™ ื“ืขื•ื•ืขืœืึธืคึผืขืจืก ื”ืึธื‘ืŸ ืฆื•ื’ืขื’ืจื™ื™ื˜ ืึท ื”ืขืœื ืคึผืขืงืœ. ืœืึธืžื™ืจ ืขืก ืืจืืคืงืืคื™ืข:

$ helm fetch rook-master/rook-ceph --untar --version 1.0.0

ืื™ืŸ ื˜ืขืงืข rook-ceph/values.yaml ืื™ืจ ืงืขื ืขืŸ ื’ืขืคึฟื™ื ืขืŸ ืคื™ืœืข ืคืึทืจืฉื™ื“ืขื ืข ืกืขื˜ื˜ื™ื ื’ืก. ื“ื™ ืžืขืจืกื˜ ื•ื•ื™ื›ื˜ื™ืง ื–ืึทืš ืื™ื– ืฆื• ืกืคึผืขืฆื™ืคื™ืฆื™ืจืŸ ื˜ืึธืœืขืจื™ื™ืฉืึทื ื– ืคึฟืึทืจ ืึทื’ืขื ืฅ ืื•ืŸ ื–ื•ื›ืŸ. ืžื™ืจ ื“ื™ืกืงืจื™ื™ื‘ื“ ืื™ืŸ ื“ืขื˜ืึทืœ ืคึฟืึทืจ ื•ื•ืึธืก ื“ื™ ื˜ืึทื™ื ื˜ / ื˜ืึธืœืขืจื™ื™ืฉืึทื ื– ืžืขืงืึทื ื™ื–ืึทื ืงืขื ืขืŸ ื–ื™ื™ืŸ ื’ืขื•ื•ื™ื™ื ื˜ ื“ืขื ืึทืจื˜ื™ืงืœ.

ืื™ืŸ ืงื•ืจืฅ, ืžื™ืจ ื˜ืึธืŸ ื ื™ื˜ ื•ื•ืขืœืŸ ื“ื™ ืงืœื™ืขื ื˜ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ืคึผืึธื“ืก ืฆื• ื–ื™ื™ืŸ ืœื™ื’ืŸ ืื•ื™ืฃ ื“ื™ ื–ืขืœื‘ืข ื ืึธื•ื“ื– ื•ื•ื™ ื“ื™ ื“ืึทื˜ืŸ ืกื˜ืึธืจื™ื“ื–ืฉ ื“ื™ืกืงืก. ื“ื™ ืกื™ื‘ื” ืื™ื– ืคึผืฉื•ื˜: ืึทื–ื•ื™ ื“ื™ ืึทืจื‘ืขื˜ ืคื•ืŸ ืจื•ืง ืื’ืขื ื˜ืŸ ื•ื•ืขื˜ ื ื™ืฉื˜ ื•ื•ื™ืจืงืŸ ื“ื™ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ื–ื™ืš.

ืึทื–ื•ื™, ืขืคืขื ืขืŸ ื“ื™ ื˜ืขืงืข rook-ceph/values.yaml ืžื™ื˜ ื“ื™ื™ืŸ ื‘ืึทืœื™ื‘ืกื˜ืข ืจืขื“ืึทืงื˜ืึธืจ ืื•ืŸ ืœื™ื™ื’ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ื‘ืœืึธืง ืื™ืŸ ื“ื™ ืกื•ืฃ:

discover:
  toleration: NoExecute
  tolerationKey: node-role/storage
agent:
  toleration: NoExecute
  tolerationKey: node-role/storage
  mountSecurityMode: Any

ืคึฟืึทืจ ื™ืขื“ืขืจ ื ืึธื“ืข ืจืขื–ืขืจื•ื•ื™ืจื˜ ืคึฟืึทืจ ื“ืึทื˜ืŸ ืกื˜ืึธืจื™ื“ื–ืฉ, ืœื™ื™ื’ืŸ ื“ื™ ืงืึธืจืึทืกืคึผืึทื ื“ื™ื ื’ ื˜ืึทื:

$ kubectl taint node ${NODE_NAME} node-role/storage="":NoExecute

ื“ืขืจื ืึธืš ื™ื ืกื˜ืึทืœื™ืจืŸ ื“ื™ ื”ืขืœื ื˜ืฉืึทืจื˜ ืžื™ื˜ ื“ืขื ื‘ืึทืคึฟืขืœ:

$ helm install --namespace ${ROOK_NAMESPACE} ./rook-ceph

ืื™ืฆื˜ ืื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ืฉืึทืคึฟืŸ ืึท ืงื ื•ื™ืœ ืื•ืŸ ืกืคึผืขืฆื™ืคื™ืฆื™ืจืŸ ื“ืขื ืึธืจื˜ ืึธืกื“:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  clusterName: "ceph"
  finalizers:
  - cephcluster.ceph.rook.io
  generation: 1
  name: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v13
  dashboard:
    enabled: true
  dataDirHostPath: /var/lib/rook/osd
  mon:
    allowMultiplePerNode: false
    count: 3
  network:
    hostNetwork: true
  rbdMirroring:
    workers: 1
  placement:
    all:
      tolerations:
      - key: node-role/storage
        operator: Exists
  storage:
    useAllNodes: false
    useAllDevices: false
    config:
      osdsPerDevice: "1"
      storeType: filestore
    resources:
      limits:
        memory: "1024Mi"
      requests:
        memory: "1024Mi"
    nodes:
    - name: host-1
      directories:
      - path: "/mnt/osd"
    - name: host-2
      directories:
      - path: "/mnt/osd"
    - name: host-3
      directories:
      - path: "/mnt/osd"

ืงืึธื ื˜ืจืึธืœื™ืจื•ื ื’ Ceph ืกื˜ืึทื˜ื•ืก - ื“ืขืจื•ื•ืึทืจื˜ืŸ ืฆื• ื–ืขืŸ HEALTH_OK:

$ kubectl -n ${ROOK_NAMESPACE} exec $(kubectl -n ${ROOK_NAMESPACE} get pod -l app=rook-ceph-operator -o name -o jsonpath='{.items[0].metadata.name}') -- ceph -s

ืื™ืŸ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืฆื™ื™ื˜, ืœืึธืžื™ืจ ืงืึธื ื˜ืจืึธืœื™ืจืŸ ืึทื– ื“ื™ ืคึผืึธื“ืก ืžื™ื˜ ื“ื™ ืงืœื™ืขื ื˜ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ื˜ืึธืŸ ื ื™ื˜ ืกื•ืฃ ืึทืจื•ื™ืฃ ืื•ื™ืฃ ื ืึธื•ื“ื– ืจืขื–ืขืจื•ื•ื™ืจื˜ ืคึฟืึทืจ Ceph:

$ kubectl -n ${APPLICATION_NAMESPACE} get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

ื“ืขืจืฆื•, ื ืึธืš ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ืงืขื ืขืŸ ื–ื™ื™ืŸ ืงืึทื ืคื™ื’ื™ืขืจื“ ื•ื•ื™ ื’ืขื•ื•ืืœื˜. ืžืขืจ ื“ืขื˜ืึทื™ืœืก ื•ื•ืขื’ืŸ ื–ื™ื™ ื–ืขื ืขืŸ ื’ืขื•ื•ื™ื–ืŸ ืื™ืŸ ื“ืึทืงื™ื•ืžืขื ื˜ื™ื™ืฉืึทืŸ. ืคึฟืึทืจ ืึทื“ืžื™ื ื™ืกื˜ืจืึทืฆื™ืข, ืžื™ืจ ืจืขืงืึธืžืขื ื“ื™ืจืŸ ืฆื• ื™ื ืกื˜ืึทืœื™ืจืŸ ื“ื™ ื“ืึทืฉื‘ืึธืจื“ ืื•ืŸ ืžื›ืฉื™ืจื™ื.

ืจื•ืง ืก ืื•ืŸ ื›ื•ืงืก: ืื™ื– ืจื•ืง ื’ืขื ื•ื’ ืคึฟืึทืจ ืึทืœืฅ?

ื•ื•ื™ ืื™ืจ ืงืขื ืขืŸ ื–ืขืŸ, ื“ื™ ืึทื ื˜ื•ื•ื™ืงืœื•ื ื’ ืคื•ืŸ Rook ืื™ื– ืื™ืŸ ืคื•ืœ ืกื•ื•ื™ื ื’. ืึธื‘ืขืจ ืขืก ื–ืขื ืขืŸ ื ืึธืš ืคึผืจืึธื‘ืœืขืžืก ื•ื•ืึธืก ื˜ืึธืŸ ื ื™ื˜ ืœืึธื–ืŸ ืื•ื ื“ื– ื’ืึธืจ ืคืึทืจืœืึธื–ืŸ ื“ื™ ืžืึทื ื•ืึทืœ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืคื•ืŸ Ceph:

  • ืงื™ื™ืŸ ืจื•ืงืŸ ื“ืจื™ื™ื•ื•ืขืจ ืงืขื ืขืŸ ื ื™ื˜ ืึทืจื•ื™ืกืคื™ืจืŸ ืžืขื˜ืจื™ืงืก ืื•ื™ืฃ ื“ื™ ื ื•ืฆืŸ ืคื•ืŸ ืžืึธื•ื ื˜ืขื“ ื‘ืœืึทืงืก, ื•ื•ืึธืก ื“ื™ืคึผืจื™ื™ื•ื•ื– ืื•ื ื“ื– ืžืึธื ื™ื˜ืึธืจื™ื ื’.
  • Flexvolum ืื•ืŸ CSI ื˜ืึธืŸ ื ื™ื˜ ื•ื•ื™ืกืŸ ื•ื•ื™ ื˜ื•ื™ืฉืŸ ื“ื™ ื’ืจื™ื™ืก ืคื•ืŸ ื•ื•ืึทืœื™ื•ืžื– (ื•ื•ื™ ืงืขื’ืŸ ืฆื• ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืจื‘ื“), ืึทื–ื•ื™ ืจื•ืง ืื™ื– ื“ื™ืคึผืจื™ื™ื•ื•ื“ ืคื•ืŸ ืึท ื ื•ืฆื™ืง (ืื•ืŸ ืžืืœ ืงืจื™ื˜ื™ืงืึทืœื™ ื“ืืจืฃ!) ื’ืขืฆื™ื™ึทื’.
  • ืจื•ืง ืื™ื– ื ืึธืš ื ื™ืฉื˜ ื•ื•ื™ ืคืœืขืงืกืึทื‘ืึทืœ ื•ื•ื™ ืจืขื’ื•ืœืขืจ ืกืขืฃ. ืื•ื™ื‘ ืžื™ืจ ื•ื•ื™ืœืŸ ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ ื“ื™ ื‘ืขืงืŸ ืคึฟืึทืจ CephFS ืžืขื˜ืึทื“ืึทื˜ืึท ืฆื• ื–ื™ื™ืŸ ืกื˜ืึธืจื“ ืื•ื™ืฃ SSD, ืื•ืŸ ื“ื™ ื“ืึทื˜ืŸ ื–ื™ืš ืฆื• ื–ื™ื™ืŸ ืกื˜ืึธืจื“ ืื•ื™ืฃ ื”ื“ื“, ืžื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ืคืึทืจืฉืจื™ื™ึทื‘ืŸ ื‘ืึทื–ื•ื ื“ืขืจ ื’ืจื•ืคึผืขืก ืคื•ืŸ ื“ืขื•ื•ื™ืกืขืก ืื™ืŸ CRUSH ืžืึทืคึผืก ืžืึทื ื™ื•ืึทืœื™.
  • ื˜ืจืึธืฅ ื“ืขืจ ืคืึทืงื˜ ืึทื– ืจื•ืง-ืกืขืฃ-ืึธืคึผืขืจืึทื˜ืึธืจ ืื™ื– ื’ืขื”ืืœื˜ืŸ ืกื˜ืึทื‘ื™ืœ, ืขืก ื–ืขื ืขืŸ ื“ืขืจื•ื•ื™ื™ึทืœ ืขื˜ืœืขื›ืข ืคืจืื‘ืœืขืžืขืŸ ื•ื•ืขืŸ ืึทืคึผื’ืจื™ื™ื“ื™ื ื’ Ceph ืคื•ืŸ ื•ื•ืขืจืกื™ืข 13 ืฆื• 14.

ืคื™ื™ื ื“ื™ื ื’ื–

"ืื™ืฆื˜ ืจื•ืง ืื™ื– ืคืืจืžืื›ื˜ ืคื•ืŸ ื“ื™ ื“ืจื•ื™ืกื ื“ื™ืง ื•ื•ืขืœื˜ ื“ื•ืจืš ืคึผืึธื ื–, ืึธื‘ืขืจ ืžื™ืจ ื’ืœื•ื™ื‘ืŸ ืึทื– ืื™ื™ืŸ ื˜ืึธื’ ื–ื™ ื•ื•ืขื˜ ืฉืคึผื™ืœืŸ ืึท ื‘ืึทืฉื˜ื™ืžืขื ื“ื™ืง ืจืึธืœืข ืื™ืŸ ื“ืขืจ ืฉืคึผื™ืœ!" (ืฆื™ื˜ื™ืจืŸ ื™ื ื•ื•ืขื ื˜ื™ื“ ืกืคึผืึทืกื™ืคื™ืงืœื™ ืคึฟืึทืจ ื“ืขื ืึทืจื˜ื™ืงืœ)

ื“ื™ Rook ืคึผืจื•ื™ืขืงื˜ ื”ืื˜ ื‘ืœื™ ื•ื•ืึทืŸ ืื•ื ื“ื–ืขืจ ื”ืขืจืฆืขืจ - ืžื™ืจ ื’ืœื•ื™ื‘ืŸ ืึทื– [ืžื™ื˜ ืึทืœืข ื–ื™ื™ึทืŸ ืคึผืจืึธืก ืื•ืŸ ืงืึธื ืก] ืขืก ื‘ืืฉื˜ื™ืžื˜ ืคืืจื“ื™ื ื˜ ื“ื™ื™ืŸ ื•ืคืžืขืจืงื–ืึทืžืงื™ื™ึทื˜.

ืื•ื ื“ื–ืขืจ ืฆื•ืงื•ื ืคึฟื˜ ืคึผืœืึทื ื– ืงืึธื›ืŸ ืึทืจืึธืคึผ ืฆื• ืžืึทื›ืŸ ืจื•ืง-ืกืขืฃ ืึท ืžืึธื“ื•ืœืข ืคึฟืึทืจ ืึทื“ื“ืึธืŸ-ืึธืคึผืขืจืึทื˜ืึธืจ, ื•ื•ืึธืก ื•ื•ืขื˜ ืžืึทื›ืŸ ื–ื™ื™ึทืŸ ื ื•ืฆืŸ ืื™ืŸ ืื•ื ื“ื–ืขืจ ืกืš ืงื•ื‘ืขืจื ืขื˜ืขืก ืงืœืึทืกื˜ืขืจื– ืืคื™ืœื• ืกื™ืžืคึผืœืขืจ ืื•ืŸ ืžืขืจ ื‘ืึทืงื•ื•ืขื.

ืคึผืก

ืœื™ื™ืขื ืขืŸ ืื•ื™ืš ืื•ื™ืฃ ืื•ื ื“ื–ืขืจ ื‘ืœืึธื’:

ืžืงื•ืจ: www.habr.com

ืœื™ื™ื’ืŸ ืึท ื‘ืึทืžืขืจืงื•ื ื’