🥇 Mus? Bash! Ntsib cov neeg ua haujlwm lub plhaub (xyuas thiab tshaj tawm video los ntawm KubeCon EU'2020)

Xyoo no, lub rooj sib tham tseem ceeb European Kubernetes - KubeCon + CloudNativeCon Europe 2020 - yog virtual. Txawm li cas los xij, qhov kev hloov pauv ntawm hom no tsis tiv thaiv peb los ntawm kev xa peb daim ntawv tshaj tawm ntev "Mus? Bash! Ntsib lub Plhaub-neeg ua haujlwm” mob siab rau peb qhov Open Source project shell-operator.

Kab lus no, tau txais kev tshoov siab los ntawm kev hais lus, nthuav qhia txoj hauv kev los ua kom yooj yim cov txheej txheem ntawm kev tsim cov tswv lag luam rau Kubernetes thiab qhia tias koj tuaj yeem ua koj tus kheej li cas nrog kev siv zog tsawg kawg nkaus siv lub plhaub-tus neeg ua haujlwm.

Qhia video ntawm tsab ntawv ceeb toom (~ 23 feeb hauv lus Askiv, pom tau tias muaj kev paub ntau dua li tsab xov xwm) thiab cov ntsiab lus tseem ceeb ntawm nws hauv daim ntawv. Mus!

Ntawm Flant peb niaj hnub ua kom zoo dua qub thiab ua haujlwm txhua yam. Niaj hnub no peb yuav tham txog lwm lub tswv yim zoo siab. Ntsib: huab-native plhaub scripting!

Txawm li cas los xij, cia peb pib nrog cov ntsiab lus uas txhua qhov no tshwm sim: Kubernetes.

Kubernetes API thiab controllers

API hauv Kubernetes tuaj yeem sawv cev raws li hom ntaub ntawv server nrog cov npe rau txhua hom khoom. Cov khoom (peev txheej) ntawm lub server no yog sawv cev los ntawm YAML cov ntaub ntawv. Ntxiv rau, tus neeg rau zaub mov muaj API yooj yim uas tso cai rau koj ua peb yam:

kom tau txais cov ntaub ntawv los ntawm nws hom thiab lub npe;
hloov cov peev txheej (qhov no, cov neeg rau zaub mov tsuas yog cov khoom "tso" - tag nrho cov tsim tsis raug lossis npaj rau lwm cov npe raug muab pov tseg);
taug qab rau cov peev txheej (qhov no, tus neeg siv tam sim ntawd tau txais nws tam sim no / hloov tshiab version).

Yog li, Kubernetes ua raws li hom ntaub ntawv server (rau YAML manifests) nrog peb txoj hauv kev (yog, yeej muaj lwm tus, tab sis peb yuav tso tseg rau tam sim no).

Qhov teeb meem yog tias lub server tsuas tuaj yeem khaws cov ntaub ntawv. Ua kom nws ua haujlwm koj xav tau xws li - lub tswv yim thib ob tseem ceeb tshaj plaws thiab tseem ceeb hauv ntiaj teb ntawm Kubernetes.

Muaj ob hom kev tswj hwm loj. Thawj zaug siv cov ntaub ntawv los ntawm Kubernetes, ua nws raws li kev sib tham hauv nested, thiab xa rov qab rau K8s. Qhov thib ob siv cov ntaub ntawv los ntawm Kubernetes, tab sis, tsis zoo li thawj hom, hloov lub xeev ntawm qee qhov kev pabcuam sab nraud.

Cia peb ua tib zoo saib cov txheej txheem ntawm kev tsim Kev Tshaj Tawm hauv Kubernetes:

Deployment Controller (nrog rau hauv kube-controller-manager) tau txais cov ntaub ntawv hais txog Deployment thiab tsim ib ReplicaSet.
ReplicaSet tsim ob lub replicas (ob pods) raws li cov ntaub ntawv no, tab sis cov pods no tseem tsis tau teem sijhawm.
Tus teem sijhawm teem caij pods thiab ntxiv cov ntaub ntawv node rau lawv YAMLs.
Kubelets hloov pauv rau cov peev txheej sab nraud (hais Docker).

Tom qab ntawd tag nrho cov kab ke no rov qab ua dua: lub kubelet xyuas cov ntim, suav cov pod cov xwm txheej thiab xa rov qab. Tus tswj hwm ReplicaSet tau txais cov xwm txheej thiab hloov kho lub xeev ntawm cov txheej txheem replica. Tib yam tshwm sim nrog Deployment Controller thiab tus neeg siv thaum kawg tau txais cov xwm txheej tshiab (tam sim no).

Shell-operator

Nws hloov tawm tias Kubernetes yog raws li kev sib koom ua haujlwm ntawm ntau tus tswj hwm (cov neeg ua haujlwm Kubernetes kuj yog cov tswj hwm). Cov lus nug tshwm sim, yuav ua li cas los tsim koj tus kheej tus neeg teb xov tooj nrog kev siv zog tsawg? Thiab ntawm no tus uas peb tsim los cawm shell-operator. Nws tso cai rau cov thawj tswj hwm los tsim lawv tus kheej cov lus uas siv cov kev paub.

Piv txwv yooj yim: luam cov lus zais

Cia peb saib ib qho piv txwv yooj yim.

Cia peb hais tias peb muaj Kubernetes pawg. Nws muaj lub npe chaw default nrog qee qhov zais cia mysecret. Tsis tas li ntawd, muaj lwm lub npe chaw nyob hauv pawg. Ib txhia ntawm lawv muaj ib daim ntawv lo rau lawv. Peb lub hom phiaj yog txhawm rau luam Secret rau hauv namespaces nrog ib daim ntawv lo.

Txoj haujlwm yog qhov nyuaj los ntawm qhov tseeb tias cov npe tshiab yuav tshwm sim hauv pawg, thiab qee qhov ntawm lawv yuav muaj daim ntawv lo no. Ntawm qhov tod tes, thaum daim ntawv lo yog deleted, Secret yuav tsum tau muab deleted. Ntxiv rau qhov no, Secret nws tus kheej kuj tuaj yeem hloov pauv: nyob rau hauv rooj plaub no, daim ntawv zais tshiab yuav tsum tau theej rau txhua lub npe nrog cov ntawv sau. Yog tias qhov zais cia raug muab tshem tawm hauv ib qho chaw npe, peb tus neeg teb xov tooj yuav tsum rov qab kho tam sim ntawd.

Tam sim no hais tias txoj haujlwm tau tsim, nws yog lub sijhawm los pib ua nws siv lub plhaub-tus neeg ua haujlwm. Tab sis ua ntej nws tsim nyog hais ob peb lo lus hais txog lub plhaub-tus neeg ua haujlwm nws tus kheej.

Lub plhaub-neeg ua haujlwm ua haujlwm li cas

Zoo li lwm yam haujlwm hauv Kubernetes, lub plhaub-tus neeg ua haujlwm khiav hauv nws tus kheej pod. Nyob rau hauv cov ntaub ntawv no /hooks executable cov ntaub ntawv khaws cia. Cov no tuaj yeem yog cov ntawv sau hauv Bash, Python, Ruby, thiab lwm yam. Peb hu xws li cov ntaub ntawv executable hooks (hooks).

Shell-operator sau npe rau Kubernetes cov xwm txheej thiab khiav cov hooks no los teb rau cov xwm txheej uas peb xav tau.

Lub plhaub-neeg ua haujlwm paub li cas tus nuv yuav khiav thiab thaum twg? Lub ntsiab lus yog tias txhua tus nuv muaj ob theem. Thaum pib, lub plhaub-neeg ua haujlwm khiav tag nrho cov hooks nrog kev sib cav --config Qhov no yog theem configuration. Thiab tom qab nws, hooks yog launched nyob rau hauv ib txwm txoj kev - nyob rau hauv teb rau cov xwm txheej uas lawv txuas. Hauv qhov kawg, tus nuv tau txais cov ntsiab lus sib khi (khi cov ntsiab lus) - cov ntaub ntawv hauv JSON hom, uas peb yuav tham txog ntau yam hauv qab no.

Ua tus neeg ua haujlwm hauv Bash

Tam sim no peb npaj txhij rau kev siv. Txhawm rau ua qhov no, peb yuav tsum sau ob txoj haujlwm (los ntawm txoj kev, peb pom zoo tsev qiv ntawv shell_lib, uas ua kom yooj yim sau hooks hauv Bash):

thawj yog xav tau rau theem configuration - nws qhia cov ntsiab lus khi;
qhov thib ob muaj lub ntsiab logic ntawm tus nuv.

#!/bin/bash

source /shell_lib.sh

function __config__() {
  cat << EOF
    configVersion: v1
    # BINDING CONFIGURATION
EOF
}

function __main__() {
  # THE LOGIC
}

hook::run "$@"

Cov kauj ruam tom ntej yog txiav txim siab yam khoom peb xav tau. Hauv peb qhov xwm txheej, peb yuav tsum taug qab:

qhov chaw zais cia rau kev hloov pauv;
txhua qhov chaw nyob hauv pawg, kom koj paub tias leej twg muaj daim ntawv lo txuas rau lawv;
lub hom phiaj secrets los xyuas kom meej tias lawv tag nrho nyob rau hauv sync nrog rau qhov chaw zais cia.

Sau npe mus rau qhov chaw zais cia

Binding configuration rau nws yog heev yooj yim. Peb qhia tias peb txaus siab rau Secret nrog lub npe mysecret hauv lub npe default:

function __config__() {
  cat << EOF
    configVersion: v1
    kubernetes:
    - name: src_secret
      apiVersion: v1
      kind: Secret
      nameSelector:
        matchNames:
        - mysecret
      namespace:
        nameSelector:
          matchNames: ["default"]
      group: main
EOF

Raws li qhov tshwm sim, tus nuv yuav tshwm sim thaum qhov chaw zais cia hloov (src_secret) thiab tau txais cov ntsiab lus txuas hauv qab no:

Raws li koj tau pom, nws muaj lub npe thiab tag nrho cov khoom.

Ua raws cov npe chaw

Tam sim no koj yuav tsum sau npe rau npe chaw. Txhawm rau ua qhov no, peb teev cov ntawv txuas hauv qab no:

- name: namespaces
  group: main
  apiVersion: v1
  kind: Namespace
  jqFilter: |
    {
      namespace: .metadata.name,
      hasLabel: (
       .metadata.labels // {} |  
         contains({"secret": "yes"})
      )
    }
  group: main
  keepFullObjectsInMemory: false

Raws li koj tuaj yeem pom, daim teb tshiab tau tshwm sim hauv kev teeb tsa nrog lub npe jqFilter. Raws li nws lub npe qhia, jqFilter lim tawm tag nrho cov ntaub ntawv tsis tsim nyog thiab tsim cov khoom JSON tshiab nrog cov teb uas txaus siab rau peb. Ib tug nuv nrog ib tug zoo xws li configuration yuav tau txais cov nram qab no binding ntsiab lus:

Nws muaj array filterResults rau txhua qhov chaw nyob hauv pawg. Boolean hloov pauv hasLabel qhia seb puas muaj daim ntawv lo txuas rau lub npe chaw. Xaiv keepFullObjectsInMemory: false qhia tias tsis tas yuav khaws cov khoom tiav hauv lub cim xeeb.

Taug qab lub hom phiaj secrets

Peb sau npe rau txhua qhov Secrets uas muaj cov lus piav qhia managed-secret: "yes" (cov no yog peb lub hom phiaj dst_secrets):

- name: dst_secrets
  apiVersion: v1
  kind: Secret
  labelSelector:
    matchLabels:
      managed-secret: "yes"
  jqFilter: |
    {
      "namespace":
        .metadata.namespace,
      "resourceVersion":
        .metadata.annotations.resourceVersion
    }
  group: main
  keepFullObjectsInMemory: false

Hauv qhov no jqFilter lim tawm tag nrho cov ntaub ntawv tsuas yog lub namespace thiab parameter resourceVersion. Qhov kawg parameter tau dhau mus rau cov lus piav qhia thaum tsim qhov zais cia: nws tso cai rau koj los sib piv cov qauv ntawm cov lus zais thiab khaws lawv mus txog hnub tim.

Ib tug nuv configured li no yuav, thaum tua, tau txais peb binding contexts piav saum toj no. Lawv tuaj yeem xav tias yog hom snapshot (snapshot) pawg.

Raws li tag nrho cov ntaub ntawv no, ib qho yooj yim algorithm tuaj yeem tsim. Nws suav nrog tag nrho cov npe chaw thiab:

yog tias hasLabel teeb meem true rau lub npe chaw tam sim no:
- piv lub ntiaj teb no zais cia nrog lub zos ib:
  - yog tias lawv zoo ib yam, nws tsis ua dab tsi;
  - yog lawv txawv - executes kubectl replace los yog create;
yog tias hasLabel teeb meem false rau lub npe chaw tam sim no:
- ua kom paub tseeb tias Secret tsis nyob hauv lub npe qhov chaw:
  - Yog hais tias lub zos Secret yog tam sim no, rho tawm nws siv kubectl delete;
  - yog tsis pom lub zos Secret, nws tsis ua dab tsi.

Kev siv cov algorithm hauv Bash koj tuaj yeem rub tawm hauv peb repositories nrog piv txwv.

Qhov ntawd yog qhov peb muaj peev xwm tsim tau ib qho yooj yim Kubernetes maub los siv 35 kab ntawm YAML config thiab txog tib tus lej ntawm Bash code! Lub plhaub-neeg ua haujlwm txoj haujlwm yog los txuas lawv ua ke.

Txawm li cas los xij, kev luam tawm tsis pub lwm tus paub tsis yog tsuas yog thaj chaw ntawm kev siv hluav taws xob. Nov yog ob peb yam piv txwv ntxiv los qhia tias nws muaj peev xwm ua tau dab tsi.

Piv txwv 1: Kev hloov pauv rau ConfigMap

Cia peb saib ntawm Kev Tshaj Tawm uas muaj peb lub plhaub. Pods siv ConfigMap khaws qee qhov kev teeb tsa. Thaum lub pods tau launched, ConfigMap nyob rau hauv ib lub xeev (cia peb hu nws v.1). Raws li, tag nrho cov pods siv qhov tshwj xeeb version ntawm ConfigMap.

Tam sim no cia peb xav tias ConfigMap tau hloov pauv (v.2). Txawm li cas los xij, cov pods yuav siv cov ntawv dhau los ntawm ConfigMap (v.1):

Kuv tuaj yeem tau txais lawv hloov mus rau ConfigMap tshiab (v.2) li cas? Cov lus teb yog yooj yim: siv tus qauv. Cia peb ntxiv cov lus piav qhia checksum rau ntu template Deployment configurations:

Raws li qhov tshwm sim, qhov kev txheeb xyuas no yuav raug sau npe nyob rau hauv tag nrho cov pods, thiab nws yuav zoo ib yam li kev xa tawm. Tam sim no koj tsuas yog yuav tsum hloov kho cov lus piav qhia thaum ConfigMap hloov pauv. Thiab lub plhaub-tus neeg teb xov tooj tuaj yeem ua ke hauv qhov no. Txhua yam koj yuav tau ua yog program ib tug nuv uas yuav sau npe rau ConfigMap thiab hloov kho cov checksum.

Yog tias tus neeg siv hloov pauv rau ConfigMap, lub plhaub-tus neeg ua haujlwm yuav pom lawv thiab rov suav cov tshev. Tom qab uas cov khawv koob ntawm Kubernetes yuav los ua si: tus orchestrator yuav tua cov pod, tsim ib tug tshiab, tos kom nws los ua. Ready, thiab txav mus rau lwm qhov. Yog li ntawd, Deployment yuav synchronize thiab hloov mus rau lub tshiab version ntawm ConfigMap.

Piv txwv 2: Ua hauj lwm nrog cov kev cai txhais lus

Raws li koj paub, Kubernetes tso cai rau koj los tsim hom kev cai ntawm cov khoom. Piv txwv li, koj tuaj yeem tsim qhov zoo MysqlDatabase. Cia peb hais tias hom no muaj ob qhov metadata tsis: name и namespace.

apiVersion: example.com/v1alpha1
kind: MysqlDatabase
metadata:
  name: foo
  namespace: bar

Peb muaj Kubernetes pawg nrog cov npe sib txawv uas peb tuaj yeem tsim MySQL databases. Hauv qhov no, shell-operator tuaj yeem siv los taug qab cov peev txheej MysqlDatabase, txuas lawv mus rau MySQL neeg rau zaub mov thiab synchronizing qhov xav tau thiab pom cov xeev ntawm pawg.

Piv txwv 3: Cluster Network Monitoring

Raws li koj paub, siv ping yog txoj hauv kev yooj yim tshaj plaws los saib xyuas lub network. Hauv qhov piv txwv no peb yuav qhia txog yuav ua li cas siv cov kev soj ntsuam zoo li no siv lub plhaub-tus neeg ua haujlwm.

Ua ntej tshaj plaws, koj yuav tsum tau sau npe rau cov nodes. Tus neeg teb xov tooj lub plhaub xav tau lub npe thiab IP chaw nyob ntawm txhua qhov ntawm. Nrog lawv pab, nws yuav ping cov nodes.

configVersion: v1
kubernetes:
- name: nodes
  apiVersion: v1
  kind: Node
  jqFilter: |
    {
      name: .metadata.name,
      ip: (
       .status.addresses[] |  
        select(.type == "InternalIP") |
        .address
      )
    }
  group: main
  keepFullObjectsInMemory: false
  executeHookOnEvent: []
schedule:
- name: every_minute
  group: main
  crontab: "* * * * *"

Parameter executeHookOnEvent: [] tiv thaiv tus nuv los ntawm kev khiav hauv kev teb rau txhua qhov xwm txheej (uas yog, hauv kev teb rau kev hloov pauv, ntxiv, tshem cov nodes). Txawm li cas los xij, nws yuav khiav (thiab hloov kho cov npe ntawm cov nodes) Teem sijhawm - txhua feeb, raws li tau hais los ntawm thaj chaw schedule.

Tam sim no cov lus nug tshwm sim, yuav ua li cas raws nraim peb paub txog cov teeb meem xws li pob ntawv poob? Cia peb saib ntawm lub code:

function __main__() {
  for i in $(seq 0 "$(context::jq -r '(.snapshots.nodes | length) - 1')"); do
    node_name="$(context::jq -r '.snapshots.nodes['"$i"'].filterResult.name')"
    node_ip="$(context::jq -r '.snapshots.nodes['"$i"'].filterResult.ip')"
    packets_lost=0
    if ! ping -c 1 "$node_ip" -t 1 ; then
      packets_lost=1
    fi
    cat >> "$METRICS_PATH" <<END
      {
        "name": "node_packets_lost",
        "add": $packets_lost,
        "labels": {
          "node": "$node_name"
        }
      }
END
  done
}

Peb rov hais dua los ntawm cov npe ntawm cov nodes, tau txais lawv cov npe thiab IP chaw nyob, ping lawv thiab xa cov txiaj ntsig mus rau Prometheus. Plhaub-tus neeg teb xov tooj tuaj yeem xa cov ntsuas mus rau Prometheus, txuag lawv mus rau ib cov ntaub ntawv nyob rau hauv raws li txoj kev teev nyob rau hauv ib puag ncig kuj sib txawv $METRICS_PATH.

Zoo li no koj tuaj yeem ua tus neeg teb xov tooj rau kev saib xyuas network yooj yim hauv pawg.

Queing mechanism

Kab lus no yuav ua tsis tiav yam tsis tau piav qhia txog lwm yam tseem ceeb uas ua rau hauv lub plhaub-tus neeg ua haujlwm. Xav txog tias nws ua qee yam kev sib ntsib hauv kev teb rau ib qho xwm txheej hauv pawg.

Yuav ua li cas yog tias, tib lub sijhawm, ib yam dab tsi tshwm sim hauv pawg? Lwm tus xwm txheej?
Lub plhaub-neeg ua haujlwm puas yuav ua lwm yam piv txwv ntawm tus nuv?
Yuav ua li cas yog hais tias, tsib txheej xwm tshwm sim nyob rau hauv pawg ib zaug?
Lub plhaub-neeg ua haujlwm puas yuav ua rau lawv sib luag?
Yuav ua li cas txog cov khoom siv xws li nco thiab CPU?

Hmoov zoo, plhaub-neeg teb xov tooj muaj ib tug built-in queuing mechanism. Tag nrho cov txheej xwm raug queued thiab ua raws li cov txheej txheem.

Cia peb piav qhia qhov no nrog cov piv txwv. Cia peb muaj ob lub hooks. Thawj qhov xwm txheej mus rau thawj tus nuv. Thaum nws ua tiav, cov kab mus rau pem hauv ntej. Peb qhov xwm txheej tom ntej no raug xa mus rau qhov sib txuas thib ob - lawv raug tshem tawm ntawm cov kab thiab nkag mus rau hauv nws hauv "khoom". Qhov ntawd yog nuv tau txais ib qho array ntawm cov xwm txheej - los yog, ntau precisely, ib tug array ntawm khi cov ntsiab lus.

Kuj cov no Cov xwm txheej tuaj yeem ua ke ua ib qho loj. Tus parameter yog lub luag haujlwm rau qhov no group nyob rau hauv kev khi configuration.

Koj tuaj yeem tsim ib tus lej ntawm cov kab / hooks thiab lawv ntau qhov sib xyaw ua ke. Piv txwv li, ib kab tuaj yeem ua haujlwm nrog ob lub hooks, lossis rov ua dua.

Txhua yam koj yuav tau ua yog configure lub teb raws li queue nyob rau hauv kev khi configuration. Yog hais tias lub npe queue tsis tau teev tseg, tus nuv yuav khiav ntawm lub default queue (default). Cov txheej txheem queuing no tso cai rau koj los daws txhua yam teeb meem kev tswj hwm kev lag luam thaum ua haujlwm nrog hooks.

xaus

Peb tau piav qhia tias lub plhaub-tus neeg ua haujlwm yog dab tsi, tau qhia tias nws tuaj yeem siv tau sai thiab siv zog tsim Kubernetes tus neeg ua haujlwm, thiab muab ntau yam piv txwv ntawm nws siv.

Cov ncauj lus kom ntxaws txog lub plhaub-tus neeg ua haujlwm, nrog rau kev qhia ceev ceev ntawm kev siv nws, muaj nyob rau hauv qhov sib thooj repositories ntawm GitHub. Tsis txhob yig tiv tauj peb nrog cov lus nug: koj tuaj yeem tham nrog lawv hauv qhov tshwj xeeb Telegram pawg (hauv Russian) los yog hauv lub rooj sab laj no (hauv lus Askiv).

Thiab yog tias koj nyiam nws, peb ib txwm zoo siab pom cov teeb meem tshiab / PR / hnub qub ntawm GitHub, qhov twg, los ntawm txoj kev, koj tuaj yeem nrhiav lwm tus nthuav tej yaam num. Ntawm lawv nws yog tsim nyog highlighting addon-operator, uas yog tus tij laug loj ntawm plhaub-neeg ua haujlwm. Qhov kev siv hluav taws xob no siv Helm kab kos rau nruab add-ons, tuaj yeem xa cov kev hloov kho tshiab thiab saib xyuas ntau yam kab ntawv / tus nqi, tswj cov txheej txheem kev teeb tsa ntawm cov kab kos, thiab tuaj yeem hloov kho lawv hauv cov lus teb rau cov xwm txheej hauv pawg.

Yeeb yaj duab thiab slides

Video los ntawm kev ua yeeb yam (~ 23 feeb):

Kev nthuav qhia ntawm daim ntawv qhia:

PS

Nyeem kuj ntawm peb blog:

Tau qhov twg los: www.hab.com

Mus? Bash! Ntsib cov neeg ua haujlwm lub plhaub (xyuas thiab tshaj tawm video los ntawm KubeCon EU'2020)