6 lom zem kab kab hauv kev ua haujlwm ntawm Kubernetes [thiab lawv cov kev daws teeb meem]

Ntau xyoo dhau los ntawm kev siv Kubernetes hauv kev tsim khoom, peb tau sau ntau cov dab neeg nthuav qhia txog yuav ua li cas kab mob hauv ntau yam kab ke ua rau tsis zoo thiab / lossis tsis tuaj yeem ua rau muaj kev cuam tshuam rau kev ua haujlwm ntawm cov ntim khoom thiab cov pods. Nyob rau hauv tsab xov xwm no, peb tau muab tso ua ke ib co ntawm cov feem ntau pom los yog nthuav sawv daws yuav. Txawm hais tias koj yeej tsis muaj hmoo txaus los ntsib cov xwm txheej zoo li no, nyeem txog cov dab neeg luv luv no - tshwj xeeb tshaj yog "thawj-tes" - yeej ib txwm lom zem, puas yog? ..

Dab Neeg 1. Supercronic thiab ib tug daig Docker

Ntawm ib qho ntawm pawg, peb ib txwm tau txais "dai" Docker, uas cuam tshuam nrog kev ua haujlwm ntawm pawg. Tib lub sijhawm, cov hauv qab no tau pom hauv Docker cav

level=error msg="containerd: start init process" error="exit status 2: "runtime/cgo: pthread_create failed: No space left on device
SIGABRT: abort
PC=0x7f31b811a428 m=0

goroutine 0 [idle]:

goroutine 1 [running]:
runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420026768 sp=0xc420026760
runtime.main() /usr/local/go/src/runtime/proc.go:127 +0x6c fp=0xc4200267c0 sp=0xc420026768
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200267c8 sp=0xc4200267c0

goroutine 17 [syscall, locked to thread]:
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

…

Hauv qhov yuam kev no, peb txaus siab tshaj plaws hauv cov lus: pthread_create failed: No space left on device. Ib txoj kev tshawb nrhiav cursory cov ntaub ntawv piav qhia tias Docker tsis tuaj yeem cuam tshuam cov txheej txheem, uas yog vim li cas nws ib ntus "hangs".

Hauv kev saib xyuas, dab tsi tshwm sim sib raug rau daim duab hauv qab no:

Ib qho xwm txheej zoo sib xws tau pom ntawm lwm cov nodes:

Ntawm tib lub nodes peb pom:

root@kube-node-1 ~ # ps auxfww | grep curl -c
19782
root@kube-node-1 ~ # ps auxfww | grep curl | head
root     16688  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     17398  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     16852  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      9473  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      4664  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     30571  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     24113  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     16475  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      7176  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      1090  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>

Nws muab tawm tias qhov kev coj cwj pwm no yog qhov tshwm sim ntawm lub plhaub taum ua haujlwm nrog supercronic (ib qho khoom siv mus rau peb siv los khiav haujlwm cron hauv pods):

 _ docker-containerd-shim 833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 /var/run/docker/libcontainerd/833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 docker-runc
|   _ /usr/local/bin/supercronic -json /crontabs/cron
|       _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true
|       |   _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true -no-pidfile
|       _ [newrelic-daemon] <defunct>
|       _ [curl] <defunct>
|       _ [curl] <defunct>
|       _ [curl] <defunct>
…

Qhov teeb meem yog qhov no: thaum pib ua haujlwm hauv supercronic, cov txheej txheem spawned los ntawm nws tsis tuaj yeem ua tiav kom raug, tig rau zombie.

Примечание: Kom meej meej, cov txheej txheem yog spawned los ntawm cron cov hauj lwm, tab sis supercronic tsis yog ib qho init system thiab tsis tuaj yeem "tau txais" cov txheej txheem uas nws cov me nyuam spawned. Thaum SIGHUP lossis SIGTERM teeb liab tshwm sim, lawv tsis dhau mus rau cov txheej txheem menyuam yaus, ua rau cov txheej txheem menyuam yaus tsis tuaj yeem txiav tawm, tshuav nyob rau hauv cov xwm txheej zombie. Koj tuaj yeem nyeem ntxiv txog txhua qhov no, piv txwv li, hauv xws li ib tsab xov xwm.

Muaj ob peb txoj hauv kev los daws cov teeb meem:

1. Raws li kev ua haujlwm ib ntus - nce tus naj npawb ntawm PIDs hauv qhov system ntawm tib lub sijhawm:

          /proc/sys/kernel/pid_max (since Linux 2.5.34)
                 This file specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID).  PIDs greater than this  value  are  not  allo‐
                 cated;  thus, the value in this file also acts as a system-wide limit on the total number of processes and threads.  The default value for this file, 32768, results in the
                 same range of PIDs as on earlier kernels

2. Los yog ua kom lub community launch txog cov dej num hauv supercronic tsis ncaj qha, tab sis siv tib yam tsi, uas muaj peev xwm ntawm gracefully txiav cov txheej txheem thiab tsis spawning zombies.

Dab neeg 2. "Zombies" thaum rho tawm ib pawg

Kubelet pib siv ntau CPU:

Tsis muaj leej twg nyiam qhov no, yog li peb armed peb tus kheej zoo tag nrho thiab pib daws qhov teeb meem. Cov txiaj ntsig ntawm kev tshawb nrhiav tau raws li hauv qab no:

• Kubelet siv ntau tshaj li ib feem peb ntawm CPU lub sij hawm rub cov ntaub ntawv nco los ntawm txhua pawg:

  

• Nyob rau hauv daim ntawv teev npe tsim tawm kernel, koj tuaj yeem pom kev sib tham txog qhov teeb meem. Hauv luv, lub ntsiab lus yog qhov ntawd ntau yam tmpfs cov ntaub ntawv thiab lwm yam zoo sib xws tsis raug tshem tawm tag nrho ntawm qhov system thaum tshem ib cgroup, lub thiaj li hu memcg zombie. Tsis ntev los sis tom qab lawv tseem yuav raug rho tawm ntawm nplooj ntawv cache, txawm li cas los xij, muaj ntau lub cim xeeb ntawm cov neeg rau zaub mov thiab cov ntsiav tsis pom lub ntsiab lus hauv nkim sijhawm tshem lawv. Yog li ntawd, lawv tseem ua ke. Vim li cas qhov no tseem tshwm sim? Qhov no yog ib tug neeg rau zaub mov nrog cron txoj hauj lwm uas tas li tsim cov hauj lwm tshiab, thiab nrog lawv cov pods tshiab. Yog li, cov cgroups tshiab yog tsim rau cov ntim hauv lawv, uas yuav raug tshem tawm sai sai.
• Vim li cas cAdvisor hauv kubelet siv sijhawm ntau? Qhov no yog ib qho yooj yim pom los ntawm kev ua kom yooj yim tshaj plaws time cat /sys/fs/cgroup/memory/memory.stat. Yog tias ntawm lub tshuab noj qab haus huv kev ua haujlwm yuav siv sijhawm 0,01 vib nas this, tom qab ntawd ntawm qhov teeb meem cron02 nws yuav siv sijhawm 1,2 vib nas this. Qhov tshaj plaws yog tias cAdvisor, uas nyeem cov ntaub ntawv los ntawm sysfs qeeb heev, sim coj mus rau hauv tus account siv lub cim xeeb hauv zombie cgroups thiab.
• Txhawm rau tshem tawm cov zombies, peb sim tshem cov caches raws li kev pom zoo los ntawm LKML: sync; echo 3 > /proc/sys/vm/drop_caches, - tab sis cov tub ntxhais tau hloov mus ua qhov nyuaj dua thiab dai lub tsheb.

Yuav ua li cas? Qhov teeb meem raug khocog lus, thiab saib cov lus piav qhia hauv tso lus) kev hloov kho tshiab ntawm lub kernel Linux mus rau version 4.16.

Dab Neeg 3. Systemd thiab nws mount

Ntxiv dua thiab, kubelet siv ntau cov peev txheej ntawm qee cov nodes, tab sis lub sijhawm no nws yog qhov nco ntau dua:

Nws tau muab tawm tias muaj teeb meem nrog systemd siv rau hauv Ubuntu 16.04, thiab nws tshwm sim thaum tswj hwm cov mounts uas tau tsim rau kev sib txuas subPath los ntawm ConfigMap'ov los yog secret'ov. Tom qab kaw lub pod cov kev pabcuam systemd thiab nws cov kev pabcuam mount tseem nyob hauv qhov system. Nyob rau tib lub sij hawm, lawv sau ntau heev. Tseem muaj teeb meem ntawm lub ncauj lus no:

1. npe 5916;
2. kubernetes #57345.

... nyob rau hauv qhov kawg ntawm uas lawv xa mus rau PR hauv systemd: #7811 (qhov teeb meem hauv systemd - #7798).

Qhov teeb meem tsis muaj lawm Ubuntu 18.04, tab sis yog tias koj xav txuas ntxiv siv Ubuntu 16.04, tej zaum koj yuav pom peb txoj kev daws teeb meem ntawm cov ncauj lus no muaj txiaj ntsig.

Yog li, peb tau ua DaemonSet hauv qab no:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: systemd-slices-cleaner
  name: systemd-slices-cleaner
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: systemd-slices-cleaner
  template:
    metadata:
      labels:
        app: systemd-slices-cleaner
    spec:
      containers:
      - command:
        - /usr/local/bin/supercronic
        - -json
        - /app/crontab
        Image: private-registry.org/systemd-slices-cleaner/systemd-slices-cleaner:v0.1.0
        imagePullPolicy: Always
        name: systemd-slices-cleaner
        resources: {}
        securityContext:
          privileged: true
        volumeMounts:
        - name: systemd
          mountPath: /run/systemd/private
        - name: docker
          mountPath: /run/docker.sock
        - name: systemd-etc
          mountPath: /etc/systemd
        - name: systemd-run
          mountPath: /run/systemd/system/
        - name: lsb-release
          mountPath: /etc/lsb-release-host
      imagePullSecrets:
      - name: antiopa-registry
      priorityClassName: cluster-low
      tolerations:
      - operator: Exists
      volumes:
      - name: systemd
        hostPath:
          path: /run/systemd/private
      - name: docker
        hostPath:
          path: /run/docker.sock
      - name: systemd-etc
        hostPath:
          path: /etc/systemd
      - name: systemd-run
        hostPath:
          path: /run/systemd/system/
      - name: lsb-release
        hostPath:
          path: /etc/lsb-release

... thiab nws siv tsab ntawv no:

#!/bin/bash

# we will work only on xenial
hostrelease="/etc/lsb-release-host"
test -f ${hostrelease} && grep xenial ${hostrelease} > /dev/null || exit 0

# sleeping max 30 minutes to dispense load on kube-nodes
sleep $((RANDOM % 1800))

stoppedCount=0
# counting actual subpath units in systemd
countBefore=$(systemctl list-units | grep subpath | grep "run-" | wc -l)
# let's go check each unit
for unit in $(systemctl list-units | grep subpath | grep "run-" | awk '{print $1}'); do
  # finding description file for unit (to find out docker container, who born this unit)
  DropFile=$(systemctl status ${unit} | grep Drop | awk -F': ' '{print $2}')
  # reading uuid for docker container from description file
  DockerContainerId=$(cat ${DropFile}/50-Description.conf | awk '{print $5}' | cut -d/ -f6)
  # checking container status (running or not)
  checkFlag=$(docker ps | grep -c ${DockerContainerId})
  # if container not running, we will stop unit
  if [[ ${checkFlag} -eq 0 ]]; then
    echo "Stopping unit ${unit}"
    # stoping unit in action
    systemctl stop $unit
    # just counter for logs
    ((stoppedCount++))
    # logging current progress
    echo "Stopped ${stoppedCount} systemd units out of ${countBefore}"
  fi
done

... thiab nws khiav txhua 5 feeb nrog kev pab los ntawm yav tas los hais supercronic. Nws Dockerfile zoo li no:

FROM ubuntu:16.04
COPY rootfs /
WORKDIR /app
RUN apt-get update && 
    apt-get upgrade -y && 
    apt-get install -y gnupg curl apt-transport-https software-properties-common wget
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" && 
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && 
    apt-get update && 
    apt-get install -y docker-ce=17.03.0*
RUN wget https://github.com/aptible/supercronic/releases/download/v0.1.6/supercronic-linux-amd64 -O 
    /usr/local/bin/supercronic && chmod +x /usr/local/bin/supercronic
ENTRYPOINT ["/bin/bash", "-c", "/usr/local/bin/supercronic -json /app/crontab"]

Zaj Dab Neeg 4: Concurrency in Pod Planning

Nws tau pom tias: yog tias peb muaj lub plhaub taum tso rau ntawm lub pob thiab nws cov duab tau muab tso rau lub sijhawm ntev heev, ces lwm lub plhaub taum uas "ntaus" ntawm tib lub node yooj yim. tsis pib rub daim duab tshiab. Hloov chaw, nws tos rau yav dhau los pod cov duab kom 'rub'. Yog li ntawd, lub plhaub taum uas twb tau npaj lawm thiab nws cov duab tuaj yeem rub tawm hauv ib feeb xwb yuav xaus rau hauv cov xwm txheej ntev. containerCreating.

Cov xwm txheej yuav zoo li no:

Normal  Pulling    8m    kubelet, ip-10-241-44-128.ap-northeast-1.compute.internal  pulling image "registry.example.com/infra/openvpn/openvpn:master"

Nws hloov tawm ntawd ib daim duab los ntawm kev sau npe qeeb tuaj yeem thaiv kev xa tawm rau node.

Hmoov tsis zoo, tsis muaj ntau txoj hauv kev tawm ntawm qhov xwm txheej:

1. Sim siv koj Docker Registry ncaj qha rau hauv pawg lossis ncaj qha nrog pawg (xws li GitLab Registry, Nexus, thiab lwm yam);
2. Siv cov khoom siv hluav taws xob xws li kraken.

Dab neeg 5. Txawb pob vim tsis nco

Thaum lub sijhawm ua haujlwm ntawm ntau daim ntawv thov, peb kuj tau txais qhov xwm txheej thaum lub node kiag li tsis muaj: SSH tsis teb, tag nrho cov saib xyuas daems poob, thiab tom qab ntawd tsis muaj dab tsi (lossis yuav luag tsis muaj dab tsi) txawv txav hauv cov cav.

Kuv yuav qhia koj hauv cov duab uas siv qhov piv txwv ntawm ib qho ntawm qhov twg MongoDB ua haujlwm.

Qhov no yog qhov zoo li saum toj no rau xwm txheej:

Thiab zoo li no - tom qab xwm txheej:

Hauv kev saib xyuas, kuj tseem muaj qhov dhia ntse, uas lub node tsis muaj:

Yog li los ntawm cov screenshots koj tuaj yeem pom tias:

1. Lub RAM ntawm lub tshuab yog ze rau khiav tawm;
2. Muaj qhov ntse dhia hauv kev noj ntawm RAM, tom qab uas nkag mus rau tag nrho lub tshuab ua haujlwm sai sai;
3. Ib txoj haujlwm loj tuaj txog ntawm Mongo, uas ua rau cov txheej txheem DBMS siv ntau lub cim xeeb thiab nquag nyeem los ntawm disk.

Yuav ua li cas yog tias nyob rau hauv Linux lub cim xeeb dawb tas lawm (qhov siab ntawm lub cim xeeb tshwm sim) thiab tsis muaj kev sib pauv, ces rau Kev tuaj txog ntawm OOM killer, tej zaum yuav muaj qhov sib npaug ntawm cov nplooj ntawv pov rau hauv nplooj ntawv cache thiab sau lawv rov qab rau disk. Qhov no yog tswj hwm los ntawm kswapd, uas ua siab loj tso tawm ntau nplooj ntawv nco li ua tau rau kev faib tom qab.

Hmoov tsis zoo, nrog rau qhov hnyav I / O load ua ke nrog me me ntawm lub cim xeeb dawb, kswapd ua lub bottleneck ntawm tag nrho cov systemvim lawv muab khi rau nws tag nrho allocations (nplooj faults) ntawm cov nplooj ntawv nco hauv qhov system. Qhov no tuaj yeem mus ntev heev yog tias cov txheej txheem tsis xav siv lub cim xeeb ntxiv lawm, tab sis fixate ntawm ntug heev ntawm OOM killer abyss.

Lo lus nug yog ntuj: yog vim li cas tus OOM killer tuaj lig? Hauv nws qhov kev rov hais dua tam sim no, OOM killer yog qhov ruam heev: nws yuav tua cov txheej txheem tsuas yog thaum qhov kev sim faib nplooj ntawv nco tsis tau, piv txwv li. yog tias nplooj ntawv yuam kev hla nrog qhov yuam kev. Qhov no tsis tshwm sim ntev txaus vim tias kswapd valiantly tso cov nplooj ntawv ntawm lub cim xeeb los ntawm kev yaug cov nplooj ntawv cache (tag nrho disk I / O ntawm lub kaw lus, qhov tseem ceeb) rov qab mus rau disk. Hauv kev nthuav dav ntxiv, nrog cov lus piav qhia ntawm cov kauj ruam tsim nyog los tshem tawm cov teeb meem zoo li no hauv cov ntsiav, koj tuaj yeem nyeem no.

Tus cwj pwm no yuav tsum txhim kho nrog lub hauv paus Linux 4.6 +.

Zaj Dab Neeg 6. Pods tau daig hauv lub xeev Pending

Nyob rau hauv ib co pawg uas muaj coob heev ntawm cov pods khiav, peb pib pom tias feem ntau ntawm lawv dai nyob rau hauv lub xeev ntev heev. Pending, txawm hais tias Docker ntim lawv tus kheej twb tau khiav ntawm cov nodes thiab koj tuaj yeem ua haujlwm nrog lawv.

Ntxiv mus, hauv describe tsis muaj dab tsi tsis ncaj ncees lawm:

  Type    Reason                  Age                From                     Message
  ----    ------                  ----               ----                     -------
  Normal  Scheduled               1m                 default-scheduler        Successfully assigned sphinx-0 to ss-dev-kub07
  Normal  SuccessfulAttachVolume  1m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
  Normal  SuccessfulMountVolume   1m                 kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "sphinx-config"
  Normal  SuccessfulMountVolume   1m                 kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "default-token-fzcsf"
  Normal  SuccessfulMountVolume   49s (x2 over 51s)  kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
  Normal  Pulled                  43s                kubelet, ss-dev-kub07    Container image "registry.example.com/infra/sphinx-exporter/sphinx-indexer:v1" already present on machine
  Normal  Created                 43s                kubelet, ss-dev-kub07    Created container
  Normal  Started                 43s                kubelet, ss-dev-kub07    Started container
  Normal  Pulled                  43s                kubelet, ss-dev-kub07    Container image "registry.example.com/infra/sphinx/sphinx:v1" already present on machine
  Normal  Created                 42s                kubelet, ss-dev-kub07    Created container
  Normal  Started                 42s                kubelet, ss-dev-kub07    Started container

Tom qab khawb ib ncig, peb tau ua qhov kev xav tias kubelet tsuas yog tsis muaj sijhawm los xa tag nrho cov ntaub ntawv hais txog lub xeev ntawm cov pods, kev ua neej nyob / kev npaj ua qauv rau API server.

Thiab tau kawm txog kev pab, peb pom cov hauv qab no tsis:

--kube-api-qps - QPS to use while talking with kubernetes apiserver (default 5)
--kube-api-burst  - Burst to use while talking with kubernetes apiserver (default 10) 
--event-qps - If > 0, limit event creations per second to this value. If 0, unlimited. (default 5)
--event-burst - Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding event-qps. Only used if --event-qps > 0 (default 10) 
--registry-qps - If > 0, limit registry pull QPS to this value.
--registry-burst - Maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0 (default 10)

Raws li pom, default values ​​yog heev me me, thiab hauv 90% lawv them tag nrho cov kev xav tau ... Txawm li cas los xij, hauv peb cov ntaub ntawv, qhov no tsis txaus. Yog li ntawd, peb teev cov nqi hauv qab no:

--event-qps=30 --event-burst=40 --kube-api-burst=40 --kube-api-qps=30 --registry-qps=30 --registry-burst=40

... thiab rov pib lub kubelets, tom qab ntawd peb pom cov duab hauv qab no ntawm cov duab rau kev nkag mus rau API server:

... thiab yog, txhua yam pib ya!

PS

Rau kev pab sau cov kab thiab npaj cov kab lus, kuv ua tsaug rau ntau tus engineers ntawm peb lub tuam txhab, thiab tshwj xeeb tshaj yog rau peb cov npoj yaig los ntawm peb pab pawg R & D Andrey Klimentyev (zuzzas).

PPS

Nyeem kuj ntawm peb blog:

• «kubectl-debug plugin rau kev debugging hauv Kubernetes pods".
• Kubernetes lub tswv yim & tricks voj voog:
  • «Hloov cov peev txheej khiav hauv pawg raws li kev tswj hwm ntawm Helm 2»;
  • «Hais txog kev faib cov nodes thiab cov loads ntawm lub vev xaib thov»;
  • «Nkag mus rau dev sites»;
  • «Txhim kho bootstrap rau cov ntaub ntawv loj".

Tau qhov twg los: www.hab.com