🥇6 inovaraidza system bugs kana uchishandisa Kubernetes [nemhinduro yavo]

Kwemakore ekushandisa Kubernetes mukugadzira, takaunganidza nyaya dzakawanda dzinonakidza dzekuti zvipembenene muzvikamu zvakasiyana zvehurongwa zvakatungamira sei kusingafadzi uye / kana kusanzwisisika mhedzisiro inokanganisa kushanda kwemidziyo nemapods. Muchinyorwa chino takaita sarudzo yezvimwe zvakajairika kana zvinonakidza. Kunyangwe iwe usina kumbobvira waita rombo rakanaka rekusangana nemamiriro akadai, kuverenga nezve mapfupi nyaya dzemutikitivha - kunyanya "kutanga-ruoko" - zvinogara zvichinakidza, handizvo here?

Nyaya 1. Supercronic uye Docker yakarembera

Pane rimwe remasumbu, isu nguva nenguva taigamuchira Docker yakaomeswa nechando, iyo yakakanganisa kushanda kwakajairika kwesumbu. Panguva imwecheteyo, zvinotevera zvakaonekwa muDocker matanda:

level=error msg="containerd: start init process" error="exit status 2: "runtime/cgo: pthread_create failed: No space left on device
SIGABRT: abort
PC=0x7f31b811a428 m=0

goroutine 0 [idle]:

goroutine 1 [running]:
runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420026768 sp=0xc420026760
runtime.main() /usr/local/go/src/runtime/proc.go:127 +0x6c fp=0xc4200267c0 sp=0xc420026768
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200267c8 sp=0xc4200267c0

goroutine 17 [syscall, locked to thread]:
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

…

Chinonyanya kutifadza pamusoro pekukanganisa uku imharidzo: pthread_create failed: No space left on device. Kudzidza Nokukurumidza zvinyorwa akatsanangura kuti Docker aisakwanisa kuforoma maitiro, ndosaka aipota achioma.

Mukutarisa, mufananidzo unotevera unoenderana nezviri kuitika:

Mamiriro akafanana anoonekwa pane dzimwe node:

Pamanodhi akafanana tinoona:

root@kube-node-1 ~ # ps auxfww | grep curl -c
19782
root@kube-node-1 ~ # ps auxfww | grep curl | head
root     16688  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     17398  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     16852  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      9473  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      4664  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     30571  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     24113  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     16475  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      7176  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      1090  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>

Zvakazoitika kuti maitiro aya ndiwo mhedzisiro yepodhi inoshanda nayo supercronic (a Go utility yatinoshandisa kumhanyisa cron mabasa mumapods):

 _ docker-containerd-shim 833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 /var/run/docker/libcontainerd/833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 docker-runc
|   _ /usr/local/bin/supercronic -json /crontabs/cron
|       _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true
|       |   _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true -no-pidfile
|       _ [newrelic-daemon] <defunct>
|       _ [curl] <defunct>
|       _ [curl] <defunct>
|       _ [curl] <defunct>
…

Dambudziko nderekuti: kana basa richiitwa mune supercronic, maitiro anokonzerwa nazvo haigoni kugumisa zvakarurama, kupinduka kuita zombie.

taura pfungwa: Kuti zvive zvakanyatsojeka, maitiro anokonzerwa necron mabasa, asi supercronic haisi init system uye haigoni "kutora" maitiro ayo vana vayo vakabereka. Kana SIGHUP kana SIGTERM masaini anosimudzwa, haapfuudzirwe kumaitiro emwana, zvichiita kuti maitiro emwana asagumise uye arambe ari mu zombie chimiro. Iwe unogona kuverenga zvakawanda nezve izvi zvese, semuenzaniso, mukati nyaya yakadaro.

Pane nzira mbiri dzekugadzirisa dambudziko:

Senguva pfupi yekushanda - wedzera huwandu hwePIDs muhurongwa pane imwe nguva panguva:

       /proc/sys/kernel/pid_max (since Linux 2.5.34)
              This file specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID).  PIDs greater than this  value  are  not  allo‐
              cated;  thus, the value in this file also acts as a system-wide limit on the total number of processes and threads.  The default value for this file, 32768, results in the
              same range of PIDs as on earlier kernels

Kana kuvhura mabasa mune supercronic kwete zvakananga, asi uchishandisa zvakafanana ini, iyo inokwanisa kumisa maitiro nemazvo uye kwete kubereka zombies.

Nyaya 2. "Zombies" pakubvisa boka

Kubelet akatanga kudya yakawanda yeCPU:

Hapana achada izvi, saka takazvigadzirira zvakakwana ndokutanga kubata nedambudziko. Mhedzisiro yekuferefeta yaive seizvi:

Kubelet anoshandisa inopfuura chikamu chimwe muzvitatu cheCPU yake nguva achidhonza ndangariro data kubva kumapoka ese:
Mutsamba yetsamba yevagadziri vekernel iwe unogona kuwana hurukuro yedambudziko. Muchidimbu, pfungwa inodzika kune iyi: akasiyana tmpfs mafaera uye zvimwe zvinhu zvakafanana hazvina kubviswa zvachose kubva kuhurongwa pakubvisa cgroup, inonzi memcg Zombie. Nokukurumidza kana kuti gare gare ivo vachabviswa kubva pacheji peji, asi pane zvakawanda zvekuyeuka pavhavha uye kernel haioni pfungwa yekupedza nguva pakudzibvisa. Ndiko kusaka vachiramba vachiunganidza. Sei izvi zviri kuitika? Iyi iseva ine cron mabasa inogara ichigadzira mabasa matsva, uye nawo mapodhi matsva. Saka, mapoka matsva anogadzirwa emidziyo mairi, ayo anozobviswa nekukurumidza.
Sei cAdvisor mukubelet ichitambisa nguva yakawanda kudaro? Izvi zviri nyore kuona nekuita kuri nyore time cat /sys/fs/cgroup/memory/memory.stat. Kana pamushini une hutano kuvhiyiwa kunotora masekonzi 0,01, ipapo pane inonetsa cron02 inotora 1,2 masekonzi. Chinhu ndechekuti cAdvisor, iyo inoverenga data kubva kusysfs zvishoma nezvishoma, inoedza kufunga nezve ndangariro inoshandiswa mu zombie cgroups.
Kuti tibvise nechisimba zombies, takaedza kubvisa cache sezvakakurudzirwa muLKML: sync; echo 3 > /proc/sys/vm/drop_caches, - asi kernel yakava yakaoma uye yakarova motokari.

Kuita sei? Dambudziko riri kugadziriswa (commit, uye netsanangudzo ona sunungura meseji) kugadzirisa iyo Linux kernel kune shanduro 4.16.

Nhoroondo 3. Systemd uye gomo rayo

Zvekare, iyo kubelet iri kupedza zviwanikwa zvakawandisa pane dzimwe node, asi panguva ino iri kupedza ndangariro yakawandisa:

Zvakazoitika kuti pane dambudziko mu systemd inoshandiswa muUbuntu 16.04, uye inoitika kana uchigadzirisa mamoiri anogadzirirwa kubatana. subPath kubva kuConfigMap kana zvakavanzika. Mushure mokunge pod yapedza basa rayo iyo systemd sevhisi uye basa rayo gomo rinosara muhurongwa. Nokufamba kwenguva, nhamba huru yavo inounganidza. Pane kunyange nyaya panyaya iyi:

#5916;
kubernetes #57345.

...yekupedzisira iyo inoreva iyo PR mu systemd: #7811 (nyaya mu systemd - #7798).

Dambudziko harisipo muUbuntu 18.04, asi kana iwe uchida kuenderera mberi uchishandisa Ubuntu 16.04, unogona kuwana yedu workaround pane iyi nyaya inobatsira.

Saka takagadzira inotevera DaemonSet:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: systemd-slices-cleaner
  name: systemd-slices-cleaner
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: systemd-slices-cleaner
  template:
    metadata:
      labels:
        app: systemd-slices-cleaner
    spec:
      containers:
      - command:
        - /usr/local/bin/supercronic
        - -json
        - /app/crontab
        Image: private-registry.org/systemd-slices-cleaner/systemd-slices-cleaner:v0.1.0
        imagePullPolicy: Always
        name: systemd-slices-cleaner
        resources: {}
        securityContext:
          privileged: true
        volumeMounts:
        - name: systemd
          mountPath: /run/systemd/private
        - name: docker
          mountPath: /run/docker.sock
        - name: systemd-etc
          mountPath: /etc/systemd
        - name: systemd-run
          mountPath: /run/systemd/system/
        - name: lsb-release
          mountPath: /etc/lsb-release-host
      imagePullSecrets:
      - name: antiopa-registry
      priorityClassName: cluster-low
      tolerations:
      - operator: Exists
      volumes:
      - name: systemd
        hostPath:
          path: /run/systemd/private
      - name: docker
        hostPath:
          path: /run/docker.sock
      - name: systemd-etc
        hostPath:
          path: /etc/systemd
      - name: systemd-run
        hostPath:
          path: /run/systemd/system/
      - name: lsb-release
        hostPath:
          path: /etc/lsb-release

... uye inoshandisa script inotevera:

#!/bin/bash

# we will work only on xenial
hostrelease="/etc/lsb-release-host"
test -f ${hostrelease} && grep xenial ${hostrelease} > /dev/null || exit 0

# sleeping max 30 minutes to dispense load on kube-nodes
sleep $((RANDOM % 1800))

stoppedCount=0
# counting actual subpath units in systemd
countBefore=$(systemctl list-units | grep subpath | grep "run-" | wc -l)
# let's go check each unit
for unit in $(systemctl list-units | grep subpath | grep "run-" | awk '{print $1}'); do
  # finding description file for unit (to find out docker container, who born this unit)
  DropFile=$(systemctl status ${unit} | grep Drop | awk -F': ' '{print $2}')
  # reading uuid for docker container from description file
  DockerContainerId=$(cat ${DropFile}/50-Description.conf | awk '{print $5}' | cut -d/ -f6)
  # checking container status (running or not)
  checkFlag=$(docker ps | grep -c ${DockerContainerId})
  # if container not running, we will stop unit
  if [[ ${checkFlag} -eq 0 ]]; then
    echo "Stopping unit ${unit}"
    # stoping unit in action
    systemctl stop $unit
    # just counter for logs
    ((stoppedCount++))
    # logging current progress
    echo "Stopped ${stoppedCount} systemd units out of ${countBefore}"
  fi
done

... uye inomhanya maminetsi ega ega mashanu uchishandisa iyo yakambotaurwa supercronic. Yayo Dockerfile inotaridzika seizvi:

FROM ubuntu:16.04
COPY rootfs /
WORKDIR /app
RUN apt-get update && 
    apt-get upgrade -y && 
    apt-get install -y gnupg curl apt-transport-https software-properties-common wget
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" && 
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && 
    apt-get update && 
    apt-get install -y docker-ce=17.03.0*
RUN wget https://github.com/aptible/supercronic/releases/download/v0.1.6/supercronic-linux-amd64 -O 
    /usr/local/bin/supercronic && chmod +x /usr/local/bin/supercronic
ENTRYPOINT ["/bin/bash", "-c", "/usr/local/bin/supercronic -json /app/crontab"]

Nyaya 4. Kukwikwidzana pakuronga mapodhi

Izvo zvakacherechedzwa kuti: kana isu tiine podhi yakaiswa pane node uye mufananidzo wayo wakapombwa kunze kwenguva yakareba kwazvo, ipapo imwe podhi "inorova" iyo imwechete node inongodaro. haitangi kudhonza mufananidzo wepodhi itsva. Pane kudaro, inomirira kusvikira mufananidzo wepodhi yapfuura wadhonzwa. Nekuda kweizvozvo, podhi yanga yatorongwa uye iyo mufananidzo waigona kunge wakatorwa muminiti imwe chete inozoguma yave muchimiro che. containerCreating.

Zviitiko zvichataridzika seizvi:

Normal  Pulling    8m    kubelet, ip-10-241-44-128.ap-northeast-1.compute.internal  pulling image "registry.example.com/infra/openvpn/openvpn:master"

Icho chinopera icho mufananidzo mumwe chete kubva kune inononoka registry inogona kuvharira kutumirwa pa node.

Zvinosuruvarisa, hapana nzira dzakawanda dzekubuda mumamiriro ezvinhu:

Edza kushandisa yako Docker Registry zvakananga musumbu kana zvakananga nesumbu (semuenzaniso, GitLab Registry, Nexus, nezvimwewo);
Shandisa zvishandiso zvakadai se kraken.

Nhau 5. Node dzinorembera nekuda kwekushaya ndangariro

Munguva yekushanda kwezvikumbiro zvakasiyana-siyana, takasanganawo nemamiriro ezvinhu apo node inorega zvachose kuwanikwa: SSH haipindure, madhimoni ose ekucherechedza anodonha, uyezve hapana (kana kuti hapana) chinonyangadza mumatanda.

Ini ndichakutaurira mumifananidzo ndichishandisa muenzaniso weimwe node paishanda MongoDB.

Izvi ndizvo zvinoita atop up to tsaona:

Uye seizvi - после tsaona:

Mukutarisa, kune zvakare kusvetuka kwakapinza, uko iyo node inorega kuwanikwa:

Saka, kubva pascreenshots zviri pachena kuti:

Iyo RAM pamushini iri pedyo kusvika kumagumo;
Iko kune kusvetuka kwakapinza mukushandiswa kweRAM, mushure mezvo kuwana kumuchina wese kunovharwa kamwe kamwe;
Basa rakakura rinosvika paMongo, iro rinomanikidza DBMS maitiro kushandisa ndangariro uye kushingaira kuverenga kubva kudhisiki.

Zvinozoitika kuti kana Linux ikapera ndangariro yemahara (yeuko yekumanikidza inomira mukati) uye pasina shanduko, ipapo. up to Kana mhondi yeOOM ichisvika, chiitiko chekuenzanisa chinogona kusimuka pakati pekukanda mapeji mucache yepeji nekuanyora achidzosera kudhisiki. Izvi zvinoitwa ne kswapd, iyo inosunungura neushingi mapeji endangariro akawanda sezvinobvira pakugovera kunotevera.

Nehurombo, nehukuru hweI / O hwakasanganiswa nehuwandu hudiki hwekurangarira kwemahara, kswapd inova iyo bhodhoro reiyo system yese, nokuti vakasungwa kwairi all the kugoverwa (kukanganisa kwepeji) yemapeji ekurangarira muhurongwa. Izvi zvinogona kuenderera kwenguva yakareba kwazvo kana maitiro asingachadi kushandisa ndangariro zvakare, asi akaiswa pamucheto weOOM-muurayi gomba.

Mubvunzo wechisikigo ndewekuti: sei mhondi yeOOM ichiuya kunonoka? Mukutaura kwayo kwazvino, muurayi weOOM akapusa zvakanyanya: inouraya maitiro chete kana kuyedza kugovera peji rekurangarira kwakundikana, i.e. kana peji yacho yakundikana. Izvi hazviitike kwenguva yakareba, nekuti kswapd yakashinga inosunungura mapeji endangariro, ichirasa cache peji (iyo dhisiki yese I/O muhurongwa, kutaura zvazviri) kudzokera kudhisiki. Mune mamwe mashoko, nekutsanangurwa kwematanho anodiwa kubvisa matambudziko akadaro mu kernel, unogona kuverenga pano.

Maitiro aya inofanira kuvandudza ine Linux kernel 4.6+.

Nyaya 6. Mapodhi anonamira muPending state

Mune mamwe masumbu, umo mune akawanda mapods anoshanda, takatanga kuona kuti mazhinji acho "akarembera" kwenguva yakareba kwazvo mudunhu. Pending, kunyangwe midziyo yeDocker pachayo yave kutomhanya pamanodhi uye inogona kushandiswa nemawoko.

Uyezve, mu describe hapana chakaipa:

  Type    Reason                  Age                From                     Message
  ----    ------                  ----               ----                     -------
  Normal  Scheduled               1m                 default-scheduler        Successfully assigned sphinx-0 to ss-dev-kub07
  Normal  SuccessfulAttachVolume  1m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
  Normal  SuccessfulMountVolume   1m                 kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "sphinx-config"
  Normal  SuccessfulMountVolume   1m                 kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "default-token-fzcsf"
  Normal  SuccessfulMountVolume   49s (x2 over 51s)  kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
  Normal  Pulled                  43s                kubelet, ss-dev-kub07    Container image "registry.example.com/infra/sphinx-exporter/sphinx-indexer:v1" already present on machine
  Normal  Created                 43s                kubelet, ss-dev-kub07    Created container
  Normal  Started                 43s                kubelet, ss-dev-kub07    Started container
  Normal  Pulled                  43s                kubelet, ss-dev-kub07    Container image "registry.example.com/infra/sphinx/sphinx:v1" already present on machine
  Normal  Created                 42s                kubelet, ss-dev-kub07    Created container
  Normal  Started                 42s                kubelet, ss-dev-kub07    Started container

Mushure mekuchera, takaita fungidziro yekuti kubelet haina nguva yekutumira ruzivo rwese nezve mamiriro emapods uye yehupenyu / yekugadzirira bvunzo kune API server.

Uye mushure mekudzidza rubatsiro, takawana zvinotevera paramita:

--kube-api-qps - QPS to use while talking with kubernetes apiserver (default 5)
--kube-api-burst  - Burst to use while talking with kubernetes apiserver (default 10) 
--event-qps - If > 0, limit event creations per second to this value. If 0, unlimited. (default 5)
--event-burst - Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding event-qps. Only used if --event-qps > 0 (default 10) 
--registry-qps - If > 0, limit registry pull QPS to this value.
--registry-burst - Maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0 (default 10)

Sekuona, default tsika idiki chaizvo, uye mu90% vanovhara zvose zvinodiwa ... Zvisinei, munyaya yedu izvi zvakanga zvisina kukwana. Naizvozvo, tinoisa zvinotevera zvakakosha:

--event-qps=30 --event-burst=40 --kube-api-burst=40 --kube-api-qps=30 --registry-qps=30 --registry-burst=40

... uye akatangazve kubelets, mushure mezvo takaona mufananidzo unotevera mumagirafu ekufona kune API server:

... uye hongu, zvinhu zvose zvakatanga kubhururuka!

PS

Nekubatsira kwavo mukuunganidza tsikidzi nekugadzirira chinyorwa ichi, ndinotaura kutenda kwangu kwakadzama kune akawanda mainjiniya ekambani yedu, uye kunyanya kune wandaishanda naye kubva kuchikwata chedu cheR&D Andrey Klimentyev (zuzzas).

PPS

Verenga zvakare pablog yedu:

«kubectl-debug plugin ye debugging muKubernetes pods".
Kubernetes matipi & matipi loop:

Source: www.habr.com

6 inovaraidza system bugs mukushanda kweKubernetes [uye mhinduro yavo]