![6 lom zem kab kab hauv kev ua haujlwm ntawm Kubernetes [thiab lawv cov kev daws teeb meem]](https://prohoster.info/wp-content/plugins/wp-fastest-cache-premium/pro/images/blank.gif)
Ntau xyoo dhau los ntawm kev siv Kubernetes hauv kev tsim khoom, peb tau sau ntau cov dab neeg nthuav qhia txog yuav ua li cas kab mob hauv ntau yam kab ke ua rau tsis zoo thiab / lossis tsis tuaj yeem ua rau muaj kev cuam tshuam rau kev ua haujlwm ntawm cov ntim khoom thiab cov pods. Nyob rau hauv tsab xov xwm no, peb tau muab tso ua ke ib co ntawm cov feem ntau pom los yog nthuav sawv daws yuav. Txawm hais tias koj yeej tsis muaj hmoo txaus los ntsib cov xwm txheej zoo li no, nyeem txog cov dab neeg luv luv no - tshwj xeeb tshaj yog "thawj-tes" - yeej ib txwm lom zem, puas yog? ..
Dab Neeg 1. Supercronic thiab ib tug daig Docker
Ntawm ib qho ntawm pawg, peb ib txwm tau txais "dai" Docker, uas cuam tshuam nrog kev ua haujlwm ntawm pawg. Tib lub sijhawm, cov hauv qab no tau pom hauv Docker cav
level=error msg="containerd: start init process" error="exit status 2: "runtime/cgo: pthread_create failed: No space left on device
SIGABRT: abort
PC=0x7f31b811a428 m=0
goroutine 0 [idle]:
goroutine 1 [running]:
runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420026768 sp=0xc420026760
runtime.main() /usr/local/go/src/runtime/proc.go:127 +0x6c fp=0xc4200267c0 sp=0xc420026768
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200267c8 sp=0xc4200267c0
goroutine 17 [syscall, locked to thread]:
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
β¦
Hauv qhov yuam kev no, peb txaus siab tshaj plaws hauv cov lus: pthread_create failed: No space left on device. Ib txoj kev tshawb nrhiav cursory piav qhia tias Docker tsis tuaj yeem cuam tshuam cov txheej txheem, uas yog vim li cas nws ib ntus "hangs".
Hauv kev saib xyuas, dab tsi tshwm sim sib raug rau daim duab hauv qab no:
Ib qho xwm txheej zoo sib xws tau pom ntawm lwm cov nodes:
Ntawm tib lub nodes peb pom:
root@kube-node-1 ~ # ps auxfww | grep curl -c
19782
root@kube-node-1 ~ # ps auxfww | grep curl | head
root 16688 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 17398 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 16852 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 9473 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 4664 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 30571 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 24113 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 16475 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 7176 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>
root 1090 0.0 0.0 0 0 ? Z Feb06 0:00 | _ [curl] <defunct>Nws muab tawm tias qhov kev coj cwj pwm no yog qhov tshwm sim ntawm lub plhaub taum ua haujlwm nrog (ib qho khoom siv mus rau peb siv los khiav haujlwm cron hauv pods):
_ docker-containerd-shim 833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 /var/run/docker/libcontainerd/833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 docker-runc
| _ /usr/local/bin/supercronic -json /crontabs/cron
| _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true
| | _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true -no-pidfile
| _ [newrelic-daemon] <defunct>
| _ [curl] <defunct>
| _ [curl] <defunct>
| _ [curl] <defunct>
β¦Qhov teeb meem yog qhov no: thaum pib ua haujlwm hauv supercronic, cov txheej txheem spawned los ntawm nws tsis tuaj yeem ua tiav kom raug, tig rau .
ΠΡΠΈΠΌΠ΅ΡΠ°Π½ΠΈΠ΅: Kom meej meej, cov txheej txheem yog spawned los ntawm cron cov hauj lwm, tab sis supercronic tsis yog ib qho init system thiab tsis tuaj yeem "tau txais" cov txheej txheem uas nws cov me nyuam spawned. Thaum SIGHUP lossis SIGTERM teeb liab tshwm sim, lawv tsis dhau mus rau cov txheej txheem menyuam yaus, ua rau cov txheej txheem menyuam yaus tsis tuaj yeem txiav tawm, tshuav nyob rau hauv cov xwm txheej zombie. Koj tuaj yeem nyeem ntxiv txog txhua qhov no, piv txwv li, hauv .
Muaj ob peb txoj hauv kev los daws cov teeb meem:
- Raws li kev ua haujlwm ib ntus - nce tus naj npawb ntawm PIDs hauv qhov system ntawm tib lub sijhawm:
/proc/sys/kernel/pid_max (since Linux 2.5.34) This file specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID). PIDs greater than this value are not alloβ cated; thus, the value in this file also acts as a system-wide limit on the total number of processes and threads. The default value for this file, 32768, results in the same range of PIDs as on earlier kernels - Los yog ua kom lub community launch txog cov dej num hauv supercronic tsis ncaj qha, tab sis siv tib yam , uas muaj peev xwm ntawm gracefully txiav cov txheej txheem thiab tsis spawning zombies.
Dab neeg 2. "Zombies" thaum rho tawm ib pawg
Kubelet pib siv ntau CPU:
Tsis muaj leej twg nyiam qhov no, yog li peb armed peb tus kheej thiab pib daws qhov teeb meem. Cov txiaj ntsig ntawm kev tshawb nrhiav tau raws li hauv qab no:
- Kubelet siv ntau tshaj li ib feem peb ntawm CPU lub sij hawm rub cov ntaub ntawv nco los ntawm txhua pawg:
- Nyob rau hauv daim ntawv teev npe tsim tawm kernel, koj tuaj yeem pom . Hauv luv, lub ntsiab lus yog qhov ntawd ntau yam tmpfs cov ntaub ntawv thiab lwm yam zoo sib xws tsis raug tshem tawm tag nrho ntawm qhov system thaum tshem ib cgroup, lub thiaj li hu zombie. Tsis ntev los sis tom qab lawv tseem yuav raug rho tawm ntawm nplooj ntawv cache, txawm li cas los xij, muaj ntau lub cim xeeb ntawm cov neeg rau zaub mov thiab cov ntsiav tsis pom lub ntsiab lus hauv nkim sijhawm tshem lawv. Yog li ntawd, lawv tseem ua ke. Vim li cas qhov no tseem tshwm sim? Qhov no yog ib tug neeg rau zaub mov nrog cron txoj hauj lwm uas tas li tsim cov hauj lwm tshiab, thiab nrog lawv cov pods tshiab. Yog li, cov cgroups tshiab yog tsim rau cov ntim hauv lawv, uas yuav raug tshem tawm sai sai.
- Vim li cas cAdvisor hauv kubelet siv sijhawm ntau? Qhov no yog ib qho yooj yim pom los ntawm kev ua kom yooj yim tshaj plaws
time cat /sys/fs/cgroup/memory/memory.stat. Yog tias ntawm lub tshuab noj qab haus huv kev ua haujlwm yuav siv sijhawm 0,01 vib nas this, tom qab ntawd ntawm qhov teeb meem cron02 nws yuav siv sijhawm 1,2 vib nas this. Qhov tshaj plaws yog tias cAdvisor, uas nyeem cov ntaub ntawv los ntawm sysfs qeeb heev, sim coj mus rau hauv tus account siv lub cim xeeb hauv zombie cgroups thiab. - Txhawm rau tshem tawm cov zombies, peb sim tshem cov caches raws li kev pom zoo los ntawm LKML:
sync; echo 3 > /proc/sys/vm/drop_caches, - tab sis cov tub ntxhais tau hloov mus ua qhov nyuaj dua thiab dai lub tsheb.
Yuav ua li cas? Qhov teeb meem raug kho, thiab saib cov lus piav qhia hauv ) los ntawm kev hloov kho Linux kernel rau version 4.16.
Dab Neeg 3. Systemd thiab nws mount
Ntxiv dua thiab, kubelet siv ntau cov peev txheej ntawm qee cov nodes, tab sis lub sijhawm no nws yog qhov nco ntau dua:
Nws tau muab tawm tias muaj teeb meem nrog lub systemd siv hauv Ubuntu 16.04, thiab nws tshwm sim thaum tswj cov mounts uas tsim los txuas. subPath los ntawm ConfigMap'ov los yog secret'ov. Tom qab kaw lub pod cov kev pabcuam systemd thiab nws cov kev pabcuam mount tseem nyob hauv qhov system. Nyob rau tib lub sij hawm, lawv sau ntau heev. Tseem muaj teeb meem ntawm lub ncauj lus no:
- ;
- .
... nyob rau hauv qhov kawg ntawm uas lawv xa mus rau PR hauv systemd: (qhov teeb meem hauv systemd - ).
Qhov teeb meem tsis nyob hauv Ubuntu 18.04 lawm, tab sis yog tias koj xav siv Ubuntu 16.04 txuas ntxiv, koj tuaj yeem pom peb qhov kev daws teeb meem ntawm cov ncauj lus no muaj txiaj ntsig.
Yog li, peb tau ua DaemonSet hauv qab no:
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
app: systemd-slices-cleaner
name: systemd-slices-cleaner
namespace: kube-system
spec:
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: systemd-slices-cleaner
template:
metadata:
labels:
app: systemd-slices-cleaner
spec:
containers:
- command:
- /usr/local/bin/supercronic
- -json
- /app/crontab
Image: private-registry.org/systemd-slices-cleaner/systemd-slices-cleaner:v0.1.0
imagePullPolicy: Always
name: systemd-slices-cleaner
resources: {}
securityContext:
privileged: true
volumeMounts:
- name: systemd
mountPath: /run/systemd/private
- name: docker
mountPath: /run/docker.sock
- name: systemd-etc
mountPath: /etc/systemd
- name: systemd-run
mountPath: /run/systemd/system/
- name: lsb-release
mountPath: /etc/lsb-release-host
imagePullSecrets:
- name: antiopa-registry
priorityClassName: cluster-low
tolerations:
- operator: Exists
volumes:
- name: systemd
hostPath:
path: /run/systemd/private
- name: docker
hostPath:
path: /run/docker.sock
- name: systemd-etc
hostPath:
path: /etc/systemd
- name: systemd-run
hostPath:
path: /run/systemd/system/
- name: lsb-release
hostPath:
path: /etc/lsb-release... thiab nws siv tsab ntawv no:
#!/bin/bash
# we will work only on xenial
hostrelease="/etc/lsb-release-host"
test -f ${hostrelease} && grep xenial ${hostrelease} > /dev/null || exit 0
# sleeping max 30 minutes to dispense load on kube-nodes
sleep $((RANDOM % 1800))
stoppedCount=0
# counting actual subpath units in systemd
countBefore=$(systemctl list-units | grep subpath | grep "run-" | wc -l)
# let's go check each unit
for unit in $(systemctl list-units | grep subpath | grep "run-" | awk '{print $1}'); do
# finding description file for unit (to find out docker container, who born this unit)
DropFile=$(systemctl status ${unit} | grep Drop | awk -F': ' '{print $2}')
# reading uuid for docker container from description file
DockerContainerId=$(cat ${DropFile}/50-Description.conf | awk '{print $5}' | cut -d/ -f6)
# checking container status (running or not)
checkFlag=$(docker ps | grep -c ${DockerContainerId})
# if container not running, we will stop unit
if [[ ${checkFlag} -eq 0 ]]; then
echo "Stopping unit ${unit}"
# stoping unit in action
systemctl stop $unit
# just counter for logs
((stoppedCount++))
# logging current progress
echo "Stopped ${stoppedCount} systemd units out of ${countBefore}"
fi
done... thiab nws khiav txhua 5 feeb nrog kev pab los ntawm yav tas los hais supercronic. Nws Dockerfile zoo li no:
FROM ubuntu:16.04
COPY rootfs /
WORKDIR /app
RUN apt-get update &&
apt-get upgrade -y &&
apt-get install -y gnupg curl apt-transport-https software-properties-common wget
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" &&
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - &&
apt-get update &&
apt-get install -y docker-ce=17.03.0*
RUN wget https://github.com/aptible/supercronic/releases/download/v0.1.6/supercronic-linux-amd64 -O
/usr/local/bin/supercronic && chmod +x /usr/local/bin/supercronic
ENTRYPOINT ["/bin/bash", "-c", "/usr/local/bin/supercronic -json /app/crontab"]Zaj Dab Neeg 4: Concurrency in Pod Planning
Nws tau pom tias: yog tias peb muaj lub plhaub taum tso rau ntawm lub pob thiab nws cov duab tau muab tso rau lub sijhawm ntev heev, ces lwm lub plhaub taum uas "ntaus" ntawm tib lub node yooj yim. tsis pib rub daim duab tshiab. Hloov chaw, nws tos rau yav dhau los pod cov duab kom 'rub'. Yog li ntawd, lub plhaub taum uas twb tau npaj lawm thiab nws cov duab tuaj yeem rub tawm hauv ib feeb xwb yuav xaus rau hauv cov xwm txheej ntev. containerCreating.
Cov xwm txheej yuav zoo li no:
Normal Pulling 8m kubelet, ip-10-241-44-128.ap-northeast-1.compute.internal pulling image "registry.example.com/infra/openvpn/openvpn:master"Nws hloov tawm ntawd ib daim duab los ntawm kev sau npe qeeb tuaj yeem thaiv kev xa tawm rau node.
Hmoov tsis zoo, tsis muaj ntau txoj hauv kev tawm ntawm qhov xwm txheej:
- Sim siv koj Docker Registry ncaj qha rau hauv pawg lossis ncaj qha nrog pawg (xws li GitLab Registry, Nexus, thiab lwm yam);
- Siv cov khoom siv hluav taws xob xws li .
Dab neeg 5. Txawb pob vim tsis nco
Thaum lub sijhawm ua haujlwm ntawm ntau daim ntawv thov, peb kuj tau txais qhov xwm txheej thaum lub node kiag li tsis muaj: SSH tsis teb, tag nrho cov saib xyuas daems poob, thiab tom qab ntawd tsis muaj dab tsi (lossis yuav luag tsis muaj dab tsi) txawv txav hauv cov cav.
Kuv yuav qhia koj hauv cov duab uas siv qhov piv txwv ntawm ib qho ntawm qhov twg MongoDB ua haujlwm.
Qhov no yog qhov zoo li saum toj no rau xwm txheej:
Thiab zoo li no - tom qab xwm txheej:
Hauv kev saib xyuas, kuj tseem muaj qhov dhia ntse, uas lub node tsis muaj:
Yog li los ntawm cov screenshots koj tuaj yeem pom tias:
- Lub RAM ntawm lub tshuab yog ze rau khiav tawm;
- Muaj qhov ntse dhia hauv kev noj ntawm RAM, tom qab uas nkag mus rau tag nrho lub tshuab ua haujlwm sai sai;
- Ib txoj haujlwm loj tuaj txog ntawm Mongo, uas ua rau cov txheej txheem DBMS siv ntau lub cim xeeb thiab nquag nyeem los ntawm disk.
Nws hloov tawm tias yog tias Linux khiav tawm ntawm lub cim xeeb dawb (nco siab teeb tsa hauv) thiab tsis muaj kev sib pauv, ces rau Kev tuaj txog ntawm OOM killer, tej zaum yuav muaj qhov sib npaug ntawm cov nplooj ntawv pov rau hauv nplooj ntawv cache thiab sau lawv rov qab rau disk. Qhov no yog tswj hwm los ntawm kswapd, uas ua siab loj tso tawm ntau nplooj ntawv nco li ua tau rau kev faib tom qab.
Hmoov tsis zoo, nrog rau qhov hnyav I / O load ua ke nrog me me ntawm lub cim xeeb dawb, kswapd ua lub bottleneck ntawm tag nrho cov systemvim lawv muab khi rau nws tag nrho allocations (nplooj faults) ntawm cov nplooj ntawv nco hauv qhov system. Qhov no tuaj yeem mus ntev heev yog tias cov txheej txheem tsis xav siv lub cim xeeb ntxiv lawm, tab sis fixate ntawm ntug heev ntawm OOM killer abyss.
Lo lus nug yog ntuj: yog vim li cas tus OOM killer tuaj lig? Hauv nws qhov kev rov hais dua tam sim no, OOM killer yog qhov ruam heev: nws yuav tua cov txheej txheem tsuas yog thaum qhov kev sim faib nplooj ntawv nco tsis tau, piv txwv li. yog tias nplooj ntawv yuam kev hla nrog qhov yuam kev. Qhov no tsis tshwm sim ntev txaus vim tias kswapd valiantly tso cov nplooj ntawv ntawm lub cim xeeb los ntawm kev yaug cov nplooj ntawv cache (tag nrho disk I / O ntawm lub kaw lus, qhov tseem ceeb) rov qab mus rau disk. Hauv kev nthuav dav ntxiv, nrog cov lus piav qhia ntawm cov kauj ruam tsim nyog los tshem tawm cov teeb meem zoo li no hauv cov ntsiav, koj tuaj yeem nyeem .
Tus cwj pwm no nrog Linux kernel 4.6+.
Zaj Dab Neeg 6. Pods tau daig hauv lub xeev Pending
Nyob rau hauv ib co pawg uas muaj coob heev ntawm cov pods khiav, peb pib pom tias feem ntau ntawm lawv dai nyob rau hauv lub xeev ntev heev. Pending, txawm hais tias Docker ntim lawv tus kheej twb tau khiav ntawm cov nodes thiab koj tuaj yeem ua haujlwm nrog lawv.
Ntxiv mus, hauv describe tsis muaj dab tsi tsis ncaj ncees lawm:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned sphinx-0 to ss-dev-kub07
Normal SuccessfulAttachVolume 1m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
Normal SuccessfulMountVolume 1m kubelet, ss-dev-kub07 MountVolume.SetUp succeeded for volume "sphinx-config"
Normal SuccessfulMountVolume 1m kubelet, ss-dev-kub07 MountVolume.SetUp succeeded for volume "default-token-fzcsf"
Normal SuccessfulMountVolume 49s (x2 over 51s) kubelet, ss-dev-kub07 MountVolume.SetUp succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
Normal Pulled 43s kubelet, ss-dev-kub07 Container image "registry.example.com/infra/sphinx-exporter/sphinx-indexer:v1" already present on machine
Normal Created 43s kubelet, ss-dev-kub07 Created container
Normal Started 43s kubelet, ss-dev-kub07 Started container
Normal Pulled 43s kubelet, ss-dev-kub07 Container image "registry.example.com/infra/sphinx/sphinx:v1" already present on machine
Normal Created 42s kubelet, ss-dev-kub07 Created container
Normal Started 42s kubelet, ss-dev-kub07 Started containerTom qab khawb ib ncig, peb tau ua qhov kev xav tias kubelet tsuas yog tsis muaj sijhawm los xa tag nrho cov ntaub ntawv hais txog lub xeev ntawm cov pods, kev ua neej nyob / kev npaj ua qauv rau API server.
Thiab tau kawm txog kev pab, peb pom cov hauv qab no tsis:
--kube-api-qps - QPS to use while talking with kubernetes apiserver (default 5)
--kube-api-burst - Burst to use while talking with kubernetes apiserver (default 10)
--event-qps - If > 0, limit event creations per second to this value. If 0, unlimited. (default 5)
--event-burst - Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding event-qps. Only used if --event-qps > 0 (default 10)
--registry-qps - If > 0, limit registry pull QPS to this value.
--registry-burst - Maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0 (default 10)Raws li pom, default values ββyog heev me me, thiab hauv 90% lawv them tag nrho cov kev xav tau ... Txawm li cas los xij, hauv peb cov ntaub ntawv, qhov no tsis txaus. Yog li ntawd, peb teev cov nqi hauv qab no:
--event-qps=30 --event-burst=40 --kube-api-burst=40 --kube-api-qps=30 --registry-qps=30 --registry-burst=40... thiab rov pib lub kubelets, tom qab ntawd peb pom cov duab hauv qab no ntawm cov duab rau kev nkag mus rau API server:
... thiab yog, txhua yam pib ya!
PS
Rau kev pab sau cov kab thiab npaj cov kab lus, kuv ua tsaug rau ntau tus engineers ntawm peb lub tuam txhab, thiab tshwj xeeb tshaj yog rau peb cov npoj yaig los ntawm peb pab pawg R & D Andrey Klimentyev ().
PPS
Nyeem kuj ntawm peb blog:
- Β«".
- Kubernetes lub tswv yim & tricks voj voog:
- «»;
- «»;
- «»;
- Β«".
Tau qhov twg los: www.hab.com