6 mau ʻōnaehana hoihoi i ka wā e hoʻohana ai iā Kubernetes [a me kā lākou hoʻonā]

I loko o nā makahiki o ka hoʻohana ʻana iā Kubernetes i ka hana ʻana, ua hōʻiliʻili mākou i nā moʻolelo hoihoi e pili ana i ke ʻano o nā pōpoki i nā ʻāpana ʻōnaehana like ʻole i alakaʻi i nā hopena maikaʻi ʻole a/a i ʻole hiki ke hoʻomaopopo ʻia e pili ana i ka hana o nā ipu a me nā pods. Ma kēia ʻatikala ua koho mākou i kekahi o nā mea maʻamau a hoihoi paha. ʻOiai inā ʻaʻole ʻoe i laki i ka hālāwai ʻana i kēlā mau kūlana, ʻo ka heluhelu ʻana e pili ana i nā moʻolelo detective pōkole - ʻoi aku ka "lima mua" - hoihoi mau, ʻaʻole anei?..

Moʻolelo 1. ʻO Supercronic a me Docker e kau ana

Ma kekahi o nā puʻupuʻu, loaʻa iā mākou kahi Docker maloʻo i kēlā me kēia manawa, i hoʻopilikia i ka hana maʻamau o ka hui. I ka manawa like, ua ʻike ʻia nā mea i loko o nā log Docker:

level=error msg="containerd: start init process" error="exit status 2: "runtime/cgo: pthread_create failed: No space left on device
SIGABRT: abort
PC=0x7f31b811a428 m=0

goroutine 0 [idle]:

goroutine 1 [running]:
runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420026768 sp=0xc420026760
runtime.main() /usr/local/go/src/runtime/proc.go:127 +0x6c fp=0xc4200267c0 sp=0xc420026768
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200267c8 sp=0xc4200267c0

goroutine 17 [syscall, locked to thread]:
runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

…

ʻO ka mea hoihoi loa iā mākou e pili ana i kēia hewa ʻo ia ka memo: pthread_create failed: No space left on device. Haʻawina wikiwiki palapala wehewehe ʻaʻole hiki iā Docker ke hana i kahi kaʻina hana, ʻo ia ke kumu i hoʻomaloʻo pinepine ai.

I ka nānā ʻana, pili kēia kiʻi i ka mea e hana nei:

ʻIke ʻia kahi kūlana like ma nā nodes ʻē aʻe:

Ma nā nodes like mākou e ʻike ai:

root@kube-node-1 ~ # ps auxfww | grep curl -c
19782
root@kube-node-1 ~ # ps auxfww | grep curl | head
root     16688  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     17398  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     16852  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      9473  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      4664  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     30571  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     24113  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root     16475  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      7176  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>
root      1090  0.0  0.0      0     0 ?        Z    Feb06   0:00      |       _ [curl] <defunct>

Ua ʻike ʻia he hopena kēia ʻano o ka hana ʻana o ka pod supercronic (kahi pono hele a mākou e hoʻohana ai e holo i nā hana cron i nā pods):

 _ docker-containerd-shim 833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 /var/run/docker/libcontainerd/833b60bb9ff4c669bb413b898a5fd142a57a21695e5dc42684235df907825567 docker-runc
|   _ /usr/local/bin/supercronic -json /crontabs/cron
|       _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true
|       |   _ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /dev/stderr --port /run/newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true -no-pidfile
|       _ [newrelic-daemon] <defunct>
|       _ [curl] <defunct>
|       _ [curl] <defunct>
|       _ [curl] <defunct>
…

ʻO ka pilikia kēia: ke holo ʻia kahi hana ma supercronic, ua hoʻomaka ke kaʻina hana hiki ole ke hoopau pono, huli ana i zombie.

i hoʻopuka: No ka ʻoi aku ka pololei, hoʻopuka ʻia nā kaʻina hana e nā hana cron, akā ʻaʻole ʻo supercronic kahi ʻōnaehana init a ʻaʻole hiki ke "hoʻokomo" i nā kaʻina hana i hana ʻia e kāna mau keiki. Ke hoʻokiʻekiʻe ʻia nā hōʻailona SIGHUP a i ʻole SIGTERM, ʻaʻole ia e hāʻawi ʻia i nā kaʻina hana o ke keiki, e hopena i ka hoʻopau ʻana o nā kaʻina keiki a noho i ke kūlana zombie. Hiki iā ʻoe ke heluhelu hou aku e pili ana i kēia mau mea a pau, no ka laʻana, ma he ʻatikala.

ʻElua mau ala e hoʻoponopono ai i nā pilikia:

1. Ma ke ʻano he hana manawa pōkole - hoʻonui i ka helu o nā PID i ka ʻōnaehana i hoʻokahi manawa i ka manawa:

          /proc/sys/kernel/pid_max (since Linux 2.5.34)
                 This file specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID).  PIDs greater than this  value  are  not  allo‐
                 cated;  thus, the value in this file also acts as a system-wide limit on the total number of processes and threads.  The default value for this file, 32768, results in the
                 same range of PIDs as on earlier kernels

2. A i ʻole e hoʻomaka i nā hana ma supercronic ʻaʻole pololei, akā hoʻohana like tini, hiki iā ia ke hoʻopau pololei i nā kaʻina hana a ʻaʻole hoʻi i nā zombies.

Moʻolelo 2. "Zombies" i ka wā e holoi ana i kahi hui

Ua hoʻomaka ʻo Kubelet e ʻai i ka nui o ka CPU:

ʻAʻohe mea makemake i kēia, no laila ua hoʻopaʻa mākou iā mākou iho ʻala a hoʻomaka e hoʻoponopono i ka pilikia. ʻO nā hopena o ka hoʻokolokolo ʻana penei:

• Hoʻohana ʻo Kubelet ma mua o ka hapakolu o kāna manawa CPU e huki ana i ka ʻikepili hoʻomanaʻo mai nā hui āpau:

  

• Ma ka papa inoa o nā mea hoʻomohala kernel hiki iā ʻoe ke loaʻa kūkākūkā o ka pilikia. I ka pōkole, hele mai ka manaʻo i kēia: ʻAʻole hoʻopau loa ʻia nā faila tmpfs a me nā mea like ʻole mai ka ʻōnaehana i ka holoi ʻana i kahi cgroup, ka mea i kapa ʻia memcg zombie. Ma hope a ma hope paha e holoi ʻia lākou mai ka cache ʻaoʻao, akā nui ka hoʻomanaʻo ma ka kikowaena a ʻaʻole ʻike ka kernel i ke kumu o ka hoʻopau manawa i ka holoi ʻana iā lākou. ʻO ia ke kumu e hōʻuluʻulu mau nei lākou. No ke aha kēia e hana nei? He kikowaena kēia me nā hana cron e hana mau i nā hana hou, a me lākou nā pods hou. No laila, hoʻokumu ʻia nā hui hou no nā pahu i loko o lākou, kahi e hoʻopau koke ʻia.
• No ke aha e pau ai ka manawa nui o ka cAdvisor ma kubelet? He maʻalahi kēia e ʻike me ka hoʻokō maʻalahi time cat /sys/fs/cgroup/memory/memory.stat. Inā ma ka mīkini olakino ka hana he 0,01 kekona, a laila ma ka cron02 pilikia he 1,2 kekona. ʻO ka mea ʻo cAdvisor, ka mea heluhelu i ka ʻikepili mai nā sysfs me ka mālie, e hoʻāʻo e noʻonoʻo i ka hoʻomanaʻo i hoʻohana ʻia i nā hui zombie.
• No ka wehe ikaika ʻana i nā zombies, ua hoʻāʻo mākou e hoʻomaʻemaʻe i nā huna huna e like me ka mea i ʻōlelo ʻia ma LKML: sync; echo 3 > /proc/sys/vm/drop_caches, - akā, ua lilo ka kernel i mea paʻakikī a ua hāʻule i ke kaʻa.

He aha ka hana? Hoʻoponopono ʻia ka pilikia (hoʻopaʻa, a no ka wehewehe ʻana e ʻike hoʻokuʻu memo) обновлением ядра Linux a hiki i ka mana 4.16.

Moolelo 3. Systemd a me kona mauna

Eia hou, ke ʻai nei ka kubelet i nā kumuwaiwai he nui loa ma kekahi mau nodes, akā i kēia manawa ke hoʻopau nui nei ka hoʻomanaʻo:

Оказалось, что есть проблема в systemd, используемом в Ubuntu 16.04, и возникает она при управлении mount’ами, которые создаются для подключения subPath mai ka ConfigMap a i ʻole nā ​​mea huna. Ma hope o ka pau ʻana o kāna hana noho ka lawelawe systemd a me kāna mauna lawelawe ma ka ʻōnaehana. I ka wā lōʻihi, e hōʻiliʻili ka nui o lākou. Aia kekahi mau pilikia e pili ana i kēia kumuhana:

1. #5916;
2. kubernetes #57345.

...ʻo ka mea hope loa e pili ana i ka PR ma systemd: #7811 (pilikia ma systemd - #7798).

Проблемы уже нет в Ubuntu 18.04, но если вы хотите и дальше использовать Ubuntu 16.04, вам может пригодиться наш workaround на эту тему.

No laila ua hana mākou i kēia DaemonSet:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: systemd-slices-cleaner
  name: systemd-slices-cleaner
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: systemd-slices-cleaner
  template:
    metadata:
      labels:
        app: systemd-slices-cleaner
    spec:
      containers:
      - command:
        - /usr/local/bin/supercronic
        - -json
        - /app/crontab
        Image: private-registry.org/systemd-slices-cleaner/systemd-slices-cleaner:v0.1.0
        imagePullPolicy: Always
        name: systemd-slices-cleaner
        resources: {}
        securityContext:
          privileged: true
        volumeMounts:
        - name: systemd
          mountPath: /run/systemd/private
        - name: docker
          mountPath: /run/docker.sock
        - name: systemd-etc
          mountPath: /etc/systemd
        - name: systemd-run
          mountPath: /run/systemd/system/
        - name: lsb-release
          mountPath: /etc/lsb-release-host
      imagePullSecrets:
      - name: antiopa-registry
      priorityClassName: cluster-low
      tolerations:
      - operator: Exists
      volumes:
      - name: systemd
        hostPath:
          path: /run/systemd/private
      - name: docker
        hostPath:
          path: /run/docker.sock
      - name: systemd-etc
        hostPath:
          path: /etc/systemd
      - name: systemd-run
        hostPath:
          path: /run/systemd/system/
      - name: lsb-release
        hostPath:
          path: /etc/lsb-release

... a hoʻohana ʻo ia i kēia ʻatikala:

#!/bin/bash

# we will work only on xenial
hostrelease="/etc/lsb-release-host"
test -f ${hostrelease} && grep xenial ${hostrelease} > /dev/null || exit 0

# sleeping max 30 minutes to dispense load on kube-nodes
sleep $((RANDOM % 1800))

stoppedCount=0
# counting actual subpath units in systemd
countBefore=$(systemctl list-units | grep subpath | grep "run-" | wc -l)
# let's go check each unit
for unit in $(systemctl list-units | grep subpath | grep "run-" | awk '{print $1}'); do
  # finding description file for unit (to find out docker container, who born this unit)
  DropFile=$(systemctl status ${unit} | grep Drop | awk -F': ' '{print $2}')
  # reading uuid for docker container from description file
  DockerContainerId=$(cat ${DropFile}/50-Description.conf | awk '{print $5}' | cut -d/ -f6)
  # checking container status (running or not)
  checkFlag=$(docker ps | grep -c ${DockerContainerId})
  # if container not running, we will stop unit
  if [[ ${checkFlag} -eq 0 ]]; then
    echo "Stopping unit ${unit}"
    # stoping unit in action
    systemctl stop $unit
    # just counter for logs
    ((stoppedCount++))
    # logging current progress
    echo "Stopped ${stoppedCount} systemd units out of ${countBefore}"
  fi
done

... a holo ia i kēlā me kēia 5 mau minuke me ka hoʻohana ʻana i ka supercronic i haʻi mua ʻia. Penei kona Dockerfile:

FROM ubuntu:16.04
COPY rootfs /
WORKDIR /app
RUN apt-get update && 
    apt-get upgrade -y && 
    apt-get install -y gnupg curl apt-transport-https software-properties-common wget
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" && 
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && 
    apt-get update && 
    apt-get install -y docker-ce=17.03.0*
RUN wget https://github.com/aptible/supercronic/releases/download/v0.1.6/supercronic-linux-amd64 -O 
    /usr/local/bin/supercronic && chmod +x /usr/local/bin/supercronic
ENTRYPOINT ["/bin/bash", "-c", "/usr/local/bin/supercronic -json /app/crontab"]

Moʻolelo 4. Ka hoʻokūkū i ka wā e hoʻonohonoho ai i nā pods

Ua ʻike ʻia: inā loaʻa iā mākou kahi pod i kau ʻia ma luna o kahi node a ua hoʻomake ʻia kona kiʻi i waho no ka manawa lōʻihi, a laila ʻo kekahi pod e "pa" i ka node hoʻokahi. ʻaʻole hoʻomaka e huki i ke kiʻi o ka pod hou. Akā, kali ia a huki ʻia ke kiʻi o ka pod mua. ʻO ka hopena, ʻo kahi pod i hoʻonohonoho mua ʻia a hiki ke kiʻi ʻia ke kiʻi i hoʻokahi minuke wale nō e pau i ke kūlana o containerCreating.

ʻO nā hanana e like me kēia:

Normal  Pulling    8m    kubelet, ip-10-241-44-128.ap-northeast-1.compute.internal  pulling image "registry.example.com/infra/openvpn/openvpn:master"

Ke hoʻololi nei ia hiki i ke kiʻi hoʻokahi mai kahi papa inoa lohi ke ālai i ka hoʻolaha ʻana node.

ʻO ka mea pōʻino, ʻaʻole nui nā ala i waho o ke kūlana:

1. E ho'āʻo e hoʻohana pololei i kāu Docker Registry i loko o ka pūʻulu a i ʻole pololei me ka pūʻulu (e laʻa, GitLab Registry, Nexus, etc.);
2. E hoʻohana i nā pono hana e like me kraken.

Moʻolelo 5. Ke kau nei nā node ma muli o ka nele o ka hoʻomanaʻo

I ka wā o ka hana ʻana o nā noi like ʻole, ua ʻike pū mākou i kahi kūlana kahi i pau ai ke komo ʻana o kahi node: ʻAʻole pane ʻo SSH, hāʻule nā ​​​​daemons kiaʻi a pau, a laila ʻaʻohe mea (a ʻaneʻane ʻaʻohe mea) anomalous i nā lāʻau.

E haʻi wau iā ʻoe i nā kiʻi e hoʻohana ana i ka laʻana o kahi node kahi i hana ai ʻo MongoDB.

ʻO kēia ke ʻano o luna i luna nā pōʻino:

A penei - mahope nā pōʻino:

I ka nānā ʻana, aia kekahi lele ʻoi, kahi e pau ai ka loaʻa ʻana o ka node:

No laila, mai nā kiʻi paʻi kiʻi e maopopo ai:

1. ʻO ka RAM ma ka mīkini kokoke i ka hopena;
2. Aia kahi lele ʻoi loa i ka hoʻohana ʻana i ka RAM, a laila hoʻopau koke ʻia ke komo ʻana i ka mīkini holoʻokoʻa;
3. Hiki mai kahi hana nui ma Mongo, e koi ana i ke kaʻina hana DBMS e hoʻohana i ka hoʻomanaʻo hou aʻe a heluhelu ikaika mai ka disk.

Оказывается, если в Linux заканчивается свободная память (наступает memory pressure) и swap’а нет, то i luna I ka hiki ʻana mai o ka mea pepehi kanaka OOM, hiki ke ala mai kahi hana kaulike ma waena o ka hoʻolei ʻana i nā ʻaoʻao i loko o ka waihona ʻaoʻao a kākau hou iā lākou i ka disk. Hana ʻia kēia e kswapd, nāna e hoʻokuʻu me ka wiwo ʻole i nā ʻaoʻao hoʻomanaʻo e like me ka hiki no ka hāʻawi ʻana ma hope.

ʻO ka mea pōʻino, me ka haʻawe I/O nui i hui pū ʻia me kahi hoʻomanaʻo manuahi liʻiliʻi, lilo ʻo kswapd i ka bottleneck o ka ʻōnaehana holoʻokoʻa, no ka mea, ua paa lakou nā mea a pau nā hoʻokaʻawale (nā ʻaoʻao hewa) o nā ʻaoʻao hoʻomanaʻo i ka ʻōnaehana. Hiki i kēia ke hoʻomau no ka manawa lōʻihi inā ʻaʻole makemake nā kaʻina hana e hoʻohana hou i ka hoʻomanaʻo, akā paʻa i ka lihi loa o ka OOM-killer abyss.

ʻO ka nīnau maʻamau: no ke aha e hele lōʻihi ai ka mea pepehi OOM? I kona ʻike ʻana i kēia manawa, he naʻaupō loa ka mea pepehi kanaka OOM: e pepehi wale ia i ke kaʻina hana ke hāʻule ka hoʻāʻo e hoʻokaʻawale i kahi ʻaoʻao hoʻomanaʻo, ʻo ia. inā hāʻule ka hewa ʻaoʻao. ʻAʻole hiki kēia i ka manawa lōʻihi, no ka mea, hoʻokuʻu wiwo ʻole ʻo kswapd i nā ʻaoʻao hoʻomanaʻo, e hoʻolei ana i ka cache ʻaoʻao (ka pā holoʻokoʻa I/O i ka ʻōnaehana, ʻoiaʻiʻo) hoʻi i ka disk. Ma nā kikoʻī hou aku, me ka wehewehe ʻana i nā ʻanuʻu e pono ai e hoʻopau i kēlā mau pilikia i ka kernel, hiki iā ʻoe ke heluhelu maanei.

ʻO kēia ʻano pono e hoʻomaikaʻi с ядром Linux 4.6 +.

Moʻolelo 6. Paʻa nā Pods i ka mokuʻāina e kali ana

I kekahi mau puʻupuʻu, kahi i nui maoli nā pods e hana ana, ua hoʻomaka mākou e ʻike i ka hapa nui o lākou "kau" no ka manawa lōʻihi loa i ka mokuʻāina. Pending, ʻoiai ke holo nei nā pahu Docker iā lākou iho ma nā nodes a hiki ke hana me ka lima.

Eia kekahi, ma describe ʻaʻohe hewa:

  Type    Reason                  Age                From                     Message
  ----    ------                  ----               ----                     -------
  Normal  Scheduled               1m                 default-scheduler        Successfully assigned sphinx-0 to ss-dev-kub07
  Normal  SuccessfulAttachVolume  1m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
  Normal  SuccessfulMountVolume   1m                 kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "sphinx-config"
  Normal  SuccessfulMountVolume   1m                 kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "default-token-fzcsf"
  Normal  SuccessfulMountVolume   49s (x2 over 51s)  kubelet, ss-dev-kub07    MountVolume.SetUp succeeded for volume "pvc-6aaad34f-ad10-11e8-a44c-52540035a73b"
  Normal  Pulled                  43s                kubelet, ss-dev-kub07    Container image "registry.example.com/infra/sphinx-exporter/sphinx-indexer:v1" already present on machine
  Normal  Created                 43s                kubelet, ss-dev-kub07    Created container
  Normal  Started                 43s                kubelet, ss-dev-kub07    Started container
  Normal  Pulled                  43s                kubelet, ss-dev-kub07    Container image "registry.example.com/infra/sphinx/sphinx:v1" already present on machine
  Normal  Created                 42s                kubelet, ss-dev-kub07    Created container
  Normal  Started                 42s                kubelet, ss-dev-kub07    Started container

Ma hope o ka ʻeli ʻana, ua manaʻo mākou ʻaʻohe manawa o ka kubelet e hoʻouna i nā ʻike āpau e pili ana i ke kūlana o nā pods a me nā hoʻāʻo ola / mākaukau i ka server API.

A ma hope o ke aʻo ʻana i ke kōkua, ʻike mākou i kēia mau ʻāpana:

--kube-api-qps - QPS to use while talking with kubernetes apiserver (default 5)
--kube-api-burst  - Burst to use while talking with kubernetes apiserver (default 10) 
--event-qps - If > 0, limit event creations per second to this value. If 0, unlimited. (default 5)
--event-burst - Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding event-qps. Only used if --event-qps > 0 (default 10) 
--registry-qps - If > 0, limit registry pull QPS to this value.
--registry-burst - Maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0 (default 10)

E like me ka mea i ʻike ʻia, liʻiliʻi nā waiwai paʻamau, a ma 90% uhi lākou i nā pono āpau ... Eia naʻe, i kā mākou hihia ʻaʻole lawa kēia. No laila, hoʻonoho mākou i kēia mau waiwai:

--event-qps=30 --event-burst=40 --kube-api-burst=40 --kube-api-qps=30 --registry-qps=30 --registry-burst=40

... a hoʻomaka hou i nā kubelets, a laila ʻike mākou i ke kiʻi aʻe ma nā pakuhi o nā kelepona i ka server API:

... a ʻae, ua hoʻomaka nā mea a pau e lele!

PS

No ko lākou kōkua ʻana i ka ʻohi ʻana i nā pōpoki a me ka hoʻomākaukau ʻana i kēia ʻatikala, ke hōʻike aku nei au i koʻu mahalo nui i nā ʻenekini he nui o kā mākou hui, a ʻoi aku hoʻi i kaʻu hoa hana mai kā mākou hui R&D Andrey Klimentyev (zuzzas).

PPS

E heluhelu pū ma kā mākou blog:

• «kubectl-debug plugin no ka hoʻopau ʻana i nā pods Kubernetes".
• Nā ʻōlelo aʻoaʻo a me nā hoʻopunipuni Kubernetes:
  • «Ka hoʻololi ʻana i nā kumuwaiwai e holo ana i kahi hui i ka hoʻokele Helm 2";
  • «E pili ana i ka hoʻokaʻawale ʻana a me nā ukana ma kahi noi pūnaewele";
  • «Loaʻa i nā pūnaewele dev";
  • «ʻO ka wikiwiki ʻana i ka bootstrap no nā ʻikepili nui".

Source: www.habr.com