Nā Kubernetes Liʻiliʻi Loa

Ua hoʻomākaukau ʻia ka unuhi ʻana o ka ʻatikala ma ka pō o ka hoʻomaka ʻana o ka papa "Nā hana a me nā mea hana DevOps".

Nā Kubernetes Liʻiliʻi Loa

Inā ʻoe e heluhelu ana i kēia, ua lohe paha ʻoe i kekahi mea e pili ana i nā Kubernetes (a inā ʻaʻole, pehea ʻoe i pau ai ma ʻaneʻi?) Akā he aha ka Kubernetes? ʻO kēia "Ka hoʻonohonoho ʻana o nā ipu ʻoihana"? A i ʻole "Cloud-Native Operating System"? He aha ke ʻano o kēia?

ʻO kaʻoiaʻiʻo, ʻaʻole wau 100% maopopo. Akā, manaʻo wau he mea hoihoi ia e ʻeli i loko o nā mea i loko a ʻike i nā mea e hana maoli nei ma Kubernetes ma lalo o kāna mau papa hana abstractions. No laila, no ka leʻaleʻa, e nānā kākou i ke ʻano o ka "hui Kubernetes" liʻiliʻi. (E ʻoi aku ka maʻalahi o kēia Kubernetes Ke Ala Paʻakikī.)

Manaʻo wau ua loaʻa iā ʻoe ka ʻike kumu o Kubernetes, Linux, a me nā ipu. ʻO nā mea a pau a mākou e kamaʻilio nei no ka noiʻi/aʻo wale nō, mai hoʻokomo i kekahi o ia mea i ka hana!

Hōʻuluʻulu manaʻo

Aia nā Kubernetes i nā mea he nui. Wahi a Wikipedia, ua like ke ano o ka hale hana:

Nā Kubernetes Liʻiliʻi Loa

Aia ma kahi o ʻewalu mau ʻāpana i hōʻike ʻia ma aneʻi, akā e haʻalele mākou i ka hapa nui o lākou. Makemake au e haʻi aku ʻo ka mea liʻiliʻi e hiki ke kapa ʻia ʻo Kubernetes he ʻekolu mau mea nui:

  • kubelet
  • kube-apiserver (e pili ana i etcd - kona waihona)
  • ka manawa holo pahu (Docker i kēia hihia)

E ʻike kākou i ka ʻōlelo a ka palapala e pili ana i kēlā me kēia o lākou (rus., Pelekania.). Ma ka mua kubelet:

He ʻelele e holo ana ma kēlā me kēia node o ka hui. Hoʻomaopopo ia e holo ana nā pahu i loko o ka pod.

He kani maʻalahi. Pehea pahu holo manawa (manawa holo pahu)?

ʻO kahi pahu holo manawa he papahana i hoʻolālā ʻia e holo i nā ipu.

ʻIke loa. Akā inā kamaʻāina ʻoe iā Docker, a laila pono ʻoe i ka manaʻo maʻamau o kāna hana. (ʻO nā kikoʻī o ka hoʻokaʻawale ʻana o nā kuleana ma waena o ka pahu pahu a me ka kubelet he maʻalahi maoli a ʻaʻole wau e komo i loko o lākou ma aneʻi.)

И kikowaena API?

ʻO API Server ka ʻāpana hoʻomalu o Kubernetes e hōʻike ana i ka Kubernetes API. ʻO ke kikowaena API ka ʻaoʻao mea kūʻai aku o ka papa mana Kubernetes

ʻO kēlā me kēia mea i hana i kekahi mea me Kubernetes, pono e launa pū me ka API a i ʻole ma o kubectl. ʻO kēia ka puʻuwai o ka mea e hana ai ʻo Kubernetes Kubernetes - ka lolo nāna e hoʻohuli i nā mauna o YAML a mākou i ʻike a aloha ai (?) i ʻōnaehana hana. Me he mea lā aia ka API i kā mākou hoʻonohonoho liʻiliʻi.

Nā ʻondlelo Mua

  • ʻO ka Linux virtual a i ʻole ka mīkini kino me ke aʻa (ke hoʻohana nei au i ka Ubuntu 18.04 ma kahi mīkini virtual).
  • A pau!

Hoʻokomo luuluu

Pono mākou e hoʻokomo iā Docker ma ka mīkini a mākou e hoʻohana ai. (ʻAʻole wau e hele i nā kikoʻī e pili ana i ka hana ʻana o Docker a me nā ipu; inā makemake ʻoe, aia nā ʻatikala kupanaha). E hoʻouka wale kāua me apt:

$ sudo apt install docker.io
$ sudo systemctl start docker

Ma hope o kēlā, pono mākou e kiʻi i nā binaries Kubernetes. ʻOiaʻiʻo, no ka hoʻomaka mua ʻana o kā mākou "cluster" pono mākou kubelet, no ka holo ʻana i nā ʻāpana kikowaena ʻē aʻe hiki iā mākou ke hoʻohana kubelet. E launa pū me kā mākou hui ma hope o ka holo ʻana, e hoʻohana pū mākou kubectl.

$ curl -L https://dl.k8s.io/v1.18.5/kubernetes-server-linux-amd64.tar.gz > server.tar.gz
$ tar xzvf server.tar.gz
$ cp kubernetes/server/bin/kubelet .
$ cp kubernetes/server/bin/kubectl .
$ ./kubelet --version
Kubernetes v1.18.5

He aha ka hopena inā holo wale mākou kubelet?

$ ./kubelet
F0609 04:03:29.105194    4583 server.go:254] mkdir /var/lib/kubelet: permission denied

kubelet pono e holo ma ke ano he kumu. Maikaʻi loa, no ka mea pono ʻo ia e hoʻokele i ka node holoʻokoʻa. E nana kakou i kona mau palena:

$ ./kubelet -h
<слишком много строк, чтобы разместить здесь>
$ ./kubelet -h | wc -l
284

Auē, nui nā koho! ʻO ka mea pōmaikaʻi, pono mākou i ʻelua o lākou. Eia kekahi o nā palena a mākou e makemake ai:

--pod-manifest-path string

Ala i ka papa kuhikuhi i loaʻa nā faila no nā pods static, a i ʻole ala i kahi faila e wehewehe ana i nā pods static. ʻAʻole mālama ʻia nā faila e hoʻomaka ana me nā kiko. (KUHI: Pono e hoʻonoho ʻia kēia koho i ka faila hoʻonohonoho i hāʻawi ʻia i ke Kubelet ma o ke koho --config. No ka ʻike hou aku, e ʻike. kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file .)

Hāʻawi kēia koho iā mākou e holo nā ʻōpala paʻa - nā pods i mālama ʻole ʻia ma o ka Kubernetes API. ʻAʻole hoʻohana pinepine ʻia nā static pods, akā maʻalahi lākou no ka hoʻoulu koke ʻana i kahi pūpū, a ʻo ia ka mea e pono ai mākou. E haʻalele mākou i kēia ʻōlelo aʻo nui (eia hou, mai holo i kēia i ka hana ʻana!) a ʻike inā hiki iā mākou ke holo ka pod.

E hana mua mākou i kahi papa kuhikuhi no nā pods static a holo kubelet:

$ mkdir pods
$ sudo ./kubelet --pod-manifest-path=pods

A laila, ma kahi puka makani ʻē aʻe / tmux / he aha, e hana mākou i kahi pod manifest:

$ cat <<EOF > pods/hello.yaml
apiVersion: v1
kind: Pod
metadata:
  name: hello
spec:
  containers:
  - image: busybox
    name: hello
    command: ["echo", "hello world!"]
EOF

kubelet hoʻomaka e kākau i kekahi mau ʻōlelo aʻo a me he mea lā ʻaʻohe mea e hana nei. Akā ʻaʻole ʻoiaʻiʻo kēlā! E nānā kākou iā Docker:

$ sudo docker ps -a
CONTAINER ID        IMAGE                  COMMAND                 CREATED             STATUS                      PORTS               NAMES
8c8a35e26663        busybox                "echo 'hello world!'"   36 seconds ago      Exited (0) 36 seconds ago                       k8s_hello_hello-mink8s_default_ab61ef0307c6e0dee2ab05dc1ff94812_4
68f670c3c85f        k8s.gcr.io/pause:3.2   "/pause"                2 minutes ago       Up 2 minutes                                    k8s_POD_hello-mink8s_default_ab61ef0307c6e0dee2ab05dc1ff94812_0
$ sudo docker logs k8s_hello_hello-mink8s_default_ab61ef0307c6e0dee2ab05dc1ff94812_4
hello world!

kubelet Ua heluhelu au i ka pod manifest a hāʻawi iā Docker i ke kauoha e hoʻomaka i ʻelua mau pahu e like me kā mākou kikoʻī. (Inā ʻoe e noʻonoʻo e pili ana i ka ipu "hoʻomaha", he hack Kubernetes - ʻike kēia blog.) E hoʻomaka ʻo Kubelet i kā mākou pahu busybox me ke kauoha i kuhikuhi ʻia a e hoʻomaka hou ia a hiki i ka pau ʻana o ka static pod.

E hoʻomaikaʻi iā ʻoe iho. Ua hele mai mākou me kekahi o nā ala huikau loa e hoʻopuka i nā kikokikona i ka pahu!

Hoʻolana etcd

ʻO kā mākou pahuhopu nui ka holo ʻana i ka Kubernetes API, akā no ka hana ʻana i ia mea pono mākou e holo mua a pēlā aku. E hoʻomaka kākou i kahi puʻupuʻu etcd liʻiliʻi ma ke kau ʻana i kāna mau hoʻonohonoho i ka papa kuhikuhi pods (e laʻa, pods/etcd.yaml):

apiVersion: v1
kind: Pod
metadata:
  name: etcd
  namespace: kube-system
spec:
  containers:
  - name: etcd
    command:
    - etcd
    - --data-dir=/var/lib/etcd
    image: k8s.gcr.io/etcd:3.4.3-0
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
  hostNetwork: true
  volumes:
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data

Inā ʻoe i hana pū me Kubernetes, pono e kamaʻāina kēia mau faila YAML iā ʻoe. ʻElua wale nō mea e ʻike ʻia ma aneʻi:

Ua kau mākou i ka waihona host /var/lib/etcd i loko o ka pod i mālama ʻia ka ʻikepili etcd ma hope o ka hoʻomaka hou ʻana (inā ʻaʻole i hana ʻia kēia, e holoi ʻia ka mokuʻāina cluster i kēlā me kēia manawa e hoʻomaka hou ai ka pod, ʻaʻole maikaʻi no ka hoʻokomo ʻana i nā Kubernetes liʻiliʻi).

Ua hoʻokomo mākou hostNetwork: true. ʻO kēia hoʻonohonoho, he mea kupanaha, hoʻonohonoho i ka etcd e hoʻohana i ka pūnaewele hoʻokipa ma kahi o ka pūnaewele kūloko o ka pod (e maʻalahi kēia i ka server API e ʻike i ka hui etcd).

Hōʻike kahi hōʻike maʻalahi e holo maoli ana etcd ma localhost a mālama i ka ʻikepili i ka disk:

$ curl localhost:2379/version
{"etcdserver":"3.4.3","etcdcluster":"3.4.0"}
$ sudo tree /var/lib/etcd/
/var/lib/etcd/
└── member
    ├── snap
    │   └── db
    └── wal
        ├── 0.tmp
        └── 0000000000000000-0000000000000000.wal

E hoʻomaka ana i ke kikowaena API

ʻOi aku ka maʻalahi o ka holo ʻana i kahi kikowaena API Kubernetes. ʻO ka palena wale nō e pono e hala --etcd-servers, hana i kāu mea i manaʻo ai:

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - name: kube-apiserver
    command:
    - kube-apiserver
    - --etcd-servers=http://127.0.0.1:2379
    image: k8s.gcr.io/kube-apiserver:v1.18.5
  hostNetwork: true

E kau i kēia faila YAML i ka papa kuhikuhi pods, a e hoʻomaka ke kikowaena API. Ke nānā nei me curl e hōʻike ana e hoʻolohe ana ka Kubernetes API ma ke awa 8080 me ka hāmama loa - ʻaʻohe pono o ka hōʻoia!

$ curl localhost:8080/healthz
ok
$ curl localhost:8080/api/v1/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/pods",
    "resourceVersion": "59"
  },
  "items": []
}

(Eia hou, mai holo i kēia ma ka hana ʻana! Ua kāhāhā iki au no ka paʻa ʻole o ka hoʻonohonoho paʻamau. Akā ke manaʻo nei au ʻo ia ka mea e maʻalahi ai ka hoʻomohala ʻana a me ka hoʻāʻo ʻana.)

A, kahaha ʻoluʻolu, hana ʻo kubectl ma waho o ka pahu me ka ʻole o nā hoʻonohonoho hou aʻe!

$ ./kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:39:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
$ ./kubectl get pod
No resources found in default namespace.

pilikia

Akā inā ʻeli ʻoe i kahi hohonu, ʻike ʻia kekahi mea e hewa ana:

$ ./kubectl get pod -n kube-system
No resources found in kube-system namespace.

Ua hala nā pods static a mākou i hana ai! ʻOiaʻiʻo, ʻaʻole ʻike ʻia kā mākou kubelet node:

$ ./kubectl get nodes
No resources found in default namespace.

He aha ka pilikia? Inā hoʻomanaʻo ʻoe i kekahi mau paukū i hala aku nei, ua hoʻomaka mākou i ka kubelet me kahi hoʻonohonoho maʻalahi o nā palena laina kauoha, no laila ʻaʻole ʻike ka kubelet pehea e hoʻopili ai i ka server API a haʻi aku i kona mokuʻāina. Ma hope o ke aʻo ʻana i ka palapala, ʻike mākou i ka hae e pili ana:

--kubeconfig string

ʻO ke ala i ka faila kubeconfig, e kuhikuhi ana pehea e hoʻopili ai i ka kikowaena API. Loaʻa --kubeconfig hiki i ke ʻano kikowaena API, ʻaʻole --kubeconfig hiki i ke ʻano hoʻopahemo.

I kēia manawa a pau, me ka ʻike ʻole, ke holo nei mākou i ke kubelet ma ke ʻano "offline mode." (Inā mākou e hoʻopaʻa haʻahaʻa, hiki iā mākou ke noʻonoʻo i kahi kubelet kūʻokoʻa ma ke ʻano he "Kubernetes liʻiliʻi liʻiliʻi", akā ʻoluʻolu loa ia). No ka hana ʻana o ka hoʻonohonoho "ʻoiaʻiʻo", pono mākou e hāʻawi i ka faila kubeconfig i ka kubelet i ʻike ʻo ia pehea e kamaʻilio ai me ka server API. ʻO ka mea pōmaikaʻi, he maʻalahi loa ia (no ka mea, ʻaʻohe o mākou pilikia me ka hōʻoia a i ʻole nā ​​palapala hōʻoia):

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: http://127.0.0.1:8080
  name: mink8s
contexts:
- context:
    cluster: mink8s
  name: mink8s
current-context: mink8s

E mālama i kēia kubeconfig.yaml, pepehi i ka hana kubelet a hoʻomaka hou me nā ʻāpana pono:

$ sudo ./kubelet --pod-manifest-path=pods --kubeconfig=kubeconfig.yaml

(Ma ke ala, inā ʻoe e hoʻāʻo e komo i ka API ma o ka curl ke holo ʻole ka kubelet, e ʻike ʻoe e holo mau ana ia! ʻAʻole ʻo Kubelet he "makua" o kāna mau pods e like me Docker, ua like ia me ka "mana. daemon.

I loko o kekahi mau minuke kubectl e hōʻike iā mākou i nā pods a me nā nodes e like me kā mākou e manaʻo ai:

$ ./kubectl get pods -A
NAMESPACE     NAME                    READY   STATUS             RESTARTS   AGE
default       hello-mink8s            0/1     CrashLoopBackOff   261        21h
kube-system   etcd-mink8s             1/1     Running            0          21h
kube-system   kube-apiserver-mink8s   1/1     Running            0          21h
$ ./kubectl get nodes -owide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
mink8s   Ready    <none>   21h   v1.18.5   10.70.10.228   <none>        Ubuntu 18.04.4 LTS   4.15.0-109-generic   docker://19.3.6

E hoʻomaikaʻi maoli mākou iā mākou iho i kēia manawa (ʻike wau ua hoʻomaikaʻi mua wau iā mākou iho) - loaʻa iā mākou kahi "hui" Kubernetes liʻiliʻi e holo ana me kahi API hana piha!

Hoʻomaka mākou ma lalo

I kēia manawa e ʻike kākou i ka mea hiki i ka API. E hoʻomaka me ka nginx pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx

Maʻaneʻi mākou e loaʻa ai kahi hewa hoihoi:

$ ./kubectl apply -f nginx.yaml
Error from server (Forbidden): error when creating "nginx.yaml": pods "nginx" is
forbidden: error looking up service account default/default: serviceaccount
"default" not found
$ ./kubectl get serviceaccounts
No resources found in default namespace.

Ma ʻaneʻi ke ʻike nei mākou i ka piha ʻole o kā mākou puni Kubernetes - ʻaʻohe a mākou moʻokāki no nā lawelawe. E hoʻāʻo hou ma ka hana lima ʻana i kahi moʻokāki lawelawe a ʻike i ka mea e hana nei:

$ cat <<EOS | ./kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
EOS
serviceaccount/default created
$ ./kubectl apply -f nginx.yaml
Error from server (ServerTimeout): error when creating "nginx.yaml": No API
token found for service account "default", retry after the token is
automatically created and added to the service account

ʻOiai mākou i hana lima i ka moʻokāki lawelawe, ʻaʻole i hana ʻia ka hōʻailona hōʻoia. Ke hoʻomau nei mākou i ka hoʻokolohua me kā mākou minimalistic "cluster", e ʻike mākou e nalowale ka hapa nui o nā mea pono e hana maʻamau. ʻO ke kikowaena API Kubernetes he mea minimalistic, me ka hapa nui o ka hāpai kaumaha a me ka hoʻonohonoho maʻalahi e hana nei i nā mea hoʻoponopono like ʻole a me nā hana hope ʻaʻole i holo.

Hiki iā mākou ke hoʻoponopono i kēia pilikia ma ka hoʻonohonoho ʻana i ke koho automountServiceAccountToken no ka moʻokāki lawelawe (no ka mea ʻaʻole pono mākou e hoʻohana iā ia):

$ cat <<EOS | ./kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
automountServiceAccountToken: false
EOS
serviceaccount/default configured
$ ./kubectl apply -f nginx.yaml
pod/nginx created
$ ./kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   0/1     Pending   0          13m

ʻO ka hope, ua ʻike ʻia ka pod! Akā, ʻaʻole ia e hoʻomaka no ka mea ʻaʻohe o mākou mea hoʻolālā (scheduler) kekahi mea nui o Kubernetes. Eia hou, ʻike mākou he mea kupanaha ka Kubernetes API - ke hana ʻoe i Pod i ka API, hoʻopaʻa inoa ʻo ia, akā ʻaʻole hoʻāʻo e noʻonoʻo i ka node e holo ai.

ʻOiaʻiʻo, ʻaʻole pono ʻoe i kahi mea hoʻonohonoho e holo i kahi pod. Hiki iā ʻoe ke hoʻohui lima i kahi node i ka hōʻike ma ka ʻāpana nodeName:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  nodeName: mink8s

(Pai mink8s i ka inoa o ka node.) Ma hope o ka holoi ʻana a hoʻopili, ʻike mākou ua hoʻomaka ka nginx a ke hoʻolohe nei i ka IP address kūloko:

$ ./kubectl delete pod nginx
pod "nginx" deleted
$ ./kubectl apply -f nginx.yaml
pod/nginx created
$ ./kubectl get pods -owide
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          30s   172.17.0.2   mink8s   <none>           <none>
$ curl -s 172.17.0.2 | head -4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

No ka hōʻoia i ka hana pono ʻana o ka pūnaewele ma waena o nā pods, hiki iā mākou ke holo curl mai kahi pod ʻē aʻe:

$ cat <<EOS | ./kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl
spec:
  containers:
  - image: curlimages/curl
    name: curl
    command: ["curl", "172.17.0.2"]
  nodeName: mink8s
EOS
pod/curl created
$ ./kubectl logs curl | head -6
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

He mea hoihoi loa ka ʻeli ʻana i loko o kēia kaiapuni a ʻike i ka hana a me ka mea ʻole. Ua ʻike au i ka hana ʻo ConfigMap a me Secret e like me ka mea i manaʻo ʻia, akā ʻaʻole hana ʻo Service and Deployment.

Спех!

Ke lōʻihi nei kēia pou, no laila e haʻi wau i ka lanakila a ʻōlelo wau he hoʻonohonoho kūpono kēia e hiki ke kapa ʻia ʻo "Kubernetes". No ka hōʻuluʻulu ʻana: ʻehā binaries, ʻelima mau laina kauoha a "wale" 45 laina o YAML (ʻaʻole. e like me nā kūlana Kubernetes) a he mau mea liʻiliʻi kā mākou e hana nei:

  • Hoʻokele ʻia nā Pods me ka hoʻohana ʻana i ka Kubernetes API maʻamau (me kekahi mau hacks)
  • Hiki iā ʻoe ke hoʻouka a mālama i nā kiʻi pahu lehulehu
  • Noho mau nā Pods a hoʻomaka hou
  • Hoʻohana maikaʻi ka ʻoihana ma waena o nā pods i loko o ka node hoʻokahi
  • ConfigMap, huna a me ka maʻalahi o ka mālama ʻana i ka hana e like me ka mea i manaʻo ʻia

Akā ʻaʻole nalo ka hapa nui o nā mea e pono ai ʻo Kubernetes, e like me:

  • Pod Scheduler
  • Hōʻoiaʻiʻo/ʻae
  • He mau node
  • Pūnaewele o nā lawelawe
  • Pūʻulu DNS kūloko
  • Nā mea hoʻomalu no nā moʻokāki lawelawe, hoʻolālā, hoʻohui pū me nā mea hoʻolako kapua a me ka hapa nui o nā mea maikaʻi ʻē aʻe a Kubernetes e lawe mai ai.

No laila he aha ka mea i loaʻa iā mākou? ʻO ka Kubernetes API, e holo ana iā ia iho, he kahua wale nō ia no mīkini pahu pahu. ʻAʻole ia e hana nui - he hana ia no nā mea hoʻoponopono like ʻole a me nā mea hoʻohana e hoʻohana ana i ka API - akā hāʻawi ia i kahi ʻano kūlike no ka automation.

E aʻo hou e pili ana i ka papa ma ka webinar manuahi.

E heluhelu hou:

Source: www.habr.com

Pākuʻi i ka manaʻo hoʻopuka