ProHoster > Pūnaewele > Nā Administration > Hoʻonohonoho i kahi pūʻulu Nomad me ka hoʻohana ʻana i ke Consul a hoʻohui pū me Gitlab

Hoʻonohonoho i kahi pūʻulu Nomad me ka hoʻohana ʻana i ke Consul a hoʻohui pū me Gitlab

Hōʻike

I kēia mau lā, ua ulu wikiwiki ka kaulana o Kubernetes - ʻoi aku ka nui o nā papahana e hoʻokō nei. Makemake au e hoʻopā aku i kahi orchestrator e like me Nomad: kūpono ia no nā papahana i hoʻohana mua i nā hoʻonā ʻē aʻe mai HashiCorp, no ka laʻana, Vault a me Consul, a ʻaʻole paʻakikī nā papahana iā lākou iho i ka ʻōlelo o ka ʻoihana. Loaʻa i kēia mea nā ʻōlelo kuhikuhi no ka hoʻokomo ʻana iā Nomad, e hui pū ana i ʻelua nodes i loko o kahi pūpū, a me ka hoʻohui ʻana iā Nomad me Gitlab.

Hoʻonohonoho i kahi pūʻulu Nomad me ka hoʻohana ʻana i ke Consul a hoʻohui pū me Gitlab

Kū hoʻāʻo

He mea liʻiliʻi e pili ana i ka papa hoʻāʻo: ʻekolu mau kikowaena virtual i hoʻohana ʻia me nā hiʻohiʻona o 2 CPU, 4 RAM, 50 Gb SSD, i hui pū ʻia i kahi pūnaewele kūloko maʻamau. ʻO kā lākou mau inoa a me nā helu IP:

  1. nomad-livelinux-01: 172.30.0.5
  2. nomad-livelinux-02: 172.30.0.10
  3. consul-livelinux-01: 172.30.0.15

Hoʻokomo ʻia ʻo Nomad, Kanikela. Ke hana ʻana i kahi pūʻulu Nomad

E hoʻomaka kākou me ka hoʻonohonoho kumu. ʻOiai ua maʻalahi ka hoʻonohonoho ʻana, e wehewehe wau no ka pono o ka ʻatikala: ua hana maoli ʻia ia mai nā kikoʻī a me nā memo no ke komo wikiwiki ʻana inā pono.

Ma mua o ka hoʻomaka ʻana i ka hoʻomaʻamaʻa, e kūkākūkā mākou i ka ʻāpana theoretical, no ka mea ma kēia pae he mea nui e hoʻomaopopo i ke ʻano o ka wā e hiki mai ana.

Loaʻa iā mākou ʻelua nomad nodes a makemake mākou e hoʻohui iā lākou i loko o kahi pūʻulu, a i ka wā e hiki mai ana e pono ai mākou i ka scaling cluster maʻalahi - no kēia e pono mākou i ke Kanikela. Me kēia hāmeʻa, lilo ka hui ʻana a me ka hoʻohui ʻana i nā node hou i mea maʻalahi loa: ʻo ka Nomad node i hana ʻia e hoʻopili i ka ʻelele Consul, a laila hoʻopili i ka pūʻulu Nomad e kū nei. No laila, i ka hoʻomaka ʻana e hoʻokomo mākou i ke kikowaena Consul, e hoʻonohonoho i ka mana http kumu no ka ʻaoʻao pūnaewele (ʻaʻole ʻae ʻia ma ke ʻano maʻamau a hiki ke kiʻi ʻia ma kahi helu waho), a me nā ʻelele o ke Kanikela ma nā kikowaena Nomad, a laila e hele wale mākou i Nomad.

He mea maʻalahi loa ka hoʻokomo ʻana i nā mea hana a HashiCorp: ʻo ka mea nui, neʻe wale mākou i ka faila binary i ka papa kuhikuhi bin, hoʻonohonoho i ka faila hoʻonohonoho o ka mea hana, a hana i kāna faila lawelawe.

E hoʻoiho i ka waihona binary Consul a wehe i loko o ka papa kuhikuhi home o ka mea hoʻohana:

root@consul-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# mv consul /usr/local/bin/

I kēia manawa, loaʻa iā mākou kahi consul binary i mākaukau no ka hoʻonohonoho hou ʻana.

No ka hana pū me ke Kanikela, pono mākou e hana i kahi kī kū hoʻokahi me ka hoʻohana ʻana i ke kauoha keygen:

root@consul-livelinux-01:~# consul keygen

E neʻe kāua i ka hoʻonohonoho ʻana i ka hoʻonohonoho Konsul, e hana ana i kahi papa kuhikuhi /etc/consul.d/ me kēia ʻano:

/etc/consul.d/
├── bootstrap
│   └── config.json

Loaʻa i ka papa kuhikuhi bootstrap kahi faila hoʻonohonoho config.json - i loko e hoʻonohonoho mākou i nā hoʻonohonoho Consul. ʻO nā mea i loko:

{
"bootstrap": true,
"server": true,
"datacenter": "dc1",
"data_dir": "/var/consul",
"encrypt": "your-key",
"log_level": "INFO",
"enable_syslog": true,
"start_join": ["172.30.0.15"]
}

E nānā kaʻawale i nā kuhikuhi nui a me nā manaʻo:

  • bootstrap: oiaio. Hiki iā mākou ke hoʻohui aunoa i nā nodes hou inā pili lākou. ʻIke wau ʻaʻole mākou e hōʻike ma aneʻi i ka helu pololei o nā nodes i manaʻo ʻia.
  • kikowaena: oiaio. E hoʻā i ke ʻano kikowaena. ʻO ke kanikela ma kēia mīkini virtual e hana ʻo ia wale nō ke kikowaena a me ka haku i kēia manawa, ʻo Nomad's VM ka mea kūʻai aku.
  • 'ike helu: dc1. E wehewehe i ka inoa o ke kikowaena ʻikepili e hana i ka pūʻulu. Pono e like me nā mea kūʻai aku a me nā kikowaena.
  • hoʻopā'ālua: kou-ki. ʻO ke kī, pono e kū hoʻokahi a kūlike i nā mea kūʻai aku a me nā kikowaena. Hana ʻia me ke kauoha consul keygen.
  • hoʻomaka_hui. Ma kēia papa inoa mākou e hōʻike i kahi papa inoa o nā IP address kahi e hoʻopili ai. I kēia manawa ke waiho wale nei mākou i kā mākou ʻōlelo ponoʻī.

I kēia manawa hiki iā mākou ke holo consul me ka hoʻohana ʻana i ka laina kauoha:

root@consul-livelinux-01:~# /usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui

He ala maikaʻi kēia e debug i kēia manawa, akā naʻe, ʻaʻole hiki iā ʻoe ke hoʻohana i kēia ʻano ma ke kumu mau no nā kumu maopopo. E hana mākou i faila lawelawe e hoʻokele i ke Kanikela ma o systemd:

root@consul-livelinux-01:~# nano /etc/systemd/system/consul.service

Nā mea i loko o ka waihona consul.service: 

[Unit]
Description=Consul Startup process
After=network.target
 
[Service]
Type=simple
ExecStart=/bin/bash -c '/usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui' 
TimeoutStartSec=0
 
[Install]
WantedBy=default.target

E hoʻokuʻu i ke Kanikela ma o systemctl:

root@consul-livelinux-01:~# systemctl start consul

E nānā: pono e holo kā mākou lawelawe, a ma ka hoʻokō ʻana i ke kauoha a nā lālā kanikela e ʻike mākou i kā mākou kikowaena:

root@consul-livelinux:/etc/consul.d# consul members
consul-livelinux    172.30.0.15:8301  alive   server  1.5.0  2         dc1  <all>

ʻO ka pae aʻe: hoʻokomo iā Nginx a hoʻonohonoho i ka proxying a me ka mana http. Hoʻokomo mākou i ka nginx ma o ka luna pūʻolo a ma ka /etc/nginx/sites-enabled directory hana mākou i kahi faila consul.conf me kēia mau mea:

upstream consul-auth {
    server localhost:8500;
}

server {

    server_name consul.doman.name;
    
    location / {
      proxy_pass http://consul-auth;
      proxy_set_header Host $host;
      auth_basic_user_file /etc/nginx/.htpasswd;
      auth_basic "Password-protected Area";
    }
}

Mai poina e hana i kahi faila .htpasswd a hana i kahi inoa inoa a me ka ʻōlelo huna no ia mea. Pono kēia mea i ʻole e loaʻa ka papa pūnaewele i nā mea a pau i ʻike i kā mākou kikowaena. Eia naʻe, i ka hoʻonohonoho ʻana iā Gitlab, pono mākou e haʻalele i kēia - inā ʻaʻole hiki iā mākou ke kau i kā mākou noi iā Nomad. Ma kaʻu papahana, aia ʻo Gitlab a me Nomad ma ka pūnaewele hina, no laila ʻaʻohe pilikia ma aneʻi.

Ma nā kikowaena ʻelua i koe, hoʻokomo mākou i nā ʻelele Consul e like me nā ʻōlelo aʻe. Hana hou mākou i nā ʻanuʻu me ka faila binary:

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# mv consul /usr/local/bin/

Ma ka hoʻohālikelike me ke kikowaena mua, hana mākou i kahi papa kuhikuhi no nā faila hoʻonohonoho /etc/consul.d me kēia ʻano:

/etc/consul.d/
├── client
│   └── config.json

Nā mea o ka waihona config.json:

{
    "datacenter": "dc1",
    "data_dir": "/opt/consul",
    "log_level": "DEBUG",
    "node_name": "nomad-livelinux-01",
    "server": false,
    "encrypt": "your-private-key",
    "domain": "livelinux",
    "addresses": {
      "dns": "127.0.0.1",
      "https": "0.0.0.0",
      "grpc": "127.0.0.1",
      "http": "127.0.0.1"
    },
    "bind_addr": "172.30.0.5", # локальный адрес вм
    "start_join": ["172.30.0.15"], # удаленный адрес консул сервера
    "ports": {
      "dns": 53
     }

E mālama i nā hoʻololi a neʻe i ka hoʻonohonoho ʻana i ka faila lawelawe, nā mea i loko:

/etc/systemd/system/consul.service:

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/client
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

Hoʻomaka mākou i ke kanikela ma ke kikowaena. I kēia manawa, ma hope o ka hoʻomaka ʻana, pono mākou e ʻike i ka lawelawe i hoʻonohonoho ʻia i nā lālā nsul. ʻO ia ke ʻano ua pili pono ʻo ia i ka pūʻulu ma ke ʻano he mea kūʻai aku. E hana hou i ka mea like ma ka lua o ke kikowaena a ma hope o kēlā hiki iā mākou ke hoʻomaka e hoʻokomo a hoʻonohonoho iā Nomad.

Hōʻike ʻia ka hoʻonohonoho kikoʻī ʻana o Nomad i kāna palapala mana. ʻElua mau ala hoʻonohonoho kuʻuna: hoʻoiho ʻana i kahi faila binary a me ka hoʻohui ʻana mai ke kumu. E koho au i ke ala mua.

i hoʻopuka: Ke ulu wikiwiki nei ka papahana, hoʻokuʻu pinepine ʻia nā mea hou. E hoʻokuʻu ʻia paha kahi mana hou i ka wā e pau ai kēia ʻatikala. No laila, ma mua o ka heluhelu ʻana, manaʻo wau e nānā i ka mana o kēia manawa o Nomad i kēia manawa a hoʻoiho iā ia. 

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/nomad/0.9.1/nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# unzip nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# mv nomad /usr/local/bin/
root@nomad-livelinux-01:~# nomad -autocomplete-install
root@nomad-livelinux-01:~# complete -C /usr/local/bin/nomad nomad
root@nomad-livelinux-01:~# mkdir /etc/nomad.d

Ma hope o ka wehe ʻana, e loaʻa iā mākou kahi faila binary Nomad ma ke kaupaona ʻana he 65 MB - pono e neʻe ʻia i /usr/local/bin.

E hana kāua i papa kuhikuhi ʻikepili no Nomad a hoʻoponopono i kāna faila lawelawe (ʻaʻole paha e noho ma ka hoʻomaka):

root@nomad-livelinux-01:~# mkdir --parents /opt/nomad
root@nomad-livelinux-01:~# nano /etc/systemd/system/nomad.service

Hoʻopili i kēia mau laina ma laila:

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity

[Install]
WantedBy=multi-user.target

Eia naʻe, ʻaʻole mākou wikiwiki e hoʻomaka i ka nomad - ʻaʻole mākou i hana i kāna faila hoʻonohonoho:

root@nomad-livelinux-01:~# mkdir --parents /etc/nomad.d
root@nomad-livelinux-01:~# chmod 700 /etc/nomad.d
root@nomad-livelinux-01:~# nano /etc/nomad.d/nomad.hcl
root@nomad-livelinux-01:~# nano /etc/nomad.d/server.hcl

Penei ka papa kuhikuhi hope:

/etc/nomad.d/
├── nomad.hcl
└── server.hcl

Pono e loaʻa i ka faila nomad.hcl kēia hoʻonohonoho hoʻonohonoho:

datacenter = "dc1"
data_dir = "/opt/nomad"

Nā mea o ka waihona server.hcl:

server {
  enabled = true
  bootstrap_expect = 1
}

consul {
  address             = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
}

bind_addr = "127.0.0.1" 

advertise {
  http = "172.30.0.5"
}

client {
  enabled = true
}

Mai poina e hoʻololi i ka faila hoʻonohonoho ma ke kikowaena lua - pono ʻoe e hoʻololi i ka waiwai o ka kuhikuhi http.

ʻO ka mea hope loa i kēia manawa, ʻo ia ka hoʻonohonoho ʻana iā Nginx no ke koho ʻana a me ka hoʻonohonoho ʻana i ka mana http. Nā mea o ka waihona nomad.conf:

upstream nomad-auth {
        server 172.30.0.5:4646;
}

server {

        server_name nomad.domain.name;
        
        location / {
	        proxy_pass http://nomad-auth;
	        proxy_set_header Host $host;
	        auth_basic_user_file /etc/nginx/.htpasswd;
		   auth_basic "Password-protected Area";
        }
        
}

I kēia manawa hiki iā mākou ke komo i ka pūnaewele pūnaewele ma o kahi pūnaewele waho. Hoʻohui a hele i ka ʻaoʻao kikowaena:

Hoʻonohonoho i kahi pūʻulu Nomad me ka hoʻohana ʻana i ke Consul a hoʻohui pū me Gitlab
Kiʻi 1. Ka papa inoa o nā kikowaena ma ka pūʻulu Nomad

Hōʻike maikaʻi ʻia nā kikowaena ʻelua i ka panel, e ʻike mākou i ka mea like i ka puka o ke kauoha kūlana nomad node:

Hoʻonohonoho i kahi pūʻulu Nomad me ka hoʻohana ʻana i ke Consul a hoʻohui pū me Gitlab
Kiʻi 2. Puka o ka nomad node kūlana kauoha

Pehea ke Kanikela? E nānā kāua. E hele i ka ʻaoʻao hoʻomalu ʻo Consul, i ka ʻaoʻao nodes:
Hoʻonohonoho i kahi pūʻulu Nomad me ka hoʻohana ʻana i ke Consul a hoʻohui pū me Gitlab
Kiʻi 3. Ka papa inoa o nā nodes ma ka hui ʻo Consul

I kēia manawa, loaʻa iā mākou kahi Nomad mākaukau e hana pū ana me ke Kanikela. I ka pae hope, e hele mākou i ka ʻāpana leʻaleʻa: hoʻonohonoho i ka lawe ʻana i nā pahu Docker mai Gitlab a i Nomad, a kamaʻilio pū kekahi o kāna mau hiʻohiʻona ʻokoʻa.

Ke hoʻokumu nei iā Gitlab Runner

No ke kau ʻana i nā kiʻi docker iā Nomad, e hoʻohana mākou i kahi mea holo kaʻawale me ka faila binary Nomad i loko (ma ʻaneʻi, ma ke ala, hiki iā mākou ke ʻike i kahi hiʻohiʻona ʻē aʻe o nā noi Hashicorp - ʻo kēlā me kēia mea he faila binary hoʻokahi). Hoʻouka iā ia i ka papa kuhikuhi runner. E hana mākou i kahi Dockerfile maʻalahi no ia me kēia ʻike:


FROM alpine:3.9
RUN apk add --update --no-cache libc6-compat gettext
COPY nomad /usr/local/bin/nomad

Ma ka papahana like mākou e hana ai .gitlab-ci.yml:

variables:
  DOCKER_IMAGE: nomad/nomad-deploy
  DOCKER_REGISTRY: registry.domain.name
 

stages:
  - build

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:latest
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}

ʻO ka hopena, e loaʻa iā mākou kahi kiʻi i loaʻa o ka mea holo Nomad ma ka Gitlab Registry, i kēia manawa hiki iā mākou ke hele pololei i ka waihona papahana, hana i kahi Pipeline a hoʻonohonoho i ka hana nomad a Nomad.

Hoʻonohonoho papahana

E hoʻomaka kākou me ka faila o ka hana no Nomad. ʻO kaʻu papahana ma kēia ʻatikala he mea kahiko loa ia: hoʻokahi hana. Penei nā mea o .gitlab-ci:

variables:
  NOMAD_ADDR: http://nomad.address.service:4646
  DOCKER_REGISTRY: registry.domain.name
  DOCKER_IMAGE: example/project

stages:
  - build
  - deploy

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad-runner/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:${CI_COMMIT_SHORT_SHA}
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}


deploy:
  stage: deploy
  image: registry.example.com/nomad/nomad-runner:latest
  script:
    - envsubst '${CI_COMMIT_SHORT_SHA}' < project.nomad > job.nomad
    - cat job.nomad
    - nomad validate job.nomad
    - nomad plan job.nomad || if [ $? -eq 255 ]; then exit 255; else echo "success"; fi
    - nomad run job.nomad
  environment:
    name: production
  allow_failure: false
  when: manual

Maʻaneʻi ka hoʻolālā lima, akā hiki iāʻoe ke hoʻonohonoho iā ia e hoʻololi i nā mea o ka papa kuhikuhi papahana. ʻElua mau ʻanuʻu ʻo Pipeline: hui kiʻi a me kona hoʻolaha ʻana i ka nomad. I ka pae mua, hōʻuluʻulu mākou i kahi kiʻi docker a paʻi i loko o kā mākou Registry, a ma ka lua e hoʻomaka mākou i kā mākou hana ma Nomad. 

job "monitoring-status" {
    datacenters = ["dc1"]
    migrate {
        max_parallel = 3
        health_check = "checks"
        min_healthy_time = "15s"
        healthy_deadline = "5m"
    }

    group "zhadan.ltd" {
        count = 1
        update {
            max_parallel      = 1
            min_healthy_time  = "30s"
            healthy_deadline  = "5m"
            progress_deadline = "10m"
            auto_revert       = true
        }
        task "service-monitoring" {
            driver = "docker"

            config {
                image = "registry.domain.name/example/project:${CI_COMMIT_SHORT_SHA}"
                force_pull = true
                auth {
                    username = "gitlab_user"
                    password = "gitlab_password"
                }
                port_map {
                    http = 8000
                }
            }
            resources {
                network {
                    port "http" {}
                }
            }
        }
    }
}

E ʻoluʻolu e hoʻomaopopo he Registry pilikino kaʻu a no ka huki ʻana i kahi kiʻi docker pono wau e komo i laila. ʻO ka hopena maikaʻi loa i kēia hihia ʻo ke komo ʻana i kahi login a me ka ʻōlelo huna i loko o Vault a laila hoʻohui pū me Nomad. Kākoʻo maoli ʻo Nomad iā Vault. Akā ʻo ka mea mua, e hoʻokomo i nā kulekele pono no Nomad ma Vault ponoʻī; hiki iā lākou ke hoʻoiho:

# Download the policy and token role
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L

# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl

# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json

I kēia manawa, i ka hana ʻana i nā kulekele pono, e hoʻohui mākou i ka hoʻohui ʻana me Vault i ka papa hana ma ka faila job.nomad:

vault {
  enabled = true
  address = "https://vault.domain.name:8200"
  token = "token"
}

Ke hoʻohana nei au i ka ʻae ʻia ma ka hōʻailona a hoʻopaʻa inoa pololei ma aneʻi, aia nō ke koho o ka wehewehe ʻana i ka hōʻailona ma ke ʻano he hoʻololi i ka wā e hoʻomaka ai i ka ʻelele nomad:

$ VAULT_TOKEN=<token> nomad agent -config /path/to/config

I kēia manawa hiki iā mākou ke hoʻohana i nā kī me Vault. He mea maʻalahi ke kumumanaʻo o ka hana: hana mākou i kahi faila ma Nomad job e mālama i nā waiwai o nā loli, no ka laʻana:

template {
                data = <<EOH
{{with secret "secrets/pipeline-keys"}}
REGISTRY_LOGIN="{{ .Data.REGISTRY_LOGIN }}"
REGISTRY_PASSWORD="{{ .Data.REGISTRY_LOGIN }}{{ end }}"

EOH
    destination = "secrets/service-name.env"
    env = true
}

Me kēia ala maʻalahi, hiki iā ʻoe ke hoʻonohonoho i ka lawe ʻana i nā ipu i ka pūʻulu Nomad a hana pū me ia i ka wā e hiki mai ana. E ʻōlelo wau i kekahi manawa ke aloha nei au iā Nomad - ʻoi aku ka maikaʻi no nā papahana liʻiliʻi kahi e hiki ai iā Kubernetes ke hoʻonui i ka paʻakikī a ʻaʻole ʻike i kona hiki piha. Hoʻohui, kūpono ʻo Nomad no ka poʻe hoʻomaka - maʻalahi ke hoʻouka a hoʻonohonoho. Eia nō naʻe, i ka wā e hoʻāʻo ai i kekahi mau papahana, loaʻa iaʻu kahi pilikia me kāna mau mana mua - ʻaʻole nui nā hana maʻamau i laila a ʻaʻole hana pololei. Eia naʻe, ke manaʻoʻiʻo nei au e hoʻomau ka ulu ʻana o Nomad a i ka wā e hiki mai ana e loaʻa iā ia nā hana e pono ai nā mea a pau.

Mea kākau: Ilya Andreev, hoʻoponopono ʻia e Alexey Zhadan a me ka hui Live Linux


Source: www.habr.com

Pākuʻi i ka manaʻo hoʻopuka