🥇Fa'atūina se fuifui Nomad e fa'aaoga ai le Konesula ma fa'atasi ma Gitlab

Faatomuaga

Talu ai nei, o le taʻutaʻua o Kubernetes ua faʻavavevave ona faʻatupulaia - o loʻo faʻateleina galuega faatino o loʻo faʻatinoina. Na ou manaʻo e paʻi atu i se tagata faʻapipiʻi pei o Nomad: e lelei mo galuega faatino ua uma ona faʻaogaina isi fofo mai HashiCorp, mo se faʻataʻitaʻiga, Vault ma Consul, ma o galuega lava latou e le faigata i tulaga o atinaʻe. O lenei mea o le a aofia ai faʻatonuga mo le faʻapipiʻiina o Nomad, faʻapipiʻi lua nodes i se fuifui, faʻapea foʻi ma le tuʻufaʻatasia o Nomad ma Gitlab.

nofoa su'ega

O sina mea itiiti e uiga i le suʻega suʻega: tolu sapalai mataʻutia o loʻo faʻaaogaina ma uiga o le 2 PPU, 4 RAM, 50 Gb SSD, faʻatasi i se fesoʻotaʻiga masani i le lotoifale. O latou igoa ma tuatusi IP:

nomad-livelinux-01: 172.30.0.5
nomad-livelinux-02: 172.30.0.10
consul-livelinux-01: 172.30.0.15

Fa'atuina o Nomad, Konesula. Fausia se fuifui Nomad

Tatou amata i le fa'apipi'i fa'avae. E ui ina faigofie le seti, ae o le a ou faʻamatalaina mo le faʻamaoni o le tusiga: na faia moni lava mai faʻataʻitaʻiga ma faʻamatalaga mo le vave maua pe a manaʻomia.

Ae tatou te leʻi amata faʻataʻitaʻiga, o le a tatou talanoaina le vaega faʻapitoa, aua i lenei laasaga e taua le malamalama i le fausaga o le lumanaʻi.

E lua a matou node nomad ma matou te mananaʻo e tuʻufaʻatasia i latou i se fuifui, ma i le lumanaʻi matou te manaʻomia foʻi le faʻapipiʻiina otometi - o le mea lea matou te manaʻomia ai le Konesula. Faatasi ai ma lenei meafaigaluega, o le faʻapipiʻiina ma le faʻaopoopoina o nodes fou e avea o se galuega faigofie tele: o le Nomad na fausia e fesoʻotaʻi ma le sui Konesula, ona faʻafesoʻotaʻi lea i le Nomad cluster o iai. O le mea lea, i le amataga o le a matou faʻapipiʻi le Consul server, faʻapipiʻi le faʻatagaina http faavae mo le upega tafaʻilagi (e aunoa ma se faʻatagaga ona o le faaletonu ma e mafai ona maua i se tuatusi fafo), faʻapea foʻi ma sui o le Konesula i latou lava i luga o Nomad servers, mulimuli ane. o le a tatou agai atu i Nomad.

O le faʻapipiʻiina o meafaigaluega a HashiCorp e matua faigofie lava: o le mea moni, matou te faʻanofo le faila binary i le lisi o talone, seti le faila o le mea faigaluega, ma fatuina lana faila tautua.

La'u mai le faila binary Consul ma tatala i totonu o le lisi o le fale a le tagata fa'aoga:

root@consul-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# mv consul /usr/local/bin/

O lea la ua i ai le matou consul binary ua saunia mo le fa'aopoopo atili.

Ina ia galulue ma le Konesula, tatou te manaʻomia le fatuina o se ki tulaga ese e faʻaaoga ai le keygen command:

root@consul-livelinux-01:~# consul keygen

Sei o tatou agai i luma i le setiina o le Consul configuration, fatuina o se directory /etc/consul.d/ ma le fausaga nei:

/etc/consul.d/
├── bootstrap
│   └── config.json

O le bootstrap directory o le a aofia ai se faila faʻatulagaina config.json - i totonu o le a matou setiina ai le Consul settings. O mea o iai:

{
"bootstrap": true,
"server": true,
"datacenter": "dc1",
"data_dir": "/var/consul",
"encrypt": "your-key",
"log_level": "INFO",
"enable_syslog": true,
"start_join": ["172.30.0.15"]
}

Se'i o tatou va'ava'ai i ta'iala autu ma o latou uiga eseese:

bootstrap: moni. Matou te fa'atagaina le fa'aopoopoina otometi o nodes fou pe a feso'ota'i. Ou te maitauina tatou te le o faʻaalia iinei le numera saʻo o nodes faʻamoemoeina.
faigaluega: moni. Fa'amalo le faiga o le server. Konesula i luga o lenei masini komepiuta o le a galue e na o le pau lea o le server ma le matai i le taimi nei, o le Nomad's VM o le a avea ma tagata faʻatau.
datacenter: dc1. Fa'ailoa le igoa ole nofoaga autu e fai ai le fuifui. E tatau ona tutusa i tagata fa'atau ma 'au'aunaga.
faʻailoga: lau-ki. Le ki, lea e tatau foi ona tulaga ese ma fetaui i luga o tagata uma ma sapalai. Fausia i le fa'aaogaina o le consul keygen command.
start_join. I lenei lisi matou te faʻaalia se lisi o tuatusi IP o le a faia ai le fesoʻotaʻiga. I le taimi nei matou te tuua na o matou lava tuatusi.

I le taimi nei e mafai ona matou taʻavale consul e faʻaaoga ai le laina o le poloaiga:

root@consul-livelinux-01:~# /usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui

O se auala lelei lea e faʻafefe ai le taimi nei, peitaʻi, o le a le mafai ona e faʻaogaina lenei metotia i se faʻaauau pea mo mafuaaga manino. Sei o tatou fatuina se faila tautua e pulea le Konesula e ala ile systemd:

root@consul-livelinux-01:~# nano /etc/systemd/system/consul.service

Anotusi o le faila consul.service:

[Unit]
Description=Consul Startup process
After=network.target
 
[Service]
Type=simple
ExecStart=/bin/bash -c '/usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui' 
TimeoutStartSec=0
 
[Install]
WantedBy=default.target

Tatala Konesula e ala ile systemctl:

root@consul-livelinux-01:~# systemctl start consul

Sei o tatou siaki: e tatau ona tamoʻe la tatou auaunaga, ma e ala i le faʻatinoina o le faʻatonuga a sui o le konesula e tatau ona tatou vaʻai i la tatou 'auʻaunaga:

root@consul-livelinux:/etc/consul.d# consul members
consul-livelinux    172.30.0.15:8301  alive   server  1.5.0  2         dc1  <all>

Laasaga e sosoo ai: faʻapipiʻi Nginx ma faʻatulagaina le sui ma le faʻatagaina http. Matou te faʻapipiʻi le nginx e ala i le pule o pusa ma i totonu o le /etc/nginx/sites-enabled directory matou te fatuina se faila faila consul.conf ma mea nei:

upstream consul-auth {
    server localhost:8500;
}

server {

    server_name consul.doman.name;
    
    location / {
      proxy_pass http://consul-auth;
      proxy_set_header Host $host;
      auth_basic_user_file /etc/nginx/.htpasswd;
      auth_basic "Password-protected Area";
    }
}

Aua ne'i galo e fai se faila .htpasswd ma fa'atupu se igoa fa'aigoa ma upu fa'aulu. O lenei mea e mana'omia ina ia le avanoa le laulau i luga ole laiga mo tagata uma e iloa la tatou vaega. Ae peitaʻi, pe a faʻatulagaina Gitlab, e tatau ona tatou lafoaʻia lenei mea - a leai o le a le mafai ona faʻaogaina la tatou talosaga i Nomad. I laʻu poloketi, o Gitlab ma Nomad o loʻo i luga o le upega tafaʻilagi, o lea e leai se faʻafitauli faʻapea iinei.

I luga o isi 'au'aunaga e lua matou te fa'apipi'i sui Konesula e tusa ai ma fa'atonuga nei. Matou te toe faia laasaga i le faila binary:

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# mv consul /usr/local/bin/

E ala i le faʻatusa ma le server muamua, matou te fatuina se lisi mo faila faila /etc/consul.d faʻatasi ai ma le fausaga o loʻo i lalo:

/etc/consul.d/
├── client
│   └── config.json

I totonu o le faila config.json:

{
    "datacenter": "dc1",
    "data_dir": "/opt/consul",
    "log_level": "DEBUG",
    "node_name": "nomad-livelinux-01",
    "server": false,
    "encrypt": "your-private-key",
    "domain": "livelinux",
    "addresses": {
      "dns": "127.0.0.1",
      "https": "0.0.0.0",
      "grpc": "127.0.0.1",
      "http": "127.0.0.1"
    },
    "bind_addr": "172.30.0.5", # локальный адрес вм
    "start_join": ["172.30.0.15"], # удаленный адрес консул сервера
    "ports": {
      "dns": 53
     }

Faasaoina suiga ma agai i luma i le setiina o le faila o le tautua, o mea e aofia ai:

/etc/systemd/system/consul.service:

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/client
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

Matou te tatalaina le konesula i luga o le server. I le taimi nei, a maeʻa le faʻalauiloaina, e tatau ona tatou vaʻai i le faʻatulagaina auaunaga i sui o le nsul. O lona uiga ua manuia le feso'ota'i atu i le fuifui o se tagata o tausia. Toe fai tutusa i luga o le server lona lua ma a maeʻa e mafai ona tatou amata faʻapipiʻi ma faʻapipiʻi Nomad.

O loʻo faʻamatalaina atili le faʻapipiʻiina o Nomad i ana faʻamaumauga aloaia. E lua auala faʻapipiʻi masani: laʻuina o se faila faila ma faʻapipiʻi mai le puna. O le a ou filifilia le auala muamua.

mataʻi: O le poloketi o loʻo atinaʻe vave, faʻafouga fou e masani ona faʻasalalau. Masalo o le a tatalaina se lomiga fou i le taimi e maeʻa ai lenei tusiga. O le mea lea, aʻo leʻi faitau, ou te fautuaina le siakiina o le taimi nei o Nomad i le taimi nei ma sii mai.

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/nomad/0.9.1/nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# unzip nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# mv nomad /usr/local/bin/
root@nomad-livelinux-01:~# nomad -autocomplete-install
root@nomad-livelinux-01:~# complete -C /usr/local/bin/nomad nomad
root@nomad-livelinux-01:~# mkdir /etc/nomad.d

A maeʻa ona tatala, o le a matou mauaina se faila faila Nomad e mamafa le 65 MB - e tatau ona siitia i /usr/local/bin.

Sei o tatou fatuina se lisi o faʻamaumauga mo Nomad ma faʻasaʻo lana faila faila (e foliga mai e le o iai i le amataga):

root@nomad-livelinux-01:~# mkdir --parents /opt/nomad
root@nomad-livelinux-01:~# nano /etc/systemd/system/nomad.service

Faapipii laina nei iina:

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity

[Install]
WantedBy=multi-user.target

Ae ui i lea, matou te le faanatinati e faʻalauiloa le nomad - matou te leʻi faia lana faila faila:

root@nomad-livelinux-01:~# mkdir --parents /etc/nomad.d
root@nomad-livelinux-01:~# chmod 700 /etc/nomad.d
root@nomad-livelinux-01:~# nano /etc/nomad.d/nomad.hcl
root@nomad-livelinux-01:~# nano /etc/nomad.d/server.hcl

O le fa'asologa mulimuli o fa'amaumauga o le a fa'apea:

/etc/nomad.d/
├── nomad.hcl
└── server.hcl

O le faila nomad.hcl e tatau ona i ai le faʻatulagaga nei:

datacenter = "dc1"
data_dir = "/opt/nomad"

Anotusi o le server.hcl faila:

server {
  enabled = true
  bootstrap_expect = 1
}

consul {
  address             = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
}

bind_addr = "127.0.0.1" 

advertise {
  http = "172.30.0.5"
}

client {
  enabled = true
}

Aua neʻi galo e sui le faila faʻapipiʻi i luga o le server lona lua - o iina e te manaʻomia ai le suia o le tau o le http directive.

O le mea mulimuli i lenei laasaga o le faʻatulagaina o Nginx mo le sui ma le faʻatulagaina o le faʻatagaina http. Anotusi o le faila nomad.conf:

upstream nomad-auth {
        server 172.30.0.5:4646;
}

server {

        server_name nomad.domain.name;
        
        location / {
	        proxy_pass http://nomad-auth;
	        proxy_set_header Host $host;
	        auth_basic_user_file /etc/nginx/.htpasswd;
		   auth_basic "Password-protected Area";
        }
        
}

Ole taimi nei e mafai ona tatou fa'aogaina le 'upega tafa'ilagi e ala i se feso'otaiga i fafo. Feso'ota'i ma alu i le 'au'aunaga itulau:

Ata 1. Lisi o 'au'aunaga i le Nomad cluster

O loʻo faʻaalia lelei uma ia 'auʻaunaga i le laulau, o le a tatou vaʻai i le mea lava e tasi i le gaioiga o le faʻatonuga o le nomad node:

Ata 2. Fa'ailo o le fa'atonuga o tulaga node

Ae faapefea Konesula? Se'i tatou va'ai. Alu i le Consul control panel, i le itulau nodes:

Ata 3. Lisi o nodes i le vaega Konesula

O lea la ua i ai le tatou Nomad saunia e galulue faʻatasi ma le Konesula. I le tulaga mulimuli, o le a tatou oʻo i le vaega malie: faʻatulagaina o le tuʻuina atu o pusa Docker mai Gitlab i Nomad, ma talanoa foi e uiga i nisi o ona uiga iloga.

Fausia Gitlab Runner

Ina ia faʻapipiʻi ata docker i Nomad, o le a matou faʻaogaina se isi tagata tamoʻe ma le Nomad binary faila i totonu (iinei, i le ala, e mafai ona matou matauina se isi vaega o talosaga Hashicorp - taʻitoʻatasi latou o se faila binary tasi). Tu'u i luga o le lisi o tagata tamo'e. Sei o tatou fatuina se Dockerfile faigofie mo ia ma mea nei:


FROM alpine:3.9
RUN apk add --update --no-cache libc6-compat gettext
COPY nomad /usr/local/bin/nomad

I le galuega lava e tasi matou te fatuina .gitlab-ci.yml:

variables:
  DOCKER_IMAGE: nomad/nomad-deploy
  DOCKER_REGISTRY: registry.domain.name
 

stages:
  - build

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:latest
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}

O se taunuuga, o le a matou maua se ata avanoa o le Nomad runner i le Gitlab Registry, o lea e mafai ona matou o saʻo i le fale teu oloa, fatuina se Pipeline ma faʻapipiʻi le Nomad's nomad job.

Fa'atulagaina o poloketi

Tatou amata i le faila o le galuega mo Nomad. O laʻu galuega faatino i lenei tusiga o le a fai si matua: o le a aofia ai le tasi galuega. O mea o lo'o i totonu o le .gitlab-ci o le a fa'apea:

variables:
  NOMAD_ADDR: http://nomad.address.service:4646
  DOCKER_REGISTRY: registry.domain.name
  DOCKER_IMAGE: example/project

stages:
  - build
  - deploy

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad-runner/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:${CI_COMMIT_SHORT_SHA}
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}


deploy:
  stage: deploy
  image: registry.example.com/nomad/nomad-runner:latest
  script:
    - envsubst '${CI_COMMIT_SHORT_SHA}' < project.nomad > job.nomad
    - cat job.nomad
    - nomad validate job.nomad
    - nomad plan job.nomad || if [ $? -eq 255 ]; then exit 255; else echo "success"; fi
    - nomad run job.nomad
  environment:
    name: production
  allow_failure: false
  when: manual

O iinei o le faʻapipiʻiina e tupu ma le lima, ae e mafai ona e faʻatulagaina e sui ai mea o loʻo i totonu o le lisi o galuega. Pipeline e lua vaega: faʻapipiʻi ata ma lona faʻapipiʻiina i tagata faimalaga. I le laasaga muamua, matou te faʻapipiʻiina se ata faʻailoga ma tulei i totonu o le matou Resitala, ma i le lona lua matou te faʻalauiloa la matou galuega i Nomad.

job "monitoring-status" {
    datacenters = ["dc1"]
    migrate {
        max_parallel = 3
        health_check = "checks"
        min_healthy_time = "15s"
        healthy_deadline = "5m"
    }

    group "zhadan.ltd" {
        count = 1
        update {
            max_parallel      = 1
            min_healthy_time  = "30s"
            healthy_deadline  = "5m"
            progress_deadline = "10m"
            auto_revert       = true
        }
        task "service-monitoring" {
            driver = "docker"

            config {
                image = "registry.domain.name/example/project:${CI_COMMIT_SHORT_SHA}"
                force_pull = true
                auth {
                    username = "gitlab_user"
                    password = "gitlab_password"
                }
                port_map {
                    http = 8000
                }
            }
            resources {
                network {
                    port "http" {}
                }
            }
        }
    }
}

Faamolemole ia matau o loʻo i ai laʻu Resitala tumaoti ma ia manuia le tosoina o se ata faʻailoga e manaʻomia ona ou ulufale i ai. O le fofo sili i lenei tulaga o le ulufale lea i se saini ma upu faataga i Vault ona tuʻufaʻatasia lea ma Nomad. E lagolagoina e Nomad le Vault. Ae muamua, seʻi o tatou faʻapipiʻi tulafono talafeagai mo Nomad i Vault lava ia; e mafai ona sii mai:

# Download the policy and token role
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L

# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl

# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json

I le taimi nei, i le faia o faiga faʻavae talafeagai, matou te faʻaopoopoina le tuʻufaʻatasia ma Vault i le poloka galuega i le job.nomad faila:

vault {
  enabled = true
  address = "https://vault.domain.name:8200"
  token = "token"
}

Ou te faʻaaogaina le faʻatagaina e ala i le faʻailoga ma resitala saʻo iinei, o loʻo i ai foi le filifiliga e faʻamaonia ai le faʻailoga o se fesuiaiga pe a amata le sui sui:

$ VAULT_TOKEN=<token> nomad agent -config /path/to/config

Ole taimi nei e mafai ona tatou faʻaogaina ki ile Vault. O le mataupu faavae o le faʻagaioiga e faigofie: matou te fatuina se faila i Nomad galuega o le a teuina ai le tau o fesuiaiga, mo se faʻataʻitaʻiga:

template {
                data = <<EOH
{{with secret "secrets/pipeline-keys"}}
REGISTRY_LOGIN="{{ .Data.REGISTRY_LOGIN }}"
REGISTRY_PASSWORD="{{ .Data.REGISTRY_LOGIN }}{{ end }}"

EOH
    destination = "secrets/service-name.env"
    env = true
}

Faatasi ai ma lenei auala faigofie, e mafai ona e faʻatulagaina le tuʻuina atu o pusa i le Nomad cluster ma galulue faʻatasi ai i le lumanaʻi. O le a ou fai atu i se isi itu ou te alofa ia Nomad - e sili atu ona talafeagai mo galuega laiti e mafai ai e Kubernetes ona mafua ai le lavelave faaopoopo ma o le a le iloa ai lona gafatia atoatoa. Ma le isi, Nomad e lelei atoatoa mo tagata amata-e faigofie ona faʻapipiʻi ma faʻapipiʻi. Ae peitaʻi, pe a faʻataʻitaʻiina nisi o galuega faatino, ou te feagai ma se faʻafitauli i ona uluaʻi faʻasologa - o le tele o galuega faʻavae e le o iai pe latou te le saʻo lelei. Ae ui i lea, ou te talitonu o le a faʻaauau pea ona atiaʻe Nomad ma i le lumanaʻi o le a maua ai galuega e manaʻomia e tagata uma.

Tusitala: Ilya Andreev, faatonu e Alexey Zhadan ma le au Live Linux

puna: www.habr.com

Fa'atūina se fuifui Nomad e fa'aaoga ai le Konesula ma fa'atasi ma Gitlab

Faatomuaga

nofoa su'ega

Fa'atuina o Nomad, Konesula. Fausia se fuifui Nomad

Fausia Gitlab Runner

Fa'atulagaina o poloketi

Faaopoopo i ai se faamatalaga faalēaogāina tali