Ukusetha iqoqo le-Nomad usebenzisa i-Consul futhi ihlanganisa ne-Gitlab

Isingeniso

Muva nje, ukuthandwa kwe-Kubernetes kuye kwakhula ngokushesha - amaphrojekthi amaningi ayawenza. Bengifuna ukuthinta i-orchestrator efana ne-Nomad: iphelele kumaphrojekthi asevele asebenzisa ezinye izixazululo ezivela ku-HashiCorp, isibonelo, i-Vault ne-Consul, futhi amaphrojekthi ngokwawo awanzima ngokwengqalasizinda. Lokhu okuqukethwe kuzoqukatha imiyalelo yokufaka i-Nomad, ukuhlanganisa ama-node amabili abe yiqoqo, kanye nokuhlanganisa i-Nomad ne-Gitlab.

Isitendi sokuhlola

Okuncane mayelana nebhentshi lokuhlola: amaseva amathathu abonakalayo asetshenziswa anezici ze-2 CPU, 4 RAM, 50 Gb SSD, ehlanganiswe ibe yinethiwekhi yendawo evamile. Amagama abo namakheli e-IP:

1. nomad-livelinux-01: 172.30.0.5
2. nomad-livelinux-02: 172.30.0.10
3. i-consul-livelinux-01: 172.30.0.15

Ukufakwa kweNomad, Consul. Ukudala iqoqo le-Nomad

Ake siqale ngokufaka okuyisisekelo. Nakuba ukusetha bekulula, ngizokuchaza ngenxa yobuqotho be-athikili: empeleni yadalwa kusukela kokusalungiswa namanothi ukuze kufinyeleleke ngokushesha lapho kudingeka.

Ngaphambi kokuba siqale ukuzijwayeza, sizoxoxa ngengxenye yethiyori, ngoba kulesi sigaba kubalulekile ukuqonda isakhiwo esizayo.

Sinama-node amabili e-nomad futhi sifuna ukuwahlanganisa abe yiqoqo, futhi esikhathini esizayo sizodinga ukukala kweqoqo okuzenzakalelayo - kulokhu sizodinga i-Consul. Ngaleli thuluzi, ukuhlanganisa kanye nokwengeza ama-node amasha kuba umsebenzi olula kakhulu: indawo edaliwe ye-Nomad ixhuma ku-ejenti ye-Consul, bese ixhuma kuqoqo elikhona le-Nomad. Ngakho-ke, ekuqaleni sizofaka iseva ye-Consul, silungise ukugunyazwa okuyisisekelo kwe-http yephaneli yewebhu (ngaphandle kokugunyazwa ngokuzenzakalelayo futhi ingafinyelelwa ekhelini langaphandle), kanye nama-Consul agents ngokwabo kumaseva e-Nomad, ngemva kwalokho. sizodlulela kuNomad kuphela.

Ukufaka amathuluzi e-HashiCorp kulula kakhulu: empeleni, sivele sihambise ifayela kanambambili kumkhombandlela womgqomo, simise ifayela lokumisa lethuluzi, futhi sakhe ifayela lalo lesevisi.

Landa ifayela kanambambili le-Consul futhi ulikhiphe ohlwini lwemibhalo lwasekhaya lomsebenzisi:

root@consul-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# mv consul /usr/local/bin/

Manje sesine-binnary ye-consul eseyenziwe ngomumo ukuze siphinde sicushwe.

Ukuze sisebenze ne-Consul, sidinga ukudala ukhiye oyingqayizivele sisebenzisa umyalo we-keygen:

root@consul-livelinux-01:~# consul keygen

Masiqhubekele phambili ekusetheni ukucushwa kwe-Consul, sakha inkomba /etc/consul.d/ ngesakhiwo esilandelayo:

/etc/consul.d/
├── bootstrap
│   └── config.json

Inkomba ye-bootstrap izoqukatha ifayela lokucushwa elithi config.json - kuyo sizosetha izilungiselelo ze-Consul. Okuqukethwe kwayo:

{
"bootstrap": true,
"server": true,
"datacenter": "dc1",
"data_dir": "/var/consul",
"encrypt": "your-key",
"log_level": "INFO",
"enable_syslog": true,
"start_join": ["172.30.0.15"]
}

Ake sibheke iziqondiso eziyinhloko nezincazelo zazo ngokwehlukana:

• i-bootstrap: iqiniso. Sivumela ukwengeza okuzenzakalelayo kwamanodi amasha uma exhunyiwe. Ngiyaqaphela ukuthi asibonisi lapha inani eliqondile lamanodi alindelekile.
• Iseva: iqiniso. Nika amandla imodi yeseva. I-Consul kulo mshini obonakalayo izosebenza njengokuphela kweseva nenkosi okwamanje, i-VM kaNomad kuzoba amakhasimende.
• idatha yedatha:dc1. Cacisa igama lesikhungo sedatha ukuze udale iqoqo. Kufanele ifane kuwo womabili amaklayenti namaseva.
• ukubethela: ukhiye wakho. Ukhiye, okufanele futhi uhluke futhi ufanise kuwo wonke amaklayenti namaseva. Kwenziwe kusetshenziswa umyalo we-consul keygen.
• qala_joyina. Kulolu hlu sibonisa uhlu lwamakheli e-IP okuzoxhunywa kuwo. Okwamanje sishiya ikheli lethu kuphela.

Kuleli qophelo singasebenzisa i-consul sisebenzisa umugqa womyalo:

root@consul-livelinux-01:~# /usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui

Lena indlela enhle yokususa iphutha manje, nokho, ngeke ukwazi ukusebenzisa le ndlela ngokuqhubekayo ngenxa yezizathu ezisobala. Masidale ifayela lesevisi ukuze siphathe i-Consul nge-systemd:

root@consul-livelinux-01:~# nano /etc/systemd/system/consul.service

Okuqukethwe kwefayela le-consul.service:

[Unit]
Description=Consul Startup process
After=network.target
 
[Service]
Type=simple
ExecStart=/bin/bash -c '/usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui' 
TimeoutStartSec=0
 
[Install]
WantedBy=default.target

Yethula i-Consul nge-systemctl:

root@consul-livelinux-01:~# systemctl start consul

Ake sihlole: isevisi yethu kufanele isebenze, futhi ngokukhipha umyalo wamalungu e-consul kufanele sibone iseva yethu:

root@consul-livelinux:/etc/consul.d# consul members
consul-livelinux    172.30.0.15:8301  alive   server  1.5.0  2         dc1  <all>

Isigaba esilandelayo: ukufaka i-Nginx nokusetha i-proxying kanye nokugunyazwa kwe-http. Sifaka i-nginx ngomphathi wephakheji futhi kusiqondisi esinikwe amandla amasayithi /etc/nginx/sites sakha ifayela lokucushwa consul.conf nokuqukethwe okulandelayo:

upstream consul-auth {
    server localhost:8500;
}

server {

    server_name consul.doman.name;
    
    location / {
      proxy_pass http://consul-auth;
      proxy_set_header Host $host;
      auth_basic_user_file /etc/nginx/.htpasswd;
      auth_basic "Password-protected Area";
    }
}

Ungakhohlwa ukwenza ifayela le-.htpasswd futhi ukhiqize igama lomsebenzisi nephasiwedi yalo. Le nto iyadingeka ukuze iphaneli yewebhu ingatholakali kuwo wonke umuntu owazi isizinda sethu. Kodwa-ke, lapho sisetha i-Gitlab, kuzodingeka sikuyeke lokhu - ngaphandle kwalokho ngeke sikwazi ukuthumela isicelo sethu ku-Nomad. Kuphrojekthi yami, kokubili i-Gitlab ne-Nomad ikuwebhu empunga kuphela, ngakho-ke ayikho inkinga enjalo lapha.

Kumaseva amabili asele sifaka ama-Consul agents ngokwemiyalelo elandelayo. Siphinda izinyathelo ngefayela kanambambili:

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# mv consul /usr/local/bin/

Ngokufanisa neseva yangaphambilini, sakha uhla lwemibhalo lwamafayela okumisa /etc/consul.d ngesakhiwo esilandelayo:

/etc/consul.d/
├── client
│   └── config.json

Okuqukethwe kwefayela le-config.json:

{
    "datacenter": "dc1",
    "data_dir": "/opt/consul",
    "log_level": "DEBUG",
    "node_name": "nomad-livelinux-01",
    "server": false,
    "encrypt": "your-private-key",
    "domain": "livelinux",
    "addresses": {
      "dns": "127.0.0.1",
      "https": "0.0.0.0",
      "grpc": "127.0.0.1",
      "http": "127.0.0.1"
    },
    "bind_addr": "172.30.0.5", # локальный адрес вм
    "start_join": ["172.30.0.15"], # удаленный адрес консул сервера
    "ports": {
      "dns": 53
     }

Londoloza izinguquko bese uqhubekela phambili ekusetheni ifayela lesevisi, elikuqukethe:

/etc/systemd/system/consul.service:

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/client
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

Sethula i-consul kuseva. Manje, ngemva kokwethulwa, kufanele sibone isevisi emisiwe kumalungu e-nsul. Lokhu kuzosho ukuthi ixhumeke ngempumelelo kuqoqo njengeklayenti. Phinda okufanayo kuseva yesibili futhi ngemva kwalokho singaqala ukufaka nokumisa i-Nomad.

Ukufakwa okuningiliziwe kweNomad kuchazwe emibhalweni yayo esemthethweni. Kunezindlela ezimbili zokufaka zendabuko: ukulanda ifayela kanambambili nokuhlanganisa kusuka emthonjeni. Ngizokhetha indlela yokuqala.

Ukubhala: Iphrojekthi ithuthuka ngokushesha okukhulu, izibuyekezo ezintsha zivame ukukhishwa. Mhlawumbe inguqulo entsha izokhishwa ngesikhathi lesi sihloko siqedwa. Ngakho-ke, ngaphambi kokufunda, ngincoma ukuthi uhlole inguqulo yamanje ye-Nomad okwamanje futhi uyilande.

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/nomad/0.9.1/nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# unzip nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# mv nomad /usr/local/bin/
root@nomad-livelinux-01:~# nomad -autocomplete-install
root@nomad-livelinux-01:~# complete -C /usr/local/bin/nomad nomad
root@nomad-livelinux-01:~# mkdir /etc/nomad.d

Ngemva kokuqaqa, sizothola ifayela kanambambili le-Nomad elinesisindo esingu-65 MB - kufanele lihanjiswe ku-/usr/local/bin.

Ake sakhe uhla lwemibhalo lwedatha lwe-Nomad futhi sihlele ifayela layo lesevisi (cishe ngeke libe khona ekuqaleni):

root@nomad-livelinux-01:~# mkdir --parents /opt/nomad
root@nomad-livelinux-01:~# nano /etc/systemd/system/nomad.service

Namathisela imigqa elandelayo lapho:

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity

[Install]
WantedBy=multi-user.target

Kodwa-ke, asijahile ukwethula i-nomad - asikakadali ifayela layo lokucushwa:

root@nomad-livelinux-01:~# mkdir --parents /etc/nomad.d
root@nomad-livelinux-01:~# chmod 700 /etc/nomad.d
root@nomad-livelinux-01:~# nano /etc/nomad.d/nomad.hcl
root@nomad-livelinux-01:~# nano /etc/nomad.d/server.hcl

Isakhiwo sohlu lokugcina sizoba kanje:

/etc/nomad.d/
├── nomad.hcl
└── server.hcl

Ifayela le-nomad.hcl kufanele libe nokucushwa okulandelayo:

datacenter = "dc1"
data_dir = "/opt/nomad"

Okuqukethwe kwefayela le-server.hcl:

server {
  enabled = true
  bootstrap_expect = 1
}

consul {
  address             = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
}

bind_addr = "127.0.0.1" 

advertise {
  http = "172.30.0.5"
}

client {
  enabled = true
}

Ungakhohlwa ukushintsha ifayela lokucushwa kuseva yesibili - lapho uzodinga ukushintsha inani lomyalelo we-http.

Into yokugcina kulesi sigaba ukulungisa i-Nginx yokwenza ummeleli nokusetha ukugunyazwa kwe-http. Okuqukethwe kwefayela le-nomad.conf:

upstream nomad-auth {
        server 172.30.0.5:4646;
}

server {

        server_name nomad.domain.name;
        
        location / {
	        proxy_pass http://nomad-auth;
	        proxy_set_header Host $host;
	        auth_basic_user_file /etc/nginx/.htpasswd;
		   auth_basic "Password-protected Area";
        }
        
}

Manje sesingakwazi ukufinyelela iphaneli yewebhu ngenethiwekhi yangaphandle. Xhuma bese uya ekhasini leziphakeli:

Isithombe 1. Uhlu lwamaseva kuqoqo le-Nomad

Womabili amaseva akhonjiswa ngempumelelo kuphaneli, sizobona into efanayo ekuphumeni komyalo wesimo se-nomad node:

Isithombe 2. Umphumela womyalo wesimo se-nomad node

Kuthiwani nge-Consul? Ake sibheke. Iya kuphaneli yokulawula ye-Consul, ekhasini lama-node:

Isithombe 3. Uhlu lwama-node ku-Consul cluster

Manje sine-Nomad elungisiwe esebenza ngokubambisana ne-Consul. Esigabeni sokugcina, sizofika engxenyeni ejabulisayo: ukusetha ukulethwa kweziqukathi ze-Docker ukusuka e-Gitlab kuya ku-Nomad, futhi siphinde sikhulume ngezinye zezici zayo ezihlukile.

Idala i-Gitlab Runner

Ukuze sikhiphe izithombe ze-docker ku-Nomad, sizosebenzisa umgijimi ohlukile onefayela kanambambili le-Nomad ngaphakathi (lapha, ngendlela, singaphawula esinye isici sezinhlelo zokusebenza ze-Hashicorp - ngazinye ziyifayela elilodwa kanambambili). Ilayishe kuhla lwemibhalo lomgijimi. Masizenzele i-Dockerfile elula ngokuqukethwe okulandelayo:

FROM alpine:3.9
RUN apk add --update --no-cache libc6-compat gettext
COPY nomad /usr/local/bin/nomad

Kuphrojekthi efanayo sidala i-.gitlab-ci.yml:

variables:
  DOCKER_IMAGE: nomad/nomad-deploy
  DOCKER_REGISTRY: registry.domain.name
 

stages:
  - build

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:latest
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}

Ngenxa yalokho, sizoba nesithombe esitholakalayo somgijimi we-Nomad ku-Gitlab Registry, manje singaya ngqo endaweni yokugcina iphrojekthi, sidale iPipeline futhi silungiselele umsebenzi kaNomad.

Ukusethwa kwephrojekthi

Ake siqale ngefayela lomsebenzi kaNomad. Iphrojekthi yami kulesi sihloko izoba ngeyakudala: izoba nomsebenzi owodwa. Okuqukethwe kwe-.gitlab-ci kuzoba kanje:

variables:
  NOMAD_ADDR: http://nomad.address.service:4646
  DOCKER_REGISTRY: registry.domain.name
  DOCKER_IMAGE: example/project

stages:
  - build
  - deploy

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad-runner/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:${CI_COMMIT_SHORT_SHA}
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}


deploy:
  stage: deploy
  image: registry.example.com/nomad/nomad-runner:latest
  script:
    - envsubst '${CI_COMMIT_SHORT_SHA}' < project.nomad > job.nomad
    - cat job.nomad
    - nomad validate job.nomad
    - nomad plan job.nomad || if [ $? -eq 255 ]; then exit 255; else echo "success"; fi
    - nomad run job.nomad
  environment:
    name: production
  allow_failure: false
  when: manual

Lapha ukuthunyelwa kwenzeka mathupha, kodwa ungakwazi ukukulungisa ukuze ushintshe okuqukethwe kunkomba yephrojekthi. Ipayipi linezigaba ezimbili: ukuhlanganiswa kwesithombe kanye nokuthunyelwa kwaso ku-nomad. Esigabeni sokuqala, sihlanganisa isithombe se-docker futhi sisiphushele kuRegistry yethu, bese kwesibili sethula umsebenzi wethu e-Nomad.

job "monitoring-status" {
    datacenters = ["dc1"]
    migrate {
        max_parallel = 3
        health_check = "checks"
        min_healthy_time = "15s"
        healthy_deadline = "5m"
    }

    group "zhadan.ltd" {
        count = 1
        update {
            max_parallel      = 1
            min_healthy_time  = "30s"
            healthy_deadline  = "5m"
            progress_deadline = "10m"
            auto_revert       = true
        }
        task "service-monitoring" {
            driver = "docker"

            config {
                image = "registry.domain.name/example/project:${CI_COMMIT_SHORT_SHA}"
                force_pull = true
                auth {
                    username = "gitlab_user"
                    password = "gitlab_password"
                }
                port_map {
                    http = 8000
                }
            }
            resources {
                network {
                    port "http" {}
                }
            }
        }
    }
}

Sicela uqaphele ukuthi ngineRegistry yangasese futhi ukuze ngidonse ngempumelelo isithombe sedokhu engidinga ukungena kuso. Isixazululo esingcono kakhulu kuleli cala ukufaka igama lokungena nephasiwedi ku-Vault bese uyihlanganisa noNomad. UNomad ngokwemvelo usekela i-Vault. Kepha okokuqala, ake sifake izinqubomgomo ezidingekayo ze-Nomad ku-Vault uqobo; zingalandwa:

# Download the policy and token role
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L

# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl

# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json

Manje, njengoba sesidale izinqubomgomo ezidingekayo, sizokwengeza ukuhlanganiswa ne-Vault kubhulokhi yomsebenzi kufayela le- job.nomad:

vault {
  enabled = true
  address = "https://vault.domain.name:8200"
  token = "token"
}

Ngisebenzisa ukugunyazwa ngethokheni futhi ngibhalisa ngqo lapha, kukhona futhi inketho yokucacisa ithokheni njengokuguquguqukayo lapho ngiqala i-ejenti ye-nomad:

$ VAULT_TOKEN=<token> nomad agent -config /path/to/config

Manje singasebenzisa okhiye nge-Vault. Umgomo wokusebenza ulula: sakha ifayela emsebenzini we-Nomad ozogcina amanani okuguquguqukayo, isibonelo:

template {
                data = <<EOH
{{with secret "secrets/pipeline-keys"}}
REGISTRY_LOGIN="{{ .Data.REGISTRY_LOGIN }}"
REGISTRY_PASSWORD="{{ .Data.REGISTRY_LOGIN }}{{ end }}"

EOH
    destination = "secrets/service-name.env"
    env = true
}

Ngale ndlela elula, ungalungiselela ukulethwa kweziqukathi kuqoqo le-Nomad futhi usebenze nalo esikhathini esizayo. Ngizothi ngezinga elithile ngizwelana noNomad - ifaneleka kakhulu kumaphrojekthi amancane lapho i-Kubernetes ingabangela ubunzima obengeziwe futhi ngeke ibone amandla ayo aphelele. Futhi, i-Nomad ilungele abaqalayo—kulula ukuyifaka nokuyilungisa. Kodwa-ke, lapho ngihlola kwamanye amaphrojekthi, ngihlangabezana nenkinga ngezinguqulo zayo zangaphambili - imisebenzi eminingi eyisisekelo ayikho noma ayisebenzi kahle. Kodwa-ke, ngikholwa ukuthi i-Nomad izoqhubeka nokuthuthuka futhi ngokuzayo izothola imisebenzi edingwa yiwo wonke umuntu.

Umbhali: U-Ilya Andreev, uhlelwe ngu-Alexey Zhadan kanye neqembu le-Live Linux


Source: www.habr.com