ProHoster > Blog > Administration > Setting up a Nomad cluster with Consul and integrating with Gitlab

Setting up a Nomad cluster with Consul and integrating with Gitlab

Introduction

Recently, the popularity of Kubernetes has been growing rapidly - more and more projects are implementing it at home. I wanted to touch on an orchestrator such as Nomad: it is perfect for projects where other solutions from HashiCorp, such as Vault and Consul, are already used, and the projects themselves are not complex in terms of infrastructure. This article will provide instructions for installing Nomad, merging two nodes into a cluster, and integrating Nomad with Gitlab.

Setting up a Nomad cluster with Consul and integrating with Gitlab

Test stand

A little about the test bench: three virtual servers with characteristics of 2 CPUs, 4 RAM, 50 Gb SSD are used, united in a common local network. Their names and IP addresses:

  1. nomad-livelinux-01: 172.30.0.5
  2. nomad-livelinux-02: 172.30.0.10
  3. consul-livelinux-01: 172.30.0.15

Installing Nomad, Consul. Creating a Nomad Cluster

Let's start with the basic installation. Despite the ease of installation, I will describe it for the sake of the integrity of the article: in fact, it was created from drafts and notes for quick access in case of need.

Before proceeding to practice, we will discuss the theoretical part, because at this stage it is important to understand the future structure.

We have two nomad nodes and we want to combine them into a cluster, also for the future we will need automatic scaling of the cluster - for this we need Consul. With this tool, clustering and adding new nodes becomes a very simple task: the created Nomad node connects to the Consul agent, after which it connects to the existing Nomad cluster. Therefore, at the beginning we will install the Consul server, configure the basic http authorization for the web panel (it is by default without authorization and can be accessed at an external address), as well as the Consul agents themselves on Nomad servers, after which we will just proceed to Nomad.

Installing HashiCorp's tools is very simple: in essence, we just move the binary file to the bin directory, set up the tool's configuration file, and create its service file.

Download the Consul binary and extract it to the user's home directory:

root@consul-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@consul-livelinux-01:~# mv consul /usr/local/bin/

We now have the consul binary ready for further customization.

To work with Consul, we need to generate a unique key using the keygen command:

root@consul-livelinux-01:~# consul keygen

Let's move on to configuring Consul, create a directory /etc/consul.d/ with the following structure:

/etc/consul.d/
├── bootstrap
│   └── config.json

The bootstrap directory will contain the config.json configuration file - in it we will set the Consul settings. Its content:

{
"bootstrap": true,
"server": true,
"datacenter": "dc1",
"data_dir": "/var/consul",
"encrypt": "your-key",
"log_level": "INFO",
"enable_syslog": true,
"start_join": ["172.30.0.15"]
}

Let's analyze separately the main directives and their meanings:

  • bootstrap: true. Enable automatic addition of new nodes if they are connected. I note that we do not indicate here the exact number of expected nodes.
  • server: true. Turn on server mode. Consul on this virtual machine will act as the only server and master at the moment, Nomad's VM will be clients.
  • datacenter: dc1. Specify the name of the data center to create the cluster. It must be identical on both clients and servers.
  • encrypt:your-key. The key, which must also be unique and match on all clients and servers. Generated using the consul keygen command.
  • start_join. In this list, we specify a list of IP addresses to which the connection will be made. At the moment, we leave only our own address.

At this point, we can start consul using the command line:

root@consul-livelinux-01:~# /usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui

This is a good way to debug now, however, it will not work on a permanent basis for obvious reasons. Let's create a service file to manage Consul via systemd:

root@consul-livelinux-01:~# nano /etc/systemd/system/consul.service

Consul.service file content: 

[Unit]
Description=Consul Startup process
After=network.target
 
[Service]
Type=simple
ExecStart=/bin/bash -c '/usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui' 
TimeoutStartSec=0
 
[Install]
WantedBy=default.target

Run Consul via systemctl:

root@consul-livelinux-01:~# systemctl start consul

We check: our service should be running, and by running the consul members command we should see our server:

root@consul-livelinux:/etc/consul.d# consul members
consul-livelinux    172.30.0.15:8301  alive   server  1.5.0  2         dc1  <all>

The next step: installing Nginx and setting up proxying, http authorization. Install nginx through the package manager and in the /etc/nginx/sites-enabled directory create a consul.conf configuration file with the following content:

upstream consul-auth {
    server localhost:8500;
}

server {

    server_name consul.doman.name;
    
    location / {
      proxy_pass http://consul-auth;
      proxy_set_header Host $host;
      auth_basic_user_file /etc/nginx/.htpasswd;
      auth_basic "Password-protected Area";
    }
}

Don't forget to create an .htpasswd file and generate a username and password for it. This item is required so that the web panel is not available to everyone who knows our domain. However, when setting up Gitlab, we will have to refuse this - otherwise we will not be able to deploy our application in Nomad. In my project, both Gitlab and Nomad are only on the gray network, so there is no such problem here.

On the remaining two servers, install Consul agents according to the following instructions. We repeat the steps with the binary file:

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/consul/1.5.0/consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# unzip consul_1.5.0_linux_amd64.zip
root@nomad-livelinux-01:~# mv consul /usr/local/bin/

By analogy with the previous server, we create a directory for configuration files /etc/consul.d with the following structure:

/etc/consul.d/
├── client
│   └── config.json

Content of config.json file:

{
    "datacenter": "dc1",
    "data_dir": "/opt/consul",
    "log_level": "DEBUG",
    "node_name": "nomad-livelinux-01",
    "server": false,
    "encrypt": "your-private-key",
    "domain": "livelinux",
    "addresses": {
      "dns": "127.0.0.1",
      "https": "0.0.0.0",
      "grpc": "127.0.0.1",
      "http": "127.0.0.1"
    },
    "bind_addr": "172.30.0.5", # локальный адрес вм
    "start_join": ["172.30.0.15"], # удаленный адрес консул сервера
    "ports": {
      "dns": 53
     }

We save the changes and proceed to setting up the service file, its contents are:

/etc/systemd/system/consul.service:

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/client
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

We start consul on the server. Now, after starting, we should see the configured service in nsul members. This will mean that he has successfully connected to the cluster as a client. Repeat the same on the second server and after that we can start installing and configuring Nomad.

A more detailed installation of Nomad is described in its official documentation. There are two traditional installation methods: downloading a binary file and compiling from source. I will choose the first way.

Note: the project is developing very quickly, new updates are often released. Perhaps, by the time the article is completed, a new version will be released. Therefore, before reading, I recommend checking the current version of Nomad at the moment and downloading it. 

root@nomad-livelinux-01:~# wget https://releases.hashicorp.com/nomad/0.9.1/nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# unzip nomad_0.9.1_linux_amd64.zip
root@nomad-livelinux-01:~# mv nomad /usr/local/bin/
root@nomad-livelinux-01:~# nomad -autocomplete-install
root@nomad-livelinux-01:~# complete -C /usr/local/bin/nomad nomad
root@nomad-livelinux-01:~# mkdir /etc/nomad.d

After unpacking, we will get a 65 MB Nomad binary file - it must be moved to /usr/local/bin.

Let's create a data directory for Nomad and edit its service file (it probably won't exist at the start):

root@nomad-livelinux-01:~# mkdir --parents /opt/nomad
root@nomad-livelinux-01:~# nano /etc/systemd/system/nomad.service

Insert the following lines there:

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity

[Install]
WantedBy=multi-user.target

However, we are in no hurry to launch nomad - we have not yet created its configuration file:

root@nomad-livelinux-01:~# mkdir --parents /etc/nomad.d
root@nomad-livelinux-01:~# chmod 700 /etc/nomad.d
root@nomad-livelinux-01:~# nano /etc/nomad.d/nomad.hcl
root@nomad-livelinux-01:~# nano /etc/nomad.d/server.hcl

The resulting directory structure will be as follows:

/etc/nomad.d/
├── nomad.hcl
└── server.hcl

The nomad.hcl file must contain the following configuration:

datacenter = "dc1"
data_dir = "/opt/nomad"

Server.hcl file content:

server {
  enabled = true
  bootstrap_expect = 1
}

consul {
  address             = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
}

bind_addr = "127.0.0.1" 

advertise {
  http = "172.30.0.5"
}

client {
  enabled = true
}

Do not forget to change the configuration file on the second server - you will need to change the value of the http directive there.

The last thing at this stage is setting up Nginx to proxy and install http authorization. Contents of nomad.conf file:

upstream nomad-auth {
        server 172.30.0.5:4646;
}

server {

        server_name nomad.domain.name;
        
        location / {
	        proxy_pass http://nomad-auth;
	        proxy_set_header Host $host;
	        auth_basic_user_file /etc/nginx/.htpasswd;
		   auth_basic "Password-protected Area";
        }
        
}

Now we can access the web panel over the external network. Connect and go to the servers page:

Setting up a Nomad cluster with Consul and integrating with Gitlab
Image 1. List of servers in a Nomad cluster

Both servers are successfully displayed in the panel, we will see the same in the output of the nomad node status command:

Setting up a Nomad cluster with Consul and integrating with Gitlab
Image 2. Nomad node status command output

What about Consul? Let's get a look. Go to the Consul control panel, to the nodes page:
Setting up a Nomad cluster with Consul and integrating with Gitlab
Image 3. List of nodes in the Consul cluster

Now we have Nomad ready, working in conjunction with Consul. In the final stage, we will get down to the most interesting part: we will set up the delivery of Docker containers from Gitlab to Nomad, and also talk about some of its other distinguishing features.

Creating a Gitlab Runner

To deploy docker images to Nomad, we will use a separate runner with Nomad's binary file inside (here, by the way, one more feature of Hashicorp applications can be noted - individually they are the only binary file). Upload it to the runner directory. Let's create a simple Dockerfile for it with the following content:


FROM alpine:3.9
RUN apk add --update --no-cache libc6-compat gettext
COPY nomad /usr/local/bin/nomad

In the same project, create .gitlab-ci.yml:

variables:
  DOCKER_IMAGE: nomad/nomad-deploy
  DOCKER_REGISTRY: registry.domain.name
 

stages:
  - build

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:latest
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}

As a result, we will have an available image of the Nomad runner in the Gitlab Registry, now we can go directly to the project repository, create a Pipeline and set up Nomad's nomad job.

Project setup

Let's start with the job's file for Nomad. My project in this article will be quite primitive: it will consist of one task. The content of .gitlab-ci will be as follows:

variables:
  NOMAD_ADDR: http://nomad.address.service:4646
  DOCKER_REGISTRY: registry.domain.name
  DOCKER_IMAGE: example/project

stages:
  - build
  - deploy

build:
  stage: build
  image: ${DOCKER_REGISTRY}/nomad-runner/alpine:3
  script:
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:${CI_COMMIT_SHORT_SHA}
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}


deploy:
  stage: deploy
  image: registry.example.com/nomad/nomad-runner:latest
  script:
    - envsubst '${CI_COMMIT_SHORT_SHA}' < project.nomad > job.nomad
    - cat job.nomad
    - nomad validate job.nomad
    - nomad plan job.nomad || if [ $? -eq 255 ]; then exit 255; else echo "success"; fi
    - nomad run job.nomad
  environment:
    name: production
  allow_failure: false
  when: manual

Here, the deployment is done manually, but you can configure it to change the contents of the project directory. Pipeline, on the other hand, consists of two stages: from the assembly of the image and its deployment to the nomad. At the first stage, we build the docker image and push it to our Registry, and at the second stage, we launch our job in Nomad. 

job "monitoring-status" {
    datacenters = ["dc1"]
    migrate {
        max_parallel = 3
        health_check = "checks"
        min_healthy_time = "15s"
        healthy_deadline = "5m"
    }

    group "zhadan.ltd" {
        count = 1
        update {
            max_parallel      = 1
            min_healthy_time  = "30s"
            healthy_deadline  = "5m"
            progress_deadline = "10m"
            auto_revert       = true
        }
        task "service-monitoring" {
            driver = "docker"

            config {
                image = "registry.domain.name/example/project:${CI_COMMIT_SHORT_SHA}"
                force_pull = true
                auth {
                    username = "gitlab_user"
                    password = "gitlab_password"
                }
                port_map {
                    http = 8000
                }
            }
            resources {
                network {
                    port "http" {}
                }
            }
        }
    }
}

Please note that I have a private Registry and for a successful pull of the docker image I need to log in to it. The best solution in this case is to conclude the login and password in Vault and then integrate it with Nomad. Nomad supports Vault natively. But first, in the Vault itself, we will set the necessary policies for Nomad, they can be downloaded:

# Download the policy and token role
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L

# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl

# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json

Now, having created the necessary policies, we will add integration with Vault in the task block in the job.nomad file:

vault {
  enabled = true
  address = "https://vault.domain.name:8200"
  token = "token"
}

I use token authorization and write it directly here, there is also an option to specify the token as a variable when starting the nomad agent:

$ VAULT_TOKEN=<token> nomad agent -config /path/to/config

Now we can use keys with Vault. The principle of operation is simple: we create a file in the Nomad job that will store the values ​​of variables, for example:

template {
                data = <<EOH
{{with secret "secrets/pipeline-keys"}}
REGISTRY_LOGIN="{{ .Data.REGISTRY_LOGIN }}"
REGISTRY_PASSWORD="{{ .Data.REGISTRY_LOGIN }}{{ end }}"

EOH
    destination = "secrets/service-name.env"
    env = true
}

With this simple approach, you can set up the delivery of containers to the Nomad cluster and work with it in the future. I will say that to some extent I sympathize with Nomad - it is more suitable for small projects where Kubernetes can cause additional difficulties and will not realize its full potential. In addition, Nomad is great for beginners - it's easy to install and configure. However, when testing on some projects, I encounter the problem of its early versions - many basic functions simply do not exist or they do not work correctly. However, I believe that Nomad will continue to develop and in the future it will acquire the functions that everyone needs.

Author: Ilya Andreev, edited by Alexey Zhadan and the Live Linux team


Source: habr.com

Add a comment