ProHoster > Blog > Administration > Deploy applications using Docker Swarm

Deploy applications using Docker Swarm

The online video content recommendation system we are working on is a closed commercial development and technically represents a multi-component cluster made up of proprietary and open-source components. The purpose of this article is to describe the implementation of a Docker Swarm clustering system for a staging environment, without disrupting our established workflow due to time constraints. The narrative presented to you is divided into two parts. The first part describes CI/CD prior to using Docker Swarm, while the second covers its implementation. Those who are not interested in reading the first part may skip directly to the second.

Part I

In a distant year, it was required to set up the CI/CD process as quickly as possible. One condition was to not use Docker for deployment of the developed components for several reasons:

  • for more reliable and stable component operation in Production (essentially a requirement to avoid virtualization)
  • the lead developers did not want to work with Docker (strange, but that was the case)
  • due to ideological reasons from the R&D management

The infrastructure, stack, and rough initial requirements for the MVP were as follows:

  • 4 Intel® X5650 servers running Debian (one more powerful machine dedicated entirely to development)
  • Development of proprietary custom components is conducted in C++, Python3
  • Main 3rd party tools used: Kafka, Clickhouse, Airflow, Redis, Grafana, Postgresql, Mysql, ...
  • Separate build and testing pipelines for components for debug and release

One of the first questions to be resolved at the initial stage is how the custom components will be deployed in any environment (CI/CD).

Third-party components were decided to be installed and updated systemically. Custom applications developed in C++ or Python can be deployed in various ways. For instance, they can be created as system packages, sent to a repository of built images, and then installed on servers. For unknown reasons, a different method was chosen: using CI, executable files of the applications are compiled, a project virtual environment is created, py-modules from requirements.txt are installed, and all these artifacts are sent along with configurations, scripts, and the accompanying application environment to the servers. Applications are then launched from a virtual user without administrator rights.

Gitlab-CI was chosen as the CI/CD system. The resulting pipeline looked approximately like this:

Deploy applications using Docker Swarm
Structurally, gitlab-ci.yml looked as follows

---
variables:
  # minimum CPU version on the servers where the cluster is deployed
  CMAKE_CPUTYPE: "westmere"

  DEBIAN: "MYREGISTRY:5000/debian:latest"

before_script:
  - eval $(ssh-agent -s)
  - ssh-add  ~/.ssh/config

stages:
  - build
  - testing
  - deploy

debug.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always
release.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always

## testing stage
tests.codestyle:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -t codestyle -b "${CI_COMMIT_REF_NAME}_codestyle"
tests.debug.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - debug.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_debug"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week
tests.release.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_release"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week

## staging stage
deploy_staging:
  stage: deploy
  environment: staging
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - cd scripts/deploy/ &&
        python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
        /bin/bash install_venv.sh -d -r ../../requirements.txt &&
        python3 prepare_init.d.py &&
        python3 deploy.py -s $CI_ENVIRONMENT_NAME
  when: manual

It is worth noting that the build and testing are performed on a custom image where all the necessary system packages are already installed, and other configurations have been made.

While each of these scripts in the jobs is interesting in its own way, I won't discuss them in detail as describing each one would take significant time, and that's not the purpose of this article. I’ll just highlight that the deployment stage consists of a sequence of script calls:

  1. createconfig.py — creates a settings.ini file with component configurations in different environments for subsequent deployment (Preproduction, Production, Testing, …)
  2. install_venv.sh — creates a virtual environment for Python components in a specified directory and copies it to remote servers
  3. prepare_init.d.py — prepares start-stop scripts for components based on a template
  4. deploy.py — distributes and restarts the new components

Time passed. The staging phase was replaced by preproduction and production. Support for the product was added on another distribution (CentOS). We gained five more powerful physical servers and a dozen virtual ones. Developers and testers found it increasingly difficult to test their tasks in an environment more or less close to a working state. At that time, it became clear that it was impossible to do without it...

Part II

Deploy applications using Docker Swarm

So, our cluster is quite a spectacle — a system of several dozen individual components, not described by Dockerfiles. Configuring it for deployment in a specific environment can only be done as a whole. Our task is to deploy the cluster in the staging environment for testing before the release.

Theoretically, there can be several simultaneously operating clusters: as many as there are tasks in a completed or nearly completed state. The capacities available in our servers allow running several clusters on each server. Each staging cluster must be isolated (there should be no overlap in ports, directories, etc.).

The most valuable resource is our time, and we had little of it.

To get started more quickly, we chose Docker Swarm due to its simplicity and flexible architecture. The first thing we did was create a manager and several nodes on the remote servers:

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
kilqc94pi2upzvabttikrfr5d     nop-test-1     Ready               Active                                  19.03.2
jilwe56pl2zvabupryuosdj78     nop-test-2     Ready               Active                                  19.03.2
j5a4yz1kr2xke6b1ohoqlnbq5 *   nop-test-3     Ready               Active              Leader              19.03.2

Next, we created a network:


$ docker network create --driver overlay --subnet 10.10.10.0/24 nw_swarm

Then, we linked Gitlab-CI and Swarm nodes for remote management of nodes from CI: installation of certificates, configuration of secret variables, and setup of the Docker service on the manager server. This article saved us a lot of time.

Next, we added jobs for creating and destroying the stack in .gitlab-ci.yml.

In .gitlab-ci.yml, a few more jobs were added

## staging stage
deploy_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
    CI_BIN_DEPENDENCIES_JOB: "release.centos.7"
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack deploy -c docker-compose.yml ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME} --with-registry-auth
    - rm -rf $DOCKER_CERT_PATH
  when: manual

## stop staging stage
stop_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack rm ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME}
    # TODO: need check that stopped
  when: manual

From the above code snippet, it's clear that two buttons (deploy_staging, stop_staging) requiring manual intervention have been added to Pipelines.

Deploy applications using Docker Swarm
The stack name corresponds to the branch name, and this uniqueness should be sufficient. Services in the stack receive unique IP addresses, while ports, directories, etc., will be isolated but identical from stack to stack (since the configuration file is the same for all stacks) — which is what we aimed for. The stack (cluster) is deployed using docker-compose.yml, which describes our cluster.

docker-compose.yml

---
version: '3'

services:
  userprop:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:
  celery_bcd:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  schedulerdb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: schedulerdb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    command: ['--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci', '--explicit_defaults_for_timestamp=1']
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  celerydb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: celerydb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  cluster:
    image: $CENTOS7
    environment:
      - CENTOS
      - CI_ENVIRONMENT_NAME
      - CI_API_V4_URL
      - CI_REPOSITORY_URL
      - CI_PROJECT_ID
      - CI_PROJECT_URL
      - CI_PROJECT_PATH
      - CI_PROJECT_NAME
      - CI_COMMIT_REF_NAME
      - CI_BIN_DEPENDENCIES_JOB
    command: >
      sudo -u myusername -H /bin/bash -c ". /etc/profile &&
        mkdir -p /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
            git clone -b $CI_COMMIT_REF_NAME $CI_REPOSITORY_URL . &&
            curl $CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=$CI_BIN_DEPENDENCIES_JOB -o artifacts.zip &&
            unzip artifacts.zip ;
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME/scripts/deploy/ &&
            python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
            /bin/bash install_venv.sh -d -r ../../requirements.txt &&
            python3 prepare_init.d.py &&
            python3 deploy.py -s $CI_ENVIRONMENT_NAME"
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    tty: true
    stdin_open: true
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

It is clear that the components are combined into one network (nw_swarm) and are accessible to each other.

The system components (based on redis, mysql) are separated from the common pool of custom components (in plans to also separate custom ones as services). The deployment stage of our cluster looks like passing CMD to our one large configured image and is practically indistinguishable from the deployment described in Part I. I will emphasize the differences:

  • git clone … — we get the files necessary to carry out the deployment (createconfig.py, install_venv.sh, etc.)
  • curl… && unzip … — we download and unpack the build artifacts (compiled utilities)

There remains only one unexplained problem: the components with a web interface are not accessible from developers' browsers. We are solving this issue with a reverse proxy as follows:

In .gitlab-ci.yml, after deploying the cluster stack, we add a line for deploying the load balancer (which only updates its configuration during commits, creating new Nginx configuration files from the template: /etc/nginx/conf.d/${CI_COMMIT_REF_NAME}.conf) — see docker-compose-nginx.yml code.

    - docker stack deploy -c docker-compose-nginx.yml ${CI_ENVIRONMENT_NAME} --with-registry-auth

docker-compose-nginx.yml

---
version: '3'

services:
  nginx:
    image: nginx:latest
    environment:
      CI_COMMIT_REF_NAME: ${CI_COMMIT_REF_NAME}
      NGINX_CONFIG: |-
            server {
                listen 8080;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:8080;
                }
            }
            server {
                listen 5555;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:5555;
                }
            }
    volumes:
      - /tmp/staging/nginx:/etc/nginx/conf.d
    command:
      /bin/bash -c "echo -e "$$NGINX_CONFIG" > /etc/nginx/conf.d/${CI_COMMIT_REF_NAME}.conf;
        nginx -g "daemon off;";
        /etc/init.d/nginx reload"
    ports:
      - 8080:8080
      - 5555:5555
      - 3000:3000
      - 443:443
      - 80:80
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

On developers' computers, we update /etc/hosts; we write the URL to Nginx:

10.50.173.106 staging_BRANCH-1831_cluster.dev

Thus, the deployment of isolated staging clusters has been implemented, and developers can now run them in any sufficient quantity for verifying their tasks.

Future plans:

  • Separate our components as services
  • Create a Dockerfile for each
  • Automatically determine less loaded nodes in the stack
  • Specify nodes by name template (instead of using ID as in the article)
  • Add a check that the stack is destroyed
  • …

Special thanks for the article.

Source: habr.com

Buy reliable website hosting with DDoS protection, VPS VDS servers 🔥 Buy reliable website hosting with DDoS protection, VPS VDS servers | ProHoster