🥇 Hoʻopili i nā noi me ka hoʻohana ʻana iā Docker Swarm

ʻO ka ʻōnaehana hoʻolaha wikiō pūnaewele a mākou e hana nei he hoʻomohala pāʻoihana pani ʻia a ʻo ia hoʻi he pūʻulu multi-component o nā ʻāpana proprietary a open source. ʻO ke kumu o ke kākau ʻana i kēia ʻatikala ʻo ia ka wehewehe ʻana i ka hoʻokō ʻana o ka ʻōnaehana docker swarm clustering no kahi kahua hoʻokūkū, me ka ʻole o ka hoʻopau ʻana i ke kaʻina hana i hoʻokumu ʻia o kā mākou kaʻina hana ma lalo o nā kūlana manawa palena. Ua māhele ʻia ka moʻolelo i hōʻike ʻia i kou maka i ʻelua ʻāpana. Hōʻike ka ʻāpana mua iā CI/CD ma mua o ka hoʻohana ʻana i ka docker swarm, a ʻo ka ʻāpana ʻelua e wehewehe i ke kaʻina hana o ka hoʻokō ʻana. ʻO ka poʻe hoihoi ʻole i ka heluhelu ʻana i ka hapa mua hiki ke neʻe maluhia i ka lua.

Māhele I

I kekahi manawa, pono e hoʻonohonoho i kahi kaʻina CI/CD i ka hikiwawe. ʻO kekahi o nā kūlana ʻaʻole ia e hoʻohana iā Docker no ka hoʻolaha ʻana kūkulu ʻia nā ʻāpana no kekahi mau kumu:

no ka ʻoi aku ka hilinaʻi a me ka paʻa o ka hana o nā ʻāpana i Production (ʻo ia hoʻi, ma ke ʻano, ke koi ʻole e hoʻohana i ka virtualization)
ʻAʻole makemake nā mea hoʻomohala alakaʻi e hana pū me Docker (ʻano ʻē, akā ʻo ia ke ʻano)
no nā kumu manaʻo o ka hoʻokele R&D

ʻO ka ʻōnaehana, hoʻopaʻa a me nā koi mua e pili ana i ka MVP penei:

4 mau kikowaena Intel® X5650 me Debian (hoʻokahi mīkini ikaika loa no ka hoʻomohala ʻana)
Hana ʻia ka hoʻomohala ʻana i kāu mau mea maʻamau ma C++, Python3
ʻO nā mea hana 3rdparty nui i hoʻohana ʻia: Kafka, Clickhouse, Airflow, Redis, Grafana, Postgresql, Mysql, ...
Nā paipu no ke kūkulu ʻana a me ka hoʻāʻo ʻana i nā ʻāpana no ka debug a me ka hoʻokuʻu ʻana

ʻO kekahi o nā nīnau mua e pono e hoʻoholo ʻia ma ka pae mua, ʻo ia ke ʻano o ka hoʻokomo ʻia ʻana o nā mea maʻamau i kekahi kaiapuni (CI/CD).

Ua hoʻoholo mākou e hoʻokomo i nā ʻāpana ʻaoʻao ʻekolu ma ka ʻōnaehana a hōʻano hou iā lākou me ka ʻōnaehana. Hiki ke kau ʻia nā noi maʻamau i kūkulu ʻia ma C++ a i ʻole Python ma nā ʻano he nui. I waena o lākou, no ka laʻana: hana ʻana i nā pūʻulu ʻōnaehana, hoʻouna iā lākou i ka waihona o nā kiʻi i hōʻiliʻili ʻia a me kā lākou hoʻokomo ʻana ma nā kikowaena. No kahi kumu i ʻike ʻole ʻia, ua koho ʻia kahi ala ʻē aʻe, ʻo ia hoʻi: me ka hoʻohana ʻana i ka CI, hoʻopili ʻia nā faila hiki ke hoʻohana ʻia, hana ʻia kahi kaiapuni virtual project, hoʻokomo ʻia nā modula py mai requirements.txt, a ua hoʻouna ʻia kēia mau artifact āpau me nā configs, scripts a ke kaiapuni noi e pili ana i nā kikowaena. A laila, hoʻomaka ʻia nā noi mai kahi mea hoʻohana virtual me ka ʻole o nā kuleana luna.

Ua koho ʻia ʻo Gitlab-CI e like me ka ʻōnaehana CI/CD. Ua like ka hopena o ka pipeline penei:

Ma ke ʻano, gitlab-ci.yml i like me kēia:

---
variables:
  # минимальная версия ЦПУ на серверах, где разворачивается кластер
  CMAKE_CPUTYPE: "westmere"

  DEBIAN: "MYREGISTRY:5000/debian:latest"

before_script:
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh && echo -e "Host *ntStrictHostKeyChecking nonn" > ~/.ssh/config

stages:
  - build
  - testing
  - deploy

debug.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always
release.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always

## testing stage
tests.codestyle:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -t codestyle -b "${CI_COMMIT_REF_NAME}_codestyle"
tests.debug.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - debug.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_debug"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week
tests.release.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_release"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week

## staging stage
deploy_staging:
  stage: deploy
  environment: staging
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - cd scripts/deploy/ &&
        python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
        /bin/bash install_venv.sh -d -r ../../requirements.txt &&
        python3 prepare_init.d.py &&
        python3 deploy.py -s $CI_ENVIRONMENT_NAME
  when: manual

He mea pono e hoʻomaopopo i ka hui ʻana a me ka hoʻāʻo ʻana e hana ʻia ma kāna kiʻi ponoʻī, kahi i hoʻokomo ʻia ai nā ʻōnaehana pono āpau a hana ʻia nā hoʻonohonoho ʻē aʻe.

ʻOiai ʻo kēlā me kēia palapala i nā hana he mea hoihoi i kona ʻano ponoʻī, ʻaʻole wau e kamaʻilio e pili ana iā lākou; ʻo ka wehewehe ʻana i kēlā me kēia o lākou e lōʻihi ka manawa a ʻaʻole kēia ke kumu o ka ʻatikala. E ʻae mai iaʻu e huki wale i kou manaʻo i ka ʻoiaʻiʻo o ka pae hoʻonohonoho ʻana i kahi ʻano o ke kāhea ʻana i nā palapala:

createconfig.py - hoʻokumu i kahi faila settings.ini me nā hoʻonohonoho no nā ʻāpana i nā ʻāpana like ʻole no ka hoʻoili ʻana ma hope (Preproduction, Production, Testing, ...)
install_venv.sh - hana i kahi kaiapuni no nā ʻāpana py i kahi papa kuhikuhi kikoʻī a kope iā ia i nā kikowaena mamao
prepare_init.d.py - hoʻomākaukau i nā palapala no nā ʻāpana hoʻomaka e pili ana i ka template
deploy.py - hoʻolālā a hoʻomaka hou i nā mea hou

Ua hala ka manawa. Ua hoʻololi ʻia ke kahua hoʻokūkū e ka preproduction a me ka hana ʻana. Ua hoʻohui ʻia ke kākoʻo no ka huahana ma kahi hoʻolaha hou (CentOS). Ua hoʻohui ʻia he 5 mau kikowaena kino ikaika a he ʻumi mau mea virtual. A ua lilo ia i mea paʻakikī no nā mea hoʻomohala a me nā mea hoʻāʻo e hoʻāʻo i kā lākou mau hana ma kahi kaiapuni kokoke i ke kūlana hana. I kēia manawa ua maopopo he mea hiki ʻole ke hana me ia ...

Mahele II

No laila, ʻo kā mākou puʻupuʻu he ʻōnaehana kupaianaha o ʻelua mau ʻāpana pākahi i wehewehe ʻole ʻia e Dockerfiles. Hiki iā ʻoe ke hoʻonohonoho iā ia no ka waiho ʻana i kahi kaiapuni kikoʻī wale nō ma ka laulā. ʻO kā mākou hana, ʻo ia ke kau ʻana i ka puʻupuʻu i loko o kahi kaiapuni e hoʻāʻo ai ma mua o ka hoʻokuʻu ʻana.

ʻO ka manaʻo, hiki ke nui nā pūʻulu e hana like ana: e like me ka nui o nā hana ma kahi mokuʻāina i hoʻopau ʻia a kokoke i ka pau ʻana. ʻO ka mana o nā kikowaena e hiki ai iā mākou ke holo i kekahi mau pūʻulu ma kēlā me kēia kikowaena. Pono e hoʻokaʻawale ʻia kēlā me kēia pūʻulu staging (ʻaʻole pono e uhi ʻia ma nā awa, nā papa kuhikuhi, a me nā mea ʻē aʻe).

ʻO kā mākou kumu waiwai nui ko mākou manawa, a ʻaʻole i loaʻa iā mākou ka nui o ia mea.

No ka hoʻomaka wikiwiki ʻana, ua koho mākou iā Docker Swarm ma muli o kona ʻano maʻalahi a maʻalahi. ʻO ka mea mua a mākou i hana ai, ʻo ia ka hana ʻana i kahi mana a me kekahi mau nodes ma nā kikowaena mamao:

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
kilqc94pi2upzvabttikrfr5d     nop-test-1     Ready               Active                                  19.03.2
jilwe56pl2zvabupryuosdj78     nop-test-2     Ready               Active                                  19.03.2
j5a4yz1kr2xke6b1ohoqlnbq5 *   nop-test-3     Ready               Active              Leader              19.03.2

A laila, hana mākou i kahi pūnaewele:


$ docker network create --driver overlay --subnet 10.10.10.0/24 nw_swarm

Ma hope aʻe, ua hoʻopili mākou iā Gitlab-CI a me Swarm nodes e pili ana i ka hoʻokele mamao o nā nodes mai CI: hoʻokomo i nā palapala hōʻoia, hoʻonohonoho i nā mea huna huna, a hoʻonohonoho pū i ka lawelawe Docker ma ka kikowaena mana. ʻO kēia 'atikala hoʻopakele iā mākou i ka manawa nui.

A laila, hoʻohui mākou i nā hana no ka hana ʻana a me ka luku ʻana i ka waihona ma .gitlab-ci .yml.

Ua hoʻohui ʻia kekahi mau hana hou i .gitlab-ci .yml

## staging stage
deploy_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
    CI_BIN_DEPENDENCIES_JOB: "release.centos.7"
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack deploy -c docker-compose.yml ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME} --with-registry-auth
    - rm -rf $DOCKER_CERT_PATH
  when: manual

## stop staging stage
stop_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack rm ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME}
    # TODO: need check that stopped
  when: manual

Mai ka ʻāpana code i luna aʻe ua maopopo ua hoʻohui ʻia nā pihi ʻelua i Pipelines (deploy_staging, stop_staging) e koi ana i ka hana lima.

Ua like ka inoa o ka waihona me ka inoa o ka lālā a ua lawa kēia ʻokoʻa. Loaʻa i nā lawelawe i loko o ka waihona i nā helu IP kūʻokoʻa, a me nā awa, nā papa kuhikuhi, etc. e hoʻokaʻawale ʻia, akā ʻo ia ka mea mai kahi hoʻopaʻa i ka hoʻopaʻa ʻana (no ka mea, ua like ka faila hoʻonohonoho no nā waihona āpau) - ʻo ia kā mākou makemake. Hoʻohana mākou i ka hoʻopaʻa ʻana (cluster). docker-compose.yml, e wehewehe ana i kā mākou pūʻulu.

docker-compose.yml

---
version: '3'

services:
  userprop:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:
  celery_bcd:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  schedulerdb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: schedulerdb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    command: ['--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci', '--explicit_defaults_for_timestamp=1']
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  celerydb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: celerydb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  cluster:
    image: $CENTOS7
    environment:
      - CENTOS
      - CI_ENVIRONMENT_NAME
      - CI_API_V4_URL
      - CI_REPOSITORY_URL
      - CI_PROJECT_ID
      - CI_PROJECT_URL
      - CI_PROJECT_PATH
      - CI_PROJECT_NAME
      - CI_COMMIT_REF_NAME
      - CI_BIN_DEPENDENCIES_JOB
    command: >
      sudo -u myusername -H /bin/bash -c ". /etc/profile &&
        mkdir -p /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
            git clone -b $CI_COMMIT_REF_NAME $CI_REPOSITORY_URL . &&
            curl $CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=$CI_BIN_DEPENDENCIES_JOB -o artifacts.zip &&
            unzip artifacts.zip ;
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME/scripts/deploy/ &&
            python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
            /bin/bash install_venv.sh -d -r ../../requirements.txt &&
            python3 prepare_init.d.py &&
            python3 deploy.py -s $CI_ENVIRONMENT_NAME"
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    tty: true
    stdin_open: true
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Maʻaneʻi hiki iāʻoe keʻike i ka piliʻana o nā mea i hoʻokahi pūnaewele (nw_swarm) a hiki ke loaʻa i kekahi i kekahi.

Hoʻokaʻawale ʻia nā ʻāpana ʻōnaehana (e pili ana i ka redis, mysql) mai ka kolamu maʻamau o nā ʻāpana maʻamau (ma nā hoʻolālā, hoʻokaʻawale ʻia nā ʻāpana maʻamau e like me nā lawelawe). ʻO ka pae hoʻolālā o kā mākou puʻupuʻu e like me ka hoʻoili ʻana i ka CMD i kā mākou kiʻi nui i hoʻonohonoho ʻia a, ma ke ʻano nui, ʻaʻohe ʻokoʻa i ka hoʻolaha ʻana i wehewehe ʻia ma ka Māhele I.

git clone... - loaʻa iā mākou nā faila e pono ai e hoʻokō (createconfig.py, install_venv.sh, etc.)
wili... && wehe i ka zip... - hoʻoiho a wehe i nā mea hana i kūkulu ʻia (nā pono i hui ʻia)

Hoʻokahi wale nō pilikia i wehewehe ʻole ʻia: ʻaʻole hiki ke loaʻa nā ʻāpana i loaʻa i kahi kikowaena pūnaewele mai nā polokalamu kele pūnaewele. Hoʻopau mākou i kēia pilikia me ka hoʻohana ʻana i ka proxy reverse, penei:

I loko o .gitlab-ci.yml, ma hope o ka hoʻouka ʻana i ka puʻupuʻu puʻupuʻu, e hoʻohui i kahi laina no ka waiho ʻana i ka mea kaulike (ʻo ia ka mea, i ka wā e hana ai, e hoʻoponopono wale i kāna hoʻonohonoho (hana i nā faila hoʻonohonoho nginx hou e like me ke ʻano: /etc/nginx/conf.d /${CI_COMMIT_REF_NAME}.conf) - e nānā i ke code docker-compose-nginx.yml)

    - docker stack deploy -c docker-compose-nginx.yml ${CI_ENVIRONMENT_NAME} --with-registry-auth

docker-compose-nginx.yml

---
version: '3'

services:
  nginx:
    image: nginx:latest
    environment:
      CI_COMMIT_REF_NAME: ${CI_COMMIT_REF_NAME}
      NGINX_CONFIG: |-
            server {
                listen 8080;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:8080;
                }
            }
            server {
                listen 5555;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:5555;
                }
            }
    volumes:
      - /tmp/staging/nginx:/etc/nginx/conf.d
    command:
      /bin/bash -c "echo -e "$$NGINX_CONFIG" > /etc/nginx/conf.d/${CI_COMMIT_REF_NAME}.conf;
        nginx -g "daemon off;";
        /etc/init.d/nginx reload"
    ports:
      - 8080:8080
      - 5555:5555
      - 3000:3000
      - 443:443
      - 80:80
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Ma nā kamepiula hoʻomohala, hōʻano hou /etc/hosts; hoʻonoho i ka url i nginx:

10.50.173.106 staging_BRANCH-1831_cluster.dev
No laila, ua hoʻokō ʻia ka hoʻonohonoho ʻana o nā pūʻulu staging kaʻawale a hiki i nā mea hoʻomohala ke hoʻolauna iā lākou i kēlā me kēia nui e hoʻāʻo ai i kā lākou mau hana.

Nā papahana e hiki mai ana:

E hoʻokaʻawale i kā mākou mau ʻāpana e like me nā lawelawe
E hana i kahi Dockerfile no kēlā me kēia
ʻIke ʻakomi i nā node liʻiliʻi i hoʻouka ʻia ma ka waihona
E wehewehe i nā node me ka hoʻohana ʻana i ka laʻana inoa (ma mua o ka hoʻohana ʻana i ka id e like me ka ʻatikala)
Hoʻohui i kahi hōʻoia ua luku ʻia ka waihona
...

Mahalo nui no ʻO kahi mea kākau.

Source: www.habr.com

E hoʻolālā i nā noi me ka hoʻohana ʻana iā Docker Swarm

Māhele I

Mahele II

Pākuʻi i ka manaʻo hoʻopuka ana pane