ProHoster > Blog > Pulega > Fa'asoa talosaga e fa'aaoga ai le Docker Swarm

Fa'asoa talosaga e fa'aaoga ai le Docker Swarm

O le faiga fa'atonuga o mea i luga ole laiga i luga ole laiga o lo'o matou galulue ai ose atina'e fa'apisinisi tapuni ma e fa'atekinisi ose fa'aputuga tele o vaega fa'apitoa ma fa'apogai matala. O le faʻamoemoega o le tusiaina o lenei tusiga o le faʻamatalaina lea o le faʻatinoina o le faʻaogaina o le docker swarm clustering system mo se faʻatulagaga faʻatulagaina, e aunoa ma le faʻalavelaveina o le faʻatulagaina o galuega a tatou faiga i lalo ole taimi faatapulaaina. O le fa'amatalaga o lo'o tu'uina atu ia te oe ua vaevaeina i ni vaega se lua. O le vaega muamua o loʻo faʻamatalaina le CI / CD aʻo leʻi faʻaogaina le docker swarm, ma le vaega lona lua o loʻo faʻamatalaina le faagasologa o le faʻatinoina. O i latou e le fiafia e faitau le vaega muamua e mafai ona saogalemu agai i le lona lua.

Vaega I

I se tasi taimi, sa i ai se manaʻoga e faʻapipiʻi se faiga CI / CD i se taimi vave e mafai ai. O se tasi o tulaga o le le faʻaaogaina o Docker mo le fa'atulagaina vaega ua atiaeina mo ni mafuaaga se tele:

  • mo le sili atu ona faʻatuatuaina ma mautu le faʻaogaina o vaega i le Gaosiga (o lona uiga, o le mea moni, o le manaʻoga e aua le faʻaogaina le virtualization)
  • taʻitaʻia atinaʻe e leʻi manaʻo e galulue ma Docker (ese, ae o le mea lena)
  • mo mafua'aga fa'ale-mafaufau ole pulega ole R&D

O mea tetele, fa'aputu ma fa'atatau muamua mana'oga mo le MVP e fa'apea:

  • 4 Intel® X5650 servers ma Debian (se tasi masini malosi atoatoa mo le atinaʻe)
  • O le atina'eina o au lava vaega masani o lo'o faia ile C++, Python3
  • Meafaigaluega 3rdparty faʻaaogaina: Kafka, Clickhouse, Airflow, Redis, Grafana, Postgresql, Mysql, ...
  • Pipeline mo le faufale ma le su'ega vaega eseese mo le debug ma le tatalaina

O se tasi o fesili muamua e manaʻomia ona foia i le laasaga muamua o le faʻafefea ona faʻapipiʻiina vaega faʻapitoa i soʻo se siosiomaga (CI / CD).

Na matou filifili e faʻapipiʻi vaega lona tolu faʻapitoa ma faʻafouina i latou. O talosaga faʻapitoa e atiaʻe ile C++ poʻo le Python e mafai ona faʻapipiʻiina i le tele o auala. Faatasi ai ma i latou, mo se faʻataʻitaʻiga: fatuina o pusa faʻapipiʻi, auina atu i le fale teu oloa o ata aoina ma latou faʻapipiʻi mulimuli ane i luga o sapalai. Mo se mafuaʻaga e leʻi iloa, na filifilia ai se isi auala, e pei o le: faʻaaogaina o le CI, faʻapipiʻiina faila faila, faʻapipiʻiina se siosiomaga faʻataʻitaʻiga, faʻapipiʻi py modules mai requirements.txt, ma o nei mea uma e lafoina faʻatasi ma configs, scripts ma le si'osi'omaga o lo'o tu'u fa'atasi i 'au'aunaga. O le isi, o talosaga e faʻalauiloa mai se tagata faʻaoga faʻapitoa e aunoa ma aia tatau pule.

Gitlab-CI na filifilia e avea ma CI / CD system. O le laina paipa na maua e pei o lenei:

Fa'asoa talosaga e fa'aaoga ai le Docker Swarm
I le fausaga, gitlab-ci.yml e pei o lenei:

---
variables:
  # минимальная версия ЦПУ на серверах, где разворачивается кластер
  CMAKE_CPUTYPE: "westmere"

  DEBIAN: "MYREGISTRY:5000/debian:latest"

before_script:
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh && echo -e "Host *ntStrictHostKeyChecking nonn" > ~/.ssh/config

stages:
  - build
  - testing
  - deploy

debug.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always
release.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always

## testing stage
tests.codestyle:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -t codestyle -b "${CI_COMMIT_REF_NAME}_codestyle"
tests.debug.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - debug.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_debug"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week
tests.release.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_release"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week

## staging stage
deploy_staging:
  stage: deploy
  environment: staging
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - cd scripts/deploy/ &&
        python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
        /bin/bash install_venv.sh -d -r ../../requirements.txt &&
        python3 prepare_init.d.py &&
        python3 deploy.py -s $CI_ENVIRONMENT_NAME
  when: manual

E taua le maitauina o le faʻapotopotoga ma le suʻega e faia i luga o lona lava ata, lea ua uma ona faʻapipiʻiina uma pusa faʻaoga talafeagai ma faia isi faʻatulagaga.

E ui lava o nei tusitusiga taʻitasi i galuega e manaia i lana lava auala, e mautinoa lava ou te le talanoa e uiga ia i latou; o le faʻamatalaina o latou taʻitasi o le a umi se taimi ma e le o le faʻamoemoega lea o le tusiga. Sei ou tosina atu lou mafaufau i le mea moni e faapea o le laasaga o le faʻapipiʻiina e aofia ai se faasologa o le valaʻau o tusitusiga:

  1. createconfig.py - faia se faila settings.ini ma faʻatulagaina mo vaega i siosiomaga eseese mo le faʻapipiʻiina mulimuli ane (Preproduction, Production, Testing, ...)
  2. install_venv.sh - fa'atupuina se si'osi'omaga fa'anatura mo vaega o le py i totonu o se lisi fa'apitoa ma kopiina i 'au'aunaga mamao
  3. prepare_init.d.py — saunia tusitusiga mo vaega amata-taofi e faavae i le mamanu
  4. deploy.py - fa'apipi'i ma toe amata vaega fou

Ua mavae le taimi. O le tulaga fa'atulagaina na suia i le mua'i gaosia ma le gaosiga. Lagolago mo le oloa ua faʻaopoopoina i luga o le isi tufatufaga (CentOS). O isi 'auʻaunaga faʻaletino malosi e 5 ma le tele o mea faʻapitoa na faʻaopoopoina. Ma ua atili ai ona faigata mo tagata atiaʻe ma suʻega e faʻataʻitaʻi a latou galuega i se siosiomaga e sili atu pe itiiti ifo i le tulaga faigaluega. O le taimi lea na manino ai e le mafai ona faia e aunoa ma ia ...

Vaega II

Fa'asoa talosaga e fa'aaoga ai le Docker Swarm

O lea la, o la matou fuifui o se faiga mataʻina o ni nai vaega taʻitoʻatasi e leʻo faʻamatalaina e Dockerfiles. E mafai ona e fa'atulagaina mo le fa'apipi'iina i se si'osi'omaga fa'apitoa na'o le lautele. O la matou galuega o le faʻapipiʻiina o le fuifui i totonu o se siosiomaga faʻatulagaina e faʻataʻitaʻi ai aʻo leʻi tuʻuina atu suʻega.

I le talitonuga, e mafai ona tele ni fuifui e galulue i le taimi e tasi: o le tele o loʻo i ai galuega i se setete maeʻa pe latalata ile maeʻa. O le malosi o le 'au'aunaga o lo'o matou fa'aogaina e mafai ai ona matou fa'atautaia ni fuifui i luga o fa'aumau ta'itasi. O fuifui ta'itasi ta'itasi e tatau ona tu'u'ese (e le tatau ona so'a i ports, directories, etc.).

O la tatou punaoa sili ona taua o lo tatou taimi, ma e le'i tele ni a tatou mea.

Mo se amataga vave, matou filifilia Docker Swarm ona o lona faigofie ma fetuutuunai fausaga fausaga. O le mea muamua na matou faia o le fatuina lea o se pule ma nisi nodes i luga o sapalai mamao:

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
kilqc94pi2upzvabttikrfr5d     nop-test-1     Ready               Active                                  19.03.2
jilwe56pl2zvabupryuosdj78     nop-test-2     Ready               Active                                  19.03.2
j5a4yz1kr2xke6b1ohoqlnbq5 *   nop-test-3     Ready               Active              Leader              19.03.2

Le isi, matou faia se feso'otaiga:


$ docker network create --driver overlay --subnet 10.10.10.0/24 nw_swarm

O le isi, matou faʻafesoʻotaʻi Gitlab-CI ma Swarm nodes i tulaga o le pulega mamao o nodes mai CI: faʻapipiʻi tusi faamaonia, faʻatulagaina suiga faalilolilo, ma faʻapipiʻi foʻi le auaunaga Docker i luga o le server pule. Le tasi lea se tusiga faasaoina i matou i le tele o taimi.

O le isi, matou faʻaopoopoina galuega mo le fatuina ma le faʻaleagaina o le faaputuga i .gitlab-ci .yml.

E tele isi galuega ua fa'aopoopo i le .gitlab-ci .yml

## staging stage
deploy_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
    CI_BIN_DEPENDENCIES_JOB: "release.centos.7"
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack deploy -c docker-compose.yml ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME} --with-registry-auth
    - rm -rf $DOCKER_CERT_PATH
  when: manual

## stop staging stage
stop_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack rm ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME}
    # TODO: need check that stopped
  when: manual

Mai le vaega o loʻo i luga o loʻo manino mai e lua faʻamau na faʻaopoopoina i Pipelines (deploy_staging, stop_staging) e manaʻomia ai le faʻatinoga o le tusi.

Fa'asoa talosaga e fa'aaoga ai le Docker Swarm
O le igoa o le faaputuga e fetaui ma le igoa o le lala ma o lenei tulaga ese e tatau ona lava. Au'aunaga i totonu o le faaputuga e maua ai tuatusi IP tulaga ese, ma ports, directories, etc. o le a faʻaesea, ae tutusa mai le faʻaputu i le faʻaputu (talu ai o le faila faʻatulagaina e tutusa mo faʻaputu uma) - o le mea lena na matou manaʻo ai. Matou te fa'apipi'iina le fa'aputuga (cluster) fa'aaoga faicker-compose.yml, lea e faamatalaina ai la tatou fuifui.

faicker-compose.yml

---
version: '3'

services:
  userprop:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:
  celery_bcd:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  schedulerdb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: schedulerdb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    command: ['--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci', '--explicit_defaults_for_timestamp=1']
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  celerydb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: celerydb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  cluster:
    image: $CENTOS7
    environment:
      - CENTOS
      - CI_ENVIRONMENT_NAME
      - CI_API_V4_URL
      - CI_REPOSITORY_URL
      - CI_PROJECT_ID
      - CI_PROJECT_URL
      - CI_PROJECT_PATH
      - CI_PROJECT_NAME
      - CI_COMMIT_REF_NAME
      - CI_BIN_DEPENDENCIES_JOB
    command: >
      sudo -u myusername -H /bin/bash -c ". /etc/profile &&
        mkdir -p /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
            git clone -b $CI_COMMIT_REF_NAME $CI_REPOSITORY_URL . &&
            curl $CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=$CI_BIN_DEPENDENCIES_JOB -o artifacts.zip &&
            unzip artifacts.zip ;
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME/scripts/deploy/ &&
            python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
            /bin/bash install_venv.sh -d -r ../../requirements.txt &&
            python3 prepare_init.d.py &&
            python3 deploy.py -s $CI_ENVIRONMENT_NAME"
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    tty: true
    stdin_open: true
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

O iinei e mafai ona e vaʻaia ai o vaega e fesoʻotaʻi e le tasi fesoʻotaʻiga (nw_swarm) ma e mafai ona maua e le tasi le isi.

O vaega faʻavae (faʻavae i luga ole redis, mysql) e vavae ese mai le vaitaele lautele o vaega faʻapitoa (i fuafuaga, o vaega faʻapitoa e vaevaeina foi e avea ma auʻaunaga). O le faʻatulagaina o la matou fuifui e foliga mai o le tuʻuina atu o le CMD i la matou ata tele faʻapipiʻi ma, i se tulaga lautele, e toetoe lava a leai se eseesega mai le faʻapipiʻiina o loʻo faʻamatalaina i le Vaega I. O le a ou faamamafaina le eseesega:

  • git clone... - matou te maua faila e manaʻomia e faʻatino ai le faʻatulagaina (createconfig.py, install_venv.sh, ma isi)
  • pipi'i... && tatala le sipa... - la'u mai ma tatala le sipuni mea fau (fa'aoga tu'ufa'atasi)

E na'o le tasi le fa'afitauli e le'i fa'amatalaina: o vaega o lo'o i ai se upega tafa'ilagi e le mafai ona maua mai i tagata su'esu'e. Matou te foia lenei faʻafitauli e faʻaaoga ai le sui sui, faʻapea:

I le .gitlab-ci.yml, a maeʻa ona faʻapipiʻi le faaputuga fuifui, faʻaopoopo se laina mo le faʻapipiʻiina o le paleni (lea, pe a faʻatinoina, naʻo le faʻafouina o lona faʻatulagaga (fausia faila fou o le nginx e tusa ai ma le mamanu: /etc/nginx/conf.d /${CI_COMMIT_REF_NAME}.conf) - va'ai le code docker-compose-nginx.yml)

    - docker stack deploy -c docker-compose-nginx.yml ${CI_ENVIRONMENT_NAME} --with-registry-auth

docker-compose-nginx.yml

---
version: '3'

services:
  nginx:
    image: nginx:latest
    environment:
      CI_COMMIT_REF_NAME: ${CI_COMMIT_REF_NAME}
      NGINX_CONFIG: |-
            server {
                listen 8080;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:8080;
                }
            }
            server {
                listen 5555;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:5555;
                }
            }
    volumes:
      - /tmp/staging/nginx:/etc/nginx/conf.d
    command:
      /bin/bash -c "echo -e "$$NGINX_CONFIG" > /etc/nginx/conf.d/${CI_COMMIT_REF_NAME}.conf;
        nginx -g "daemon off;";
        /etc/init.d/nginx reload"
    ports:
      - 8080:8080
      - 5555:5555
      - 3000:3000
      - 443:443
      - 80:80
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

I luga o komepiuta atiaʻe, faʻafouina /etc/hosts; seti le url i le nginx:

10.50.173.106 staging_BRANCH-1831_cluster.dev

O lea la, ua fa'atinoina le fa'apipi'iina o fuifui tu'ufua ma ua mafai nei e le au atina'e ona fa'alauiloa i so'o se aofa'iga e lava e fa'ata'ita'i ai a latou galuega.

Fuafuaga i le lumana'i:

  • Vaevae a tatou vaega e fai ma tautua
  • Fausia se Dockerfile mo taʻitasi
  • E otometi lava ona su'esu'e pona e itiiti ifo le uta i totonu o le faaputuga
  • Fa'ama'oti nodes e fa'aaoga ai se fa'ata'ita'iga igoa (nai lo le fa'aogaina o id e pei ona i totonu o le tusiga)
  • Fa'aopoopo se siaki ua fa'aumatia le fa'aputu
  • ...

Faafetai faapitoa mo tusiga.

puna: www.habr.com

Faaopoopo i ai se faamatalaga