🥇Disa zvikumbiro uchishandisa Docker Swarm

Iyo yepamhepo vhidhiyo yemukati yekurudziro sisitimu yatiri kushanda pairi yakavharwa yekushambadzira uye tekinoroji izhinji-chikamu cluster yevaridzi uye yakavhurika sosi zvikamu. Chinangwa chekunyora chinyorwa ichi ndechekutsanangura kuitwa kweiyo docker swarm clustering system yepuratifomu, pasina kukanganisa kufambiswa kwekushanda kwemaitiro edu pasi penguva shoma. Rondedzero yaunzwa kwauri yakakamurwa kuita zvikamu zviviri. Chikamu chekutanga chinotsanangura CI / CD usati washandisa docker swarm, uye chikamu chechipiri chinotsanangura maitiro ekuzviita. Vaya vasingafariri kuverenga chikamu chekutanga vanogona kuenda kune yechipiri vakachengeteka.

Chikamu I

Pane imwe nguva, paive nekudikanwa kwekumisikidza CI / CD maitiro nekukurumidza sezvinobvira. Imwe yemamiriro ezvinhu yaive yekusashandisa Docker yekutumirwa zvikamu zvinogadziriswa nekuda kwezvikonzero zvakawanda:

kune yakavimbika uye yakagadzikana mashandiro ezvikamu muKugadzira (kureva, muchidimbu, chinodiwa kusashandisa virtualization)
vanotungamira vanogadzira vaisada kushanda neDocker (zvinoshamisa, asi ndozvazvaive)
nekuda kwezvikonzero zvepfungwa zveR&D manejimendi

Zvivako, stack uye fungidziro yekutanga zvinodiwa zveMVP zvaive zvinotevera:

4 Intel® X5650 maseva ane Debian (mumwe muchina une simba wakazara wekusimudzira)
Kuvandudzwa kwezvimiro zvako zvetsika kunoitwa muC ++, Python3
Main 3rd party maturusi anoshandiswa: Kafka, Clickhouse, Airflow, Redis, Grafana, Postgresql, Mysql, ...
Mapaipi ekuvaka uye ekuyedza zvikamu zvakasiyana kuti zvigadzirise uye kuburitswa

Imwe yemibvunzo yekutanga inoda kugadziriswa padanho rekutanga nderekuti zvikamu zvetsika zvichaiswa sei mune chero nharaunda (CI/CD).

Isu takasarudza kuisa yechitatu-bato zvikamu systemic uye nekuzvivandudza systemically. Tsika dzekushandisa dzakagadzirwa muC ++ kana Python dzinogona kuiswa munzira dzinoverengeka. Pakati pavo, semuenzaniso: kugadzira masisitimu mapakeji, kuvatumira kune repository yemifananidzo yakaunganidzwa uye yavo yakatevera yekumisikidza pamaseva. Nekuda kwechikonzero chisati chazivikanwa, imwe nzira yakasarudzwa, iyo: kushandisa CI, mafaera ekushandisa anounganidzwa, chaiyo purojekiti nharaunda inogadzirwa, py modules kubva kuzvinodiwa.txt inoiswa, uye zvese izvi zvigadzirwa zvinotumirwa pamwe chete ne configs, zvinyorwa uye. iyo inoperekedza application nharaunda kune maseva. Tevere, zvikumbiro zvinotangwa kubva kune chaiwo mushandisi asina kodzero yemutungamiriri.

Gitlab-CI yakasarudzwa seCI/CD system. Iyo pombi yakaguma yakaita seizvi:

Nechimiro, gitlab-ci.yml yaitaridzika seizvi:

---
variables:
  # минимальная версия ЦПУ на серверах, где разворачивается кластер
  CMAKE_CPUTYPE: "westmere"

  DEBIAN: "MYREGISTRY:5000/debian:latest"

before_script:
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh && echo -e "Host *ntStrictHostKeyChecking nonn" > ~/.ssh/config

stages:
  - build
  - testing
  - deploy

debug.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always
release.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always

## testing stage
tests.codestyle:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -t codestyle -b "${CI_COMMIT_REF_NAME}_codestyle"
tests.debug.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - debug.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_debug"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week
tests.release.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_release"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week

## staging stage
deploy_staging:
  stage: deploy
  environment: staging
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - cd scripts/deploy/ &&
        python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
        /bin/bash install_venv.sh -d -r ../../requirements.txt &&
        python3 prepare_init.d.py &&
        python3 deploy.py -s $CI_ENVIRONMENT_NAME
  when: manual

Zvinokosha kucherechedza kuti kuungana uye kuyedzwa kunoitwa pamufananidzo wayo, apo ese anodiwa system mapakeji akatoiswa uye mamwe marongero anogadzirwa.

Kunyangwe chimwe nechimwe chezvinyorwa izvi mumabasa chichinakidza nenzira yaro, ini handitaure nezvavo; kutsanangura chimwe nechimwe chazvo zvinotora nguva yakawanda uye ichi hachisi chinangwa chechinyorwa. Rega ndingokwevera pfungwa dzako kune chokwadi chekuti nhanho yekuendesa ine kutevedzana kwekufona zvinyorwa:

createconfig.py - inogadzira faira resetting.ini rine zvigadziriso zvezvikamu munzvimbo dzakasiyana dzekuzotumirwa kunotevera (Preproduction, Production, Testing, ...)
install_venv.sh -Inogadzira nharaunda chaiyo yezvikamu zvepy mune yakatarwa dhairekitori uye inoikopa kumaseva ari kure
prepare_init.d.py - inogadzirira zvinyorwa zvekutanga-kumisa zvikamu zvinoenderana netemplate
deploy.py - inoshandisa uye inotangazve zvinhu zvitsva

Nguva yakafamba. Danho rekuita rakatsiviwa nepreproduction nekugadzirwa. Tsigiro yechigadzirwa yakawedzerwa pane imwezve kugovera (CentOS). Mamwe maseva mashanu ane simba emuviri uye gumi nemaviri chaiwo akawedzerwa. Uye zvakawedzera kuomera vagadziri uye vaedzi kuti vaedze mabasa avo munzvimbo iri pedyo kana kushoma kunharaunda yekushanda. Panguva iyi zvakava pachena kuti hazvaigona kuita pasina iye ...

Chikamu II

Saka, cluster yedu inoshamisa sisitimu yezvishoma gumi nemaviri zvikamu zvisina kutsanangurwa neDockerfiles. Iwe unogona kuzvigadzirisa kuti iendeswe kune imwe nharaunda chete muhuwandu. Basa redu ndere kuendesa cluster munzvimbo yekutamba kuti iedze isati yaburitswa isati yaburitswa.

Tichifunga, panogona kuve nemasumbu akati wandei anoshanda panguva imwe chete: akawanda sekunge paine mabasa ari muchimiro chakapedzwa kana pedyo nekupedzwa. Simba remaseva atiinawo rinotitendera kuti timhanye masumbu akati wandei pane imwe neimwe server. Imwe neimwe dhizaini sumbu inofanirwa kuve yega (hapafanirwe kuve nekupindirana mumachiteshi, madhairekitori, nezvimwewo).

Chishandiso chedu chinonyanya kukosha inguva yedu, uye takanga tisina zvakawanda zvayo.

Kutanga nekukurumidza, isu takasarudza Docker Swarm nekuda kwekureruka kwayo uye inochinjika dhizaini. Chinhu chekutanga chatakaita kugadzira maneja uye node akati wandei pamaseva ari kure:

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
kilqc94pi2upzvabttikrfr5d     nop-test-1     Ready               Active                                  19.03.2
jilwe56pl2zvabupryuosdj78     nop-test-2     Ready               Active                                  19.03.2
j5a4yz1kr2xke6b1ohoqlnbq5 *   nop-test-3     Ready               Active              Leader              19.03.2

Zvadaro, takagadzira network:


$ docker network create --driver overlay --subnet 10.10.10.0/24 nw_swarm

Tevere, takabatanidza maGitlab-CI uye Swarm node maererano nekutonga kure kwemanodhi kubva kuCI: kuisa zvitupa, kuseta zvakavanzika zvakasiyana, uye zvakare kumisikidza iyo Docker sevhisi pane manejimendi server. Uyu chinyorwa akatiponesa nguva yakawanda.

Zvadaro, takawedzera mabasa ekugadzira nekuparadza stack mu .gitlab-ci .yml.

Mamwe mabasa akati wandei akawedzerwa ku .gitlab-ci .yml

## staging stage
deploy_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
    CI_BIN_DEPENDENCIES_JOB: "release.centos.7"
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack deploy -c docker-compose.yml ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME} --with-registry-auth
    - rm -rf $DOCKER_CERT_PATH
  when: manual

## stop staging stage
stop_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack rm ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME}
    # TODO: need check that stopped
  when: manual

Kubva pane iri pamusoro kodhi chidimbu zviri pachena kuti mabhatani maviri akawedzerwa kuPipelines (deploy_staging, stop_staging) inoda manyore kuita.

Zita remastaki rinoenderana nezita rebazi uye kusarudzika uku kunofanira kukwana. Masevhisi ari mu stack anogashira yakasarudzika IP kero, uye madoko, madhairekitori, nezvimwe. ichave yakaparadzaniswa, asi yakafanana kubva kune stack kusvika kune stack (sezvo iyo faira yekumisikidza yakafanana kune ese mastacks) - ndizvo zvataida. Isu tinoshandisa stack (cluster) tichishandisa docker-kunyora.yml, iyo inotsanangura sumbu redu.

docker-kunyora.yml

---
version: '3'

services:
  userprop:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:
  celery_bcd:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  schedulerdb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: schedulerdb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    command: ['--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci', '--explicit_defaults_for_timestamp=1']
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  celerydb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: celerydb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  cluster:
    image: $CENTOS7
    environment:
      - CENTOS
      - CI_ENVIRONMENT_NAME
      - CI_API_V4_URL
      - CI_REPOSITORY_URL
      - CI_PROJECT_ID
      - CI_PROJECT_URL
      - CI_PROJECT_PATH
      - CI_PROJECT_NAME
      - CI_COMMIT_REF_NAME
      - CI_BIN_DEPENDENCIES_JOB
    command: >
      sudo -u myusername -H /bin/bash -c ". /etc/profile &&
        mkdir -p /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
            git clone -b $CI_COMMIT_REF_NAME $CI_REPOSITORY_URL . &&
            curl $CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=$CI_BIN_DEPENDENCIES_JOB -o artifacts.zip &&
            unzip artifacts.zip ;
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME/scripts/deploy/ &&
            python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
            /bin/bash install_venv.sh -d -r ../../requirements.txt &&
            python3 prepare_init.d.py &&
            python3 deploy.py -s $CI_ENVIRONMENT_NAME"
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    tty: true
    stdin_open: true
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Pano iwe unogona kuona kuti zvikamu zvakabatana netiweki imwe (nw_swarm) uye zvinowanikwa kune mumwe nemumwe.

Zvikamu zveSistimu (zvichibva pane redis, mysql) zvakapatsanurwa kubva padziva rakawanda rezvikamu zvetsika (muzvirongwa, zvikamu zvetsika zvakakamurwawo sevhisi). Nhanho yekuendesa yeboka redu inoita sekuendesa CMD kumufananidzo wedu mukuru wakagadziridzwa uye, kazhinji, haina kumbosiyana nekutumirwa kunotsanangurwa muChikamu I. Ndichasimbisa kusiyana:

git clone... - tinowana mafaera anodiwa kuita deployment (createconfig.py, install_venv.sh, nezvimwewo)
curl... && unzip... - Dhawunirodha uye unzip zvivakwa zvekuvaka (zvakaunganidzwa zvinoshandiswa)

Pane dambudziko rimwe chete risati ratsanangurwa: zvikamu zvine web interface hazviwanikwe kubva kune vanogadzira. Isu tinogadzirisa dambudziko iri tichishandisa reverse proxy, nekudaro:

Mu .gitlab-ci.yml, mushure mekutumira mutsara wechikwata, wedzera mutsara wekuisa balancer (iyo, kana yazvipira, inongogadzirisa kugadzirisa kwayo (inogadzira mafaira matsva ekugadzirisa nginx maererano ne template: /etc/nginx/conf.d /${CI_COMMIT_REF_NAME}.conf) - ona kodhi docker-compose-nginx.yml)

    - docker stack deploy -c docker-compose-nginx.yml ${CI_ENVIRONMENT_NAME} --with-registry-auth

docker-compose-nginx.yml

---
version: '3'

services:
  nginx:
    image: nginx:latest
    environment:
      CI_COMMIT_REF_NAME: ${CI_COMMIT_REF_NAME}
      NGINX_CONFIG: |-
            server {
                listen 8080;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:8080;
                }
            }
            server {
                listen 5555;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:5555;
                }
            }
    volumes:
      - /tmp/staging/nginx:/etc/nginx/conf.d
    command:
      /bin/bash -c "echo -e "$$NGINX_CONFIG" > /etc/nginx/conf.d/${CI_COMMIT_REF_NAME}.conf;
        nginx -g "daemon off;";
        /etc/init.d/nginx reload"
    ports:
      - 8080:8080
      - 5555:5555
      - 3000:3000
      - 443:443
      - 80:80
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Pamakombiyuta ekuvandudza, gadzirisa /etc/hosts; isa url ku nginx:

10.50.173.106 staging_BRANCH-1831_cluster.dev
Saka, kuendeswa kwemasumbu ega ega kwaitwa uye vanogadzira vanogona ikozvino kuamisa mune chero huwandu hwakakwana kuyedza mabasa avo.

Zvirongwa zveramangwana:

Kuparadzanisa zvikamu zvedu sesevhisi
Gadzira Dockerfile yega yega
Zvionere otomatiki node dzisina kuremerwa mustack
Rondedzera node uchishandisa zita template (pane kushandisa id sezviri muchinyorwa)
Wedzera cheki kuti stack yaparadzwa
...

Kutenda kwakakosha chinyorwa.

Source: www.habr.com

Shandisa zvikumbiro uchishandisa Docker Swarm

Chikamu I

Chikamu II

Voeg kukanzura mhinduro