ProHoster > Блог > Kev tswj hwm > Xa cov ntawv thov nrog Docker Swarm

Xa cov ntawv thov nrog Docker Swarm

Cov lus pom zoo hauv online video uas peb tab tom ua haujlwm yog kev tsim kho kev lag luam kaw thiab yog technically ib pawg ntau yam ntawm cov tswv cuab thiab qhib qhov chaw. Lub hom phiaj ntawm kev sau tsab xov xwm no yog piav qhia txog kev siv cov docker swarm clustering system rau ib qhov chaw staging yam tsis muaj kev cuam tshuam rau kev tsim ua haujlwm ntawm peb cov txheej txheem hauv lub sijhawm txwv. Cov lus piav qhia rau koj mloog tau muab faib ua ob ntu. Thawj ntu piav qhia txog CI / CD ua ntej siv docker swarm, thiab qhov thib ob piav qhia txog cov txheej txheem ntawm nws qhov kev siv. Cov uas tsis nyiam nyeem thawj ntu tuaj yeem txav mus rau qhov thib ob.

Nqe I

Rov qab rau xyoo nyob deb, nyob deb, nws yog ib qho tsim nyog los teeb tsa cov txheej txheem CI / CD kom sai li sai tau. Ib qho ntawm cov xwm txheej tsis yog siv Docker rau kev xa tawm tsim cov khoom siv rau ntau yam laj thawj:

  • rau ntau txhim khu kev qha thiab ruaj khov kev ua haujlwm ntawm cov khoom hauv Kev Tsim Khoom (uas yog, qhov tseeb, qhov yuav tsum tsis txhob siv virtualization)
  • cov thawj coj tsim tawm tsis xav ua haujlwm nrog Docker (weird, tab sis qhov ntawd yog li cas)
  • raws li kev xav txog kev xav ntawm R&D kev tswj hwm

Infrastructure, pawg thiab kwv yees thawj qhov yuav tsum tau ua rau MVP tau nthuav tawm raws li hauv qab no:

  • 4 Intel® X5650 servers nrog Debian (ib lub tshuab muaj zog dua yog tsim tiav)
  • Kev txhim kho ntawm tus kheej cov khoom siv yog ua tiav hauv C ++, Python3
  • Cov cuab yeej tseem ceeb thib 3 siv: Kafka, Clickhouse, Airflow, Redis, Grafana, Postgresql, Mysql, ...
  • Cov kav dej rau kev tsim thiab kuaj cov khoom sib cais rau kev debug thiab tso tawm

Ib qho ntawm thawj cov lus nug uas yuav tsum tau hais nyob rau hauv thawj theem yog yuav ua li cas cov kev cai Cheebtsam yuav tsum deployed nyob rau hauv ib puag ncig (CI / CD).

Peb tau txiav txim siab los nruab cov khoom thib peb hauv qhov system thiab hloov kho lawv cov txheej txheem. Cov ntawv thov kev cai tsim hauv C ++ lossis Python tuaj yeem xa mus rau ntau txoj hauv kev. Ntawm lawv, piv txwv li: tsim cov pob khoom, xa lawv mus rau qhov chaw cia khoom ntawm cov duab tsim thiab tom qab ntawd txhim kho lawv ntawm servers. Rau qhov tsis paub yog vim li cas, lwm txoj kev raug xaiv, uas yog: siv CI, daim ntawv thov ua tiav cov ntaub ntawv raug muab tso ua ke, ib puag ncig virtual project tsim, py modules tau teeb tsa los ntawm cov tseev kom muaj.txt, thiab tag nrho cov khoom cuav no raug xa mus nrog rau configs, scripts thiab cov nrog rau daim ntawv thov ib puag ncig rau servers. Tom ntej no, cov ntawv thov raug tso tawm ua tus neeg siv virtual tsis muaj cai tswj hwm.

Gitlab-CI tau raug xaiv los ua CI / CD system. Cov kav dej uas tau tshwm sim zoo li no:

Xa cov ntawv thov nrog Docker Swarm
Tus qauv, gitlab-ci.yml zoo li qhov no

---
variables:
  # минимальная версия ЦПУ на серверах, где разворачивается кластер
  CMAKE_CPUTYPE: "westmere"

  DEBIAN: "MYREGISTRY:5000/debian:latest"

before_script:
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh && echo -e "Host *ntStrictHostKeyChecking nonn" > ~/.ssh/config

stages:
  - build
  - testing
  - deploy

debug.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always
release.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always

## testing stage
tests.codestyle:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -t codestyle -b "${CI_COMMIT_REF_NAME}_codestyle"
tests.debug.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - debug.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_debug"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week
tests.release.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_release"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week

## staging stage
deploy_staging:
  stage: deploy
  environment: staging
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - cd scripts/deploy/ &&
        python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
        /bin/bash install_venv.sh -d -r ../../requirements.txt &&
        python3 prepare_init.d.py &&
        python3 deploy.py -s $CI_ENVIRONMENT_NAME
  when: manual

Nws yog ib qho tsim nyog sau cia tias kev sib dhos thiab kev sim ua tiav ntawm nws tus kheej cov duab, qhov twg tag nrho cov txheej txheem tsim nyog tau raug teeb tsa thiab lwm qhov chaw tau ua.

Txawm hais tias txhua yam ntawm cov ntawv no hauv cov haujlwm yog qhov nthuav rau nws tus kheej, tab sis kuv yuav tsis tham txog lawv.Cov lus piav qhia ntawm lawv txhua tus yuav siv sijhawm ntau thiab qhov no tsis yog lub hom phiaj ntawm tsab xov xwm. Kuv tsuas yog kos koj cov xim rau qhov tseeb tias qhov kev xa tawm theem muaj ib ntu ntawm kev hu cov ntawv sau:

  1. tsimconfig.py - tsim cov ntaub ntawv settings.ini nrog cov chaw teeb tsa hauv ntau qhov chaw rau kev xa tawm tom ntej (Preproduction, Production, Testing, ...)
  2. install_venv.sh - tsim ib puag ncig virtual rau py Cheebtsam hauv cov npe tshwj xeeb thiab luam nws mus rau cov chaw taws teeb tswj
  3. npaj_init.d.py - npaj cov ntawv pib-nres rau cov khoom raws li tus qauv
  4. deploy.py - decomposes thiab restarts tshiab Cheebtsam

Lub sij hawm dhau mus. Cov theem theem tau hloov los ntawm preproduction thiab ntau lawm. Ntxiv kev txhawb nqa rau cov khoom ntawm ib qho kev faib tawm ntxiv (CentOS). Ntxiv 5 lub cev muaj zog ntau dua thiab ntau lub tshuab virtual. Thiab nws tau dhau los ua nyuaj rau cov neeg tsim khoom thiab cov neeg sim ntsuas lawv cov dej num hauv ib puag ncig ntau dua los yog tsawg dua nyob ze rau lub xeev ua haujlwm. Lub sijhawm no, nws tau pom tseeb tias nws tsis tuaj yeem ua tsis muaj nws ...

Ntu II

Xa cov ntawv thov nrog Docker Swarm

Yog li, peb pawg yog qhov zoo tshaj plaws ntawm ob peb lub kaum ob cais cais uas tsis tau piav qhia los ntawm Dockerfiles. Koj tsuas tuaj yeem teeb tsa nws rau kev xa mus rau ib puag ncig tshwj xeeb. Peb lub luag haujlwm yog xa cov pawg mus rau hauv ib puag ncig theem los sim nws ua ntej kev sim ua ntej tso tawm.

Raws li kev xav, tuaj yeem muaj ntau pawg ua haujlwm ib txhij: ntau npaum li muaj cov haujlwm hauv lub xeev ua tiav lossis ze rau kev ua tiav. Lub peev xwm ntawm cov servers ntawm peb qhov pov tseg tso cai rau peb khiav ntau pawg ntawm txhua tus neeg rau zaub mov. Txhua qhov kev sib koom ua ke yuav tsum raug cais tawm (yuav tsum tsis muaj kev sib tshuam hauv cov chaw nres nkoj, cov npe, thiab lwm yam).

Peb cov peev txheej tseem ceeb tshaj plaws yog peb lub sijhawm, thiab peb tsis muaj ntau ntawm nws.

Txhawm rau pib sai dua, peb xaiv Docker Swarm vim nws txoj kev yooj yim thiab kev ua haujlwm yooj yim. Thawj qhov uas peb tau ua yog tsim tus thawj tswj hwm thiab ob peb ntawm ntawm cov chaw taws teeb servers:

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
kilqc94pi2upzvabttikrfr5d     nop-test-1     Ready               Active                                  19.03.2
jilwe56pl2zvabupryuosdj78     nop-test-2     Ready               Active                                  19.03.2
j5a4yz1kr2xke6b1ohoqlnbq5 *   nop-test-3     Ready               Active              Leader              19.03.2

Tom ntej no, tsim ib lub network:


$ docker network create --driver overlay --subnet 10.10.10.0/24 nw_swarm

Tom ntej no, peb txuas Gitlab-CI thiab Swarm nodes nyob rau hauv cov nqe lus ntawm kev tswj chaw taws teeb ntawm cov nodes los ntawm CI: txhim kho daim ntawv pov thawj, teeb tsa kev sib txawv tsis pub lwm tus paub, thiab teeb tsa Docker kev pabcuam ntawm tus tswj server. Qhov no ib tsab xov xwm cawm peb ntau lub sijhawm.

Tom ntej no, peb ntxiv pawg tsim thiab kev puas tsuaj rau .gitlab-ci .yml.

Ob peb txoj haujlwm ntxiv tau ntxiv rau .gitlab-ci .yml

## staging stage
deploy_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
    CI_BIN_DEPENDENCIES_JOB: "release.centos.7"
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack deploy -c docker-compose.yml ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME} --with-registry-auth
    - rm -rf $DOCKER_CERT_PATH
  when: manual

## stop staging stage
stop_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack rm ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME}
    # TODO: need check that stopped
  when: manual

Los ntawm cov kab lus saum toj no, koj tuaj yeem pom tias ob lub nyees khawm (deploy_staging, stop_staging) tau ntxiv rau Pipelines, yuav tsum tau ua haujlwm.

Xa cov ntawv thov nrog Docker Swarm
Lub npe pawg sib phim lub npe ceg thiab qhov kev sib txawv no yuav tsum txaus. Cov kev pabcuam hauv pawg tau txais qhov tshwj xeeb ip chaw nyob, thiab cov chaw nres nkoj, cov npe, thiab lwm yam. yuav raug rho tawm, tab sis tib yam los ntawm pawg rau pawg (vim cov ntaub ntawv teeb tsa yog tib yam rau txhua pawg) - qhov peb xav tau. Peb xa cov pawg (cluster) siv docker-compose.yml, uas piav txog peb pawg.

docker-compose.yml

---
version: '3'

services:
  userprop:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:
  celery_bcd:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  schedulerdb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: schedulerdb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    command: ['--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci', '--explicit_defaults_for_timestamp=1']
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  celerydb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: celerydb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  cluster:
    image: $CENTOS7
    environment:
      - CENTOS
      - CI_ENVIRONMENT_NAME
      - CI_API_V4_URL
      - CI_REPOSITORY_URL
      - CI_PROJECT_ID
      - CI_PROJECT_URL
      - CI_PROJECT_PATH
      - CI_PROJECT_NAME
      - CI_COMMIT_REF_NAME
      - CI_BIN_DEPENDENCIES_JOB
    command: >
      sudo -u myusername -H /bin/bash -c ". /etc/profile &&
        mkdir -p /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
            git clone -b $CI_COMMIT_REF_NAME $CI_REPOSITORY_URL . &&
            curl $CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=$CI_BIN_DEPENDENCIES_JOB -o artifacts.zip &&
            unzip artifacts.zip ;
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME/scripts/deploy/ &&
            python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
            /bin/bash install_venv.sh -d -r ../../requirements.txt &&
            python3 prepare_init.d.py &&
            python3 deploy.py -s $CI_ENVIRONMENT_NAME"
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    tty: true
    stdin_open: true
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Ntawm no koj tuaj yeem pom tias cov khoom sib txuas los ntawm ib lub network (nw_swarm) thiab muaj rau ib leeg.

Cov Cheebtsam (raws li redis, mysql) raug cais tawm los ntawm lub pas dej dav dav ntawm cov khoom siv kev cai (hauv cov phiaj xwm thiab cov kev cai raug muab faib ua cov kev pabcuam). Lub sijhawm xa tawm ntawm peb pawg zoo li dhau CMD rau hauv peb cov duab teeb tsa loj thiab, feem ntau, xyaum tsis txawv ntawm qhov kev xa tawm tau piav qhia hauv Tshooj I. Kuv yuav qhia txog qhov sib txawv:

  • ua clone... - tau cov ntaub ntawv xav tau los siv (createconfig.py, install_venv.sh, thiab lwm yam)
  • curl... && unzip... - download tau thiab unzip tsim artifacts (sib sau cov nqi hluav taws xob)

Tsuas muaj ib qho teeb meem tseem tsis tau piav qhia: cov khoom siv uas muaj lub vev xaib cuam tshuam tsis tuaj yeem nkag tau los ntawm cov neeg tsim tawm browsers. Peb daws qhov teeb meem no siv cov npe rov qab, yog li:

Hauv .gitlab-ci.yml, tom qab xa cov pawg pawg, peb ntxiv cov kab ntawm deploying lub balancer (uas, thaum cog lus, tsuas yog hloov kho nws cov configuration (tsim tshiab nginx configuration ntaub ntawv raws li tus qauv: /etc/nginx/conf. d/${CI_COMMIT_REF_NAME}.conf) - saib docker-compose-nginx.yml code)

    - docker stack deploy -c docker-compose-nginx.yml ${CI_ENVIRONMENT_NAME} --with-registry-auth

docker-compose-nginx.yml

---
version: '3'

services:
  nginx:
    image: nginx:latest
    environment:
      CI_COMMIT_REF_NAME: ${CI_COMMIT_REF_NAME}
      NGINX_CONFIG: |-
            server {
                listen 8080;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:8080;
                }
            }
            server {
                listen 5555;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:5555;
                }
            }
    volumes:
      - /tmp/staging/nginx:/etc/nginx/conf.d
    command:
      /bin/bash -c "echo -e "$$NGINX_CONFIG" > /etc/nginx/conf.d/${CI_COMMIT_REF_NAME}.conf;
        nginx -g "daemon off;";
        /etc/init.d/nginx reload"
    ports:
      - 8080:8080
      - 5555:5555
      - 3000:3000
      - 443:443
      - 80:80
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Ntawm cov khoos phis tawj txhim kho, hloov kho /etc/hosts; sau url rau nginx:

10.50.173.106 staging_BRANCH-1831_cluster.dev

Yog li, kev xa tawm ntawm cov kab ke sib cais tau raug coj los siv thiab cov neeg tsim khoom tam sim no tuaj yeem khiav lawv hauv txhua tus lej txaus los xyuas lawv cov haujlwm.

Cov phiaj xwm yav tom ntej:

  • Cais peb cov khoom raws li kev pabcuam
  • Muaj rau txhua Dockerfile
  • Tsis siv neeg ntes tsawg loaded nodes nyob rau hauv pawg
  • Qhia cov nodes los ntawm lub npe qauv (tsis yog siv id li hauv kab lus)
  • Ntxiv ib daim tshev uas cov pawg puas lawm
  • ...

Ua tsaug tshwj xeeb rau tsab xov xwm.

Tau qhov twg los: www.hab.com

Ntxiv ib saib