ProHoster > Блог > Administrasi > Nyebarake aplikasi nggunakake Docker Swarm

Nyebarake aplikasi nggunakake Docker Swarm

Sistem rekomendasi konten video online sing lagi ditindakake yaiku pangembangan komersial sing ditutup lan sacara teknis minangka klompok multi-komponen saka komponen proprietary lan open source. Tujuan nulis artikel iki yaiku kanggo njlèntrèhaké implementasi sistem clustering docker swarm kanggo platform pementasan, tanpa ngganggu alur kerja sing wis ditemtokake ing proses kita ing kahanan wektu sing winates. Narasi sing diwènèhaké marang kowé dipérang dadi rong bagéan. Bagean kapisan nggambarake CI/CD sadurunge nggunakake swarm docker, lan bagean kapindho nggambarake proses implementasine. Sing ora kasengsem maca bagean pisanan bisa kanthi aman pindhah menyang kaloro.

Bagean I

Biyen, ana perlu kanggo nyiyapake proses CI/CD sakcepete bisa. Salah sawijining syarat yaiku ora nggunakake Docker kanggo penyebaran komponen sing dikembangaké kanggo sawetara alasan:

  • kanggo operasi komponen sing luwih dipercaya lan stabil ing Produksi (yaiku, intine, syarat ora nggunakake virtualisasi)
  • pangembang utama ora pengin nggarap Docker (aneh, nanging kaya ngono)
  • kanggo alasan ideologi manajemen R&D

Prasarana, tumpukan lan prakiraan prasyarat awal kanggo MVP yaiku:

  • 4 server Intel® X5650 karo Debian (siji mesin sing luwih kuat kanggo pangembangan)
  • Pangembangan komponen khusus dhewe ditindakake ing C ++, Python3
  • Alat pihak katelu utama sing digunakake: Kafka, Clickhouse, Airflow, Redis, Grafana, Postgresql, Mysql, ...
  • Pipeline kanggo mbangun lan nguji komponen kanthi kapisah kanggo debug lan rilis

Salah sawijining pitakonan pisanan sing kudu dirampungake ing tahap awal yaiku carane komponen khusus bakal disebarake ing lingkungan apa wae (CI / CD).

Kita mutusake kanggo nginstal komponen pihak katelu kanthi sistemik lan nganyari kanthi sistemik. Aplikasi khusus sing dikembangake ing C ++ utawa Python bisa disebarake kanthi pirang-pirang cara. Antarane wong-wong mau, contone: nggawe paket sistem, ngirim menyang repositori gambar diklumpukake lan instalasi sakteruse ing server. Kanggo alasan sing durung dingerteni, cara liya dipilih, yaiku: nggunakake CI, file eksekusi aplikasi dikompilasi, lingkungan proyek virtual digawe, modul py saka requirements.txt diinstal, lan kabeh artefak kasebut dikirim bebarengan karo konfigurasi, skrip lan lingkungan aplikasi sing diiringi menyang server. Sabanjure, aplikasi diluncurake saka pangguna virtual tanpa hak administrator.

Gitlab-CI dipilih minangka sistem CI/CD. Pipa asil katon kaya iki:

Nyebarake aplikasi nggunakake Docker Swarm
Secara struktural, gitlab-ci.yml katon kaya iki:

---
variables:
  # минимальная версия ЦПУ на серверах, где разворачивается кластер
  CMAKE_CPUTYPE: "westmere"

  DEBIAN: "MYREGISTRY:5000/debian:latest"

before_script:
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh && echo -e "Host *ntStrictHostKeyChecking nonn" > ~/.ssh/config

stages:
  - build
  - testing
  - deploy

debug.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always
release.debian:
  stage: build
  image: $DEBIAN
  script:
    - cd builds/release && ./build.sh
    paths:
      - bin/
      - builds/release/bin/
    when: always

## testing stage
tests.codestyle:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -t codestyle -b "${CI_COMMIT_REF_NAME}_codestyle"
tests.debug.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - debug.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_debug"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week
tests.release.debian:
  stage: testing
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - /bin/bash run_tests.sh -e codestyle/test_pylint.py -b "${CI_COMMIT_REF_NAME}_debian_release"
  artifacts:
    paths:
      - run_tests/username/
    when: always
    expire_in: 1 week

## staging stage
deploy_staging:
  stage: deploy
  environment: staging
  image: $DEBIAN
  dependencies:
    - release.debian
  script:
    - cd scripts/deploy/ &&
        python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
        /bin/bash install_venv.sh -d -r ../../requirements.txt &&
        python3 prepare_init.d.py &&
        python3 deploy.py -s $CI_ENVIRONMENT_NAME
  when: manual

Wigati dicathet yen perakitan lan tes ditindakake ing gambar dhewe, ing ngendi kabeh paket sistem sing dibutuhake wis diinstal lan setelan liyane digawe.

Sanajan saben skrip ing proyek kasebut menarik kanthi cara dhewe, aku mesthi ora bakal ngomong babagan iki; njlèntrèhaké saben wong bakal njupuk wektu sing cukup lan iki dudu tujuan artikel kasebut. Ayo kula narik kawigaten sampeyan yen tataran panyebaran kasusun saka urutan skrip nelpon:

  1. createconfig.py — nggawe file settings.ini kanthi setelan kanggo komponen ing lingkungan sing beda kanggo panyebaran sabanjure (Preproduksi, Produksi, Pengujian, ...)
  2. install_venv.sh - nggawe lingkungan virtual kanggo komponen py ing direktori tartamtu lan nyalin menyang server remot
  3. prepare_init.d.py - nyiapake skrip kanggo komponen wiwitan-mandeg adhedhasar cithakan
  4. deploy.py - deploys lan miwiti maneh komponen anyar

Wektu liwati. Tahap pementasan diganti dening praproduksi lan produksi. Dhukungan kanggo produk wis ditambahake ing siji distribusi liyane (CentOS). Liyane 5 server fisik kuat lan rolas virtual wis ditambahake. Lan dadi saya angel kanggo pangembang lan penguji kanggo nyoba tugas ing lingkungan sing luwih cedhak karo negara kerja. Ing wektu iki dadi cetha yen ora bisa ditindakake tanpa dheweke ...

Bagean II

Nyebarake aplikasi nggunakake Docker Swarm

Dadi, kluster kita minangka sistem spektakuler saka pirang-pirang komponen individu sing ora diterangake dening Dockerfiles. Sampeyan bisa ngatur kanggo panyebaran menyang lingkungan tartamtu mung ing umum. Tugas kita yaiku masang kluster menyang lingkungan pementasan kanggo nyoba sadurunge tes pra-rilis.

Secara teoritis, bisa uga ana sawetara kluster sing bisa digunakake bebarengan: minangka akeh sing ana tugas ing negara rampung utawa cedhak rampung. Daya saka server ing pembuangan kita ngidini kita kanggo mbukak sawetara klompok ing saben server. Saben kluster pementasan kudu diisolasi (ora ana tumpang tindih ing port, direktori, lsp).

Sumber daya kita sing paling berharga yaiku wektu, lan kita ora duwe akeh.

Kanggo wiwitan sing luwih cepet, kita milih Docker Swarm amarga kesederhanaan lan arsitektur sing fleksibel. Babagan pisanan sing ditindakake yaiku nggawe manajer lan sawetara simpul ing server remot:

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
kilqc94pi2upzvabttikrfr5d     nop-test-1     Ready               Active                                  19.03.2
jilwe56pl2zvabupryuosdj78     nop-test-2     Ready               Active                                  19.03.2
j5a4yz1kr2xke6b1ohoqlnbq5 *   nop-test-3     Ready               Active              Leader              19.03.2

Sabanjure, kita nggawe jaringan:


$ docker network create --driver overlay --subnet 10.10.10.0/24 nw_swarm

Sabanjure, kita nyambungake simpul Gitlab-CI lan Swarm babagan manajemen simpul jarak jauh saka CI: nginstal sertifikat, nyetel variabel rahasia, lan uga nyetel layanan Docker ing server manajemen. Iki artikel nylametake kita akeh wektu.

Sabanjure, kita nambah proyek kanggo nggawe lan numpes tumpukan ing .gitlab-ci .yml.

Sawetara proyek liyane wis ditambahake menyang .gitlab-ci .yml

## staging stage
deploy_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
    CI_BIN_DEPENDENCIES_JOB: "release.centos.7"
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack deploy -c docker-compose.yml ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME} --with-registry-auth
    - rm -rf $DOCKER_CERT_PATH
  when: manual

## stop staging stage
stop_staging:
  stage: testing
  before_script:
    - echo "override global 'before_script'"
  image: "REGISTRY:5000/docker:latest"
  environment: staging
  dependencies: []
  variables:
    DOCKER_CERT_PATH: "/certs"
    DOCKER_HOST: tcp://10.50.173.107:2376
    DOCKER_TLS_VERIFY: 1
  script:
    - mkdir -p $DOCKER_CERT_PATH
    - echo "$TLSCACERT" > $DOCKER_CERT_PATH/ca.pem
    - echo "$TLSCERT" > $DOCKER_CERT_PATH/cert.pem
    - echo "$TLSKEY" > $DOCKER_CERT_PATH/key.pem
    - docker stack rm ${CI_ENVIRONMENT_NAME}_${CI_COMMIT_REF_NAME}
    # TODO: need check that stopped
  when: manual

Saka fragmen kode ing ndhuwur, jelas yen rong tombol wis ditambahake menyang Pipelines (deploy_staging, stop_staging) sing mbutuhake tumindak manual.

Nyebarake aplikasi nggunakake Docker Swarm
Jeneng tumpukan cocog karo jeneng cabang lan keunikan iki kudu cukup. Layanan ing tumpukan nampa alamat IP unik, lan port, direktori, lsp. bakal diisolasi, nanging padha saka tumpukan kanggo tumpukan (wiwit file konfigurasi padha kanggo kabeh tumpukan) - sing kita wanted. We masang tumpukan (cluster) nggunakake docker-compose.yml, sing nggambarake klompok kita.

docker-compose.yml

---
version: '3'

services:
  userprop:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:
  celery_bcd:
    image: redis:alpine
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  schedulerdb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: schedulerdb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    command: ['--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci', '--explicit_defaults_for_timestamp=1']
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  celerydb:
    image: mariadb:latest
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
      MYSQL_DATABASE: celerydb
      MYSQL_USER: ****
      MYSQL_PASSWORD: ****
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

  cluster:
    image: $CENTOS7
    environment:
      - CENTOS
      - CI_ENVIRONMENT_NAME
      - CI_API_V4_URL
      - CI_REPOSITORY_URL
      - CI_PROJECT_ID
      - CI_PROJECT_URL
      - CI_PROJECT_PATH
      - CI_PROJECT_NAME
      - CI_COMMIT_REF_NAME
      - CI_BIN_DEPENDENCIES_JOB
    command: >
      sudo -u myusername -H /bin/bash -c ". /etc/profile &&
        mkdir -p /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME &&
            git clone -b $CI_COMMIT_REF_NAME $CI_REPOSITORY_URL . &&
            curl $CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=$CI_BIN_DEPENDENCIES_JOB -o artifacts.zip &&
            unzip artifacts.zip ;
        cd /storage1/$CI_COMMIT_REF_NAME/$CI_PROJECT_NAME/scripts/deploy/ &&
            python3 createconfig.py -s $CI_ENVIRONMENT_NAME &&
            /bin/bash install_venv.sh -d -r ../../requirements.txt &&
            python3 prepare_init.d.py &&
            python3 deploy.py -s $CI_ENVIRONMENT_NAME"
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    tty: true
    stdin_open: true
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Ing kene sampeyan bisa ndeleng manawa komponen disambungake dening siji jaringan (nw_swarm) lan bisa diakses saben liyane.

Komponen sistem (adhedhasar redis, mysql) dipisahake saka blumbang umum komponen khusus (ing rencana, komponen khusus uga dipérang dadi layanan). Tahap panyebaran kluster kita katon kaya nransfer CMD menyang siji gambar sing dikonfigurasi gedhe lan, ing umum, meh ora beda karo penyebaran sing diterangake ing Part I. Aku bakal nandheske bedane:

  • klone git... - kita entuk file sing dibutuhake kanggo nindakake penyebaran (createconfig.py, install_venv.sh, lsp)
  • curl... && unzip... - download lan unzip artefak mbangun (utilitas kompilasi)

Mung ana siji masalah sing durung diterangake: komponen sing duwe antarmuka web ora bisa diakses saka browser pangembang. Kita ngatasi masalah iki nggunakake proxy mbalikke, supaya:

Ing .gitlab-ci.yml, sawise deploying tumpukan kluster, nambah baris kanggo deploying balancer (kang, nalika setya, mung nganyari konfigurasi (nggawe file konfigurasi nginx anyar miturut cithakan: /etc/nginx/conf.d /${CI_COMMIT_REF_NAME}.conf) - deleng kode docker-compose-nginx.yml)

    - docker stack deploy -c docker-compose-nginx.yml ${CI_ENVIRONMENT_NAME} --with-registry-auth

docker-compose-nginx.yml

---
version: '3'

services:
  nginx:
    image: nginx:latest
    environment:
      CI_COMMIT_REF_NAME: ${CI_COMMIT_REF_NAME}
      NGINX_CONFIG: |-
            server {
                listen 8080;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:8080;
                }
            }
            server {
                listen 5555;
                server_name staging_${CI_COMMIT_REF_NAME}_cluster.dev;

                location / {
                    proxy_pass http://staging_${CI_COMMIT_REF_NAME}_cluster:5555;
                }
            }
    volumes:
      - /tmp/staging/nginx:/etc/nginx/conf.d
    command:
      /bin/bash -c "echo -e "$$NGINX_CONFIG" > /etc/nginx/conf.d/${CI_COMMIT_REF_NAME}.conf;
        nginx -g "daemon off;";
        /etc/init.d/nginx reload"
    ports:
      - 8080:8080
      - 5555:5555
      - 3000:3000
      - 443:443
      - 80:80
    deploy:
      replicas: 1
      placement:
        constraints: [node.id == kilqc94pi2upzvabttikrfr5d]
      restart_policy:
        condition: none
    networks:
      nw_swarm:

networks:
  nw_swarm:
    external: true

Ing komputer pangembang, nganyari /etc/hosts; setel url menyang nginx:

10.50.173.106 staging_BRANCH-1831_cluster.dev

Dadi, panyebaran kluster pementasan terisolasi wis dileksanakake lan pangembang saiki bisa ngluncurake kanthi jumlah sing cukup kanggo nyoba tugase.

Rencana masa depan:

  • Pisahake komponen kita minangka layanan
  • Nggawe Dockerfile kanggo saben
  • Ndeteksi simpul sing kurang dimuat kanthi otomatis ing tumpukan
  • Nemtokake simpul nggunakake cithakan jeneng (tinimbang nggunakake id kaya ing artikel)
  • Tambah mriksa sing tumpukan wis numpes
  • ...

Matur nuwun khusus kanggo artikel.

Source: www.habr.com

Add a comment