Artikel iki bakal dadi kapentingan kanggo penguji lan pangembang, nanging utamane kanggo spesialis otomatisasi sing ngadhepi masalah nyetel GitLab CI/CD kanggo tes integrasi ing kahanan sumber daya infrastruktur sing ora cukup lan / utawa ora ana wadhah. platform orkestrasi. Aku bakal pitutur marang kowe carane nyiyapake panyebaran lingkungan test nggunakake docker nulis ing siji GitLab shell runner lan supaya nalika deploying sawetara lingkungan, layanan dibukak ora ngganggu siji liyane.
Isi
Prasyarat
-
Ing praktikku, asring kedadeyan yen tes integrasi "dirawat" ing proyek. Lan asring masalah pisanan lan paling penting yaiku pipa CI, ing ngendi tes integrasi dikembangake layanan (e) ditindakake ing lingkungan dev/tataran. Iki nyebabake sawetara masalah:
- Amarga cacat ing layanan tartamtu sajrone tes integrasi, sirkuit tes bisa rusak amarga data sing rusak. Ana kasus nalika ngirim panjalukan kanthi format JSON sing rusak nabrak layanan kasebut, sing ndadekake stand kasebut ora bisa digunakake.
- Slowdown saka sirkuit test minangka data test mundhak. Aku iku ndadekake ora pangertèn kanggo njlèntrèhaké conto karo reresik / muter maneh database. Ing praktikku, aku durung nemoni proyek sing prosedur iki lancar.
- Risiko ngganggu fungsi sirkuit test nalika nyoba setelan sistem umum. Contone, pangguna / grup / sandhi / kabijakan aplikasi.
- Data tes saka tes otomatis nggawe urip angel kanggo penguji manual.
Sawetara bakal ujar manawa tes otomatis sing apik kudu ngresiki data sawise awake dhewe. Aku duwe argumentasi marang:
- Stand dinamis trep banget kanggo digunakake.
- Ora saben obyek bisa dibusak saka sistem liwat API. Contone, telpon kanggo mbusak obyek ora dileksanakake amarga mbantah logika bisnis.
- Nalika nggawe obyek liwat API, jumlah metadata sing akeh bisa digawe, sing bakal dadi masalah kanggo mbusak.
- Yen tes duwe ketergantungan ing antarane awake dhewe, mula proses ngresiki data sawise nganakake tes dadi pusing.
- Tambahan (lan, ing mratelakake panemume, ora sabdho) telpon kanggo API.
- Lan argumen utama: nalika data tes wiwit dibusak langsung saka database. Iki malih dadi sirkus PK/FK nyata! Kita krungu saka pangembang: "Aku mung nambah / mbusak / ngganti jeneng tandha, kenapa 100500 tes integrasi kejiret?"
Miturut pendapatku, solusi sing paling optimal yaiku lingkungan dinamis.
- Akeh wong nggunakake docker-compose kanggo mbukak lingkungan test, nanging sawetara wong nggunakake docker-compose nalika nindakake testing integrasi ing CI/CD. Lan ing kene aku ora njupuk akun kubernetes, swarm lan platform orkestrasi wadhah liyane. Ora saben perusahaan duwe. Luwih becik yen docker-compose.yml universal.
- Sanajan kita duwe pelari QA dhewe, kepiye kita bisa mesthekake yen layanan sing diluncurake liwat docker-compose ora ngganggu saben liyane?
- Kepiye cara ngumpulake log layanan sing diuji?
- Carane ngresiki runner?
Aku duwe pelari GitLab dhewe kanggo proyekku lan aku nemoni pitakonan kasebut sajrone pembangunan kanggo . Luwih tepat, nalika nindakake tes integrasi. Ing ngisor iki kita bakal ngrampungake masalah kasebut kanthi nggunakake conto saka proyek iki.
GitLab Shell Runner
Kanggo runner, Aku nyaranake mesin virtual Linux karo 4 vCPU, 4 GB RAM, 50 GB HDD.
Ana akeh informasi babagan nyiyapake gitlab-runner ing Internet, kanthi ringkes:
- Mlebet mesin liwat SSH
-
Yen sampeyan duwe kurang saka 8 GB RAM, aku nyaranake supaya pembunuh OOM ora teka lan mateni tugas kita amarga kekurangan RAM. Iki bisa kedadeyan yen luwih saka 5 tugas diluncurake bebarengan. Tugas bakal maju luwih alon, nanging ajeg.
Conto karo pembunuh OOM
Yen sampeyan ndeleng ing log tugas
bash: line 82: 26474 Killed, banjur mung nglakokaké ing runnersudo dmesg | grep 26474[26474] 1002 26474 1061935 123806 339 0 0 java Out of memory: Kill process 26474 (java) score 127 or sacrifice child Killed process 26474 (java) total-vm:4247740kB, anon-rss:495224kB, file-rss:0kB, shmem-rss:0kBLan yen gambar katon kaya iki, banjur nambah swap utawa nambah RAM.
- Instal , , , gawe.
- Nambahake pangguna
gitlab-runnermenyang grupdockersudo groupadd docker sudo usermod -aG docker gitlab-runner - gitlab-runner.
-
Bukak kanggo nyunting
/etc/gitlab-runner/config.tomllan nambahconcurrent=20 [[runners]] request_concurrency = 10Iki bakal ngidini sampeyan mbukak tugas paralel ing siji pelari. Waca liyane .
Yen sampeyan duwe mesin luwih kuat, contone 8 vCPU, 16 GB RAM, banjur nomer iki bisa digawe paling 2 kaping luwih gedhe. Nanging kabeh gumantung apa sing bakal diluncurake ing pelari iki lan jumlahe.
Wis cukup.
Nyiyapake docker-compose.yml
Tugas utama yaiku docker-compose.yml universal, sing bisa digunakake para pangembang / penguji kanthi lokal lan ing pipa CI.
Kaping pisanan, kita nggawe jeneng layanan unik kanggo CI. Salah sawijining variabel unik ing GitLab CI yaiku variabel CI_JOB_ID. Yen sampeyan nemtokake container_name kanthi teges "service-${CI_JOB_ID:-local}", banjur ing kasus:
- yen
CI_JOB_IDora ditetepake ing variabel lingkungan,
banjur jeneng layanan bakalservice-local - yen
CI_JOB_IDditetepake ing variabel lingkungan (contone 123),
banjur jeneng layanan bakalservice-123
Kapindho, kita nggawe jaringan umum kanggo layanan sing diluncurake. Iki menehi kita isolasi tingkat jaringan nalika mbukak macem-macem lingkungan tes.
networks:
default:
external:
name: service-network-${CI_JOB_ID:-local}Bener, iki langkah pertama kanggo sukses =)
Conto sandi docker-compose.yml karo komentar
version: "3"
# Для корректной работы web (php) и fmt нужно,
# чтобы контейнеры имели общий исполняемый контент.
# В нашем случае, это директория /var/www/testrail
volumes:
static-content:
# Изолируем окружение на сетевом уровне
networks:
default:
external:
name: testrail-network-${CI_JOB_ID:-local}
services:
db:
image: mysql:5.7.22
# Каждый container_name содержит ${CI_JOB_ID:-local}
container_name: "testrail-mysql-${CI_JOB_ID:-local}"
environment:
MYSQL_HOST: db
MYSQL_DATABASE: mydb
MYSQL_ROOT_PASSWORD: 1234
SKIP_GRANT_TABLES: 1
SKIP_NETWORKING: 1
SERVICE_TAGS: dev
SERVICE_NAME: mysql
networks:
- default
migration:
image: registry.gitlab.com/touchbit/image/testrail/migration:latest
container_name: "testrail-migration-${CI_JOB_ID:-local}"
links:
- db
depends_on:
- db
networks:
- default
fpm:
image: registry.gitlab.com/touchbit/image/testrail/fpm:latest
container_name: "testrail-fpm-${CI_JOB_ID:-local}"
volumes:
- static-content:/var/www/testrail
links:
- db
networks:
- default
web:
image: registry.gitlab.com/touchbit/image/testrail/web:latest
container_name: "testrail-web-${CI_JOB_ID:-local}"
# Если переменные TR_HTTP_PORT или TR_HTTPS_PORTS не определены,
# то сервис поднимается на 80 и 443 порту соответственно.
ports:
- ${TR_HTTP_PORT:-80}:80
- ${TR_HTTPS_PORT:-443}:443
volumes:
- static-content:/var/www/testrail
links:
- db
- fpm
networks:
- defaultContoh run lokal
docker-compose -f docker-compose.yml up -d
Starting testrail-mysql-local ... done
Starting testrail-migration-local ... done
Starting testrail-fpm-local ... done
Recreating testrail-web-local ... doneNanging ora kabeh gampang banget nalika diluncurake ing CI.
Nyiyapake Makefile
Aku nggunakake Makefile amarga trep banget kanggo manajemen lingkungan lokal lan ing CI. Komentar online liyane
# У меня в проектах все вспомогательные вещи лежат в директории `.indirect`,
# в том числе и `docker-compose.yml`
# Использовать bash с опцией pipefail
# pipefail - фейлит выполнение пайпа, если команда выполнилась с ошибкой
SHELL=/bin/bash -o pipefail
# Останавливаем контейнеры и удаляем сеть
docker-kill:
docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml kill
docker network rm network-$${CI_JOB_ID:-testrail} || true
# Предварительно выполняем docker-kill
docker-up: docker-kill
# Создаем сеть для окружения
docker network create network-$${CI_JOB_ID:-testrail}
# Забираем последние образы из docker-registry
docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml pull
# Запускаем окружение
# force-recreate - принудительное пересоздание контейнеров
# renew-anon-volumes - не использовать volumes предыдущих контейнеров
docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
# Ну и, на всякий случай, вывести что там у нас в принципе запущено на машинке
docker ps
# Коллектим логи сервисов
docker-logs:
mkdir ./logs || true
docker logs testrail-web-$${CI_JOB_ID:-local} >& logs/testrail-web.log
docker logs testrail-fpm-$${CI_JOB_ID:-local} >& logs/testrail-fpm.log
docker logs testrail-migration-$${CI_JOB_ID:-local} >& logs/testrail-migration.log
docker logs testrail-mysql-$${CI_JOB_ID:-local} >& logs/testrail-mysql.log
# Очистка раннера
docker-clean:
@echo Останавливаем все testrail-контейнеры
docker kill $$(docker ps --filter=name=testrail -q) || true
@echo Очистка докер контейнеров
docker rm -f $$(docker ps -a -f --filter=name=testrail status=exited -q) || true
@echo Очистка dangling образов
docker rmi -f $$(docker images -f "dangling=true" -q) || true
@echo Очистка testrail образов
docker rmi -f $$(docker images --filter=reference='registry.gitlab.com/touchbit/image/testrail/*' -q) || true
@echo Очистка всех неиспользуемых volume
docker volume rm -f $$(docker volume ls -q) || true
@echo Очистка всех testrail сетей
docker network rm $(docker network ls --filter=name=testrail -q) || true
docker ps
Mriksa
nggawe docker-up
$ make docker-up
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill
Killing testrail-web-local ... done
Killing testrail-fpm-local ... done
Killing testrail-mysql-local ... done
docker network rm network-${CI_JOB_ID:-testrail} || true
network-testrail
docker network create network-${CI_JOB_ID:-testrail}
d2ec063324081c8bbc1b08fd92242c2ea59d70cf4025fab8efcbc5c6360f083f
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull
Pulling db ... done
Pulling migration ... done
Pulling fpm ... done
Pulling web ... done
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
Recreating testrail-mysql-local ... done
Recreating testrail-fpm-local ... done
Recreating testrail-migration-local ... done
Recreating testrail-web-local ... done
docker ps
CONTAINER ID PORTS NAMES
a845d3cb0e5a 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp testrail-web-local
19d8ef001398 9000/tcp testrail-fpm-local
e28840a2369c 3306/tcp, 33060/tcp testrail-migration-local
0e7900c23f37 3306/tcp testrail-mysql-local
nggawe docker-log
$ make docker-logs
mkdir ./logs || true
mkdir: cannot create directory ‘./logs’: File exists
docker logs testrail-web-${CI_JOB_ID:-local} >& logs/testrail-web.log
docker logs testrail-fpm-${CI_JOB_ID:-local} >& logs/testrail-fpm.log
docker logs testrail-migration-${CI_JOB_ID:-local} >& logs/testrail-migration.log
docker logs testrail-mysql-${CI_JOB_ID:-local} >& logs/testrail-mysql.log
Nyiyapake .gitlab-ci.yml
Mlaku tes integrasi
Integration:
stage: test
tags:
- my-shell-runner
before_script:
# Аутентифицируемся в registry
- docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
# Генерируем псевдоуникальные TR_HTTP_PORT и TR_HTTPS_PORT
- export TR_HTTP_PORT=$(shuf -i10000-60000 -n1)
- export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1)
# создаем директорию с идентификатором задачи
- mkdir ${CI_JOB_ID}
# копируем в созданную директорию наш docker-compose.yml
# чтобы контекст был разный для каждой задачи
- cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml
script:
# поднимаем наше окружение
- make docker-up
# запускаем тесты исполняемым jar (у меня так)
- java -jar itest.jar --http-port ${TR_HTTP_PORT} --https-port ${TR_HTTPS_PORT}
# или в контейнере
- docker run --network=testrail-network-${CI_JOB_ID:-local} --rm itest
after_script:
# собираем логи
- make docker-logs
# останавливаем окружение
- make docker-kill
artifacts:
# сохраняем логи
when: always
paths:
- logs
expire_in: 30 daysMinangka asil nglakokake tugas kasebut, direktori log ing artefak bakal ngemot log layanan lan tes. Sing trep banget yen ana kesalahan. Saben tes ing podo karo nulis log dhewe, nanging aku bakal ngomong babagan iki kanthi kapisah.
Reresik pelari
Tugas bakal diluncurake mung miturut jadwal.
stages:
- clean
- build
- test
Clean runner:
stage: clean
only:
- schedules
tags:
- my-shell-runner
script:
- make docker-cleanSabanjure, pindhah menyang proyek GitLab -> CI / CD -> Jadwal -> Jadwal Anyar lan tambahake jadwal anyar
asil
Bukak 4 tugas ing GitLab CI
Ing log saka tugas pungkasan karo tes integrasi kita waca wadhah saka tugas beda
CONTAINER ID NAMES
c6b76f9135ed testrail-web-204645172
01d303262d8e testrail-fpm-204645172
2cdab1edbf6a testrail-migration-204645172
826aaf7c0a29 testrail-mysql-204645172
6dbb3fae0322 testrail-web-204645084
3540f8d448ce testrail-fpm-204645084
70fea72aa10d testrail-mysql-204645084
d8aa24b2892d testrail-web-204644881
6d4ccd910fad testrail-fpm-204644881
685d8023a3ec testrail-mysql-204644881
1cdfc692003a testrail-web-204644793
6f26dfb2683e testrail-fpm-204644793
029e16b26201 testrail-mysql-204644793
c10443222ac6 testrail-web-204567103
04339229397e testrail-fpm-204567103
6ae0accab28d testrail-mysql-204567103
b66b60d79e43 testrail-web-204553690
033b1f46afa9 testrail-fpm-204553690
a8879c5ef941 testrail-mysql-204553690
069954ba6010 testrail-web-204553539
ed6b17d911a5 testrail-fpm-204553539
1a1eed057ea0 testrail-mysql-204553539Log sing luwih rinci
$ docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ export TR_HTTP_PORT=$(shuf -i10000-60000 -n1)
$ export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1)
$ mkdir ${CI_JOB_ID}
$ cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml
$ make docker-up
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill
docker network rm testrail-network-${CI_JOB_ID:-local} || true
Error: No such network: testrail-network-204645172
docker network create testrail-network-${CI_JOB_ID:-local}
0a59552b4464b8ab484de6ae5054f3d5752902910bacb0a7b5eca698766d0331
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull
Pulling web ... done
Pulling fpm ... done
Pulling migration ... done
Pulling db ... done
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
Creating volume "204645172_static-content" with default driver
Creating testrail-mysql-204645172 ...
Creating testrail-mysql-204645172 ... done
Creating testrail-migration-204645172 ... done
Creating testrail-fpm-204645172 ... done
Creating testrail-web-204645172 ... done
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c6b76f9135ed registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 13 seconds ago Up 1 second 0.0.0.0:51148->80/tcp, 0.0.0.0:25426->443/tcp testrail-web-204645172
01d303262d8e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 16 seconds ago Up 13 seconds 9000/tcp testrail-fpm-204645172
2cdab1edbf6a registry.gitlab.com/touchbit/image/testrail/migration:latest "docker-entrypoint.s…" 16 seconds ago Up 13 seconds 3306/tcp, 33060/tcp testrail-migration-204645172
826aaf7c0a29 mysql:5.7.22 "docker-entrypoint.s…" 18 seconds ago Up 16 seconds 3306/tcp testrail-mysql-204645172
6dbb3fae0322 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 36 seconds ago Up 22 seconds 0.0.0.0:44202->80/tcp, 0.0.0.0:20151->443/tcp testrail-web-204645084
3540f8d448ce registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 38 seconds ago Up 35 seconds 9000/tcp testrail-fpm-204645084
70fea72aa10d mysql:5.7.22 "docker-entrypoint.s…" 40 seconds ago Up 37 seconds 3306/tcp testrail-mysql-204645084
d8aa24b2892d registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up 53 seconds 0.0.0.0:31103->80/tcp, 0.0.0.0:43872->443/tcp testrail-web-204644881
6d4ccd910fad registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp testrail-fpm-204644881
685d8023a3ec mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp testrail-mysql-204644881
1cdfc692003a registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up About a minute 0.0.0.0:44752->80/tcp, 0.0.0.0:23540->443/tcp testrail-web-204644793
6f26dfb2683e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp testrail-fpm-204644793
029e16b26201 mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp testrail-mysql-204644793
c10443222ac6 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:57123->80/tcp, 0.0.0.0:31657->443/tcp testrail-web-204567103
04339229397e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204567103
6ae0accab28d mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204567103
b66b60d79e43 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:56321->80/tcp, 0.0.0.0:58749->443/tcp testrail-web-204553690
033b1f46afa9 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204553690
a8879c5ef941 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204553690
069954ba6010 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:32869->80/tcp, 0.0.0.0:16066->443/tcp testrail-web-204553539
ed6b17d911a5 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204553539
1a1eed057ea0 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204553539Kabeh tugas rampung kasil
Artefak tugas ngemot log layanan lan tes
Kabeh katon ayu, nanging ana nuansa. Pipeline bisa dibatalake kanthi paksa nalika tes integrasi lagi mlaku, lan kontainer sing mlaku ora bakal mandheg. Saka wektu kanggo wektu sampeyan kudu ngresiki runner. Sayange, tugas kanggo perbaikan ing GitLab CE isih ana ing status
Nanging kita wis nambahake peluncuran tugas miturut jadwal, lan ora ana sing nglarang kita mbukak kanthi manual.
Pindhah menyang proyek kita -> CI / CD -> Jadwal lan jalanake tugas kasebut Clean runner
Total:
- Kita duwe siji pelari cangkang.
- Ora ana konflik antarane tugas lan lingkungan.
- Kita mbukak tugas kanthi tes integrasi kanthi paralel.
- Sampeyan bisa nindakake tes integrasi kanthi lokal utawa ing wadhah.
- Log layanan lan tes diklumpukake lan dipasang ing tugas pipa.
- Sampeyan bisa ngresiki runner saka gambar Docker lawas.
Wektu persiyapan yaiku ~ 2 jam.
Sing kabeh, bener. Aku bakal seneng nampa umpan balik.
Source: www.habr.com