Atik sa a pral nan enterè tou de tèsteur ak devlopè, men li fèt sitou pou espesyalis automatisation ki fè fas ak pwoblèm nan mete kanpe GitLab CI/CD pou tès entegrasyon nan kondisyon nan resous enfrastrikti ensifizan ak/oswa absans la nan yon orchestration veso. platfòm. Mwen pral di w ki jan yo mete kanpe deplwaman an nan anviwònman tès lè l sèvi avèk Docker konpoze sou yon sèl kourè koki GitLab epi pou ke lè deplwaye plizyè anviwònman, sèvis yo lanse pa entèfere youn ak lòt.
Content
Предпосылки
-
Nan pratik mwen an, li souvan rive ke tès entegrasyon yo te "trete" sou pwojè yo. Epi souvan pwoblèm nan premye ak pi enpòtan se tiyo a CI, nan ki tès entegrasyon ap devlope sèvis (yo) fèt nan yon anviwònman dev/etap. Sa a te lakòz anpil pwoblèm:
- Akòz defo nan yon sèvis patikilye pandan pwosesis tès entegrasyon, kous tès la ka domaje pa done kase. Te gen ka lè voye yon demann ak yon fòma JSON kase te fè aksidan sèvis la, ki te rann kanpe la konplètman inoperable.
- Ralentissement nan kous tès la kòm done tès yo ogmante. Mwen panse ke li pa gen okenn sans dekri yon egzanp ak netwaye / woule tounen baz done a. Nan pratik mwen an, mwen pa te rankontre yon pwojè kote pwosedi sa a te ale san pwoblèm.
- Risk pou deranje fonksyonalite sikwi tès la lè w teste anviwònman sistèm jeneral yo. Pa egzanp, itilizatè/gwoup/modpas/politik aplikasyon an.
- Done tès ki soti nan tès otomatik yo fè lavi difisil pou tèsteur manyèl yo.
Gen kèk ki pral di ke bon autotests ta dwe netwaye done yo apre tèt yo. Mwen gen agiman kont:
- Stands dinamik yo trè pratik pou itilize.
- Se pa tout objè yo ka retire nan sistèm nan atravè API a. Pou egzanp, yon apèl pou efase yon objè pa te aplike paske li kontredi lojik biznis.
- Lè w ap kreye yon objè atravè API a, yo ka kreye yon gwo kantite metadata, ki se pwoblèm pou efase.
- Si tès yo gen depandans nan mitan tèt yo, Lè sa a, pwosesis la nan netwaye done apre yo fin fè tès yo vin tounen yon tèt fè mal.
- Lòt (e, nan opinyon mwen, pa jistifye) apèl nan API a.
- Ak agiman prensipal la: lè done tès yo kòmanse otorize dirèkteman nan baz done a. Sa ap tounen yon vrè sirk PK/FK! Nou tande pale de devlopè yo: "Mwen jis ajoute / retire / bay yon non yon siy, poukisa yo te kenbe 100500 tès entegrasyon?"
Dapre mwen, solisyon ki pi bon an se yon anviwònman dinamik.
- Anpil moun sèvi ak docker-compose pou kouri yon anviwònman tès, men kèk moun sèvi ak docker-compose lè y ap fè tès entegrasyon nan CI/CD. Ak isit la mwen pa pran an kont kubernetes, swarm ak lòt platfòm orchestration veso. Se pa tout konpayi ki genyen yo. Li ta bon si docker-compose.yml te inivèsèl.
- Menm si nou gen pwòp kourè QA nou an, ki jan nou ka asire w ke sèvis lanse atravè docker-compose pa entèfere youn ak lòt?
- Ki jan yo kolekte mòso bwa nan sèvis teste yo?
- Ki jan yo netwaye kourè a?
Mwen gen pwòp GitLab kourè mwen pou pwojè mwen yo epi mwen te rankontre kesyon sa yo pandan devlopman pou . Plis jisteman, lè w ap kouri tès entegrasyon. Anba a nou pral rezoud pwoblèm sa yo lè l sèvi avèk egzanp nan pwojè sa a.
GitLab Shell Runner
Pou yon kourè, mwen rekòmande yon machin vityèl Linux ak 4 vCPU, 4 GB RAM, 50 GB HDD.
Gen anpil enfòmasyon sou konfigirasyon gitlab-runner sou entènèt la, kidonk yon ti tan:
- Konekte nan machin nan atravè SSH
-
Si ou gen mwens pase 8 GB RAM, Lè sa a, mwen rekòmande pou asasen an OOM pa vini epi touye travay nou akòz mank RAM. Sa ka rive lè plis pase 5 travay yo te lanse ansanm. Travay yo ap pwogrese pi dousman, men piti piti.
Egzanp ak OOM asasen
Si ou wè nan mòso bwa yo travay
bash: line 82: 26474 Killed, Lè sa a, jis egzekite sou kourè asudo dmesg | grep 26474[26474] 1002 26474 1061935 123806 339 0 0 java Out of memory: Kill process 26474 (java) score 127 or sacrifice child Killed process 26474 (java) total-vm:4247740kB, anon-rss:495224kB, file-rss:0kB, shmem-rss:0kBMen, si foto a sanble yon bagay tankou sa a, Lè sa a, swa ajoute swap oswa ajoute RAM.
- Enstale , , , fè.
- Ajoute yon itilizatè
gitlab-runnernan gwoup ladockersudo groupadd docker sudo usermod -aG docker gitlab-runner - gitlab-kourè.
-
Louvri pou koreksyon
/etc/gitlab-runner/config.tomlepi ajouteconcurrent=20 [[runners]] request_concurrency = 10Sa a pral pèmèt ou kouri travay paralèl sou yon sèl kourè. Li piplis .
Si ou gen yon machin ki pi pwisan, pou egzanp 8 vCPU, 16 GB RAM, Lè sa a, nimewo sa yo ka fè omwen 2 fwa pi gwo. Men, tout depann sou ki sa egzakteman yo pral lanse sou kourè sa a ak nan ki kantite.
Li ase.
Prepare docker-compose.yml
Travay prensipal la se yon docker-compose.yml inivèsèl, ki devlopè / tèsteur ka itilize tou de lokalman ak nan tiyo CI a.
Premye a tout, nou fè non sèvis inik pou CI. Youn nan varyab inik nan GitLab CI se varyab la CI_JOB_ID. Si ou presize container_name ak siyifikasyon "service-${CI_JOB_ID:-local}", Lè sa a, nan ka a:
- si
CI_JOB_IDpa defini nan varyab anviwònman an,
Lè sa a, non sèvis la pralservice-local - si
CI_JOB_IDdefini nan varyab anviwònman (pa egzanp 123),
Lè sa a, non sèvis la pralservice-123
Dezyèmman, nou kreye yon rezo komen pou sèvis lanse. Sa a ban nou izolasyon rezo-nivo lè w ap kouri plizyè anviwònman tès.
networks:
default:
external:
name: service-network-${CI_JOB_ID:-local}Aktyèlman, sa a se premye etap la nan siksè =)
Egzanp docker-compose.yml mwen ak kòmantè
version: "3"
# Для корректной работы web (php) и fmt нужно,
# чтобы контейнеры имели общий исполняемый контент.
# В нашем случае, это директория /var/www/testrail
volumes:
static-content:
# Изолируем окружение на сетевом уровне
networks:
default:
external:
name: testrail-network-${CI_JOB_ID:-local}
services:
db:
image: mysql:5.7.22
# Каждый container_name содержит ${CI_JOB_ID:-local}
container_name: "testrail-mysql-${CI_JOB_ID:-local}"
environment:
MYSQL_HOST: db
MYSQL_DATABASE: mydb
MYSQL_ROOT_PASSWORD: 1234
SKIP_GRANT_TABLES: 1
SKIP_NETWORKING: 1
SERVICE_TAGS: dev
SERVICE_NAME: mysql
networks:
- default
migration:
image: registry.gitlab.com/touchbit/image/testrail/migration:latest
container_name: "testrail-migration-${CI_JOB_ID:-local}"
links:
- db
depends_on:
- db
networks:
- default
fpm:
image: registry.gitlab.com/touchbit/image/testrail/fpm:latest
container_name: "testrail-fpm-${CI_JOB_ID:-local}"
volumes:
- static-content:/var/www/testrail
links:
- db
networks:
- default
web:
image: registry.gitlab.com/touchbit/image/testrail/web:latest
container_name: "testrail-web-${CI_JOB_ID:-local}"
# Если переменные TR_HTTP_PORT или TR_HTTPS_PORTS не определены,
# то сервис поднимается на 80 и 443 порту соответственно.
ports:
- ${TR_HTTP_PORT:-80}:80
- ${TR_HTTPS_PORT:-443}:443
volumes:
- static-content:/var/www/testrail
links:
- db
- fpm
networks:
- defaultEgzanp kouri lokal
docker-compose -f docker-compose.yml up -d
Starting testrail-mysql-local ... done
Starting testrail-migration-local ... done
Starting testrail-fpm-local ... done
Recreating testrail-web-local ... doneMen, se pa tout bagay ki tèlman senp ak lansman nan CI.
Prepare Makefile la
Mwen sèvi ak Makefile paske li trè pratik tou de pou jesyon anviwònman lokal ak nan CI. Plis kòmantè sou entènèt
# У меня в проектах все вспомогательные вещи лежат в директории `.indirect`,
# в том числе и `docker-compose.yml`
# Использовать bash с опцией pipefail
# pipefail - фейлит выполнение пайпа, если команда выполнилась с ошибкой
SHELL=/bin/bash -o pipefail
# Останавливаем контейнеры и удаляем сеть
docker-kill:
docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml kill
docker network rm network-$${CI_JOB_ID:-testrail} || true
# Предварительно выполняем docker-kill
docker-up: docker-kill
# Создаем сеть для окружения
docker network create network-$${CI_JOB_ID:-testrail}
# Забираем последние образы из docker-registry
docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml pull
# Запускаем окружение
# force-recreate - принудительное пересоздание контейнеров
# renew-anon-volumes - не использовать volumes предыдущих контейнеров
docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
# Ну и, на всякий случай, вывести что там у нас в принципе запущено на машинке
docker ps
# Коллектим логи сервисов
docker-logs:
mkdir ./logs || true
docker logs testrail-web-$${CI_JOB_ID:-local} >& logs/testrail-web.log
docker logs testrail-fpm-$${CI_JOB_ID:-local} >& logs/testrail-fpm.log
docker logs testrail-migration-$${CI_JOB_ID:-local} >& logs/testrail-migration.log
docker logs testrail-mysql-$${CI_JOB_ID:-local} >& logs/testrail-mysql.log
# Очистка раннера
docker-clean:
@echo Останавливаем все testrail-контейнеры
docker kill $$(docker ps --filter=name=testrail -q) || true
@echo Очистка докер контейнеров
docker rm -f $$(docker ps -a -f --filter=name=testrail status=exited -q) || true
@echo Очистка dangling образов
docker rmi -f $$(docker images -f "dangling=true" -q) || true
@echo Очистка testrail образов
docker rmi -f $$(docker images --filter=reference='registry.gitlab.com/touchbit/image/testrail/*' -q) || true
@echo Очистка всех неиспользуемых volume
docker volume rm -f $$(docker volume ls -q) || true
@echo Очистка всех testrail сетей
docker network rm $(docker network ls --filter=name=testrail -q) || true
docker ps
Tcheke
fè docker-up
$ make docker-up
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill
Killing testrail-web-local ... done
Killing testrail-fpm-local ... done
Killing testrail-mysql-local ... done
docker network rm network-${CI_JOB_ID:-testrail} || true
network-testrail
docker network create network-${CI_JOB_ID:-testrail}
d2ec063324081c8bbc1b08fd92242c2ea59d70cf4025fab8efcbc5c6360f083f
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull
Pulling db ... done
Pulling migration ... done
Pulling fpm ... done
Pulling web ... done
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
Recreating testrail-mysql-local ... done
Recreating testrail-fpm-local ... done
Recreating testrail-migration-local ... done
Recreating testrail-web-local ... done
docker ps
CONTAINER ID PORTS NAMES
a845d3cb0e5a 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp testrail-web-local
19d8ef001398 9000/tcp testrail-fpm-local
e28840a2369c 3306/tcp, 33060/tcp testrail-migration-local
0e7900c23f37 3306/tcp testrail-mysql-local
fè docker-mòso bwa
$ make docker-logs
mkdir ./logs || true
mkdir: cannot create directory ‘./logs’: File exists
docker logs testrail-web-${CI_JOB_ID:-local} >& logs/testrail-web.log
docker logs testrail-fpm-${CI_JOB_ID:-local} >& logs/testrail-fpm.log
docker logs testrail-migration-${CI_JOB_ID:-local} >& logs/testrail-migration.log
docker logs testrail-mysql-${CI_JOB_ID:-local} >& logs/testrail-mysql.log
Prepare .gitlab-ci.yml
Kouri tès entegrasyon
Integration:
stage: test
tags:
- my-shell-runner
before_script:
# Аутентифицируемся в registry
- docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
# Генерируем псевдоуникальные TR_HTTP_PORT и TR_HTTPS_PORT
- export TR_HTTP_PORT=$(shuf -i10000-60000 -n1)
- export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1)
# создаем директорию с идентификатором задачи
- mkdir ${CI_JOB_ID}
# копируем в созданную директорию наш docker-compose.yml
# чтобы контекст был разный для каждой задачи
- cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml
script:
# поднимаем наше окружение
- make docker-up
# запускаем тесты исполняемым jar (у меня так)
- java -jar itest.jar --http-port ${TR_HTTP_PORT} --https-port ${TR_HTTPS_PORT}
# или в контейнере
- docker run --network=testrail-network-${CI_JOB_ID:-local} --rm itest
after_script:
# собираем логи
- make docker-logs
# останавливаем окружение
- make docker-kill
artifacts:
# сохраняем логи
when: always
paths:
- logs
expire_in: 30 daysKòm yon rezilta nan kouri tankou yon travay, anyè mòso bwa nan zafè yo pral gen mòso bwa sèvis ak tès yo. Ki trè pratik nan ka ta gen erè. Chak tès nan paralèl ekri pwòp boutèy demi lit, men mwen pral pale sou sa separeman.
Netwaye kourè a
Travay la pral lanse sèlman dapre yon orè.
stages:
- clean
- build
- test
Clean runner:
stage: clean
only:
- schedules
tags:
- my-shell-runner
script:
- make docker-cleanApre sa, ale nan pwojè GitLab nou an -> CI/CD -> Orè -> Nouvo Orè epi ajoute yon nouvo orè.
Rezilta
Lanse 4 travay nan GitLab CI
Nan mòso bwa yo nan dènye travay la ak tès entegrasyon nou wè veso ki soti nan travay diferan
CONTAINER ID NAMES
c6b76f9135ed testrail-web-204645172
01d303262d8e testrail-fpm-204645172
2cdab1edbf6a testrail-migration-204645172
826aaf7c0a29 testrail-mysql-204645172
6dbb3fae0322 testrail-web-204645084
3540f8d448ce testrail-fpm-204645084
70fea72aa10d testrail-mysql-204645084
d8aa24b2892d testrail-web-204644881
6d4ccd910fad testrail-fpm-204644881
685d8023a3ec testrail-mysql-204644881
1cdfc692003a testrail-web-204644793
6f26dfb2683e testrail-fpm-204644793
029e16b26201 testrail-mysql-204644793
c10443222ac6 testrail-web-204567103
04339229397e testrail-fpm-204567103
6ae0accab28d testrail-mysql-204567103
b66b60d79e43 testrail-web-204553690
033b1f46afa9 testrail-fpm-204553690
a8879c5ef941 testrail-mysql-204553690
069954ba6010 testrail-web-204553539
ed6b17d911a5 testrail-fpm-204553539
1a1eed057ea0 testrail-mysql-204553539Log plis detay
$ docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ export TR_HTTP_PORT=$(shuf -i10000-60000 -n1)
$ export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1)
$ mkdir ${CI_JOB_ID}
$ cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml
$ make docker-up
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill
docker network rm testrail-network-${CI_JOB_ID:-local} || true
Error: No such network: testrail-network-204645172
docker network create testrail-network-${CI_JOB_ID:-local}
0a59552b4464b8ab484de6ae5054f3d5752902910bacb0a7b5eca698766d0331
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull
Pulling web ... done
Pulling fpm ... done
Pulling migration ... done
Pulling db ... done
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
Creating volume "204645172_static-content" with default driver
Creating testrail-mysql-204645172 ...
Creating testrail-mysql-204645172 ... done
Creating testrail-migration-204645172 ... done
Creating testrail-fpm-204645172 ... done
Creating testrail-web-204645172 ... done
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c6b76f9135ed registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 13 seconds ago Up 1 second 0.0.0.0:51148->80/tcp, 0.0.0.0:25426->443/tcp testrail-web-204645172
01d303262d8e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 16 seconds ago Up 13 seconds 9000/tcp testrail-fpm-204645172
2cdab1edbf6a registry.gitlab.com/touchbit/image/testrail/migration:latest "docker-entrypoint.s…" 16 seconds ago Up 13 seconds 3306/tcp, 33060/tcp testrail-migration-204645172
826aaf7c0a29 mysql:5.7.22 "docker-entrypoint.s…" 18 seconds ago Up 16 seconds 3306/tcp testrail-mysql-204645172
6dbb3fae0322 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 36 seconds ago Up 22 seconds 0.0.0.0:44202->80/tcp, 0.0.0.0:20151->443/tcp testrail-web-204645084
3540f8d448ce registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 38 seconds ago Up 35 seconds 9000/tcp testrail-fpm-204645084
70fea72aa10d mysql:5.7.22 "docker-entrypoint.s…" 40 seconds ago Up 37 seconds 3306/tcp testrail-mysql-204645084
d8aa24b2892d registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up 53 seconds 0.0.0.0:31103->80/tcp, 0.0.0.0:43872->443/tcp testrail-web-204644881
6d4ccd910fad registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp testrail-fpm-204644881
685d8023a3ec mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp testrail-mysql-204644881
1cdfc692003a registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up About a minute 0.0.0.0:44752->80/tcp, 0.0.0.0:23540->443/tcp testrail-web-204644793
6f26dfb2683e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp testrail-fpm-204644793
029e16b26201 mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp testrail-mysql-204644793
c10443222ac6 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:57123->80/tcp, 0.0.0.0:31657->443/tcp testrail-web-204567103
04339229397e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204567103
6ae0accab28d mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204567103
b66b60d79e43 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:56321->80/tcp, 0.0.0.0:58749->443/tcp testrail-web-204553690
033b1f46afa9 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204553690
a8879c5ef941 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204553690
069954ba6010 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:32869->80/tcp, 0.0.0.0:16066->443/tcp testrail-web-204553539
ed6b17d911a5 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204553539
1a1eed057ea0 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204553539Tout travay konplete avèk siksè
Zafè travay yo genyen mòso bwa sèvis ak tès yo
Tout bagay sanble bèl, men gen yon nuans. Pipeline ka fòse-anile pandan tès entegrasyon yo ap kouri, nan ka sa a kouri kontenè yo pa pral sispann. De tan zan tan ou bezwen netwaye kourè a. Malerezman, travay pou amelyorasyon nan GitLab CE toujou nan sitiyasyon
Men, nou te ajoute lansman yon travay selon yon orè, epi pèsonn pa entèdi nou kouri li manyèlman.
Ale nan pwojè nou an -> CI/CD -> Orè epi kouri travay la Clean runner
Total:
- Nou gen yon sèl kourè kokiy.
- Pa gen okenn konfli ant travay ak anviwònman an.
- Nou kouri travay ak tès entegrasyon an paralèl.
- Ou ka fè tès entegrasyon swa lokalman oswa nan yon veso.
- Yo kolekte mòso bwa sèvis ak tès yo epi yo tache ak travay tiyo a.
- Li posib pou netwaye kourè a soti nan vye imaj Docker.
Tan konfigirasyon se ~ 2 èdtan.
Sa a tout, aktyèlman. Mwen pral kontan resevwa fidbak.
Sous: www.habr.com