🥇GitLab Shell Runner. Vzporedno izvajanje testiranih storitev s pomočjo Docker Compose

Ta članek bo zanimiv tako za testere kot za razvijalce, vendar je bolj usmerjen na avtomatizatorje, ki so se soočili s težavo nastavitve GitLab CI/CD za izvajanje integracijskih testov v pogojih pomanjkanja infrastrukturnih virov in/ali odsotnosti platforme za orkestracijo kontejnerjev. Opisal bom, kako nastaviti uvajanje testnih okolij s pomočjo docker compose na enem samem GitLab shell runnerju in tako, da si ob uvajanju več okolij zagonjeni storitveni elementi ne bodo povzročali težav.

Vsebina

Predpostavke
GitLab Shell Runner
Priprava docker-compose.yml
Priprava Makefile
Priprava .gitlab-ci.yml
- Zagon integracijskih testov
- Čiščenje runnerja
Результат.

Predpostavke

V svoji praksi sem pogosto naletel na "reševanje" integracijskega testiranja pri projektih. In pogosto je prva in največja težava CI pipeline, v katerem se integracijsko testiranje razvijajoče storitev(i) izvaja v dev/stage okolju. To je povzročilo kar nekaj težav:
- Zaradi napak v neki storitvi se med integracijskim testiranjem testni obris lahko poškoduje s pokvarjenimi podatki. Bile so situacije, ko je pošiljanje zahteve s pokvarjenim JSON-om povzročilo, da je storitev prenehala delovati, kar je pripravilo celotni stand v neuporabno stanje.
- Upočasnjevanje delovanja testnega obrisa z rastjo testnih podatkov. Mislim, da ni smiselno opisovati primera z čiščenjem/vrnitvijo DB. V svoji praksi nisem srečal projekta, kjer bi ta postopek potekal gladko.
- Tveganje, da se prekine delovanje testnega obrisa pri testiranju skupnih nastavitev sistema. Na primer, uporabnik/skupina/geslo/politika aplikacije.
- Testni podatki iz avtomatskih testov otežujejo delo ročnim testerjem.
Nekdo bo rekel, da bi morali dobri avtomatski testi očistiti podatke za seboj. Imam argumente proti:
- Dinamična stojala so zelo priročna za uporabo.
- Ne morete vsak predmet odstraniti iz sistema prek API. Na primer, klic za odstranitev predmeta ni izveden, ker je v nasprotju s poslovno logiko.
- Pri ustvarjanju predmeta prek API se lahko ustvari ogromna količina metapodatkov, ki jih je težko odstraniti.
- Če imajo testi medsebojno odvisnost, potem se postopek čiščenja podatkov po izvajanju testov spremeni v težavo.
- Dodatni (in, po mojem mnenju, neopravdani) klici na API.
- In the end, the main argument is that when test data starts to be cleaned directly from the database, it turns into a real PK/FK circus! Developers often say, 'I just added/removed/renamed a table; why did 100500 integration tests fail?'
In my opinion, the most optimal solution is a dynamic environment.
Many people use docker-compose to launch the test environment, but few utilize docker-compose for conducting integration testing in CI/CD. Here, I am not considering Kubernetes, Swarm, or other container orchestration platforms as they are not available in every company. It would be great if the docker-compose.yml was universal.
If we do have our own QA runner, how do we ensure that the services launched via docker-compose do not interfere with each other?
How do we collect logs from the tested services?
How do we clean the runner?

I have my own GitLab runner for my projects, and I encountered these questions during development. Java client za TestRail. More specifically, during the launch of integration tests. We will resolve these issues with examples from this project.

Nazaj na vsebino

GitLab Shell Runner

For the runner, I recommend a Linux virtual machine with 4 vCPU, 4 GB RAM, and 50 GB HDD.
There is a lot of information on the internet about setting up gitlab-runner, so briefly:

Log into the machine via SSH
If you have less than 8 GB RAM, I recommend creating a 10 GB swap, to prevent the OOM killer from terminating our tasks due to insufficient RAM. This can happen when more than 5 tasks are run simultaneously. The tasks will run a bit slower but will be stable.
Example with the OOM killer
If you see in the task logs bash: line 82: 26474 Killed, simply execute on the runner sudo dmesg | grep 26474
```
[26474]  1002 26474  1061935   123806     339        0             0 java
Out of memory: Kill process 26474 (java) score 127 or sacrifice child
Killed process 26474 (java) total-vm:4247740kB, anon-rss:495224kB, file-rss:0kB, shmem-rss:0kB
```
And if the picture looks something like this, then either add swap, or increase the RAM.

Namestimo gitlab-runner, docker, docker-compose, make.

Adding a user gitlab-runner to the group docker

sudo groupadd docker
sudo usermod -aG docker gitlab-runner

Registriramo gitlab-runner.
Open for editing /etc/gitlab-runner/config.toml and add
```
concurrent=20
[[runners]]
  request_concurrency = 10
```
This will allow launching parallel tasks on one runner. For further reading, tukaj.
If you have a more powerful machine, for example, 8 vCPU, 16 GB RAM, these numbers can be increased by at least 2 times. But it all depends on what exactly will be run on this runner and in what quantity.

To je dovolj.

Nazaj na vsebino

Priprava docker-compose.yml

Glavna naloga je univerzalni docker-compose.yml, ki ga lahko razvijalci/testerji uporabljajo tako lokalno kot v CI pipeline.

Najprej ustvarimo edinstvene nazive storitev za CI. Ena od edinstvenih spremenljivk v GitLab CI je spremenljivka CI_JOB_ID. Če navedemo container_name z vrednostjo "service-{CI_JOB_ID:-local}", potem v primeru:

če CI_JOB_ID ni določena v spremenljivkah okolja,
bodo imena storitev service-local
če CI_JOB_ID določena v spremenljivkah okolja (na primer 123),
bodo imena storitev service-123

Nato ustvarimo skupno omrežje za zagnane storitve. To nam daje izolacijo na ravni omrežja pri zagonu več testnih okolij.

networks:
  default:
    external:
      name: service-network-${CI_JOB_ID:-local}

To je pravzaprav prvi korak do uspeha =)

Primer mojega docker-compose.yml z komentarji

version: "3"

# Za pravilno delovanje web (php) in fmt morajo 
# imeti vsebinski kontejnerji skupno izvršljivo vsebino.
# V našem primeru je to imenik /var/www/testrail
volumes:
  static-content:

# Izoliramo okolje na omrežni ravni
networks:
  default:
    external:
      name: testrail-network-${CI_JOB_ID:-local}

services:
  db:
    image: mysql:5.7.22
    # Ime vsakega kontejnerja vsebuje ${CI_JOB_ID:-local}
    container_name: "testrail-mysql-${CI_JOB_ID:-local}"
    environment:
      MYSQL_HOST: db
      MYSQL_DATABASE: mydb
      MYSQL_ROOT_PASSWORD: 1234
      SKIP_GRANT_TABLES: 1
      SKIP_NETWORKING: 1
      SERVICE_TAGS: dev
      SERVICE_NAME: mysql
    networks:
    - default

  migration:
    image: registry.gitlab.com/touchbit/image/testrail/migration:latest
    container_name: "testrail-migration-${CI_JOB_ID:-local}"
    links:
    - db
    depends_on:
    - db
    networks:
    - default

  fpm:
    image: registry.gitlab.com/touchbit/image/testrail/fpm:latest
    container_name: "testrail-fpm-${CI_JOB_ID:-local}"
    volumes:
    - static-content:/var/www/testrail
    links:
    - db
    networks:
    - default

  web:
    image: registry.gitlab.com/touchbit/image/testrail/web:latest
    container_name: "testrail-web-${CI_JOB_ID:-local}"
    # Če nista določeni spremenljivki TR_HTTP_PORT ali TR_HTTPS_PORTS,
    # se storitev dvigne na 80 in 443 portu.
    ports:
      - ${TR_HTTP_PORT:-80}:80
      - ${TR_HTTPS_PORT:-443}:443
    volumes:
      - static-content:/var/www/testrail
    links:
      - db
      - fpm
    networks:
      - default

Primer lokalnega zagona

docker-compose -f docker-compose.yml up -d
Zagon   testrail-mysql-local     ... končano
Zagon   testrail-migration-local ... končano
Zagon   testrail-fpm-local       ... končano
Ponovno ustvarjanje testrail-web-local       ... končano

Toda z zagonom v CI ni vse tako preprosto.

Nazaj na vsebino

Priprava Makefile

Uporabljam Makefile, saj je to zelo priročno tako za lokalno upravljanje okolja kot tudi v CI. Spodaj so komentarji v vrsticah

# У меня в проектах все вспомогательные вещи лежат в директории `.indirect`,
# в том числе и `docker-compose.yml`

# Использовать bash с опцией pipefail 
# pipefail - фейлит выполнение пайпа, если команда выполнилась с ошибкой
SHELL=/bin/bash -o pipefail

# Останавливаем контейнеры и удаляем сеть
docker-kill:
    docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml kill
    docker network rm network-$${CI_JOB_ID:-testrail} || true

# Предварительно выполняем docker-kill 
docker-up: docker-kill
    # Создаем сеть для окружения 
    docker network create network-$${CI_JOB_ID:-testrail}
    # Забираем последние образы из docker-registry
    docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml pull
    # Запускаем окружение
    # force-recreate - принудительное пересоздание контейнеров
    # renew-anon-volumes - не использовать volumes предыдущих контейнеров
    docker-compose -f $${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
    # Ну и, на всякий случай, вывести что там у нас в принципе запущено на машинке
    docker ps

# Коллектим логи сервисов
docker-logs:
    mkdir ./logs || true
    docker logs testrail-web-$${CI_JOB_ID:-local}       >& logs/testrail-web.log
    docker logs testrail-fpm-$${CI_JOB_ID:-local}       >& logs/testrail-fpm.log
    docker logs testrail-migration-$${CI_JOB_ID:-local} >& logs/testrail-migration.log
    docker logs testrail-mysql-$${CI_JOB_ID:-local}     >& logs/testrail-mysql.log

# Очистка раннера
docker-clean:
    @echo Останавливаем все testrail-контейнеры
    docker kill $$(docker ps --filter=name=testrail -q) || true
    @echo Очистка докер контейнеров
    docker rm -f $$(docker ps -a -f --filter=name=testrail status=exited -q) || true
    @echo Очистка dangling образов
    docker rmi -f $$(docker images -f "dangling=true" -q) || true
    @echo Очистка testrail образов
    docker rmi -f $$(docker images --filter=reference='registry.gitlab.com/touchbit/image/testrail/*' -q) || true
    @echo Очистка всех неиспользуемых volume
    docker volume rm -f $$(docker volume ls -q) || true
    @echo Очистка всех testrail сетей
    docker network rm $(docker network ls --filter=name=testrail -q) || true
    docker ps

Proveravamo.

make docker-up

$ make docker-up 
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill
Killing testrail-web-local   ... done
Killing testrail-fpm-local   ... done
Killing testrail-mysql-local ... done
docker network rm network-${CI_JOB_ID:-testrail} || true
network-testrail
docker network create network-${CI_JOB_ID:-testrail}
d2ec063324081c8bbc1b08fd92242c2ea59d70cf4025fab8efcbc5c6360f083f
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull
Pulling db        ... done
Pulling migration ... done
Pulling fpm       ... done
Pulling web       ... done
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
Recreating testrail-mysql-local ... done
Recreating testrail-fpm-local       ... done
Recreating testrail-migration-local ... done
Recreating testrail-web-local       ... done
docker ps
CONTAINER ID  PORTS                                     NAMES
a845d3cb0e5a  0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp  testrail-web-local
19d8ef001398  9000/tcp                                  testrail-fpm-local
e28840a2369c  3306/tcp, 33060/tcp                       testrail-migration-local
0e7900c23f37  3306/tcp                                  testrail-mysql-local

make docker-logs

$ make docker-logs
mkdir ./logs || true
mkdir: cannot create directory './logs': File exists
docker logs testrail-web-${CI_JOB_ID:-local}       >& logs/testrail-web.log
docker logs testrail-fpm-${CI_JOB_ID:-local}       >& logs/testrail-fpm.log
docker logs testrail-migration-${CI_JOB_ID:-local} >& logs/testrail-migration.log
docker logs testrail-mysql-${CI_JOB_ID:-local}     >& logs/testrail-mysql.log

Nazaj na vsebino

Priprava .gitlab-ci.yml

Zagon integracijskih testov

Integration:
  stage: test
  tags:
    - my-shell-runner
  before_script:
    # Authenticate with the registry
    - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
    # Generate pseudo-unique TR_HTTP_PORT and TR_HTTPS_PORT
    - export TR_HTTP_PORT=$(shuf -i10000-60000 -n1)
    - export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1)
    # Create a directory with the job ID
    - mkdir ${CI_JOB_ID}
    # Copy our docker-compose.yml to the created directory
    # to have different context for each job
    - cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml
  script:
    # Bring up our environment
    - make docker-up
    # Run tests with executable jar (that's how I do it)
    - java -jar itest.jar --http-port ${TR_HTTP_PORT} --https-port ${TR_HTTPS_PORT}
    # Or in a container
    - docker run --network=testrail-network-${CI_JOB_ID:-local} --rm itest
  after_script:
    # Collect logs
    - make docker-logs
    # Stop the environment
    - make docker-kill
  artifacts:
    # Save logs
    when: always
    paths:
      - logs
    expire_in: 30 days

As a result of running such a task, the logs directory in the artifacts will contain the logs of services and tests. This is very convenient in case of errors. Each test writes its log in parallel, but I will talk about this separately.

Nazaj na vsebino

Čiščenje runnerja

The task will only run on a schedule.

stages:
- clean
- build
- test

Clean runner:
  stage: clean
  only:
    - schedules
  tags:
    - my-shell-runner
  script:
    - make docker-clean

Next, go to our GitLab project -> CI/CD -> Schedules -> New Schedule and add a new schedule.

Nazaj na vsebino

Результат.

Launching 4 tasks in GitLab CI

V dnevniku zadnje naloge z integracijskimi testi vidimo kontejnerje iz različnih nalog.

ID KONTEJNERJA  IMENA
c6b76f9135ed  testrail-web-204645172
01d303262d8e  testrail-fpm-204645172
2cdab1edbf6a  testrail-migration-204645172
826aaf7c0a29  testrail-mysql-204645172
6dbb3fae0322  testrail-web-204645084
3540f8d448ce  testrail-fpm-204645084
70fea72aa10d  testrail-mysql-204645084
d8aa24b2892d  testrail-web-204644881
6d4ccd910fad  testrail-fpm-204644881
685d8023a3ec  testrail-mysql-204644881
1cdfc692003a  testrail-web-204644793
6f26dfb2683e  testrail-fpm-204644793
029e16b26201  testrail-mysql-204644793
c10443222ac6  testrail-web-204567103
04339229397e  testrail-fpm-204567103
6ae0accab28d  testrail-mysql-204567103
b66b60d79e43  testrail-web-204553690
033b1f46afa9  testrail-fpm-204553690
a8879c5ef941  testrail-mysql-204553690
069954ba6010  testrail-web-204553539
ed6b17d911a5  testrail-fpm-204553539
1a1eed057ea0  testrail-mysql-204553539

Podroben dnevnik

$ docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
WARNING! Uporaba --password preko CLI je nezadovoljiva. Uporabite --password-stdin.
WARNING! Vaša geslo bo shranjeno nešifrirano v /home/gitlab-runner/.docker/config.json.
Konfigurirajte pomočnika za poverilnice, da odstranite to opozorilo. Glejte
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Prijava uspešna
$ export TR_HTTP_PORT=$(shuf -i10000-60000 -n1)
$ export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1)
$ mkdir ${CI_JOB_ID}
$ cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml
$ make docker-up
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill
docker network rm testrail-network-${CI_JOB_ID:-local} || true
Napaka: Ni takšne mreže: testrail-network-204645172
docker network create testrail-network-${CI_JOB_ID:-local}
0a59552b4464b8ab484de6ae5054f3d5752902910bacb0a7b5eca698766d0331
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull
Pridobivanje splet ... končano
Pridobivanje fpm ... končano
Pridobivanje migracija ... končano
Pridobivanje db ... končano
docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
Ustvarjanje volumna "204645172_static-content" z privzeto voznikom
Ustvarjanje testrail-mysql-204645172 ... 
Ustvarjanje testrail-mysql-204645172 ... končano
Ustvarjanje testrail-migration-204645172 ... končano
Ustvarjanje testrail-fpm-204645172 ... končano
Ustvarjanje testrail-web-204645172 ... končano
docker ps
ID KONTEJNERJA        SLIKA                                                          UKAZ                    USTVARJENO              STATUS              PORTI                                           IMENA
c6b76f9135ed        registry.gitlab.com/touchbit/image/testrail/web:latest         "nginx -g 'daemon of…"   13 sekund nazaj       Zagon 1 sekundo         0.0.0.0:51148->80/tcp, 0.0.0.0:25426->443/tcp   testrail-web-204645172
01d303262d8e        registry.gitlab.com/touchbit/image/testrail/fpm:latest         "docker-php-entrypoi…"   16 sekund nazaj       Zagon 13 sekund       9000/tcp                                        testrail-fpm-204645172
2cdab1edbf6a        registry.gitlab.com/touchbit/image/testrail/migration:latest   "docker-entrypoint.s…"   16 sekund nazaj       Zagon 13 sekund       3306/tcp, 33060/tcp                             testrail-migration-204645172
826aaf7c0a29        mysql:5.7.22                                                   "docker-entrypoint.s…"   18 sekund nazaj       Zagon 16 sekund       3306/tcp                                        testrail-mysql-204645172
6dbb3fae0322        registry.gitlab.com/touchbit/image/testrail/web:latest         "nginx -g 'daemon of…"   36 sekund nazaj       Zagon 22 sekund       0.0.0.0:44202->80/tcp, 0.0.0.0:20151->443/tcp   testrail-web-204645084
3540f8d448ce        registry.gitlab.com/touchbit/image/testrail/fpm:latest         "docker-php-entrypoi…"   38 sekund nazaj       Zagon 35 sekund       9000/tcp                                        testrail-fpm-204645084
70fea72aa10d        mysql:5.7.22                                                   "docker-entrypoint.s…"   40 sekund nazaj       Zagon 37 sekund       3306/tcp                                        testrail-mysql-204645084
d8aa24b2892d        registry.gitlab.com/touchbit/image/testrail/web:latest         "nginx -g 'daemon of…"   Približno minuto nazaj   Zagon 53 sekund       0.0.0.0:31103->80/tcp, 0.0.0.0:43872->443/tcp   testrail-web-204644881
6d4ccd910fad        registry.gitlab.com/touchbit/image/testrail/fpm:latest         "docker-php-entrypoi…"   Približno minuto nazaj   Zagon Približno minuto   9000/tcp                                        testrail-fpm-204644881
685d8023a3ec        mysql:5.7.22                                                   "docker-entrypoint.s…"   Približno minuto nazaj   Zagon Približno minuto   3306/tcp                                        testrail-mysql-204644881
1cdfc692003a        registry.gitlab.com/touchbit/image/testrail/web:latest         "nginx -g 'daemon of…"   Približno minuto nazaj   Zagon Približno minuto   0.0.0.0:44752->80/tcp, 0.0.0.0:23540->443/tcp   testrail-web-204644793
6f26dfb2683e        registry.gitlab.com/touchbit/image/testrail/fpm:latest         "docker-php-entrypoi…"   Približno minuto nazaj   Zagon Približno minuto   9000/tcp                                        testrail-fpm-204644793
029e16b26201        mysql:5.7.22                                                   "docker-entrypoint.s…"   Približno minuto nazaj   Zagon Približno minuto   3306/tcp                                        testrail-mysql-204644793
c10443222ac6        registry.gitlab.com/touchbit/image/testrail/web:latest         "nginx -g 'daemon of…"   Pred 5 urami          Zagon 5 ur          0.0.0.0:57123->80/tcp, 0.0.0.0:31657->443/tcp   testrail-web-204567103
04339229397e        registry.gitlab.com/touchbit/image/testrail/fpm:latest         "docker-php-entrypoi…"   Pred 5 urami          Zagon 5 ur          9000/tcp                                        testrail-fpm-204567103
6ae0accab28d        mysql:5.7.22                                                   "docker-entrypoint.s…"   Pred 5 urami          Zagon 5 ur          3306/tcp                                        testrail-mysql-204567103
b66b60d79e43        registry.gitlab.com/touchbit/image/testrail/web:latest         "nginx -g 'daemon of…"   Pred 5 urami          Zagon 5 ur          0.0.0.0:56321->80/tcp, 0.0.0.0:58749->443/tcp   testrail-web-204553690
033b1f46afa9        registry.gitlab.com/touchbit/image/testrail/fpm:latest         "docker-php-entrypoi…"   Pred 5 urami          Zagon 5 ur          9000/tcp                                        testrail-fpm-204553690
a8879c5ef941        mysql:5.7.22                                                   "docker-entrypoint.s…"   Pred 5 urami          Zagon 5 ur          3306/tcp                                        testrail-mysql-204553690
069954ba6010        registry.gitlab.com/touchbit/image/testrail/web:latest         "nginx -g 'daemon of…"   Pred 5 urami          Zagon 5 ur          0.0.0.0:32869->80/tcp, 0.0.0.0:16066->443/tcp   testrail-web-204553539
ed6b17d911a5        registry.gitlab.com/touchbit/image/testrail/fpm:latest         "docker-php-entrypoi…"   Pred 5 urami          Zagon 5 ur          9000/tcp                                        testrail-fpm-204553539
1a1eed057ea0        mysql:5.7.22                                                   "docker-entrypoint.s…"   Pred 5 urami          Zagon 5 ur          3306/tcp                                        testrail-mysql-204553539

Vse naloge so bile uspešno zaključene

Artefakti naloge vsebujejo dnevnike storitev in testov

Zdi se, da je vse v redu, vendar obstaja ena malenkost. Pipeline se lahko prisilno prekine med izvajanjem integracijskih testov, v tem primeru pa zagnani vsebniki ne bodo ustavljeni. Občasno je potrebno očistiti izvajalca. Žal je naloga za izboljšanje v GitLab CE še vedno v statusu Open

Toda dodali smo možnost zagon naloge po razporedu, in nihče nam ne prepoveduje, da jo zaženemo ročno.
Preidemo v naš projekt -> CI/CD -> Razporedi in zaženemo nalogo Očisti izvajalca