Our division creates fully automatic pipelines for bringing new versions of applications to the production environment. Of course, this requires automated functional tests. Under the cut is a story about how, starting with testing in a single thread on a local machine, we reached a multi-threaded launch of autotests on Selenoid in the assembly pipeline with an Allure report on GitLab pages and, as a result, we got a cool automation tool that future users can use. commands.
Where did we start
To implement autotests and integrate them into the pipeline, we needed an automation framework that can be flexibly changed to our needs. Ideally, I wanted to get a single standard for the autotest engine, adapted for embedding autotests into the pipeline. For implementation, we have chosen the following technologies:
- Java
- maven,
- selenium,
- Cucumber+JUNIT 4,
- allure,
- gitlab.
Why this particular set? Java is one of the most popular languages ββfor autotests, and besides, all team members know it. Selenium is the obvious solution. Cucumber, among other things, was supposed to increase the confidence in the results of autotests from the departments involved in manual testing.
Single threaded tests
In order not to reinvent the wheel, we took developments from various repositories on GitHub as the basis of the framework and adapted them for ourselves. We created a repository for the main library with the core of the autotest framework and a repository with a Gold example of implementing autotests on our core. Each team had to take a Gold image and develop tests in it, adapting it to their project. Deployed in a GitLab-CI bank, on which we configured:
- daily launches of all written autotests for each project;
- runs in the build pipeline.
At first, there were few tests, and they went in one stream. A single-threaded launch on the GitLab Windows runner suited us quite well: the tests loaded the test bench very little and almost did not utilize resources.
Over time, there were more and more autotests, and we thought about running them in parallel, when a full run began to take about three hours. Other problems also appeared:
- we couldn't make sure the tests were stable;
- tests that ran multiple times in a row on the local machine sometimes crashed in CI.
An example of setting up autotests:
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<skipTests>${skipTests}</skipTests>
<testFailureIgnore>false</testFailureIgnore>
<argLine>
-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
-Dcucumber.options="--tags ${TAGS} --plugin io.qameta.allure.cucumber2jvm.AllureCucumber2Jvm --plugin pretty"
</argLine>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-maven</artifactId>
<version>2.9</version>
</plugin>
</plugins>
Allure report example
Runner load during tests (8 cores, 8 GB RAM, 1 thread)
Advantages of single-threaded tests:
- easy to set up and run;
- launches in CI are practically no different from local launches;
- tests do not affect each other;
- minimum resource requirements for the runner.
Cons of single-threaded tests:
- take a very long time to complete;
- long stabilization of tests;
- inefficient use of runner resources, extremely low utilization.
Tests on JVM forks
Since we did not take care of thread-safe code when implementing the base framework, the most obvious way to run in parallel was to for maven. The plugin is easy to set up, but for correct parallel operation, autotests need to be run in separate browsers. Nothing to do, I had to use Selenoid.
Selenoid server was raised on a machine with 32 cores and 24 GB of RAM. The limit was set at 48 browsers - 1,5 threads per core and about 400 MB of RAM. As a result, the test time was reduced from three hours to 40 minutes. Speeding up the runs helped solve the stability problem: now we could quickly run new autotests 20-30 times until we were sure that they were running stably.
The first drawback of the solution was the high resource utilization of runners with a small number of parallel threads: on 4 cores and 8 GB of RAM, the tests worked stably in no more than 6 threads. The second minus: the plugin generates runner classes for each scenario, no matter how many of them are run.
Important! Don't pass a tagged variable into argLine, for example like this:
<argLine>-Dcucumber.options="--tags ${TAGS} --plugin io.qameta.allure.cucumber2jvm.AllureCucumber2Jvm --plugin pretty"</argLine>
β¦
Mvn βDTAGS="@smoke"
If you pass the tag in this way, the plugin will generate runners for all tests, that is, it will try to run all tests, skipping them immediately after launch and creating many JVM forks in the process.
Correctly throw a variable with a tag into Tags in plugin settings, see example below. In other methods we have tested, there are problems connecting the Allure plugin.
Runtime example for 6 short tests with incorrect setup:
[INFO] Total time: 03:17 min
An example of the test run time, if you directly pass the tag to mvn... βDcucumber.options:
[INFO] Total time: 44.467 s
An example of setting up autotests:
<profiles>
<profile>
<id>parallel</id>
<build>
<plugins>
<plugin>
<groupId>com.github.temyers</groupId>
<artifactId>cucumber-jvm-parallel-plugin</artifactId>
<version>5.0.0</version>
<executions>
<execution>
<id>generateRunners</id>
<phase>generate-test-sources</phase>
<goals>
<goal>generateRunners</goal>
</goals>
<configuration>
<tags>
<tag>${TAGS}</tag>
</tags>
<glue>
<package>stepdefs</package>
</glue>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
<configuration>
<forkCount>12</forkCount>
<reuseForks>false</reuseForks>
<includes>**/*IT.class</includes>
<testFailureIgnore>false</testFailureIgnore>
<!--suppress UnresolvedMavenProperty -->
<argLine>
-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" -Dcucumber.options="--plugin io.qameta.allure.cucumber2jvm.AllureCucumber2Jvm TagPFAllureReporter --plugin pretty"
</argLine>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</profile>
Allure report example (most unstable test, 4 reruns)
Runner load during tests (8 cores, 8 GB RAM, 12 threads)
Pros:
- easy setup - you just need to add a plugin;
- the ability to simultaneously run a large number of tests;
- acceleration of test stabilization due to item 1.
Cons:
- multiple OS/containers required;
- high resource consumption per fork;
- The plugin has been deprecated and is no longer supported.
How to beat instability
Test benches are not perfect, just like autotests themselves. Not surprisingly, we now have a number of flacky tests. Came to the rescue, which out of the box supports restarting failed tests. You need to update the plugin version to at least 2.21 and write one line with the number of restarts in the pom file or pass it as an argument to Maven.
An example of setting up autotests:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
<configuration>
β¦.
<rerunFailingTestsCount>2</rerunFailingTestsCount>
β¦.
</configuration>
</plugin>
Or at startup: mvn ... -Dsurefire.rerunFailingTestsCount=2 ...
Alternatively, set the Maven options for the PowerShell script (PS1):
Set-Item Env:MAVEN_OPTS "-Dfile.encoding=UTF-8 -Dsurefire.rerunFailingTestsCount=2"Pros:
- no need to waste time analyzing an unstable test when it crashes;
- can mitigate test bench stability problems.
Cons:
- you can skip floating defects;
- run time increases.
Parallel tests with Cucumber 4 library
The number of tests grew every day. We again thought about speeding up runs. In addition, I wanted to embed as many tests as possible into the application assembly pipeline. The critical factor was the too long generation of runners when running in parallel using the Maven plugin.
At that time, Cucumber 4 had already been released, so we decided to rewrite the kernel for this version. In the release notes, we were promised a parallel launch at the thread level. Theoretically, this should have been:
- significantly speed up the run of autotests by increasing the number of threads;
- eliminate the loss of time for the generation of runners for each autotest.
It turned out to be not so difficult to optimize the framework for multi-threaded autotests. Cucumber 4 runs each individual test in a dedicated thread from start to finish, so some common static things were simply converted to ThreadLocal variables.
The main thing when converting using Idea refactoring is to check the places where the variable was compared (for example, checking for null). In addition, you need to put the Allure plugin in the annotation of the Junit Runner class.
An example of setting up autotests:
<profile>
<id>parallel</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<useFile>false</useFile>
<testFailureIgnore>false</testFailureIgnore>
<parallel>methods</parallel>
<threadCount>6</threadCount>
<perCoreThreadCount>true</perCoreThreadCount>
<argLine>
-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
</argLine>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</profile>
Allure report example (most unstable test, 5 reruns)
Runner load during tests (8 cores, 8 GB RAM, 24 threads)
Pros:
- low resource consumption;
- native support from Cucumber - no additional tools needed;
- the ability to run more than 6 threads per processor core.
Cons:
- you need to ensure that the code supports multi-threaded execution;
- the entry threshold increases.
Allure reports in GitLab pages
After the introduction of multi-threaded launch, we began to spend much more time analyzing reports. At that time, we had to upload each report as an artifact in GitLab, then download it, unpack it. It is not very convenient and long. And if someone else wants to see the report at home, then he will need to do the same operations. We wanted to get feedback faster, and we found a way out - GitLab pages. This is a built-in feature that is available out of the box in all recent versions of GitLab. Allows you to deploy static sites on your server and access them via a direct link.
All screenshots with Allure reports are made in GitLab pages. Script for deploying a report on GitLab pages - on Windows PowerShell (before that, you need to run autotests):
New-Item -ItemType directory -Path $testresulthistory | Out-Null
try {Invoke-WebRequest -Uri $hst -OutFile $outputhst}
Catch{echo "fail copy history"}
try {Invoke-WebRequest -Uri $hsttrend -OutFile $outputhsttrnd}
Catch{echo "fail copy history trend"}
mvn allure:report
#mvn assembly:single -PzipAllureReport
xcopy $buildlocationtargetsiteallure-maven-plugin* $buildlocationpublic /s /i /Y
With the result that
So, if you have been thinking about whether you need Thread safe code in the Cucumber autotest framework, now the answer is obvious - with Cucumber 4 it is easy to implement, thereby significantly increasing the number of threads running simultaneously. With this method of running tests, the question is already about the performance of the machine with Selenoid and the test bench.
Practice has shown that running autotests on threads allows you to minimize resource consumption with the best performance. As can be seen from the graphs, a 2x increase in threads does not lead to a similar acceleration in passing performance tests. Nevertheless, we were able to add more than 200 automated tests to the application build, which, even with 5 reruns, are completed in about 24 minutes. This allows you to get quick feedback from them, and if necessary, make changes and repeat the procedure again.
Source: habr.com