One of the challenges that multi-product software vendors often face is the duplication of skills among engineers β developers, testers, and infrastructure administrators β in almost every team. This also affects costly engineers β specialists in load testing.
Instead of focusing on their core responsibilities and leveraging their unique expertise to establish the load testing process, select methodologies, optimize metric values, and write automated tests according to load profiles, engineers often find themselves building testing infrastructure from scratch, setting up load tools, integrating them into CI systems, configuring monitoring, and generating reports.
Solutions to some organizational issues in testing, which we apply at Positive Technologies, can be found in . In this one, I will discuss the possibility of integrating load tests into the overall CI pipeline using the concept of 'load testing as a service.' You will learn how and which Docker images for load sources can be used in the CI pipeline; how to connect load sources to your CI project using a build template; what a demo pipeline for running load tests and publishing results looks like. This article may be useful for software testing engineers and CI automation engineers considering the architecture of their load testing system.
The Essence of the Concept
The load testing as a service concept implies the ability to integrate load tools like Apache JMeter, Yandex.Tank, and proprietary frameworks into any continuous integration system. The demo example will be for GitLab CI, but the principles outlined are applicable to all CI systems.
Load testing as a service is a centralized service for conducting load testing. Load tests are executed in dedicated agent pools, and results are automatically published to GitLab Pages, Influx DB, and Grafana or to test reporting systems (TestRail, ReportPortal, etc.). Automation and scaling are implemented as simply as adding and parameterizing a standard gitlab-ci.yml template in the GitLab CI project.
The advantage of this approach is that the entire CI infrastructure, load agents, Docker images of load sources, testing pipelines, and report publications are supported by the centralized automation department (DevOps engineers), allowing load testing engineers to focus their efforts on test development and result analysis without dealing with infrastructure issues.
For the sake of simplicity, let's assume that the target application or server to be tested has already been deployed and configured in advance (automated scripts in Python, SaltStack, Ansible, etc. can be used for this). Then the entire concept of load testing as a service is divided into three stages: preparation, testing, report publishing. More details in the diagram (all images are clickable):
Key concepts and definitions in load testing
When conducting load tests, we try to adhere to , using appropriate terminology and recommended metrics. I'll provide a brief list of key concepts and definitions in load testing.
Load agent β a virtual machine where the load source application (Apache JMeter, Yandex.Tank, or a custom load module) will be run.
Test goal β the server or application installed on the server that will be subjected to the load.
Test case β a set of parameterized steps: user actions and expected responses to these actions, with fixed network requests and responses depending on the specified parameters.
Load profile or plan β in (4.2.4, p. 43) load profiles define critical metrics for a specific test and variations in load parameters throughout the test. You can see examples of profiles in the illustration.
Test β a scenario with a predefined set of parameters.
Test plan β a set of tests and load profile.
Test run β one iteration of executing a single test with a fully completed load scenario and a resulting report.
Network request β an HTTP request sent from the agent to the target.
Network response β The HTTP response sent from the target to the agent.
HTTP response code (HTTP responses status) β a standard response code from the application server.
Transaction (transaction) β the complete cycle of 'request - response'. A transaction is considered from the beginning of sending the request (request) to the completion of receiving the response (response).
Transaction status (transactions status) β whether the 'request - response' cycle was successfully completed. If there was any error in this cycle, the entire transaction is considered unsuccessful.
Response time (latency) β the time from the end of sending the request (request) to the beginning of receiving the response (response).
Load metrics (metrics) β characteristics of the loaded service and load agent determined during load testing.
Key metrics for measuring load parameters
Some of the most commonly used and recommended in the methodology (pp. 36, 52) metrics are listed in the table below. Similar metrics for the agent and target are indicated in one row.
Metrics for the load agent
Metrics of the target system or application tested under load
QuantityΒ Β vCPU and memory RAM,
Disk β 'hardware' characteristics of the load agent
CPU, Memory, Disk usage β the dynamics of CPU, memory, and disk load
during testing. Usually measured as a percentage of
the maximum available values
Network throughput (on load agent) β the bandwidth
of the network interface on the server
where the load agent is installed.
Usually measured in bytes per second (bps)
Network throughput(on target) β the bandwidth of the network interface
on the target server. Usually measured in bytes per second (bps)
Virtual usersβ the number of virtual users
implementing load scenarios and
simulating real user actions
Virtual users status, Passed/Failed/Total β the number of successful and
unsuccessful statuses of virtual users' operation
for load scenarios, as well as their total number.
It is generally expected that all users were able to complete
all their tasks specified in the load profile.
Any error will mean that a real user will not be able to
complete their task when working with the system.
Requests per second (minute)β the number of network requests per second (or minute).
An important characteristic of the load agent: how many requests it can generate.
This is essentially a simulation of interactions with the application by virtual users.
Responses per second (minute)
β the number of network responses per second (or minute).
An important characteristic of the target service: how many
responses were generated and sent to requests from
the load agent.
HTTP responses statusβ the number of different response codes
from the application server, received by the load agent.
For example, 200 OK means a successful request,
while 404 indicates that the resource was not found.
Latency (response time) β the time from the end of
sending a request to the beginning of receiving a response.
Usually measured in milliseconds (ms)
Transaction response timeβ the time of one complete transaction,
the end of the 'request β response' cycle.
This is the time from the start of sending the request
to the completion of receiving the response.
Transaction time can be measured in seconds (or minutes)
in several ways: the minimum,
maximum, average, and, for example, the 90th percentile.
Minimum and maximum readings are the extreme
states of system performance.
The ninetieth percentile is used most often,
as it represents the majority of users
comfortably operating at the threshold of system performance.
Transactions per second (minute) β the number of complete
transactions per second (minute),
that is, how many requests the application was able to accept and
process, and provide responses.
This is essentially the system's throughput.
Transactions status , Passed / Failed / Total β the number of
successful, unsuccessful, and total transactions.
For real users, an unsuccessful
transaction would actually mean
the inability to work with the system under load.
The basic scheme of load testing
The basic scheme of load testing is very simple and consists of three main stages, which I have already mentioned: Prepare β Test β Report, that is, preparing the testing objectives and setting parameters for the load sources, then conducting load tests, and finally, generating and publishing the test report.
Notes on the scheme:
- QA.Tester β an expert in load testing,
- Target β the target application for which behavior under load needs to be understood.
Classifier of entities, stages, and steps in the scheme
Stages and steps
What happens
Input
Output
Prepare: testing preparation stage
LoadParameters
Assignment and initialization
by the user
of load parameters,
metric selection and
test plan preparation
(load profile)
User parameters for
load agent initialization
Test Plan
Testing goal
VM
Cloud deployment
of a virtual machine with
the required specifications
VM parameters for the load agent
Automation scripts for
VM creation
Configured VM in
the cloud
Env
OS setup and preparation
of the environment for
load agent operation
Environment parameters for
the load agent
Automation scripts for
environment settings
Prepared environment:
OS, services, and applications,
necessary for operation
the load agent
LoadAgents
Installation, configuration, and parameterization
of the load agent.
Or downloading the Docker image with
a pre-configured load source
Docker image of the load source
(JMeter, JM, or custom framework)
Configuration parameters
the load agent
Configured and ready
to operate load agent
Test: execution stage of load tests. Sources are load agents deployed in dedicated agent pools for GitLab CI
Load
Launching the load agent
with the selected test plan
and load parameters
User parameters
for initialization
the load agent
Test Plan
Testing goal
Execution logs
of load tests
System logs
Dynamics of target and load agent metric changes
RunAgents
Execution by the load agent
of test scenarios
according to the
load profile
Interaction of the load agent
with the testing target
Test Plan
Testing goal
Logs
Collection of raw logs
during load testing:
records of load agent actions,
status of the testing target
and the VM on which the agent is running
Execution logs
of load tests
System logs
Metrics
Collection of raw metrics during testing
Dynamics of target metric changes
and the load agent
Report: stage of preparing the testing report
Generator
Processing collected
raw metrics and logs by the load system and
monitoring system
Generating the report in
human-readable format,
possibly with elements
of analytics
Dynamics of metric changes
Execution logs
of load tests
System logs
of the target and load agent
Processed raw logs
in a format suitable for
export to external storage
Static report on load,
suitable for human analysis
Publishing the report
Publish
on load
testing in an external
service
service
Processed "raw"
logs in a suitable format
for export to external
storage
Reports saved in external
storage about
load, suitable
for human analysis
Connecting load sources in the CI template
Let's move on to the practical part. I want to show how in some projects at we implemented the concept of load testing as a service.
First, our DevOps engineers created a dedicated pool of agents in GitLab CI to run load tests. To avoid confusion in the templates with other, such as build, pools, we added tags to these agents, : load. Any other understandable tags can be used. They are set of GitLab CI Runners.
How to determine the required hardware capacity? The specifications of the load agents β sufficient number of vCPUs, RAM, and Disk β can be calculated based on the need for Docker, Python (for Yandex.Tank), GitLab CI agent, and Java (for Apache JMeter) to be running on the agent. For Java under JMeter, it is also recommended to use at least 512 MB of RAM and, at most, .
Thus, based on our experience, we recommend using for load agents at least: 4 vCPUs, 4 GB of RAM, 60 GB of SSD. Network card bandwidth is determined based on the load profile requirements.
We primarily use two load sources β Docker images of Apache JMeter and Yandex.Tank.
is an open-source tool from Yandex for conducting load testing. Its modular architecture is based on a high-performance asynchronous hit-based HTTP request generator called Phantom. The Tank includes built-in resource monitoring of the tested server via SSH, can automatically stop tests based on specified conditions, and can display results both in the console and as graphs, with the capability to connect custom modules for extended functionality. By the way, we used Tank before it became mainstream. The article "" discusses the history of how in 2013 we conducted load testing of β one of our company's products.
JMeter is an open-source tool for load testing developed by Apache. It can be effectively used for testing both static and dynamic web applications. JMeter supports a wide range of protocols and interaction methods with applications: HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, etc.), SOAP/REST Web services, FTP, TCP, LDAP, SMTP(S), POP3(S), and IMAP(S), databases via JDBC, can execute shell commands, and work with Java objects. JMeter has an IDE for creating, debugging, and executing test plans. There is also a CLI for operation in the command line of any Java-compatible OS (Linux, Windows, Mac OS X). The tool can dynamically generate an HTML report on the testing.
For convenience within our company, to allow testers to modify and add environments themselves, we created Docker image builds of load sources on GitLab CI with publication to an internal This makes it faster and easier to integrate them into pipelines for load testing. To learn how to do a docker push to the registry via GitLab CI, see .
We took this as the base Docker file for Yandex.Tank:
Dockerfile
1 | FROM direvius/yandex-tank
2 | ENTRYPOINT [""]And for Apache JMeter this:
Dockerfile
1 | FROM vmarrazzo/jmeter
2 | ENTRYPOINT [""]You can read about how our continuous integration system is structured in the article "Β».
Template and Pipeline
An example template for conducting load tests is available in the project . In you can find instructions on using the template. In the template itself (file ) there are notes about what each step is responsible for.
The template is very simple and demonstrates three stages of load testing, as described above in the diagram: preparation, testing, and reporting. These are handled by : Prepare, Test, and Report..
- The stage should be used for preliminary setup of testing targets or checking their availability. The environment for the load sources does not need to be configured, as they are pre-built as Docker images and uploaded to the Docker registry: you just need to specify the required version at the Test stage. However, you can rebuild them and create your own modified images.
- The is used to specify the load source, run tests, and save test artifacts. Any load source can be chosen: Yandex.Tank, Apache JMeter, your own, or all together. To disable unnecessary sources, simply comment out or delete the job. Entry points for load sources:
- the launch parameters for Yandex.Tank are specified in the file.,
- the launch parameters for Apache JMeter are specified in the file .
Note: The configuration template is used to set up interaction with the CI system and does not imply the placement of test logic in it. For tests, an entry point is specified where the controlling bash script resides. The method for running tests, generating reports, and the test scripts themselves must be implemented by QA engineers. In the demo example, for both load sources, a simple test uses the request of the main page of Yandex. Scenarios and test parameters are located in the directory .
- At the stage you need to describe the ways of publishing the test results obtained at the Test stage to external storage, for example, to GitLab Pages or special reporting systems. For GitLab Pages, it is required that after the tests are completed, the directory ./public is not empty and contains at least the index.html file. You can read about the nuances of how the GitLab Pages service works .
Examples of how to export data:
- from JMeter to ,
- from Yandex.Tank to .
Instructions for setting up publication:
- HTML static files in ,
- in InfluxDB and then in .
In the demo example, the pipeline with load tests and two load sources (the unnecessary one can be disabled) looks like this:
Apache JMeter can generate HTML reports on its own, so it is more advantageous to save it in GitLab Pages using built-in tools. Hereβs what an Apache JMeter report looks like:
In the demo example for Yandex.Tank, you will only see a in the section for GitLab Pages. During testing, Tank can save the results to an InfluxDB database, and from there they can be displayed, for example, in Grafana (configuration is done in the file ). Hereβs what a Tank report looks like in Grafana:
Summary
In this article, I explored the concept of "load testing as a service." The main idea is to utilize a pre-configured infrastructure of load agent pools, Docker images for load sources, reporting systems, and an integrated pipeline in GitLab CI based on a simple .gitlab-ci.yml template. This is all supported by a small team of automation engineers and replicated upon request by product teams. I hope this will assist you in preparing and implementing a similar scheme in your company. Thank you for your attention!
P. S. I want to extend a big thank you to my colleagues, Sergey Kurbanov and Nikolai Yusev, for their technical assistance in implementing the load testing as a service concept in our company.
Author: β Deputy Head of the Development Technologies and Processes (DevOps) Department at Positive Technologies
Source: habr.com
