One of the problems that multi-product software vendors often face is the duplication of the competencies of engineers - developers, testers, and infrastructure administrators - on almost every team. This also applies to expensive engineers - specialists in the field of load testing.
Instead of doing their direct duties and using their unique experience to build a load testing process, choose a methodology, optimal metrics and write autotests in accordance with load profiles, engineers often have to deploy test infrastructure from scratch, configure load tools, and embed them themselves in CI systems, set up monitoring and publication of reports.
You can find solutions to some organizational problems in testing that we use at Positive Technologies in . And in this one, I will talk about the possibility of integrating load tests into a common CI pipeline using the concept of βload testing as a serviceβ (load testing as a service). You will learn how and which docker images of load sources can be used in the CI pipeline; how to connect load sources to your CI project using a build template; what the demo pipeline looks like for running load tests and publishing the results. The article may be useful for software testing engineers and automation engineers in CI who are thinking about the architecture of their load system.
The essence of the concept
The concept of load testing as a service implies the ability to integrate load tools Apache JMeter, Yandex.Tank and your own frameworks into an arbitrary continuous integration system. The demo will be for GitLab CI, but the principles are common to all CI systems.
Load testing as a service is a centralized service for load testing. Load tests are run in dedicated agent pools, the results are published automatically in GitLab Pages, Influx DB and Grafana or in test reporting systems (TestRail, ReportPortal, etc.). Automation and scaling are implemented as simply as possible - by adding and parameterizing the usual gitlab-ci.yml template in the GitLab CI project.
The advantage of this approach is that the entire CI infrastructure, load agents, docker images of load sources, testing pipelines, and publishing reports are maintained by a centralized automation department (DevOps engineers), while load testing engineers can focus their efforts on test development and analysis of their results, without dealing with infrastructure issues.
For simplicity of description, we will assume that the target application or server under test has already been deployed and configured in advance (automated scripts in Python, SaltStack, Ansible, etc. can be used for this). Then the whole concept of load testing as a service fits into three stages: preparation, testing, publication of reports. More details on the diagram (all pictures are clickable):
Basic concepts and definitions in load testing
When carrying out load tests, we try to adhere to , use the appropriate terminology and recommended metrics. I will give a short list of the main concepts and definitions in load testing.
Load agent - a virtual machine on which the application will be launched - the load source (Apache JMeter, Yandex.Tank or a self-written load module).
Test goal (target) - server or application installed on the server that will be subject to load.
Test scenario (test case) - a set of parameterized steps: user actions and expected reactions to these actions, with fixed network requests and responses, depending on the specified parameters.
Profile or load plan (profile) - In (Section 4.2.4, p. 43) load profiles define metrics that are critical for a particular test and options for changing load parameters during the test. You can see examples of profiles in the figure.
Test β a script with a predetermined set of parameters.
Test plan (test-plan) - a set of tests and a load profile.
Testran (testrun) - one iteration of running one test with a fully executed load scenario and the received report.
Network request (request) β An HTTP request sent from an agent to a target.
Network response (response) β An HTTP response sent from the target to the agent.
HTTP response code (HTTP responses status) - standard response code from the application server.
A transaction is a complete request-response cycle. A transaction is counted from the start of sending a request (request) to the completion of receiving a response (response).
Transaction status - whether it was possible to successfully complete the request-response cycle. If there was any error in this cycle, then the entire transaction is considered unsuccessful.
Response time (latency) - the time from the end of sending a request (request) to the beginning of receiving a response (response).
Load metrics β the characteristics of the loaded service and the load agent determined in the process of load testing.
Basic metrics for measuring load parameters
Some of the most commonly used and recommended in the methodology (p. 36, 52) the metrics are shown in the table below. Similar metrics for agent and target are listed on the same line.
Metrics for the load agent
Metrics of the target system or application being tested under load
Quantity vCPU and memory RAM,
Disk - "iron" characteristics of the load agent
CPU, Memory, Disk usage - dynamics of CPU, memory and disk loading
in the process of testing. Usually measured as a percentage of
maximum available values
network throughput (on load agent) - throughput
network interface on the server,
where the load agent is installed.
Usually measured in bytes per second (bps)
network throughput(on target) - network interface bandwidth
on the target server. Usually measured in bytes per second (bps)
Virtual users- the number of virtual users,
implementing load scenarios and
imitating real user actions
Virtual users status, Passed/Failed/Total β number of successful and
unsuccessful statuses of virtual users
for load scenarios, as well as their total number.
It is generally expected that all users were able to complete
all your tasks specified in the load profile.
Any error will mean that a real user will not be able to
solve your problem when working with the system
Requests per second (minute)- the number of network requests per second (or minute).
An important characteristic of a load agent is how many requests it can generate.
In fact, this is an imitation of access to the application by virtual users
Responses per second (minute)
- the number of network responses per second (or minute).
An important characteristic of the target service: how much
generate and send responses to queries with
loading agent
HTTP response statusβ number of different response codes
from the application server received by the load agent.
For example, 200 OK means a successful call,
and 404 - that the resource was not found
Latency (response time) - time from the end
sending a request (request) before starting to receive a response (response).
Usually measured in milliseconds (ms)
Transaction response timeβ time of one full transaction,
completion of the request-response cycle.
This is the time from the start of sending the request (request)
until completion of receiving a response (response).
Transaction time can be measured in seconds (or minutes)
in several ways: consider the minimum,
maximum, average and, for example, the 90th percentile.
The minimum and maximum readings are extreme
system performance status.
The ninetieth percentile is the most commonly used,
as it shows most of the users,
comfortably operating at the threshold of system performance
Transactions per second (minute) - the number of complete
transactions per second (minute),
that is, how much the application was able to accept and
process requests and issue responses.
In fact, this is the throughput of the system
Transaction status , Passed / Failed / Total - number
successful, unsuccessful and the total number of transactions.
For real users unsuccessful
the transaction will actually mean
inability to work with the system under load
Load Testing Schematic Diagram
The concept of load testing is very simple and consists of three main stages, which I have already mentioned: Prepare-Test-Report, that is, preparing test goals and setting parameters for load sources, then executing load tests and, at the end, generating and publishing a test report.
Schematic notes:
- QA.Tester is an expert in load testing,
- Target is the target application for which you want to know its behavior under load.
Classifier of entities, stages and steps in the diagram
Stages and steps
What's going on
What's at the entrance
What is the output
Prepare: preparation stage for testing
LoadParameters
Setting and initialization
user
load parameters,
choice of metrics and
test plan preparation
(load profile)
Custom options for
load agent initialization
Test plan
Purpose of testing
VM
Cloud deployment
virtual machine with
required characteristics
VM settings for load agent
Automation scripts for
VM creation
VM configured in
the cloud
Approx
OS setup and preparation
environment for
load agent work
Environment settings for
load agent
Automation scripts for
environment settings
Prepared environment:
OS, services and applications,
necessary for work
load agent
LoadAgents
Installation, configuration and parameterization
loading agent.
Or downloading a docker image from
preconfigured load source
Load source docker image
(YAT, JM or self-written framework)
Settings
load agent
Set up and ready
to work load agent
Test: stage of execution of load tests. Sources are load agents deployed in dedicated agent pools for GitLab CI
Load
Starting the Load Agent
with selected test plan
and load parameters
User Options
for initialization
load agent
Test plan
Purpose of testing
Execution logs
load tests
System logs
Dynamics of changes in goal metrics and load agent
Run Agents
Agent Execution
loads of test scripts
in accordance with
load profile
Load Agent Interaction
for the purpose of testing
Test plan
Purpose of testing
Logs
Collection of "raw" logs
during load testing:
load agent activity records,
state of the test target
and the VM running the agent
Execution logs
load tests
System logs
Metrics
Collecting "raw" metrics during testing
Dynamics of changes in goal metrics
and load agent
Report: test report preparation stage
generator
Processing collected
loading system and
monitoring system "raw"
metrics and logs
Formation of a report in
human readable form
possible with elements
Analysts
Execution logs
load tests
System logs
Dynamics of changes in metrics
target and load agent
Processed "raw" logs
in a format suitable for
uploads to external storage
Static load report,
human-readable
Publish
Publication of the report
about load
testing in external
service
Processed "raw"
logs in a suitable format
for unloading to external
repositories
Saved in external
storage reports on
load, suitable
for human analysis
Connecting load sources in CI template
Let's move on to the practical part. I want to show how on some projects in the company we have implemented the concept of load testing as a service.
First, with the help of our DevOps engineers, we created a dedicated pool of agents in GitLab CI to run load tests. In order not to confuse them in templates with others, such as assembly pools, we added tags to these agents, : load. You can use any other understandable tags. They ask GitLab CI Runners.
How to find out the required power by hardware? The characteristics of load agents - a sufficient number of vCPU, RAM and Disk - can be calculated based on the fact that Docker, Python (for Yandex.Tank), GitLab CI agent, Java (for Apache JMeter) should be running on the agent. For Java under JMeter, it is also recommended to use a minimum of 512 MB of RAM and, as an upper limit, .
Thus, based on our experience, we recommend using at least 4 vCPUs, 4 GB RAM, 60 GB SSD for load agents. The throughput of the network card is determined based on the requirements of the load profile.
We mainly use two load sources - Apache JMeter and Yandex.Tank docker images.
is an open source tool from Yandex for load testing. Its modular architecture is based on Phantom's high-performance asynchronous hit-based HTTP request generator. The tank has a built-in monitoring of the resources of the server under test via the SSH protocol, can automatically stop the test under specified conditions, can display the results both in the console and in the form of graphs, you can connect your modules to it to expand functionality. By the way, we used the Tank when it was not yet mainstream. In the article "Β» you can read the story of how we performed load testing with it in 2013 is one of our company's products.
is an open source load testing tool from Apache. It can be used equally well for testing both static and dynamic web applications. JMeter supports a huge number of protocols and ways to interact with applications: HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, etc.), SOAP / REST Webservices, FTP, TCP, LDAP, SMTP(S), POP3(S) ) and IMAP(S), databases via JDBC, can execute shell commands and work with Java objects. JMeter has an IDE for creating, debugging and executing test plans. There is also a CLI for command line operation on any Java compatible operating system (Linux, Windows, Mac OS X). The tool can dynamically generate an HTML test report.
For ease of use within our company, for the ability of the testers themselves to change and add the environment, we made builds of docker images of load sources on GitLab CI with publication to the internal . This makes it faster and easier to connect them in pipelines for load tests. How to do docker push to registry via GitLab CI - see .
We took this basic docker file for Yandex.Tank:
Dockerfile
1 | FROM direvius/yandex-tank
2 | ENTRYPOINT [""]And for Apache JMeter this one:
Dockerfile
1 | FROM vmarrazzo/jmeter
2 | ENTRYPOINT [""]You can read how our continuous integration system works in the article "Β».
Template and pipeline
An example of a template for conducting load tests is available in the project . In You can read the instructions for using the template. In the template itself (file ) there are notes about what each step is responsible for.
The template is very simple and demonstrates the three stages of load testing described in the diagram above: preparing, testing, and publishing reports. Responsible for this : Prepare, Test and Report.
- Stage should be used to preconfigure test targets or check their availability. The environment for load sources does not need to be configured, they are pre-built as docker images and posted in the docker registry: just specify the desired version at the Test stage. But you can rebuild them and make your own modified images.
- Stage used to specify the load source, run tests, and store test artifacts. You can choose any load source: Yandex.Tank, Apache JMeter, your own, or all together. To disable unnecessary sources, just comment out or delete the job. Entry points for load sources:
- Launch parameters for Yandex.Tank are specified in the .,
- Apache JMeter startup parameters are specified in the file .
Note: The assembly configuration template is used to set up interaction with the CI system and does not imply placement of test logic in it. For tests, the entry point is specified, where the control bash script is located. The method of running tests, generating reports, and the test scripts themselves must be implemented by QA engineers. In the demo, for both load sources, the Yandex main page request is used as the simplest test. Scripts and test parameters are in the directory .
- At the stage you need to describe how to publish the test results obtained at the Test stage to external storages, for example, to GitLab Pages or special reporting systems. GitLab Pages requires that the ./public directory be non-empty and contain at least an index.html file after the tests complete. You can read about the nuances of the GitLab Pages service. .
Examples of how to export data:
- from JMeter to ,
- from Yandex.Tank to .
Posting setup instructions:
- HTML statics in ,
- to InfluxDB and then to .
In the demo example, the pipeline with load tests and two load sources (you can disable the unnecessary one) looks like this:
Apache JMeter can generate an HTML report itself, so it is more profitable to save it in GitLab Pages using standard tools. This is how the Apache JMeter report looks like:
In the demo example for Yandex.Tank, you will only see in the section for GitLab Pages. During testing, the Tank can save the results to the InfluxDB database, and from there they can be displayed, for example, in Grafana (configuration is done in the file ). This is how Tank's report looks in Grafana:
Summary
In the article, I talked about the concept of "load testing as a service" (load testing as a service). The main idea is to use the infrastructure of pre-configured pools of load agents, docker images of load sources, reporting systems and a pipeline that combines them in GitLab CI based on a simple .gitlab-ci.yml template (example ). All this is supported by a small team of automation engineers and replicated at the request of product teams. I hope this will help you in preparing and implementing a similar scheme in your company. Thank you for attention!
PS I want to say a big thank you to my colleagues, Sergey Kurbanov and Nikolai Yusev, for technical assistance with the implementation of the concept of load testing as a service in our company.
Author: - Deputy Head of Technology and Development Processes (DevOps) at Positive Technologies
Source: habr.com