In microservice architecture projects, CI/CD is moving from a nice opportunity to a must. Automated testing is an integral part of continuous integration, which, when done intelligently, can give the team many pleasant evenings with family and friends. Otherwise, the project runs the risk of never being completed.
It is possible to cover the entire microservice code with unit tests with mock objects, but this only partially solves the problem and leaves many questions and difficulties, especially when testing work with data. As always, the most acute ones are testing the consistency of data in a relational database, testing work with cloud services and incorrect assumptions when writing mock objects.
All this and a little more is solved by testing the whole microservice in a Docker container. The undoubted advantage for ensuring the validity of the tests is that the same Docker images that go into production are tested.
Automation of this approach presents a number of problems, the solution of which will be described below:
- conflicts of parallel tasks in one docker host;
- conflicts of identifiers in the database during test iterations;
- waiting for microservices to be ready;
- consolidation and output of logs to external systems;
- testing outgoing HTTP requests;
- web socket testing (using SignalR);
- testing OAuth authentication and authorization.
This article is based on at SECR 2019. So for those who are too lazy to read, .
In this article, I will tell you how to use a script to run the service under test, the database and Amazon AWS services in Docker, then the tests on Postman and, after they are completed, stop and delete the created containers. Tests are run whenever the code changes. In this way, we make sure that each version works correctly with the AWS database and services.
The same script is run by both the developers themselves on their Windows desktops and the Gitlab CI server under Linux.
In order for the introduction of new tests to be justified, it should not require the installation of additional tools either on the development machine or on the server where the tests are run on commit. Docker solves this problem.
The test must run on a local server for the following reasons:
- The network is not completely reliable. Out of a thousand requests, one may fail;
In this case, the automatic test will not pass, the work will stop, you will have to look for the reason in the logs; - Too frequent requests are not allowed by some third-party services.
In addition, it is undesirable to use the stand, because:
- A booth can be broken not only by bad code running on it, but also by data that the correct code cannot process;
- No matter how hard we try to roll back all the changes made by the test during the course of the test itself, something can go wrong (otherwise, why the test?).
About the project and organization of the process
Our company was developing a microservice web application running on Docker in the Amazon AWS cloud. The project has already used unit tests, but often there were errors that the unit tests did not detect. It was required to test the whole microservice along with the database and Amazon services.
The project uses a standard continuous integration process that includes testing the microservice with each commit. After assigning a task, the developer makes changes to the microservice, tests it manually and runs all available automated tests. If necessary, the developer changes the tests. If no problems are found, a commit is made to the branch of this task. After each commit, tests are automatically run on the server. Merging into a common branch and launching automatic tests on it occurs after a successful review. If the tests on the common branch pass, the service is automatically updated in the test environment on the Amazon Elastic Container Service (bench). The stand is necessary for all developers and testers, and it is undesirable to break it. Testers in this environment test a fix or a new feature by running manual tests.
Project architecture
The application consists of more than ten services. Some of them are written in .NET Core and some in NodeJs. Each service runs in a Docker container in Amazon Elastic Container Service. Everyone has their own Postgres database, and some also have Redis. There are no shared bases. If several services need the same data, then this data is transmitted to each of these services via SNS (Simple Notification Service) and SQS (Amazon Simple Queue Service) at the time of their change, and the services save them to their separate databases.
SQS and SNS
SQS allows you to queue messages over HTTPS and read messages from the queue.
If several services read the same queue, then each message comes to only one of them. This is useful when running multiple instances of the same service to distribute the load between them.
If you want each message to be delivered to multiple services, each recipient must have its own queue, and SNS is needed to duplicate messages to multiple queues.
In SNS, you create a topic and subscribe to it, for example, an SQS queue. You can send messages to topic. In this case, the message is sent to each queue subscribed to this topic. SNS does not have a method for reading messages. If during debugging or testing you need to know what is sent to SNS, then you can create an SQS queue, subscribe it to the desired topic and read the queue.
API Gateways
Most services are not directly accessible from the Internet. Access is through API Gateway, which checks access rights. This is also our service, and there are tests for it too.
Real time notifications
Application uses to show real-time notifications to the user. This is implemented in the notification service. It is accessible directly from the Internet and works with OAuth itself, because integrating Web socket support into the Gateway turned out to be impractical compared to integrating OAuth and the notification service.
Known approach to testing
Unit tests substitute mock objects for things like a database. If a microservice, for example, tries to create an entry in a table with a foreign key, and the entry to which this key refers does not exist, then the request cannot be executed. Unit tests cannot detect this.
Π it is proposed to use an in-memory base and implement mock objects.
An in-memory database is one of the DBMS that the Entity Framework supports. It is designed specifically for testing. Data in such a database is stored only until the end of the process using it. It does not need to create tables, and data integrity is not checked.
Mock objects model the class being replaced only as far as the test developer understands how it works.
How to get Postgres to automatically start and run the migration when you run the test is not covered in the article from Microsoft. My solution does this and, in addition, no code specifically for tests is added to the microwarehouse itself.
Let's move on to the solution
During the development process, it became clear that unit tests were not enough to find all the problems in a timely manner, so it was decided to approach this issue from a different angle.
Setting up a test environment
The first task is to deploy the test environment. Steps that are required to start the microservice:
- Set up the service under test for a local environment, the environment variables specify the details for connecting to the database and AWS;
- Start Postgres and migrate by starting Liquibase.
In relational DBMS, before writing data to the database, you need to create a data schema, in other words, tables. When updating an application, the tables must be converted to the form used by the new version, and, preferably, without data loss. This is called migration. Creating tables in an initially empty database is a special case of migration. Migration can be built into the application itself. Both .NET and NodeJS have migration frameworks. In our case, for security reasons, microservices are deprived of the right to change the data schema, and the migration is performed using Liquibase. - Launch Amazon LocalStack. This is an implementation of AWS services to run at home. There is a ready-made image for LocalStack in Docker Hub.
- Run a script to create the necessary entities in LocalStack. Shell scripts use the AWS CLI.
For project testing, use . It was before, but it was launched manually and tested the application already deployed on the stand. This tool allows you to make arbitrary HTTP(S) requests and check if the responses match your expectations. Requests are combined into a collection, and the entire collection can be run.
How the automated test works
During the test, everything works in Docker: the service being tested, and Postgres, and the migration tool, and Postman, or rather, its console version - Newman.
Docker solves a number of problems:
- Independence from host configuration;
- Installing dependencies: docker downloads images from Docker Hub;
- Returning the system to its original state: just remove the containers.
docker-compose unites containers into a virtual network isolated from the Internet, in which containers find each other by domain names.
The test is controlled by a shell script. We use git-bash to run the test on Windows. Thus, one script is enough for both Windows and Linux. Git and Docker are installed by all developers on the project. Installing Git on Windows installs git-bash, so everyone has that too.
The script performs the following steps:
- Building docker images
docker-compose build - Starting the DB and LocalStack
docker-compose up -d <ΠΊΠΎΠ½ΡΠ΅ΠΉΠ½Π΅Ρ> - Database migration and LocalStack preparation
docker-compose run <ΠΊΠΎΠ½ΡΠ΅ΠΉΠ½Π΅Ρ> - Running the service under test
docker-compose up -d <ΡΠ΅ΡΠ²ΠΈΡ> - Run test (Newman)
- Stop all containers
docker-compose down - Posting results to Slack
We have a chat where messages with a green checkmark or a red cross and a link to the log are received.
These steps involve the following Docker images:
- The service being tested is the same image as for production. The configuration for the test is through environment variables.
- For Postgres, Redis and LocalStack, pre-built images from Docker Hub are used. There are also ready-made images for Liquibase and Newman. We build ours on their skeleton, adding our files there.
- A prebuilt AWS CLI image is used to provision the LocalStack and an image containing the script is built from it.
Using , you don't have to build a Docker image just to add files to a container. However, volumes are not suitable for our environment because the Gitlab CI tasks themselves run in containers. Docker can be controlled from such a container, but volumes only mount folders from the host system, not from another container.
Problems to face
Waiting for readiness
When the container with the service is running, it does not mean that it is ready to accept connections. You have to wait for a connection to continue.
This problem is sometimes solved using a script. , which waits for a TCP connection to be established. However, LocalStack may throw a 502 Bad Gateway error. In addition, it consists of many services, and if one of them is ready, it does not say anything about the rest.
Solution: LocalStack provisioning scripts that wait for a 200 response from both SQS and SNS.
Parallel Task Conflicts
Multiple tests can run simultaneously on the same Docker host, so container and network names must be unique. Moreover, tests from different branches of the same service can also run simultaneously, so it is not enough to write your own names in each compose file.
Solution: The script sets a unique value for the COMPOSE_PROJECT_NAME variable.
Windows features
When using Docker on Windows, there are a number of things that I want to bring to your attention, because this experience is important in understanding the causes of errors.
- Shellscripts in a container must have Linux line endings.
The CR character for a shell is a syntax error. It's hard to tell from the error message what the problem is. When editing such scripts on Windows, you need a proper text editor. In addition, the version control system must be configured properly.
This is how git is configured:
git config core.autocrlf input- Git-bash emulates standard Linux folders and when calling an exe file (including docker.exe) replaces absolute Linux paths with Windows paths. However, this does not make sense for paths not on the local machine (or paths in a container). This behavior is not disabled.
Solution: append an extra slash to the beginning of the path: //bin instead of /bin. Linux understands such paths, for him several slashes are the same as one. But git-bash does not recognize such paths and does not try to convert them.
Log output
When running tests, I would like to see logs from both Newman and the service being tested. Since the events of these logs are interconnected, combining them in one console is much more convenient than two separate files. Newman run through docker-compose run, and so its output ends up in the console. It remains to make sure that the output of the service also gets there.
The original solution was to do docker-compose-up no flag -d, but, using the capabilities of the shell, send this process to the background:
docker-compose up <service> &This worked until I needed to send logs from docker to a third party service. docker-compose-up Stopped outputting logs to the console. However, the team worked docker attach.
Solution:
docker attach --no-stdin ${COMPOSE_PROJECT_NAME}_<ΡΠ΅ΡΠ²ΠΈΡ>_1 &Identifier conflict during test iterations
Tests are run in multiple iterations. The database is not cleared. Records in the database have unique IDs. If we write down specific IDs in requests, we will get a conflict at the second iteration.
To avoid it, either IDs must be unique, or all objects created by the test must be deleted. Some objects cannot be deleted, as required.
Solution: generate GUIDs with scripts in Postman.
var uuid = require('uuid');
var myid = uuid.v4();
pm.environment.set('myUUID', myid);Then in the query use the character {{myUUID}}, which will be replaced by the value of the variable.
Interaction through LocalStack
If the service under test reads or writes to the SQS queue, then the test itself must also work with this queue to check this.
Solution: requests from Postman to LocalStack.
The AWS Services API is documented, allowing you to make requests without an SDK.
If the service writes to the queue, then we read it and check the contents of the message.
If the service sends messages to the SNS, at the preparation stage, the LocalStack also creates a queue and subscribes to this SNS topic. Then it all comes down to the above.
If the service needs to read a message from the queue, then in the previous test step we write this message to the queue.
Testing HTTP requests coming from the microservice under test
Some services work over HTTP with something other than AWS, and some AWS features are not implemented in LocalStack.
Solution: in these cases can help , which has a ready image in . The expected requests and their responses are configured by the HTTP request. The API is documented, so we make requests from Postman.
Testing OAuth Authentication and Authorization
We use OAuth and . The test requires an OAuth provider that we can run locally.
All interaction of the service with the OAuth provider comes down to two requests: first, the configuration is requested /.well-known/openid-configuration, and then the public key (JWKS) is requested at the address from the configuration. All of this is static content.
Solution: Our test OAuth provider is a static content server and two files on it. The token is generated once and committed to Git.
SignalR Test Features
Postman does not work with web sockets. A special tool was created to test SignalR.
A SignalR client can be more than just a browser. There is a client library for it under .NET Core. A client written in .NET Core establishes a connection, authenticates, and waits for a certain sequence of messages. If an unexpected message is received or the connection is lost, the client exits with code 1. If the last expected message is received, exits with code 0.
Newman is working with the client at the same time. Several clients are launched to check that messages are delivered to everyone who needs them.
To run multiple clients use the option --scale on the docker-compose command line.
Before starting Postman, the script waits for all clients to establish a connection.
We have already encountered the problem of waiting for a connection. But there were servers, and here is the client. We need a different approach.
Solution: the client in the container uses the mechanism to inform the script on the host of its status. The client creates a file at a specific path, say /healthcheck, as soon as the connection is established. The HealthCheck script in the dockerfile looks like this:
HEALTHCHECK --interval=3s CMD if [ ! -e /healthcheck ]; then false; fiTeam docker inspection shows the container's normal status, health status, and exit code.
After Newman completes, the script checks that all containers with the client have terminated, moreover, with a code of 0.
Happinnes exists
After we overcame the difficulties described above, we had a set of stable working tests. In the tests, each service works as a single entity, interacting with the database and with Amazon LocalStack.
These tests protect a team of 30+ developers from errors in an application with complex interactions of 10+ microservices with frequent deployments.
Source: habr.com