I propose to read the transcript of the report by Alexander Sigachev from Inventos "Development and testing process with Docker + Gitlab CI"
Those who are just starting to implement the development and testing process based on Docker + Gitlab CI often ask basic questions. Where to begin? How to organize? How to test?
This report is good because it talks in a structured way about the development and testing process using Docker and Gitlab CI. The report itself is from 2017. I think that from this report you can learn the basics, methodology, idea, experience of use.
Who cares, please under the cat.
My name is Alexander Sigachev. I work for Inventos. I will tell you about my experience of using Docker and how we are gradually implementing it on projects in the company.
Presentation topic: Development process using Docker and Gitlab CI.
This is my second talk about Docker. At the time of the first report, we only used Docker in Development on developer machines. The number of employees who used Docker was about 2-3 people. Gradually, experience was gained and we moved a little further. Link to our .
What will be in this report? We will share our experience about what rake we have collected, what problems we have solved. Not everywhere it was beautiful, but allowed to move on.
Our motto is: dock everything we can get our hands on.
What problems are we solving?
When there are several teams in a company, the programmer is a shared resource. There are stages when a programmer is pulled out of one project and given for some time to another project.
In order for the programmer to quickly understand, he needs to download the source code of the project and launch the environment as soon as possible, which will allow him to move further solving the problems of this project.
Usually, if you start from scratch, then there is little documentation in the project. Information on how to set up is available only to old-timers. Employees set up their workplace on their own in one or two days. To speed this up, we used Docker.
The next reason is the standardization of settings in Development. In my experience, developers always take the initiative. In every fifth case, a custom domain is entered, for example, vasya.dev. Sitting next to him is his neighbor Petya, whose domain is petya.dev. They develop a website or some component of the system using this domain name.
When the system grows and these domain names start to get into configurations, then a Development environment conflict arises and the site path is rewritten.
The same happens with database settings. Someone doesn't bother with security and works with an empty root password. At the installation stage, MySQL asked someone for a password and the password turned out to be 123. It often happens that the database config is constantly changing depending on the developer's commit. Someone corrected, someone did not correct the config. There were tricks when we took out some kind of test config in .gitignore and each developer had to install the database. This made it difficult to get started. It is necessary, among other things, to remember about the database. The database must be initialized, a password must be entered, a user must be registered, a table must be created, and so on.
Another problem is different versions of libraries. It often happens that a developer works with different projects. There is a Legacy project that started five years ago (from 2017 - ed. note). At the time of launch, we started with MySQL 5.5. There are also modern projects where we try to implement more modern versions of MySQL, for example, 5.7 or older (in 2017 - ed. note)
Anyone who works with MySQL knows that these libraries bring dependencies with them. It is rather problematic to run 2 bases together. At least, old clients are problematic to connect to the new database. This in turn creates several problems.
The next problem is when a developer works on a local machine, he uses local resources, local files, local RAM. All interaction at the time of developing a solution to problems is carried out within the framework of the fact that it works on one machine. An example is when we have backend servers in Production 3, and the developer saves files to the root directory and from there nginx takes files to respond to the request. When such a code gets into Production, it turns out that the file is present on one of the 3 servers.
The direction of microservices is developing now. When we divide our large applications into some small components that interact with each other. This allows you to select technologies for a specific stack of tasks. It also allows you to share work and responsibilities between developers.
Frondend-developer, developing on JS, has almost no influence on Backend. The backend developer, in turn, develops, in our case, Ruby on Rails and does not interfere with Frondend. The interaction is performed using the API.
As a bonus, with the help of Docker, we were able to recycle resources on Staging. Each project, due to its specifics, required certain settings. Physically, it was necessary to allocate either a virtual server and configure them separately, or to share some kind of variable environment and projects could, depending on the version of the libraries, influence each other.
Tools. What do we use?
- Docker itself. The Dockerfile describes the dependencies of a single application.
- Docker-compose is a bundle that brings together a few of our Docker applications.
- We use GitLab to store the source code.
- We use GitLab-CI for system integration.
The report consists of two parts.
The first part will talk about how Docker was run on developers' machines.
The second part will talk about how to interact with GitLab, how we run tests and how we roll out to Staging.
Docker is a technology that allows (using a declarative approach) to describe the necessary components. This is an example Dockerfile. Here we declare that we are inheriting from the official Ruby:2.3.0 Docker image. It contains Ruby version 2.3 installed. We install the required build libraries and NodeJS. We describe that we create a directory /app. Set the app directory as the working directory. In this directory we place the required minimal Gemfile and Gemfile.lock. We then build the projects that install this dependency image. We indicate that the container will be ready to listen on external port 3000. The last command is the command that directly launches our application. If we execute the project start command, the application will try to run and run the specified command.
This is a minimal example of a docker-compose file. In this case, we show that there is a connection between two containers. This is directly into the database service and the web service. Our web applications in most cases require some kind of database as a backend for storing data. Since we are using MySQL, the example is with MySQL - but nothing prevents us from using some other database (PostgreSQL, Redis).
We take from the official source from the Docker hub the image of MySQL 5.7.14 without changes. We collect the image that is responsible for our web application from the current directory. It collects an image for us during the first launch. Then it runs the command that we are executing here. If we go back, we will see that the launch command via Puma has been defined. Puma is a service written in Ruby. In the second case, we override. This command can be arbitrary depending on our needs or tasks.
We also describe that we need to forward a port on our developer host machine from 3000 to 3000 on the container port. This is done automatically using iptables and its mechanism, which is directly embedded in Docker.
The developer can also, as before, access any available IP address, for example, 127.0.0.1 is the local or external IP address of the machine.
The last line says that the web container depends on the db container. When we call the start of the web container, docker-compose will first start the database for us. Already at the start of the database (in fact, after the launch of the container! This does not guarantee the readiness of the database) will launch the application, our backend.
This avoids errors when the database is not up and saves resources when we stop the database container, thus freeing resources for other projects.
What gives us the use of database dockerization on the project. We fix the version of MySQL for all developers. This avoids some errors that may occur when versions diverge, when the syntax, configuration, default settings change. This allows you to specify a common hostname for the database, login, password. We are moving away from the zoo of names and conflicts in the config files that we had earlier.
We have the opportunity to use a more optimal config for the Development environment, which will differ from the default. MySQL is configured for weak machines by default and its performance out of the box is very poor.
Docker allows you to use the Python, Ruby, NodeJS, PHP interpreter of the desired version. We get rid of the need to use some kind of version manager. Previously, Ruby used an rpm package that allowed you to change the version depending on the project. It also allows, thanks to the Docker container, to smoothly migrate the code and version it together with dependencies. We have no problem understanding the version of both the interpreter and the code. To update the version, lower the old container and raise the new container. If something went wrong, we can lower the new container, raise the old container.
After building the image, the containers in both Development and Production will be the same. This is especially true for large installations.
On Frontend we use JavaScipt and NodeJS.
Now we have the last project on ReacJS. The developer ran everything in the container and developed using hot-reload.
Next, the JavaScipt assembly task is launched and the code compiled into statics is given through nginx saving resources.
Here I have given the scheme of our last project.
What tasks were solved? We had a need to build a system with which mobile devices interact. They receive data. One possibility is to send push notifications to this device.
What have we done for this?
We divided into the application such components as: the admin part on JS, the backend, which works through the REST interface under Ruby on Rails. The backend interacts with the database. The result that is generated is given to the client. The admin panel interacts with the backend and the database via the REST interface.
We also had a need to send push notifications. Before that, we had a project that implemented a mechanism that is responsible for delivering notifications to mobile platforms.
We have developed the following scheme: an operator from the browser interacts with the admin panel, the admin panel interacts with the backend, the task is to send Push notifications.
Push notifications interact with another component that is implemented in NodeJS.
Queues are built and then notifications are sent according to their mechanism.
Two databases are drawn here. At the moment, with the help of Docker, we use 2 independent databases that are not related to each other in any way. In addition, they have a common virtual network, and physical data is stored in different directories on the developer's machine.
The same but in numbers. This is where code reuse is important.
If earlier we talked about reusing code in the form of libraries, then in this example, our service that responds to Push notifications is reused as a complete server. It provides an API. And our new development already interacts with it.
At that time, we were using version 4 of NodeJS. Now (in 2017 - ed. note) in recent developments we use version 7 of NodeJS. There is no problem in new components to involve new versions of libraries.
If necessary, you can refactor and raise the NodeJS version from the Push notification service.
And if we can maintain API compatibility, then it will be possible to replace it with other projects that were previously used.
What do you need to add Docker? We add a Dockerfile to our repository, which describes the necessary dependencies. In this example, the components are broken down logically. This is the minimum set of a backend developer.
When creating a new project, we create a Dockerfile, describe the desired ecosystem (Python, Ruby, NodeJS). In docker-compose, it describes the necessary dependency - the database. We describe that we need a database of such and such a version, store data there and there.
We use a separate third container with nginx to serve static. It is possible to upload pictures. Backend puts them in a pre-prepared volume, which is also mounted in a container with nginx, which gives the static.
To store the nginx, mysql configuration, we added a Docker folder in which we store the necessary configs. When a developer does a git clone of a repository on his machine, he already has a project ready for local development. There is no question what port or what settings to apply.
Next, we have several components: admin, inform-API, push notifications.
In order to start all this, we created another repository, which we called dockerized-app. At the moment we use several repositories before each component. They are just logically different - in GitLab it looks like a folder, but on the developer's machine, a folder for a specific project. One level down are the components that will be combined.
This is an example of just the contents of dockerized-app. We also bring the Docker directory here, in which we fill in the configurations required for the interactions of all components. There is a README.md that briefly describes how to run the project.
Here we have applied two docker-compose files. This is done in order to be able to run in steps. When a developer works with the core, he does not need push notifications, he simply launches a docker-compose file and, accordingly, the resource is saved.
If there is a need to integrate with push notifications, then docker-compose.yaml and docker-compose-push.yaml are launched.
Since docker-compose.yaml and docker-compose-push.yaml are in a folder, a single virtual network is automatically created.
Description of components. This is a more advanced file that is responsible for the collection of components. What is remarkable here? Here we introduce the balancer component.
This is a ready-made Docker image that runs nginx and an application that listens on the Docker socket. Dynamic, as containers are turned on and off, it regenerates the nginx config. We distribute the handling of components by third-level domain names.
For the Development environment, we use the .dev domain - api.informer.dev. Applications with a .dev domain are available on the developer's local machine.
Further, configs are transferred to each project and all projects are launched together at the same time.
Graphically, it turns out that the client is our browser or some tool with which we make requests to the balancer.
The domain name balancer determines which container to contact.
It can be nginx, which gives the admin JS. This can be nginx, which gives the API, or static files, which are given to nginx in the form of image uploads.
The diagram shows that the containers are connected by a virtual network and hidden behind a proxy.
On the developer's machine, you can access the container knowing the IP, but in principle we do not use this. There is practically no need for direct access.
Which example to look at to dockerize your application? In my opinion, a good example is the official docker image for MySQL.
It's quite challenging. There are many versions. But its functionality allows you to cover many needs that may arise in the process of further development. If you spend time and figure out how it all interacts, then I think you will have no problems in self-implementation.
Hub.docker.com usually contains links to github.com, which contains raw data directly from which you can build the image yourself.
Further in this repository there is a docker-endpoint.sh script, which is responsible for the initial initialization and for further processing of the application launch.
Also in this example, there is the ability to configure using environment variables. By defining an environment variable when running a single container or through docker-compose, we can say that we need to set an empty password for docker to root on MySQL or whatever we want.
There is an option to create a random password. We say that we need a user, we need to set a password for the user, and we need to create a database.
In our projects, we slightly unified the Dockerfile, which is responsible for initialization. There we corrected it to our needs to make it just an extension of the user rights that the application uses. This allowed us to simply create a database from the application console later on. Ruby applications have a command for creating, modifying, and deleting databases.
This is an example of what a specific version of MySQL looks like on github.com. You can open the Dockerfile and see how the installation is going on there.
docker-endpoint.sh is the script responsible for the entry point. During the initial initialization, some preparation steps are required, and all these actions are taken out just in the initialization script.
We pass to the second part.
To store the source codes, we switched to gitlab. This is a fairly powerful system that has a visual interface.
One of the components of Gitlab is Gitlab CI. It allows you to describe a sequence of commands that will later be used to organize a code delivery system or run automatic testing.
Gitlab CI 2 talk - report from Ruby Russia club - quite detailed and perhaps it will interest you.
Now we will look at what is required in order to activate Gitlab CI. In order to start Gitlab CI, we just need to put the .gitlab-ci.yml file in the root of the project.
Here we describe that we want to execute a sequence of states like a test, deploy.
We execute scripts that directly call docker-compose to build our application. This is just a backend example.
Next, we say that it is necessary to run migrations to change the database and run tests.
If the scripts are executed correctly and does not return an error code, then the system proceeds to the second stage of the deployment.
The deployment stage is currently implemented for staging. We did not organize a zero-downtime restart.
We forcibly extinguish all containers, and then we raise all the containers again, collected at the first stage during testing.
We are running for the current environment variable the database migrations that were written by the developers.
There is a note that this applies only to the master branch.
When changing other branches is not executed.
It is possible to organize rollouts by branches.
To organize this further, we need to install Gitlab Runner.
This utility is written in Golang. It is a single file, as is common in the Golang world, which does not require any dependencies.
On startup, we register the Gitlab Runner.
We get the key in the Gitlab web interface.
Then we call the initialization command on the command line.
Set up Gitlab Runner interactively (Shell, Docker, VirtualBox, SSH)
The code on Gitlab Runner will execute on every commit, depending on the .gitlab-ci.yml setting.
How it looks visually in Gitlab in the web interface. After we have connected GItlab CI, we have a flag that shows the state of the build at the moment.
We see that a commit was made 4 minutes ago, which passed all the tests and did not cause any problems.
We can take a closer look at the builds. Here we see that two states have already passed. Testing status and deployment status on staging.
If we click on a specific build, then there will be a console output of the commands that were run in the process according to .gitlab-ci.yml.
This is what our product history looks like. We see that there were successful attempts. When tests are submitted, it does not proceed to the next step and the staging code is not updated.
What tasks did we solve on staging when we implemented docker? Our system consists of components and we had a need to restart, only part of the components that were updated in the repository, and not the entire system.
To do this, we had to smash everything into separate folders.
After we did this, we had a problem with the fact that Docker-compose creates its own network space for each daddy and does not see the neighbor's components.
In order to get around, we created the network in Docker manually. It was written in Docker-compose that you use such a network for this project.
Thus, every component that starts with this mesh sees components in other parts of the system.
The next issue is splitting staging across multiple projects.
Since in order for all this to look beautiful and as close as possible to production, it is good to use port 80 or 443, which is used everywhere in the WEB.
How did we solve it? We have assigned one Gitlab Runner to all major projects.
Gitlab allows you to run several distributed Gitlab Runners, which will simply take all the tasks in turn in a chaotic manner and run them.
So that we donβt have a house, we limited the group of our projects to one Gitlab Runner, which copes without problems with our volumes.
We moved nginx-proxy into a separate startup script and added grids for all projects in it.
Our project has one grid, and the balancer has several grids by project names. It can proxy further by domain names.
Our requests come through the domain on port 80 and are resolved into a container group that serves this domain.
What other problems were there? This is what all containers run as root by default. This is root unequal to the root host of the system.
However, if you enter the container, it will be root and the file that we create in this container gets root rights.
If the developer entered the container and did some commands there that generate files, then left the container, then he has a file in his working directory that he does not have access to.
How can it be solved? You can add users who will be in the container.
What problems arose when we added the user?
When creating a user, we often do not have the same group ID (UID) and user ID (GID).
To solve this problem in the container, we use users with ID 1000.
In our case, this coincided with the fact that almost all developers use the Ubuntu OS. And on Ubuntu, the first user has an ID of 1000.
Do we have plans?
Read the Docker documentation. The project is actively developing, the documentation is changing. The data that was received two or three months ago is already slowly becoming outdated.
Some of the problems that we solved are quite possibly already solved by standard means.
So I want to go further already to go directly to the orchestration.
One example is Docker's built-in mechanism called Docker Swarm, which comes out of the box. I want to run something in production based on Docker Swarm technology.
Spawning containers makes it inconvenient to work with logs. Now the logs are isolated. They are scattered across containers. One of the tasks is to make convenient access to the logs through the web interface.
Source: habr.com