We developed DevOps as best we could. There were 8 of us, and Vasya was the coolest on Windows. Suddenly, Vasya left, and I had the task of launching a new project that is supplied by Windows development. When I poured the entire Windows development stack onto the table, I realized that the situation is a pain ...
This is how the story begins Alexandra Sinchinova on . When the leading Windows specialist left the company, Alexander wondered what to do now. Switch to Linux, of course! Alexander will tell how he managed to create a precedent and transfer part of Windows development to Linux using the example of a implemented project for 100 end users.
How to easily deliver a project to RPM using TFS, Puppet, Linux .NET core? How to maintain project database versioning if the development hears the words Postgres and Flyway for the first time, and the deadline is the day after tomorrow? How to integrate with Docker? How to motivate .NET developers to move away from Windows and smoothies in favor of Puppet and Linux? How to resolve ideological conflicts if there is no strength, no desire, no resources to serve Windows in production? About this, as well as about Web Deploy, testing, CI, about the practices of using TFS in existing projects, and, of course, about broken crutches and working solutions, in the transcript of Alexander's report.
So, Vasya left, the task is on me, the developers are waiting impatiently with pitchforks. When I finally realized that Vasya could not be returned, I got down to business. To begin with, I estimated the percentage of Win VM in our fleet. The score was not in favor of Windows.
Since we are actively developing DevOps, I realized that something needs to be changed in the approach of taking out a new application. There was only one solution - if possible, transfer everything to Linux. Google helped me - at that time .Net had already been ported to Linux, and I realized that this was the solution!
Why .NET core in conjunction with Linux?
There were several reasons for this. Between βpaying moneyβ and βnot payingβ, the majority will choose the latter - like me. An MSDB license costs about $1, and maintenance of a fleet of Windows virtual machines runs into the hundreds of dollars. For a large company, this is a big expense. That's why saving β first reason. Not the most important, but one of the most significant.
Windows virtual machines take up more resources than their Linux brethren - they are heavy. Given the size of a large company, we chose Linux.
The system is simply integrated into an existing CI. We consider ourselves progressive DevOps, we use Bamboo, Jenkins and GitLab CI, so most of us are on Linux.
The last reason is convenient accompaniment. We needed to lower the entry threshold for "maintainers" - guys who understand the technical part, ensure continuity and serve services from the second line. They were already familiar with the Linux stack, so it is much easier for them to understand the new product, support and maintain it, rather than spend additional resources to deal with similar software functionality for the Windows platform.
Requirements
First and foremost - convenience of the new solution for developers. Not all of them were ready for change, especially after the spoken word Linux. Developers want their favorite Visual Studio, TFS with build autotests and smoothies. How the delivery to production takes place is not important to them. Therefore, we decided not to change the usual process and leave everything unchanged for Windows development.
Need a new project embed into existing CI. The rails were already there and all the work had to be done taking into account the parameters of the configuration management system, accepted delivery standards and monitoring systems.
Ease of maintenance and operation, as a condition for the minimum entry threshold for all new participants from different departments and the support department.
Deadline - yesterday.
Win development group
What was the Windows team doing then?
Now I can confidently say that IdentityServer4 is a cool free alternative to ADFS with similar features, or whatever Entity Framework Core - a paradise for a developer, where you can not bother writing SQL scripts, but describe queries in the database in terms of OOP. But then, when discussing the action plan, I looked at this stack as Sumerian cuneiform, recognizing only PostgreSQL and Git.
At that time, we actively used Puppet as a configuration management system. In most of our projects we have used GitLab CI, Elastic, balancing high-loaded services using HAProxy kept track of everything Zabbix, bundles grafana ΠΈ Prometheus, Jaeger, and all this was spinning on pieces of iron HP c ESXi on VMware. Everyone knows - a classic of the genre.
Let's look and try to understand what happened before we started all these interventions.
What happened
TFS is a fairly powerful system that not only delivers code from the developer to the final production machine, but also has a set for very flexible integration with various services - to provide CI at the cross-platform level.
Previously, these were solid vents. TFS used several Build Agents that built many projects. Each agent has 3-4 workers to parallelize tasks and optimize the process. Further, according to the release plans, TFS delivered the freshly baked Build to the Windows application server.
Where did we want to go
We use TFS for delivery and development, and we run the application on the Linux Application server, and there is some magic between them. This Magic Box and there is the salt of the work ahead. Before I take it apart, I'll take a step aside and say a few words about the app.
Project
The application provides functionality for handling prepaid cards.
Client
There were two types of users. First accessed by logging in with an SSL SHA-2 certificate. At second was accessed by username and password.
HAProxy
Then the client request went to HAProxy, which solved the following tasks:
- primary authorization;
- termination of SSL;
- tuning HTTP requests;
- request translation.
Verification of the client certificate went along the chain. We - authority and we can afford it, as we ourselves issue certificates to customers of the service.
Pay attention to the third point, a little later we will return to it.
BACKEND
The backend was planned to be made on Linux. The backend interacts with the database, loads the required list of privileges, and then, depending on what privileges the authorized user has, provides access to sign financial documents and send them for execution, or generate some kind of report.
Savings with HAProxy
In addition to the two contexts that each of the clients walked, there was also an identity context. IdentityServer4 just allows you to log in, this is a free and powerful analogue for ADFS β Active Directory Federation Services.
The request in identity was processed in several steps. First step - client hit the backend, which communicated with this server and checked for the presence of a token for the client. If it didnβt find it, the request returned back to the context from which it came, but with a redirect, and with a redirect it went to identity.
The second step - the request got to the authorization page in IdentityServer, where the client registered, and the long-awaited token appeared in the IdentityServer database.
Third step - client redirected back to the context from which it came.
IdentityServer4 has a feature: it returns the response to the return request via HTTP. No matter how we struggled with setting up the server, no matter how enlightened the documentation was, each time we received the initial client request with a URL that came via HTTPS, and IdentityServer returned the same context, but with HTTP. We were shocked! And we transferred all this through the identity context to HAProxy, and in the headers we had to modify the HTTP protocol to HTTPS.
What is the improvement and where did you save?
We saved money by using a free solution for authorizing a group of users, resources, because we did not take out IdentityServer4 as a separate node in a separate segment, but used it together with the backend on the same server where the application backend is running.
How should it work
So, as I promised - Magic Box. We already understand that we are guaranteed to move towards Linux. Let's formulate the specific tasks that needed to be solved.
Puppet manifests. To deliver and manage the configuration of the service and application, you had to write cool recipes. The pencil roll eloquently shows how quickly and efficiently it was done.
Delivery method. The standard is RPM. Everyone understands that in Linux there is no way without it, but the project itself after assembly was a set of executable DLL files. There were about 150 of them, the project is quite heavy. The only harmonious solution is to pack this binary into RPM and deploy the application from it.
Versioning. We had to release very often, and we had to decide how to form the package name. It's a question of the level of integration with TFS. We had a build agent on Linux. When TFS sends a task to a worker - worker - on the Build agent, it also passes a bunch of variables to it, which fall into the environment of the handler process. These environment variables are passed the Build name, version name, and other variables. Read more about this in the "building an RPM package" section.
Setting up TFS came down to setting up Pipeline. Previously, we collected all Windows projects on Windows agents, and now a Linux agent appears - a Build agent that needs to be included in the build group, enriched with some artifacts, to say what type of projects will be built on this Build agent, and somehow modify Pipeline.
IdentityServer. ADFS is not our way, we drown for Open Source.
Let's go through the components.
Magic Box
Consists of four parts.
Linux Build Agent. Linux, because we build for it - it's logical. This part was done in three steps.
- Set up workers and not one, as the distributed work on the project was assumed.
- Install .NET Core 1.x. Why 1.x when 2.0 is already available in the standard repository? Because when we started development, the stable version was 1.09, and it was decided to make the project for it.
- Git 2.x.
RPM repository. RPM packages needed to be stored somewhere. We were supposed to use the same corporate RPM repository that is available to all Linux hosts. And so they did. The repository server is configured webhook which downloaded the required RPM package from the specified location. The package version of the webhook was reported by the Build agent.
gitlab. Attention! GitLab is used here not by developers, but by the operations department to control application versions, package versions, control the status of all Linux machines, and it stores the recipe - all Puppet manifests.
Puppet - resolves all controversial points and delivers exactly the configuration that we want from Gitlab.
We start to dive. How is DLL delivered to RPM?
Delivering DDL to RPM
Let's say we have a .NET development rockstar. It uses Visual Studio and creates a release branch. After that, it uploads it to Git, and Git here is a TFS entity, that is, it is the application repository that the developer works with.
Then TFS sees that a new commit has arrived. Which app? In the TFS settings there is a label with what resources this or that Build agent has. In this case, it sees that we are building a .NET Core project and selects a Linux Build agent from the pool.
Build-agent receives the sources, downloads the necessary dependencies from the .NET repository, npm, etc. and after building the application itself and subsequent packaging, submits the RPM package to the RPM repository.
On the other hand, the following happens. The engineer of the operations department is directly involved in the rollout of the project: he changes the versions of packages in yesterday in the repository where the application recipe is stored, after which Puppet triggers Yum, pulls a new package from the repository, and the new version of the application is ready to use.
In words, everything is simple, but what happens inside on the Build agent itself?
Packaging DLL RPMs
Received project sources and build task from TFS. Build agent starts building the project itself from sources. The assembled project is available as a set dll files, which are packed in a zip archive to reduce the load on the file system.
ZIP archive is thrown out to the build directory of the RPM package. Next, the Bash script initializes the environment variables, finds the Build version, the project version, the path to the build directory, and runs RPM-build. When the build is complete, the package is published to local repository, which is located on the Build agent.
Further, from the Build agent to the server in the RPM repository JSON request is sent with the name of the version and build. Webhook, which I mentioned earlier, downloads this same package from the local repository on the Build agent and makes the new build available for installation.
Why such a scheme for delivering a package to the RPM repository? Why can't you immediately push the built package to the repository? The fact is that this is a condition for ensuring security. This scenario limits the possibility of unauthorized people uploading RPM packages to a server that is accessible to all Linux machines.
Database versioning
At the development consultation, it turned out that the guys were closer to MS SQL, but in most non-Windows projects we already used PostgreSQL with might and main. Since we have already decided to abandon everything paid, we began to use PostgreSQL here as well.
In this part, I want to tell you how we versioned the database and how we chose between Flyway and Entity Framework Core. Consider their pros and cons.
Cons
Flyway only goes one way, we we can't roll back is a significant disadvantage. You can compare with Entity Framework Core in other ways - in terms of developer convenience. You remember that we put this at the forefront, and the main criterion was not to change anything for Windows development.
For flyway us some kind of wrapper neededso that the guys do not write SQL queries. They are much closer to operate in terms of OOP. We wrote instructions for working with database objects, formed an SQL query and executed. The new version of the database is ready, rolled - everything is fine, everything works.
Entity Framework Core has a minus - under heavy loads, it builds non-optimal SQL queries, and the drawdown on the database can be significant. But since we do not have a highly loaded service, we do not calculate the load in hundreds of RPS, we accepted these risks and delegated the problem to the future us.
pros
Entity Framework Core works out of the box and is easy to develop, and Flyway easily integrates into existing CI. But we do it conveniently for developers :)
Roll-on procedure
Puppet sees that a change in the version of the packages is coming, among which, the one that is responsible for the migration. First, it installs a package that contains migration scripts and database-related functionality. After that, the application that works with the database is restarted. Next comes the installation of the remaining components. The order in which packages are installed and applications are launched is described in the Puppet manifest.
Applications use sensitive data, such as tokens, database passwords, all this is pulled into the config from Puppet master, where they are stored in encrypted form.
TFS Issues
After we decided and realized that everything really works for us, I decided to see what was happening with the assemblies in TFS in general for the Win development department on other projects - whether we are going / releasing quickly or not, and found significant problems with speed .
One of the main projects is going to 12-15 minutes - it's a long time, you can't live like that. A quick analysis showed a terrible drawdown in I / O, and this is on arrays.
After analyzing component by component, I identified three foci. First - kaspersky antivirus, which scans sources on all Windows Build agents. Second - Windows indexer. It was not disabled, and everything was indexed on the Build agents in real time during the deployment process.
Third - npm install. It turned out that in most Pipelines we used this scenario. Why is he bad? The Npm install procedure is run when building the dependency tree in package-lock.json, where the versions of the packages that will be used to build the project are fixed. The downside is that Npm install pulls the latest versions of packages from the Internet each time, and this is a lot of time in the case of a large project.
Developers sometimes experiment on a local machine to test the performance of a particular part or an entire project. Sometimes it turned out that everything was cool locally, but they collected it, rolled it out - nothing worked. We begin to understand what the problem is - yeah, different versions of packages with dependencies.
Solution
- Sources in AV exceptions.
- Disable indexing.
- Go to npm ci.
The advantage of npm ci is that we collect the dependency tree once, and we get the opportunity to provide the developer up-to-date list of packages, with which he can experiment locally as much as he wants. This saves time developers who write the code.
Configuration
Now a little about the repository configuration. Historically we have used Nexus to manage repositories, including Internal REPO. This internal repository comes with all the components that we use for internal purposes, for example, self-written monitoring.
We also use NuGet, as it caches better than other package managers.
Experience the Power of Effective Results
After we optimized the build agents, the average build time was reduced from 12 minutes to 7.
If we count all the machines that we could use for Windows, but switched to Linux in this project, we saved about $ 10. And this is only on licenses, and if you include content, more.
Plans
For the next quarter, we planned to work on optimizing code delivery.
Switching to a prebuild Docker image. TFS is a cool thing with many plugins that allow you to integrate into Pipeline, including building on a trigger, for example, a Docker image. We want to make this trigger on the same package-lock.json. If the composition of the components that are used to build the project somehow changes, we build a new Docker image. It is later used to deploy the container with the built application. Now this is not the case, but we plan to switch to a microservice architecture in Kubernetes, which is actively developing in our company and has been servicing production solutions for a long time.
Summary
I urge everyone to throw out Windows, but it's not because I can't cook it. The reason is that most Opensource solutions are Linux stack. Are you good save on resources. In my opinion, the future belongs to Open Source solutions on Linux with a powerful community.
Speaker Profile Alexander Sinchinov .
is a conference on the integration of development, testing and operation processes for professionals from professionals. That is why the project that Alexander spoke about? implemented and working, and two successful releases were made on the day of the performance. On On May 27 and 28 there will be even more such cases from practitioners. You can still jump into the last car and or not in a hurry ticket. See you in Skolkovo!
Source: habr.com