We developed DevOps as best we could. There were 8 of us, and Vasya was the coolest one. WindowsSuddenly Vasya left, and I had the task of launching a new project that supplies Windows-development. When I dumped the entire stack on the table Windows-developments, then I realized that the situation is painful...
This is how the story begins Alexandra Sinchinova + When the leading specialist left the company Windows, Alexander wondered what to do now. Go to LinuxOf course! Alexander will tell you how he managed to set a precedent and translate part Windows developments on Linux using the example of a completed project for 100,000 end users.
How to easily and effortlessly deliver a project to RPM using TFS, Puppet, Linux .NET Core? How do you maintain project database versioning if the developer is hearing the words Postgres and Flyway for the first time, and the deadline is the day after tomorrow? How do you integrate with Docker? How do you motivate .NET developers to abandon Windows and smoothies for Puppet and LinuxHow to resolve ideological conflicts if you serve Windows Don't have the energy, desire, or resources to get into production? Alexander's presentation discusses this, as well as Web Deployment, testing, CI, TFS practices in existing projects, and, of course, broken workarounds and working solutions.
So, Vasya left, the task was mine, and the developers were waiting impatiently with pitchforks. When I finally realized that Vasya wasn't coming back, I got down to business. First, I assessed the Win VM percentage in our fleet. The odds were against him. Windows.
Since we're actively developing DevOps, I realized we needed to change something in the approach to deploying new applications. There was only one solution: move everything to native, if possible. LinuxGoogle helped me - at that time .Net had already been ported to Linux, and I realized that this was the solution!
Why .NET core in conjunction with Linux?
There were several reasons for this. Given the choice between paying and not paying, most people would choose the latter—as did I. A license for MSDB costs around $1,000, and maintaining a fleet of virtual machines Windows amounts to hundreds of dollars. For a large company, this is a significant expense. Therefore, saving first reason. Not the most important, but one of the most significant.
Virtual machines Windows take up more resources than their brothers from Linux they are heavyConsidering the scale of the large company, we chose Linux.
The system is simply integrated into an existing CIWe consider ourselves progressive DevOps, we use Bamboo, Jenkins and GitLab CI, so most of our work is done on Linux.
The last reason is convenient accompaniment. We needed to lower the barrier to entry for "maintainers"—those who understand the technical side, ensure uptime, and maintain services from the second line. They were already familiar with the stack. Linux, so it is much easier for them to understand, support and maintain a new product than to spend additional resources to understand similar functionality of software for Windows platform.
Requirements
First and foremost - convenience of the new solution for developersNot all of them were ready for change, especially after the word was spoken LinuxDevelopers want their favorite Visual Studio, TFS with automated build tests, and smoothies. How they deliver to production isn't important to them. That's why we decided not to change the familiar process and leave it for Windows- developments remain unchanged.
Need a new project embed into existing CI. The rails were already there and all the work had to be done taking into account the parameters of the configuration management system, accepted delivery standards and monitoring systems.
Ease of maintenance and operation, as a condition for the minimum entry threshold for all new participants from different departments and the support department.
Deadline - yesterday.
Win development group
What was the team working with then? Windows?
Now I can confidently say that IdentityServer4 is a cool free alternative to ADFS with similar features, or whatever Entity Framework Core - a paradise for a developer, where you can not bother writing SQL scripts, but describe queries in the database in terms of OOP. But then, when discussing the action plan, I looked at this stack as Sumerian cuneiform, recognizing only PostgreSQL and Git.
At that time, we actively used Puppet as a configuration management system. In most of our projects we have used GitLab CI, Elastic, balancing high-loaded services using HAProxy kept track of everything Zabbix, bundles grafana и Prometheus, Jaeger, and all this was spinning on pieces of iron HP c ESXi + VMware. Everyone knows - a classic of the genre.
Let's look and try to understand what happened before we started all these interventions.
What happened
TFS is a fairly powerful system that not only delivers code from the developer to the final production machine, but also has a set for very flexible integration with various services - to provide CI at the cross-platform level.
Previously, these were all windows. TFS used several Build agents, which built multiple projects. Each agent had 3-4 workers to parallelize tasks and optimize the process. Then, according to the release plans, TFS delivered the freshly baked Build to Windows-application server.
Where did we want to go
We use TFS for delivery and development, and launch the application on Linux Application server, and there's some magic between them. This Magic Box and there is the salt of the work ahead. Before I take it apart, I'll take a step aside and say a few words about the app.
Project
The application provides functionality for handling prepaid cards.
Client
There were two types of users. There's a accessed by logging in with an SSL SHA-2 certificate. At second was accessed by username and password.
HAProxy
Then the client request went to HAProxy, which solved the following tasks:
- primary authorization;
- termination of SSL;
- tuning HTTP requests;
- request translation.
Verification of the client certificate went along the chain. We - authority and we can afford it, as we ourselves issue certificates to customers of the service.
Pay attention to the third point, a little later we will return to it.
Backend
The backend was planned to be done on LinuxThe backend interacts with the database, loads the required list of privileges, and then, depending on the privileges of the authorized user, grants access to sign financial documents and send them for execution, or generate a report.
Savings with HAProxy
In addition to the two contexts that each of the clients walked, there was also an identity context. IdentityServer4 just allows you to log in, this is a free and powerful analogue for ADFS Active Directory Federation Services.
The request in identity was processed in several steps. First step - client hit the backend, which communicated with this server and checked for the presence of a token for the client. If it didn’t find it, the request returned back to the context from which it came, but with a redirect, and with a redirect it went to identity.
The second step - the request got to the authorization page in IdentityServer, where the client registered, and the long-awaited token appeared in the IdentityServer database.
Third step - client redirected back to the context from which it came.
IdentityServer4 has a feature: it returns the response to the return request via HTTP. No matter how we struggled with setting up the server, no matter how enlightened the documentation was, each time we received the initial client request with a URL that came via HTTPS, and IdentityServer returned the same context, but with HTTP. We were shocked! And we transferred all this through the identity context to HAProxy, and in the headers we had to modify the HTTP protocol to HTTPS.
What is the improvement and where did you save?
We saved money by using a free solution for authorizing a group of users, resources, because we did not take out IdentityServer4 as a separate node in a separate segment, but used it together with the backend on the same server where the application backend is running.
How should it work
So, as I promised – Magic Box. We already understand that we are definitely moving in the direction LinuxLet's formulate the specific tasks that needed to be solved.
Puppet manifests. To deliver and manage the configuration of the service and application, you had to write cool recipes. The pencil roll eloquently shows how quickly and efficiently it was done.
Delivery method. The standard is RPM. Everyone understands that in Linux There was no way around it, but the project itself, after being built, was a collection of executable DLL files. There were about 150 of them, making the project quite heavy. The only sustainable solution was to package these binaries into RPM and deploy the application from there.
Versioning. We were going to release very frequently, and we had to decide how to form the package name. This was a question of the level of integration with TFS. We had a build agent on LinuxWhen TFS sends a task to a worker on a Build agent, it also passes it a bunch of variables that are stored in the worker's environment. These environment variables contain the Build name, version name, and other variables. More details are available in the "Building an RPM Package" section.
Setting up TFS It all boiled down to setting up the Pipeline. Previously, we used to build on Windows-all agents Windows-projects, and now it appears Linux-agent — a Build agent that needs to be included in the build group, enriched with some artifacts, specified what type of projects will be built on this Build agent, and the Pipeline modified in some way.
IdentityServer. ADFS is not our way, we drown for Open Source.
Let's go through the components.
Magic Box
Consists of four parts.
Linux Build agent. Linux, because we're compiling for it—it makes sense. This part was completed in three steps.
- Set up workers and not one, as the distributed work on the project was assumed.
- Install .NET Core 1.x. Why 1.x when 2.0 is already available in the standard repository? Because when we started development, the stable version was 1.09, and it was decided to make the project for it.
- Git 2.x.
RPM repository. RPM packages needed to be stored somewhere. We were supposed to use the same corporate RPM repository that was accessible to everyone. Linux hosts. That's what we did. The repository server is configured webhook which downloaded the required RPM package from the specified location. The package version of the webhook was reported by the Build agent.
gitlab. Please note: GitLab is not used by developers here, but by the operations department to control application versions, package versions, and the status of all Linux-machines and it stores the recipe - all Puppet manifests.
Puppet - resolves all controversial points and delivers exactly the configuration that we want from Gitlab.
We start to dive. How is DLL delivered to RPM?
Delivering DDL to RPM
Let's say we have a .NET development rockstar. It uses Visual Studio and creates a release branch. After that, it uploads it to Git, and Git here is a TFS entity, that is, it is the application repository that the developer works with.
After that, TFS sees that a new commit has arrived. Which application? TFS settings indicate which resources each Build Agent has. In this case, it sees that we're building a .NET Core project and selects Linux Build agent from the pool.
Build-agent receives the sources, downloads the necessary dependencies from the .NET repository, npm, etc. and after building the application itself and subsequent packaging, submits the RPM package to the RPM repository.
On the other hand, the following happens. The engineer of the operations department is directly involved in the rollout of the project: he changes the versions of packages in yesterday in the repository where the application recipe is stored, after which Puppet triggers Yum, pulls a new package from the repository, and the new version of the application is ready to use.
In words, everything is simple, but what happens inside on the Build agent itself?
Packaging DLL RPMs
Received project sources and build task from TFS. Build agent starts building the project itself from sources. The assembled project is available as a set dll files, which are packed in a zip archive to reduce the load on the file system.
ZIP archive is thrown out to the build directory of the RPM package. Next, the Bash script initializes the environment variables, finds the Build version, the project version, the path to the build directory, and runs RPM-build. When the build is complete, the package is published to local repository, which is located on the Build agent.
Further, from the Build agent to the server in the RPM repository JSON request is sent with the name of the version and build. Webhook, which I mentioned earlier, downloads this same package from the local repository on the Build agent and makes the new build available for installation.
Why this particular delivery scheme for the RPM repository? Why can't the built package be pushed to the repository immediately? This is a security requirement. This scenario limits the possibility of unauthorized people uploading RPM packages to a publicly accessible server. Linux-cars.
Database versioning
At the development consultation, it turned out that the guys are closer to MS SQL, but in most cases non-Windows We were already using PostgreSQL extensively for several projects. Since we'd already decided to move away from paid software, we started using PostgreSQL here as well.
In this part, I want to tell you how we versioned the database and how we chose between Flyway and Entity Framework Core. Consider their pros and cons.
Cons
Flyway only goes one way, we we can't roll back — this is a significant drawback. It can be compared with Entity Framework Core in other ways—from a developer convenience perspective. You remember that we made this a top priority, and the main criterion was not to change anything for Windows-developments.
For flyway us some kind of wrapper neededso that the guys do not write SQL queries. They are much closer to operate in terms of OOP. We wrote instructions for working with database objects, formed an SQL query and executed. The new version of the database is ready, rolled - everything is fine, everything works.
Entity Framework Core has a minus - under heavy loads, it builds non-optimal SQL queries, and the drawdown on the database can be significant. But since we do not have a highly loaded service, we do not calculate the load in hundreds of RPS, we accepted these risks and delegated the problem to the future us.
pros
Entity Framework Core works out of the box and is easy to develop, and Flyway easily integrates into existing CI. But we do it conveniently for developers :)
Roll-on procedure
Puppet sees that a change in the version of the packages is coming, among which, the one that is responsible for the migration. First, it installs a package that contains migration scripts and database-related functionality. After that, the application that works with the database is restarted. Next comes the installation of the remaining components. The order in which packages are installed and applications are launched is described in the Puppet manifest.
Applications use sensitive data, such as tokens, database passwords, all this is pulled into the config from Puppet master, where they are stored in encrypted form.
TFS Issues
After we decided and realized that everything really works for us, I decided to see what was happening with the assemblies in TFS in general for the Win development department on other projects - whether we are going / releasing quickly or not, and found significant problems with speed .
One of the main projects is going to 12-15 minutes - it's a long time, you can't live like that. A quick analysis showed a terrible drawdown in I / O, and this is on arrays.
After analyzing component by component, I identified three foci. First - kaspersky antivirus, which is on everyone Windows Build agents scan the source code. The second one is Windows indexer. It was not disabled, and everything was indexed on the Build agents in real time during the deployment process.
Third - npm install. It turned out that in most Pipelines we used this scenario. Why is he bad? The Npm install procedure is run when building the dependency tree in package-lock.json, where the versions of the packages that will be used to build the project are fixed. The downside is that Npm install pulls the latest versions of packages from the Internet each time, and this is a lot of time in the case of a large project.
Developers sometimes experiment on a local machine to test the performance of a particular part or an entire project. Sometimes it turned out that everything was cool locally, but they collected it, rolled it out - nothing worked. We begin to understand what the problem is - yeah, different versions of packages with dependencies.
Solution
- Sources in AV exceptions.
- Disable indexing.
- Go to npm ci.
The advantage of npm ci is that we collect the dependency tree once, and we get the opportunity to provide the developer up-to-date list of packages, with which he can experiment locally as much as he wants. This saves time developers who write the code.
Configuration
Now a little about the repository configuration. Historically we have used Nexus to manage repositories, including Internal REPO. This internal repository comes with all the components that we use for internal purposes, for example, self-written monitoring.
We also use NuGet, as it caches better than other package managers.
Experience the Power of Effective Results
After we optimized the build agents, the average build time was reduced from 12 minutes to 7.
If we count all the machines we could use for Windows, but translated into Linux In this project, we saved about $10,000. And that's just on licenses; if you take into account the content, it's even more.
Plans
For the next quarter, we planned to work on optimizing code delivery.
Switching to a prebuild Docker image. TFS is a cool thing with many plugins that allow you to integrate into Pipeline, including building on a trigger, for example, a Docker image. We want to make this trigger on the same package-lock.json. If the composition of the components that are used to build the project somehow changes, we build a new Docker image. It is later used to deploy the container with the built application. Now this is not the case, but we plan to switch to a microservice architecture in Kubernetes, which is actively developing in our company and has been servicing production solutions for a long time.
Summary
I urge everyone to throw it out Windows, but it's not because I don't know how to cook it. The reason is that most open source solutions are Linux-stack. Are you good save on resourcesIn my opinion, the future lies in Open Source solutions. Linux with a powerful community.
Speaker Profile Alexander Sinchinov .
is a conference on the integration of development, testing and operation processes for professionals from professionals. That is why the project that Alexander spoke about? implemented and working, and two successful releases were made on the day of the performance. On On May 27 and 28 there will be even more such cases from practitioners. You can still jump into the last car and or not in a hurry ticket. See you in Skolkovo!
Source: habr.com
