We once installed an electronic document management system for a customer. And then to another object. And one more. And the fourth, and the fifth. We got carried away so much that we reached 10 distributed objects. Powerfully turned out ... especially when we got to the delivery of changes. As part of the delivery to the productive circuit for 5 scenarios of the testing system, as a result, it took 10 hours and 6-7 employees. Such costs forced us to make deliveries as infrequently as possible. After three years of operation, we could not stand it and decided to spice up the project with a pinch of DevOps.
Now all testing takes 3 hours, and 3 people participate in it: an engineer and two testers. The improvements are clearly quantifiable and lead to a reduction in the beloved TTM. In our experience, there are far more customers who can benefit from DevOps than those who even know about it. Therefore, in order to bring DevOps closer to the people, we have developed a simple constructor, which we will discuss in more detail in this post.
Now let's talk in more detail. In one energy company, a technical document management system is being deployed at 10 large facilities. In projects of this scale, it is not easy to swim out without DevOps, because a large proportion of manual labor greatly delays the work and also reduces quality - all manual work is fraught with errors. On the other hand, there are projects where there is only one installation, but everything needs to work automatically, constantly and without fail - for example, the same document management systems in large monolithic organizations. Otherwise, someone will make the setting manually, forget about the deployment instructions - and as a result, the setting will be lost on the sale and everything will collapse.
Usually we work with the customer through a contract, in which case our interests diverge a bit. The customer looks at the project strictly within the budget and TOR. It can be difficult to explain to him the benefits of various DevOps practices that are not included in the TOR. And if he is interested in quick releases with added business value, in building an automation pipeline?
Alas, in work with a pre-approved cost, this interest is not always possible to find. In our practice, there was a case when we had to pick up the development for an unscrupulous and sloppy contractor. It was terrible: there are no up-to-date source codes, the code base of the same system on different installations is different, the documentation was partially absent, partially it was of terrible quality. Of course, the customer did not have control over the source code, assembly, releases, etc.
So far, not everyone knows about DevOps, but itβs worth talking about its advantages, about real resource savings, and all customersβ eyes light up. So requests that include DevOps are becoming more and more over time. Here, in order to easily speak the same language with customers, we need to quickly connect business issues and DevOps practices that will help build the right development pipeline.
So, we have a set of problems on the one hand, there are DevOps knowledge, practices and tools on the other. Why not share your experience with everyone?
Create a DevOps constructor
Agile has its own manifesto. ITIL has its own methodology. DevOps is less fortunate - it has not yet acquired templates and standards. Although determine the degree of maturity of companies based on an assessment of their development and operation methods.
Fortunately, the notorious Gartner company in 2014 and analyzed key DevOps practices and the relationships between them. Based on this, I released an infographic:
We took it as the basis of our . Each of the four areas has a set of tools - we have compiled them into a database, identified the most popular ones, identified integration points and suitable optimization mechanisms. Total turned out 36 practices and 115 tools, a quarter of which are open source or free software. Next, we will talk about what we have done in each area and, for example, how it is implemented in the project for creating technical document management, with which we started the post.
Processes
In the notorious EDMS project, the technical documentation management system was deployed according to the same scheme at each of the 10 sites. The installation includes 4 servers: database server, application server, full-text indexing and content management. In the installation, they work within a single node, they are located in the data center at the facilities. All objects differ slightly in infrastructure, but this does not interfere with global interaction.
First, according to DevOps practices, we automated the infrastructures locally, then brought the delivery to the test circuit, and then to the customerβs production. Each process was worked out step by step. The environment settings are fixed in the system of source codes, taking into account which the distribution kit is built for automatic updating. In case of changes in the configuration, engineers just need to make the appropriate changes to the version control system - and then the automatic update will work without problems.
This approach greatly simplifies the testing process. Previously, there were testers in the project who did nothing but manually update the stands. Now they just come, see that everything is working and do more useful things. Each update is tested automatically - from the surface level to business scenario automation. The results are posted as separate reports in TestRail.
culture
Continuous experimentation is best explained through test design. Testing a system that doesn't exist yet is creative work. When writing a test plan, you need to understand how to test correctly, which branches to go through. And also find a balance between time and budget to determine the optimal number of reviews. It is important to choose exactly the necessary tests, think about how the user will interact with the system, take into account the environment and possible external factors. Continuous experimentation is indispensable.
Now about the culture of interaction. Previously, there were two opposing sides - engineers and developers. The developers said: βWe don't care how it will run. You are engineers, you are smart, make sure that it is operated without failures.. The engineers answered: βYou developers are too careless. Let's be more careful, and we will put your releases less often. Because every time you put a leaky code for us, and itβs not clear how we should interact. β. This is a problem of cultural interaction, which is built differently from the point of view of DevOps. Here you have both engineers and developers as part of a single team that is focused on constantly changing, but at the same time reliable software.
On the scale of one team, specialists are tuned to help each other. As it was before? For example, some thick deployment instructions were being prepared, 50 pages long. The engineer read it, did not understand something, cursed and asked the developer to comment at three in the morning. The developer commented and also cursed - in the end, no one was happy. Plus, of course, there were some mistakes, because you canβt remember everything in the instructions. And now the engineer, together with the developer, is writing a script for the automated deployment of the application software infrastructure. And they talk to each other almost in the same language.
People
The size of the team is determined by the scale of the update. The team is recruited during the formation of the delivery, it includes those who wish from the general project team. Then an update plan is written with those responsible for each stage, as the team reports back as it is completed. All team members are interchangeable. As part of the team, we also have a developer on a safety net, but he almost never has to connect.
Technologies
There are a few points highlighted in the technology scheme, but under them there are a bunch of technologies - you could publish a whole book with their descriptions. So we will highlight the most interesting.
Infrastructure as Code
Now, probably, you will not surprise anyone with this concept, but earlier descriptions of infrastructures left much to be desired. , the test environments were unique, they were groomed and cherished, dust particles were blown off them.
Now no one is afraid to experiment. There are basic images of virtual machines, there are ready-made scenarios for deploying environments. All templates and scripts are stored in the version control system and are updated promptly. Previously, when it was necessary to deliver some package to the stand, a configuration gap appeared. Now you just need to add a line in the source code.
In addition to infrastructure scenarios and pipelines, the Documentation as a Code approach is also used for documentation. Thanks to this, it is easy to connect new people to the project, introduce them to the system using the functions described, for example, in the test plan, and also reuse test cases.
Continuous supply and monitoring
about DevOps, we talked about how we chose tools for implementing continuous delivery and monitoring. Often there is no need to rewrite anything - it is enough to use previously written scripts, correctly build integration between components and create a common management console. And all processes can be launched by one button or schedule.
There are different concepts in English, Continuous Delivery and Continuous Deployment. Both can be translated as "continuous supply", but in fact there is a slight difference between them. In our project on the technical workflow of a distributed energy company, rather, Delivery is used - when the installation for production is on command. In Deployment, installation occurs automatically. Continuous Delivery in this project has generally become central part of DevOps.
In general, by collecting certain parameters, you can clearly understand how DevOps practices are useful. And bring it to the management, which loves numbers very much. The total number of launches, the execution time of the script steps, the percentage of successful launches - all this directly affects everyone's favorite time to market, that is, the time from committing to the version control system to releasing the version in a productive environment. With the implementation of the right tools, engineers receive valuable indicators by mail, and the project manager sees them on the dashboard. So you can immediately evaluate the benefits of new tools. And you can try them on your infrastructure using the DevOps constructor.
Who needs our ?
Let's not pretend: for a start, it became useful to us. As we said, you need to speak the same language with the customer, and with the help of the DevOps designer, we can quickly sketch out the basis for such a conversation. Business professionals will be able to assess what they need and thus develop faster. We tried to make the constructor as detailed as possible, added a bunch of descriptions so that any user understands what he chooses.
The format of the constructor allows you to take into account the developments already available in the company for building processes, automation. There is no need to tear everything down and rebuild if you can only choose solutions that integrate well with existing processes that can simply fill in the gaps.
Perhaps your development has already moved to a higher level and our tool will seem too "captain's". But we find it useful for ourselves and hope that it will be useful to some of the readers. We remind on the constructor - if anything, you get the scheme immediately after entering the initial data. We will be grateful for feedback and additions.
Source: habr.com