State of DevOps in Russia 2020

How to understand the state of something?

You can rely on your opinion, formed from various sources of information, for example, publications on websites or experience. You can ask colleagues, acquaintances. Another option is to look at the topics of the conferences: the program committee are active representatives of the industry, so we trust them in choosing relevant topics. A separate area is research and reports. But there is a problem. Research on the state of DevOps is conducted annually in the world, reports are published by foreign companies, and there is almost no information about Russian DevOps.

But the day has come when such a study was conducted, and today we will talk about the results. The state of DevOps in Russia was studied jointly by the companies "Express 42" and "Ontico". Express 42 helps technology companies implement and develop DevOps practices and tools and was one of the first to talk about DevOps in Russia. The authors of the study, Igor Kurochkin and Vitaly Khabarov, are engaged in analysis and consulting at Express 42, while having a technical background from operation and experience in different companies. For 8 years, colleagues have looked at dozens of companies and projects - from startups to enterprises - with different problems, as well as different cultural and engineering maturity.

In their report, Igor and Vitaly told what problems were in the process of research, how they solved them, as well as how DevOps research is conducted in principle and why Express 42 decided to conduct its own. Their report can be viewed here.

DevOps Research

The conversation was started by Igor Kurochkin.

We regularly ask the audience at DevOps conferences, “Have you read the DevOps status report for this year?” Few raise their hands, and our study showed that only a third study them. If you have never seen such reports, let's say right away that they are all very similar. Most often there are phrases like: "Compared to last year ..."

Here we have the first problem, and after it two more:

1. We don't have data for last year. The state of DevOps in Russia is of no interest to anyone;
2. Methodology. It is not clear how to test hypotheses, how to build questions, how to analyze, compare results, find connections;
3. Terminology. All reports are in English, translation is required, a common DevOps framework has not yet been invented and everyone comes up with their own.

Let's take a look at how DevOps state analyzes have been done around the world.

Historical information

DevOps research has been conducted since 2011. Puppet, a developer of configuration management systems, was the first to conduct them. At that time, it was one of the main tools for describing the infrastructure in the form of code. Until 2013, these studies were simply closed surveys and no public reports.

In 2013, IT Revolution appeared, the publisher of all major books on DevOps. Together with Puppet, they prepared the first State of DevOps publication, where 4 key metrics appeared for the first time. The following year, ThoughtWorks, a consulting firm known for its regular technology radars on industry practices and tools, got involved. And in 2015, a section with methodology was added, and it became clear how they perform the analysis.

In 2016, the authors of the study, having created their own company DORA (DevOps Research and Assessment), published an annual report. The following year, DORA and Puppet released their last joint report.

And then something interesting began:

In 2018, the companies split and two independent reports were released: one from Puppet, the second from DORA in conjunction with Google. DORA has continued to leverage its methodology with key metrics, performance profiles, and engineering practices that impact key metrics and company-wide performance. And Puppet offered its own approach with a description of the process and the evolution of DevOps. But the story did not take root, in 2019 Puppet abandoned this methodology and released a new version of the reports, which listed the key practices and how they affect DevOps from their point of view. Then another event happened: Google bought DORA, and together they released another report. You may have seen him.

This year, things got complicated. Puppet is known to have launched its own survey. They did it a week earlier than us, and it has already ended. We took part in it and looked at what topics they are interested in. Now Puppet is doing its analysis and preparing to publish the report.

But there is still no announcement from DORA and Google. In May, when the survey usually began, information came that Nicole Forsgren, one of the founders of DORA, had moved to another company. Therefore, we assumed that there would be no research and report from DORA this year.

How are things in Russia?

We haven't done DevOps research. We spoke at conferences, retelling other people's findings, and Raiffeisenbank translated "State of DevOps" for 2019 (you can find their announcement on Habré), many thanks to them. And it's all.

Therefore, we conducted our own research in Russia using DORA methodologies and findings. We used the report of colleagues from Raiffeisenbank for our research, including for synchronization of terminology and translation. And industry-relevant questions were taken from DORA reports and this year's Puppet questionnaire.

Research Process

The report is only the final part. The entire research process consists of four major steps:

During the preparation phase, we interviewed industry experts and prepared a list of hypotheses. On their basis, questions were compiled and a survey was launched for the whole of August. Then we analyzed and prepared the report itself. For DORA, this process takes 6 months. We met within 3 months, and now we understand that we barely had enough time: only by performing the analysis do you understand what questions you need to ask.

Participants

All foreign reports begin with a portrait of the participants, and most of them are not from Russia. The percentage of Russian respondents fluctuates between 5 and 1% from year to year, and this does not allow any conclusions to be drawn.

Map from the Accelerate State of DevOps 2019 report:

In our study, we managed to interview 889 people - this is quite a lot (DORA polls about a thousand people annually in its reports) and here we have achieved the goal:

True, not all of our participants reached the end: the percentage of completion turned out to be slightly less than half. But even this was enough to obtain a representative sample and conduct an analysis. DORA does not disclose fill percentages in its reports, so there is no comparison here.

Industries and positions

Our respondents represent a dozen industries. Half work in information technology. This is followed by financial services, trade, telecommunications and others. Among the positions are specialists (developer, tester, operation engineer) and management staff (heads of teams, groups, areas, directors):

One in two works for a medium-sized company. Every third person works in large companies. Most work in teams of up to 9 people. Separately, we asked about the main activities, and the majority are somehow related to the operation, and about 40% are engaged in development:

So we collected information for comparison and analysis of representatives of different industries, companies, teams. My colleague Vitaly Khabarov will tell about the analysis.

Analysis and comparison

Vitaly Khabarov: Many thanks to all the participants who completed our survey, filled out questionnaires and provided us with data for further analysis and testing of our hypotheses. And thanks to our clients and customers, we have a wealth of experience that has helped us identify industry concerns and formulate the hypotheses that we tested in our research.

Unfortunately, you can’t just take a list of questions on the one hand and data on the other, somehow compare them, say: “Yes, everything works like that, we were right” and disperse. No, methodology and statistical methods are needed to be sure that we are not mistaken and our conclusions are reliable. Then we can build our further work based on these data:

Key Metrics

We took the DORA methodology as a basis, which they described in detail in the book “Accelerate State of DevOps”. We checked whether the key metrics are suitable for the Russian market, whether they can be used in the same way as DORA uses to answer the question: “How does the industry in Russia correspond to the foreign industry?”

Key metrics:

1. Deployment frequency. How often is a new version of the application deployed to the production environment (planned changes, excluding hotfixes and incident response)?
2. Delivery time. What is the average time between committing a change (writing functionality as code) and deploying the change to the production environment?
3. Recovery time. How long does it take on average to restore an application to a production environment after an incident, service degradation, or discovery of a bug that affects application users?
4. Unsuccessful changes. What percentage of deployments in the production environment lead to application degradation or incidents and require remediation (rollback of changes, development of a hotfix or patch)?

DORA in its research has found a connection between these metrics and organizational performance. We also test it in our study.

But to make sure that the four key metrics can influence something, you need to understand - are they somehow related to each other? DORA answered in the affirmative with one caveat: the relationship between unsuccessful changes (Change Failure Rate) and three other metrics is slightly weaker. We got about the same picture. If delivery time, deployment frequency, and recovery time correlate with each other (we established this correlation through the Pearson correlation and through the Chaddock scale), then there is no such strong correlation with unsuccessful changes.

In principle, most of the respondents tend to answer that they have a rather small number of incidents in production. Although we will see later that there is still a significant difference between the groups of respondents in terms of unsuccessful changes, we cannot yet use this metric for this division.

We attribute this to the fact that (as it turned out during the analysis and communication with some of our customers) there is a slight difference in the perception of what is considered an incident. If we managed to restore the performance of our service during the technical window, can this be considered an incident? Probably not, because we fixed everything, we are great. Can we consider it an incident if we had to reroll our application 10 times in a normal, familiar mode for us? It seems not. Therefore, the question of the relationship of unsuccessful changes with other metrics remains open. We will refine it further.

Important here is that we found a significant correlation between delivery times, recovery times, and deployment frequency. Therefore, we took these three metrics to further divide the respondents into performance groups.

How much to hang in grams?

We used hierarchical cluster analysis:

• We distribute respondents over an n-dimensional space, where the coordinate of each respondent is their answers to questions.
• Each respondent is declared a small cluster.
• We combine the two clusters closest to each other into one larger cluster.
• We find the next pair of clusters and combine them into a larger cluster.

This is how we group all our respondents into the number of clusters we need. With the help of a dendrogram (a tree of connections between clusters), we see the distance between two neighboring clusters. All that remains for us is to set a certain distance limit between these clusters and say: "These two groups are quite distinguishable from each other because the distance between them is gigantic."

But there is a hidden problem here: we have no restrictions on the number of clusters - we can get 2, 3, 4, 10 clusters. And the first idea was - why not divide all our respondents into 4 groups, as DORA does. But we found that the differences between these groups become insignificant, and we cannot be sure that the respondent really belongs to his group, and not to the neighboring one. We cannot yet divide the Russian market into four groups. Therefore, we settled on three profiles between which there is a statistically significant difference:

Next, we determined the profile by clusters: we took the median for each metric for each group and compiled a table of performance profiles. In fact, we got the performance profiles of the average participant in each group. We have identified three efficiency profiles: Low, Medium, High:

Here we confirmed our hypothesis that 4 key metrics are suitable for determining the performance profile, and they work both in the Western and Russian markets. There is a difference between the groups and it is statistically significant. I emphasize that there is a significant difference between the performance profiles in terms of the metric of unsuccessful changes in terms of the average, even though we did not initially divide the respondents by this parameter.

Then the question arises: how to use all this?

How to use

If we take any team, 4 key metrics and apply it to the table, then in 85% of cases we will not get a complete match - this is just an average participant, and not what is in reality. We are all (and every team) slightly different.

We checked: we took our respondents and the DORA performance profile, and looked at how many respondents fit this or that profile. We found that only 16% of the respondents definitely fell into one of the profiles. All the rest are scattered somewhere in between:

This means that the efficiency profile has a limited scope. To understand where you are in the first approximation, you can use this table: “Oh, it seems we are closer to Medium or High!” If you understand where to go next, this may be enough. But if your goal is constant, continuous improvement, and you want to know more exactly where to develop and what to do, then additional funds are needed. We called them calculators:

• DORA calculator
• Calculator Express 42* (in development)
• Own development (you can create your own internal calculator).

What are they needed for? To understand:

• Is the team within our organization up to our standards?
• If not, can we help it, speed it up within the framework of the expertise that our company has?
• If so, can we do even better?

You can also use them to collect statistics within the company:

• What teams do we have?
• Divide teams into profiles;
• See: Oh, these commands are underperforming (they don’t pull out a little), but these are cool: they deploy every day, without errors, they have a lead time of less than an hour.

And then you can find out that within our company there is the necessary expertise and tools for those teams that are not yet up to par.

Or, if you understand that you feel great inside the company, you are better than many, then you can look a little wider. This is just the Russian industry: can we get the necessary expertise in the Russian industry in order to accelerate ourselves? The Express 42 calculator will help here (it is under development). If you have outgrown the Russian market, then look at DORA calculator and to the world market.

Fine. And if you are in the Elit group on the DORA calculator, what should you do? There is no good solution here. You are most likely at the forefront of the industry, and further acceleration and reliability is possible through internal R&D and spending more resources.

Let's move on to the sweetest - comparison.

Comparison

We initially wanted to compare Russian industry with Western industry. If we compare directly, we see that we have fewer profiles, and they are a little more mixed with each other, the borders are a little more blurred:

Our Elite performers are hidden among the High performers, but they are there - these are the elite, unicorns who have reached significant heights. In Russia, the difference between the Elite profile and the High profile is not yet significant enough. We think that in the future this separation will occur due to an increase in engineering culture, the quality of implementation of engineering practices and expertise within companies.

If we move on to a direct comparison within the Russian industry, we can see that the High profile teams are better in all respects. We also confirmed our hypothesis that there is a relationship between these metrics and organizational performance: High profile teams are much more likely to not only achieve goals, but also exceed them.
Let's become High profile teams and not stop there:

But this year is special, and we decided to check how companies are doing in a pandemic: High profile teams are doing much better and feeling better than the industry average:

• 1,5-2 times more likely to release new products,
• 2 times more likely to improve the reliability and / or performance of the application infrastructure.

That is, the competencies that they already had helped them develop faster, launch new products, modify existing products, thereby conquering new markets and new users:

What else helped our teams?

Engineering practices

I will tell you about the significant findings for each practice that we tested. Perhaps something else helped the teams, but we are talking about DevOps. And within DevOps, we see a difference among teams of different profiles.

Platform as a Service

We did not find a significant relationship between platform age and team profile: Platforms appeared at about the same time for both Low-teams and High-teams. But for the latter, the platform provides, on average, more services and more programming interfaces for control through program code. And platform teams are more likely to help their developers and teams use the platform, solve their problems and platform-related incidents more often, and educate other teams.

Infrastructure as code

Everything is pretty standard here. We found a relationship between the automation of the work of the infrastructure code and how much information is stored inside the infrastructure repository. The High profile commands store more information in the repositories: this is the infrastructure configuration, CI / CD pipeline, environment settings and build parameters. They store this information more often, work better with infrastructure code, and automate more processes and tasks for working with infrastructure code.

Interestingly, we did not see a significant difference in infrastructure tests. I attribute this to the fact that High profile teams have more test automation in general. Perhaps they should not be distracted separately by infrastructure tests, but rather those tests with which they check applications, and thanks to them they already see what and where they have broken.

Integration and Delivery

The most boring section, because we confirmed that the more automation you have, the better you work with the code, the more likely you are to get better results.

Architecture

We wanted to see how microservices affect performance. In truth, they do not, since the use of microservices is not associated with an increase in performance indicators. Microservices are used for both High profile commands and Low profile commands.

But what is significant is that for High-teams, the transition to a microservice architecture allows them to independently develop their services and roll out. If the architecture allows developers to act autonomously, without waiting for someone external to the team, then this is a key competency for increasing speed. In this case, microservices help. And just their implementation does not play a big role.

How did we discover all this?

We had an ambitious plan to fully replicate the DORA methodology, but lacked the resources. If DORA uses a lot of sponsorship and their research takes half a year, we did our research in a short time. We wanted to build a DevOps model like DORA does, and we will do that in the future. So far we have limited ourselves to heat maps:

We looked at the distribution of engineering practices across teams in each profile and found that High profile teams, on average, were more likely to use engineering practices. You can read more about all this in our report.

For a change, let's switch from complex statistics to simple ones.

What else have we discovered?

Tools

We observe that most of the commands are used by the OS of the Linux family. But Windows is still in trend - at least a quarter of our respondents noted the use of one or another of its versions. It seems that the market has this need. Therefore, you can develop these competencies and make presentations at conferences.

Among the orchestrators, it's not a secret for anyone, Kubernetes is in the lead (52%). The next in line orchestrator is Docker Swarm (about 12%). The most popular CI systems are Jenkins and GitLab. The most popular configuration management system is Ansible, followed by our beloved Shell.

Amazon is currently the leading cloud hosting provider. The share of Russian clouds is gradually increasing. Next year it will be interesting to see how Russian cloud providers will feel, whether their market share will increase. They are, you can use them, and that's good:

I pass the floor to Igor, who will give some more statistics.

Dissemination of practices

Igor Kurochkin: Separately, we asked respondents to indicate how the considered engineering practices are distributed in the company. In most companies, there is a mixed approach, consisting of a different set of patterns, and pilot projects are very popular. We also saw a slight difference between the profiles. Representatives of the High profile more often use the “Initiative from below” pattern, when small teams of specialists change work processes, tools, and share successful practices with other teams. At Medium, this is a top-down initiative that affects the entire company through the creation of communities and centers of excellence:

Agile and DevOps

The question of the connection between Agile and DevOps is often discussed in the industry. This issue is also raised in the State of Agile Report for 2019/2020, so we decided to compare how Agile and DevOps activities are connected in companies. We found that DevOps without Agile is rare. For half of the respondents, the spread of Agile began much earlier, and about 20% observed the simultaneous start, and one of the signs of a Low profile will be the absence of Agile and DevOps practices:

Command topologies

At the end of last year, the book "Team topologies”, which proposes a framework for describing command topologies. It became interesting to us whether it is applicable to Russian companies. And we asked the question: “What patterns do you find?”.

Infrastructure teams are observed in half of the respondents, as well as separate teams for development, testing and operation. Separate DevOps teams noted 45%, among which representatives of High are more common. Next comes cross-functional teams, which are also more common at High. Separate SRE commands appear in the High, Medium profiles and are rarely seen in the Low profile:

DevQaOps ratio

We saw this question on FaceBook from the team leader of the Skyeng platform team - he was interested in the ratio of developers, testers and administrators in companies. We asked it and looked at the responses based on profiles: High profile representatives have fewer test and operations engineers for each developer:

Plans for 2021 year

In the plans for the next year, the respondents noted the following activities:

Here you can see the intersection with the DevOps Live 2020 conference. We carefully reviewed the program:

• Infrastructure as a product
• DevOps transformation
• Distribution of DevOps practices
• DevSecOps
• Case clubs and discussions

But the time of our presentation is not enough to cover all the topics. Left behind the scenes:

• Platform as a service and as a product;
• Infrastructure as code, environments and clouds;
• Continuous Integration and Delivery;
• Architecture;
• DevSecOps patterns;
• Platform and cross-functional teams.

Photos we got a voluminous, 50 pages, and you can see it in more detail.

Summing up

We hope that our research and report will inspire you to experiment with new approaches to development, testing, and operations, as well as help you navigate, compare yourself with other participants in the study, and identify areas where you can improve your own approaches.

Results of the first study of the state of DevOps in Russia:

• Key metrics. We have found that key metrics (delivery time, deployment frequency, recovery time, and change failures) are suitable for analyzing the effectiveness of development, testing, and operations processes.
• Profiles High, Medium, Low. Based on the collected data, we can distinguish statistically different groups of High, Medium, Low with distinctive features in terms of metrics, practices, processes and tools. Representatives of the High profile show better results than Low. They are more likely to achieve and exceed their goals.
• Indicators, pandemic and plans for 2021. A special indicator this year is how companies coped with the pandemic. The High representatives fared better, experienced increased user engagement, and the main reasons for success were efficient development processes and a strong engineering culture.
• DevOps practices, tools and their development. The main plans of companies for the next year include the development of DevOps practices and tools, the introduction of DevSecOps practices, and changes in the organizational structure. And the effective implementation and development of DevOps practices is carried out with the help of pilot projects, the formation of communities and centers of excellence, initiatives at the upper and lower levels of the company.

We'd love to hear your feedback, stories, feedback. We thank everyone who participated in the study and look forward to your participation next year.

Source: habr.com