The question “how to implement devops” has been on the table for many years, but there are not so many good materials. Sometimes you get advertised by not-so-smart consultants who need to sell their time, no matter how. Sometimes these are vague, extremely general words about how the ships of mega-corporations plow the expanses of the universe. The question arises: what are we to do with this? Dear author, can you clearly formulate your ideas in a list?
All this comes from the fact that there is not much real practice and understanding of the outcome of the transformation of the company's culture. Cultural change is a long-term thing, the results of which will not appear in a week or a month. We need someone who is old enough to have seen companies rise and fall over the years.
John Willis one of the fathers of DevOps. John has decades of experience working with a huge number of companies. Recently, John began to notice for himself the specific patterns that take place in working with each of them. Using these archetypes, John guides companies on the true path of DevOps transformation. Read more about these archetypes in the translation of his report from the DevOops 2018 conference.
About the speaker:
More than 35 years in IT management, participated in the creation of the predecessor of OpenCloud at Canonical, participated in 10 startups, two of which he sold to Dell and Docker. He is currently Vice President of DevOps and Digital Practices at SJ Technologies.
What follows is a story from John's point of view.
My name is John Willis and I'm easiest to find on Twitter . I have the same alias on Gmail and GitHub. A You can find video recordings of my reports and presentations for them.
I have many meetings with CIOs of various large companies. They very often complain that they do not understand what DevOps is, and everyone who tries to explain it to them is talking about something of their own. Another common complaint is that DevOps does not work, although it seems that the directors are doing everything as they were explained to them. We are talking about large companies that are more than a hundred years old. After talking with them, I came to the conclusion that for many problems, not high-tech solutions, but relatively low-tech solutions are best suited. For weeks I just talked to people from different departments. What you see in the very first picture in the post is my last project, the room looked like this after three days of work.
What is DevOps?
Indeed, if you ask 10 different people, they will give 10 different answers. But here's what's interesting: all of these ten answers will be correct. There is no wrong answer here. I've been involved in DevOps quite deeply, for about 10 years, I was the first American on the first DevOpsDay. I won't say that I'm smarter than everyone involved in DevOps, but there is hardly anyone who has put so much effort into it. I believe that DevOps occurs when human capital and technology come together. We often forget about the human dimension, although we talk a lot about all kinds of cultures.
Now we have a lot of data, five years of academic research, testing of theories has been established on an industrial scale. These studies tell us the following: if you combine some behavioral patterns in an organizational culture, you can get a speedup of 2000 times. This acceleration corresponds to the same improvement in stability. It's a quantitative measure of the value that DevOps can bring to any company. A couple of years ago, I spoke about DevOps to the CEO of a Fortune 5000 company. When I was preparing for the presentation, I was very nervous, because I had to present my many years of experience in 5 minutes.
I ended up giving the following definition of DevOps: it is a set of practices and patterns that allow you to turn human capital into high-performing organizational capital. An example is how Toyota has been operating for the last 50 or 60 years.
(Hereinafter, such diagrams are provided not as a reference, but as an illustration. Their content will be different for each new company. However, the picture can be viewed and enlarged separately
One of the most successful such practices is value stream mapping. Several good books have been written about this, the most successful being Karen Martin. But over the past year, I have come to the conclusion that even this approach is too high-tech. It certainly has many advantages, I used it a lot. But when a CEO asks you why his company can't switch to a new track, it's too early to talk about value stream mapping. There are many much more fundamental questions that need to be answered first.
It seems to me that the mistake many of my colleagues make is that they simply give the company a five-point guideline and then come back six months later and see what happened. Even a good scheme like value stream mapping has, shall we say, blind spots. After hundreds of interviews with directors of various companies, I have developed a certain pattern that allows you to decompose the problem into components, and now we will discuss each of these components in order. Before applying any technological solutions, I use this pattern, and as a result, all my walls are hung with diagrams. I recently worked with a mutual fund and ended up with 100-150 of these schemes.
Bad culture eats good breakfast approaches
The main idea is this: no Lean, Agile, SAFE and DevOps will help if the culture of the organization itself is bad. It's like diving to the depths without scuba gear or operating without an x-ray. In other words, to paraphrase Drucker and Deming: a bad organizational culture will swallow up any good system and not choke.
To solve this main problem, you need to take the following steps:
- Make All Work Visible: you need to make all the work visible. Not in the sense that it must be displayed on some screen, but in the sense that it must be observable.
- Consolidated Work Management Systems: management systems need to be consolidated. In the problem of "tribal" knowledge and institutional knowledge in 9 cases out of 10 people are the bottleneck. In the book the problem was one man, Brent, who delayed the project by three years. And I see these Brents everywhere. To debug these bottlenecks, I use the next two items on our list.
- Theory of Constraints Methodology: the theory of restrictions.
- Collaboration hacks: cooperation hacks.
- Toyota Kata (): I won't talk much about Toyota Kata. If interested, on my github almost every one of these topics.
- Market Oriented Organization: market oriented organization.
- Shift-left auditors: audit early in the cycle.
I start working with an organization very simply: I go to the company and talk to employees. As you can see, no high technology. All you need is something to write on. I gather several teams in one room and analyze what they tell me in terms of my 7 archetypes. And then I give them a marker themselves and ask them to write on the board everything that they have said aloud so far. Usually at this kind of meetings there is one person who writes everything down, and at best he manages to write down 10% of the discussion. With my method, this figure can be raised to somewhere around 40%.
(Separately, this illustration can be )
My approach is based on the work of William Schneider (William Schneider, ). The approach is based on the idea that any organization can be decomposed into four squares. This scheme for me is usually the result of working with those hundreds of other schemes that arise in the analysis of the organization. Suppose we have an organization with a high level of control, but at the same time with low competence. This is an extremely undesirable option: when everyone walks along the line, but at the same time no one knows what to do.
A slightly better option with a high level of both control and competence. If such a company has a profit, then perhaps DevOps is not needed for it. It is most interesting to work with a company that has a high level of control, low competence and cooperation, but at the same time a high level of culture (cultivation). This means that the company has a lot of people who like working there, and the turnover of the workforce is low.
(Separately, this illustration can be )
It seems to me that methods with hard-coded recommendations end up getting in the way of the truth. In particular, value stream mapping has many rules about how information should be structured. In the early stages of work, which I am now talking about, no one needs these rules. If a person with a marker in his hands describes the real situation in the company on the board, this is the best way to understand the state of affairs. Such information does not reach the directors. At this moment, it is stupid to cut off a person and say that he has drawn some kind of arrow incorrectly. At this stage, it is better to use simple rules, for example: a multi-level abstraction can be created simply by using multi-colored markers.
I repeat, no high tech. Black marker depicts objective reality, how everything works. With a red marker, people note what exactly they do not like about the current state of affairs. It is important that they write it, not me. When I go to the CIO after a meeting, I don't come up with a list of 10 things to fix. I'm looking to find connections between what people in the company say and existing, proven patterns. Finally, the blue marker suggests possible solutions to the problem.
(Separately, this illustration can be )
An example of this approach is now depicted above. At the beginning of this year, I worked with one bank. The security staff there were convinced they weren't allowed to come to design and requirement reviews.
(Separately, this illustration can be )
And then we talked to people from other departments and it turned out that about 8 years ago, software developers put out security workers because they slowed down work. And then it turned into a ban, which was taken for granted. Although in fact there was no ban.
Our meeting went in an extremely confusing course: for about three hours, five different teams could not explain to me in any way what was happening between the code and the assembly. And this, it would seem, is the simplest thing. Most DevOps consultants assume up front that everyone already knows this.
Then the person in charge of IT governance, who had been silent for four hours, suddenly came to life when we got to his topic, and kept us busy for quite a long time. In the end, I asked him what he thought of the meeting, and I will never forget his answer. He said: “I used to think that our bank had only two ways to deliver software, but now I know that there are five of them, and I didn’t even know about three.”
(Separately, this illustration can be )
The last meeting at this bank was with the investment software team. It was with her that it turned out that writing diagrams with a marker on a sheet is better than on a board, and even better than on a smartboard.
The photos you see are what the hotel conference room looked like on the fourth day of our meeting. And we used these schemes to search for patterns, that is, archetypes.
So, I ask questions to the workers, they write down the answers with markers of three colors (black, red and blue). I analyze their answers for archetypes. Now let's discuss all the archetypes in order.
1. Make All Work Visible: Make work visible
Most of the companies I work with have a very high percentage of unknown work. For example, this is when one employee comes to another and simply asks to do something. Large organizations can have 60% unplanned work. And up to 40% of the work is not documented in any way. If it were Boeing, then I would never have boarded their plane again in my life. If only half of the work is documented, then it is not known whether this work is being done correctly or not. All other methods are useless - there is no point in trying to automate anything, because the known 50% can be just the most coherent and clear part of the work, the automation of which will not give great results, and all the most terrible is in the invisible half. In the absence of documentation, it is impossible to find any kind of hacks and hidden work, not to find bottlenecks, those same Brents that I have already mentioned. There is a wonderful book by Dominica DeGrandis . She brings out five different "time leaks" (thieves of time):
- Too Much Work in Process (WIP)
- Unknown Dependencies
- unplanned work
- conflicting priorities
- Neglected Work
This is a very valuable analysis and the book is great, but all this advice is useless if only 50% of the data is visible. The methods proposed by Dominica can be applied if an accuracy of more than 90% is achieved. I'm talking about situations where a boss gives a subordinate a 15-minute task, and it takes him three days; but the boss doesn't really know that this subordinate is dependent on four or five other people.
The Phoenix Project is a wonderful story about a project three years late. One of the heroes is threatened with dismissal because of this, and he meets with another character, who is presented as a kind of Socrates. It helps to figure out what exactly went wrong. It turns out that the company has one system administrator, whose name is Brent, and all work somehow goes through him. At one of the meetings, one of the subordinates is asked: why does each half-hour task take a week? The answer is a very simplified exposition of queuing theory and Little's law, and in this exposition it turns out that at 90% employment, each hour of work takes 9 hours. Each job needs to be sent to seven other people, so that hour becomes 63 hours, 7 times 9. I'm saying that in order to use Little's law or any complicated queuing theory, you must at least have data.
Therefore, when I talk about visibility, I do not mean that everything is on the screen, but that it is necessary to at least have data. When they are, it often turns out that there is a very large amount of unplanned work that for some reason is sent to Brent, although there is no need for this. And Brent is a great guy, he never says no, but at the same time he does not tell anyone how he does his job.
When the work is visible, you can neatly classify the data (which is what Dominika does in the photo), you can apply the abstraction of five time leaks and automate.
2. Consolidate Work Management Systems: Task Management
The archetypes I'm talking about are a kind of pyramid. If the first is done correctly, then the second is already a kind of add-on. Many of these don't work for startups, but should be kept in mind for large companies like the Fortune 5000. The last company I worked for had 10 ticketing systems. One team had Remedy, another wrote some kind of system of its own, a third used Jira, someone completely dispensed with email. The same problem arises if the company has 30 different pipelines, but I do not have time to discuss all such cases.
I discuss with people exactly how tickets are created, what happens to them next, how they are bypassed. The most interesting thing is that people at our meetings speak quite sincerely. I asked how many people are setting "minor / no impact" for tickets that should have been given "major impact". It turned out that almost everyone does this. I am not engaged in denunciation and in every possible way I try not to identify people. When someone sincerely confesses something to me, I do not betray the person. But when virtually everyone bypasses the system, it means that all security is, in essence, a decoration. Therefore, no conclusions can be drawn from the data of this system.
To solve the issue with tickets, you need to select one main system. If you use Jira, just use Jira. If there is any alternative, let it be only that. The bottom line is that tickets should be seen as just another step in the development process. Any activity must have a ticket that must go through the development workflow. Tickets are sent to the team, who post them on the storyboard and are then responsible for them.
This applies to all departments, including infrastructure and operations. In this case, you can get at least some plausible idea about the state of affairs. Once this process is in place, it suddenly becomes easy to establish who is responsible for each application. Because now we get not 50%, but 98% of new services. If this basic process works, then the accuracy improves throughout the system.
Services pipeline
Again, this only applies to large corporations. If you are a new company in a new field, roll up your sleeves and work with your Travis CI or CircleCI. As far as Fortune 5000 companies are concerned, the case that happened with the bank where I worked is indicative. Google came to them and showed them diagrams of old IBM systems. The guys from Google asked with incomprehension - where is the source code for this? And there is no source code, not even a GUI. This is the reality that large organizations have to deal with: 40-year-old bank records on an ancient mainframe. One of my clients uses Kubernetes containers with Circuit Breaker patterns, plus Chaos Monkey, all for the KeyBank application. But these containers are eventually connected to the COBOL application.
The guys from Google were completely sure that they would solve all the problems of my client, and then they began to ask questions: what is an IBM datapipe? They answer: it's a connector. What does she connect to? To the Sperry system. And what's that? And so on. At first glance, it seems: what kind of DevOps can be here? But actually, it's possible. There are delivery systems that allow you to outsource the workflow to delivery teams.
3. Theory of Constraints: Theory of Constraints
Let's move on to the third archetype: institutional/"tribal" knowledge. As a rule, in any organization there are several people who know everything and manage everything. These are the ones who have been in the organization the longest and who know all the workarounds.
When this is revealed on the diagram, I specifically circle such people with a marker: for example, it turns out that a certain Lou is present at all meetings. And it's clear to me: this is a local Brent. When the CIO chooses between me in a T-shirt and sneakers and a suit-clad IBM guy, I'm chosen because I can tell the CIO things that the other guy won't and the CIO might not like to hear about. I tell them the bottleneck in their company is someone named Fred and someone named Lou. This bottleneck needs to be unleashed, their knowledge needs to be extracted from them one way or another.
To solve this kind of problem, I can, for example, suggest using Slack. A smart director will ask - why? Usually in such cases, DevOps consultants answer: because everyone does it. If the director is really smart, he will say: so what. And that's where the dialogue ends. And my answer is: because there are four bottlenecks in the company, Fred, Lou, Susie and Jane. To make their knowledge institutionalized, one must first introduce Slack. All your wikis are complete rubbish because no one knows they exist. If the engineering team is doing external and internal development and everyone needs to know that you can contact the external development team or the infrastructure team with questions. That's when Lou or Fred will probably have time to connect to the wiki. And then someone in Slack might ask why, say, step 5 doesn't work. And then Lou or Fred will fix the instructions on the wiki. If you set up this process, then a lot of things will fall into place by themselves.
This is my main idea: in order to recommend any high technologies, you must first put in order the foundation for them, and this can be done with the low-tech solutions just described. If you start with high technologies and do not explain why they are needed, then, as a rule, it does not end with anything good. One of our clients is using Azure ML, a very cheap and easy solution. Somewhere around 30% of their questions were answered by the self-learning machine itself. And this thing was written by operators who were not involved in data science, statistics or mathematics. This is indicative. The cost of such a solution is minimal.
4. Collaboration hacks: Collaboration hacks
The fourth archetype is that isolation must be fought. Most people already know this: isolation breeds enmity. If each department is on its own floor, and people do not intersect with each other in any way, except in the elevator, then enmity between them arises very easily. And if, on the contrary, people are in the same room with each other, she immediately leaves. When someone throws a certain general accusation, for example, such and such an interface never works, there is nothing easier to deconstruct such an accusation. The programmers who have written the interface need only start asking specific questions, and it will soon become clear that, for example, the user simply used the tool incorrectly.
There are many ways to overcome isolation. I was once asked to advise a bank in Australia, I refused to do this because I have two children and a wife. All I could do to help them was to recommend graphical storytelling to them. This is something that is proven to work. Another interesting way is lean coffee meetings. In a large organization, this is a great way to disseminate knowledge. In addition, you can hold internal devopsdays, hackathons, and so on.
5. Coaching Kata
As I warned at the very beginning, today I will not talk about it. If you are interested, you can see .
There is also a good talk on the subject by Mike Rother:
6. Market Oriented: a market-oriented organization
There are different problems here. For example, "I" people, "T" people, and "E" people. "I" people are those who do only one thing. They usually exist in organizations with isolated divisions. "T" is if a person is good at one thing, but also excels at some other things. "E" or even "comb" is when a person has a lot of skills.
This is where Conway's law comes in.), which in the most simplified form can be stated as follows: if three teams are involved in the compiler, then the result will be a three-part compiler. Therefore, if there is a high level of isolation inside the organization, then even Kubernetes, Circuit breaker, API extensibility and other fancy things in this organization will be arranged in the same way as the organization itself. Strictly according to Conway and to spite all of you young geeks.
The solution to this problem has been described many times. There are, for example, organizational archetypes described by Fernando Fernandez. The problematic architecture I was just talking about, with isolation, is function-oriented architecture. The second type is the worst, matrix architecture, there is a mess of the other two. The third is what is seen in most startups, and large companies also try to match this type. It is a market-oriented organization. Here we are optimizing to achieve the fastest response to customer requests. This is sometimes referred to as flat organization.
Many people describe this structure in different ways, I like the wording build/run teams, in Amazon they call it two pizza teams. In this structure, all "I" people are grouped around one service, and gradually they become closer to the "T" type, and if the right management is established, they can even become "E". The first counter-argument here is that there are unnecessary elements in such a structure. Why do you need a tester in every department when you can have a dedicated department of testers? To which I answer: the extra cost in this case is the price for the whole organization to become an “E” type in the future. In this structure, the tester gradually learns about networks, architecture, design, and so on. As a result, each member of the organization is fully aware of everything that happens in the organization. If you want to know how this scheme works in industry, read .
7. Shift-left auditors: audit early in the cycle. Compliance with safety rules for show
This is when your actions do not pass, so to speak, the smell test. The people who work for you are not stupid. If, as in the example above, they set minor / no impact everywhere, this went on for three years, and no one noticed anything, then everyone knows perfectly well that the system is not working. Or another example is the change advisory board, where every, say, environment, you need to submit reports. There is a group of people working there (not very well paid, by the way) who, in theory, should know how the system as a whole works. And over the past five years, you've probably noticed that our systems are insanely complex. And five or six people have to decide on a change that they didn't make and that they don't know anything about.
Of course, this approach does not work. I have to get rid of such things, because these people do not protect the system. The decision must be made by the team itself, because the team must be responsible for it. Otherwise, a paradoxical situation arises when a manager who has never written code in his life tells the programmer how long it should take to write code. One company I worked with had 7 different boards that looked at every change, including an architecture board, a product board, and so on. There was even a mandatory waiting period, although one employee told me that in ten years of work, no one ever rejected the changes made by this person during this mandatory period.
Auditors need to be called to yourself, and not get rid of them. Tell them that you are writing immutable binary containers that, if all tests pass, remain unchanged forever. Tell them that you have pipeline as code and explain what that means. Show them the following scheme: an immutable read-only binary in a container that passes all vulnerability tests; and then not only no one touches it - they don’t even touch the system that creates the pipeline, since it is also created dynamically. I have clients, Capital One, who are using Vault to create something like a blockchain. The auditor does not need to be shown “recipes” from Chef, it is enough to show the blockchain, from which it is clear what happened to the Jira ticket in production and who is responsible for it.
According to , created in 2018 by Sonatype, had 2017 billion OSS download requests in 87.
The losses incurred due to vulnerabilities are prohibitive. Moreover, the figures that you now see above do not include opportunity costs. In a nutshell about what DevSecOps is. I want to say right away that I'm not interested in talking about how successful this name is. The point is that since DevOps has been very successful, you should try to add security to this pipeline.
An example of such a sequence:
This is not a recommendation for specific products, although I like them all. I gave them as an example to show that DevOps, originally based on the paradigm of organization in industry, allows you to automate every stage of working on a product.
And there's no reason why we couldn't take the same approach to security.
Сonclusion
As a conclusion, I will give some tips for DevSecOps. You need to include auditors in the process of creating your systems, spend time on their education. Auditors need to cooperate. Further, it is necessary to conduct an absolutely ruthless fight against false positives. Even with the most expensive vulnerability scanning tool, you can end up creating extremely bad habits with your developers if you don't know what the signal-to-noise ratio is. Developers will be overwhelmed with events and will simply delete them. If you've heard of the Equifax story, that's pretty much what happened there, where the signal of the highest level of danger was ignored. In addition, vulnerabilities should be explained in a way that makes it clear how they affect the business. For example, you can say that this is the same vulnerability as in the Equifax story. Security vulnerabilities should be treated in the same way as other software issues, that is, they should be included in the overall DevOps process. You need to work with them through Jira, Kanban, etc. Developers should not think that someone else will do it - on the contrary, everyone should do it. Finally, you need to spend energy on educating people.
Useful links
Here are a few talks from the DevOops conference that you might find useful:
- Sergey Berdnikov, Artyom Kalichkin - Success story, or "Dev+DevOps+Ops" (, )
- Baruch Sadogursky, Leonid Igolnik — DevOps on a Scale: Greek Tragedy in Three Acts (, )
- Alexander Titov, Kirill Tolkachev —
- Timothy Lister-
Look in DevOops 2020 Moscow - there is also a lot of interesting things.
Source: habr.com