The definition of DevOps is very complex, so you have to start the discussion about it again every time. Only on Habré there are a thousand publications on this topic. But if you are reading this, then you probably know what DevOps is. Because I am not. Hello my Name Is Alexander Titov (@) and we'll just talk about DevOps and I'll share my experience.
I thought for a long time how to make my story useful, so there will be many questions here - those that I ask myself, and those that I ask the clients of our company. By answering these questions, understanding becomes better. I will tell you why DevOps is needed from my point of view, what it is, again, from my point of view, and how to understand that you are moving towards DevOps again from my point of view. The last point will be through questions. By answering them to yourself, you can understand if your company is moving towards DevOps or if there are problems in some way.
At one time I walked the waves of mergers and acquisitions. First, I worked at a small startup called Qik, then it was bought by a slightly larger company, Skype, which was later bought by a slightly larger company, Microsoft. At that moment, a vision became available to me of how the idea of DevOps is being transformed in companies of different sizes. After that, it became interesting for me to look at DevOps from the point of view of the market, and my colleagues and I organized the company Express 42. For 6 years we have been moving along the waves of the market.
Among other things, I am one of the organizers of the DevOps Moscow community and the organizer of DevOps -Days 2017, but I did not organize 2018. Express 42 works with many companies. We grow DevOps there, watch how it happens, draw conclusions, analyze, tell our findings to everyone, train people in DevOps practices. In general, in every possible way we are growing experience and expertise in this sense.
Why DevOps
The first question that haunts everyone and always - why? Many people think that DevOps is just automation or something similar that every company already had.
- We had Continuous Integration - this means we already had DevOps, and why do we need all this tops? They have fun abroad, but they interfere with our work!
For 9 years of community and methodology development, it has already become clear that this is still not marketing glitter, but it is still not entirely clear why it is needed. Like any tool and process, DevOps has specific goals that it ultimately achieves.
All this is due to the fact that the world is changing. He moves away from the enterprise approach, when companies move straight to the dream, as our St. Petersburg classic sang, from point A to point B according to a certain strategy, with a certain structure built for this.
In principle, everything in IT should be built under this approach. Here IT is used exclusively for process automation.
Automation does not often change, because when a company is rolling along a well-worn rut - what is there to change? It works, don't touch it. Now approaches in the world are changing, and the one that is called the word Agile says that the end point B is not immediately visible.
When a company goes through the market, works with a client, it constantly explores the market and changes the end point B. Moreover, the more often the company changes its direction, the more successful it is in the end, because it chooses more market niches.
The strategy is demonstrated by an interesting company that I recently learned about. One Box Shave is a delivery service for razors and shaving accessories by subscription in a box. They know how to customize their "box" for different clients. This is done by a certain software, which then sends the order to the Korean factory that produces the goods.
This product was bought by Unilever for $1 billion. Now it competes with Gillette and has eaten away a significant share of consumers in the American market. One Box Shave says:
- 4 blades? Are you seriously? Why do you need it - it does not improve the quality of shaving. Specially selected cream, fragrance and a quality razor with two blades solve much more problems than these stupid 4 Gillette blades! So soon we will reach 10?
So the world is changing. Unilever claims to have a cool IT system that allows them to do this. In the end it looks like a concept time to market, about which no one has already spoken.
Time-to-market is not about how often we deploy. You can often deploy, but the release cycles will be long. If the three-month release cycles are superimposed on each other, shifting by a week, it turns out that the company seems to be deploying once a week. It takes 3 months from idea to final implementation.
Time-to-market is about minimizing the time from idea to final implementation.
In this case, software interacts with the market. This is how One Box Shave's website interacts with the client. They do not have sellers - just a site where the visitor clicks and leaves wishes. Accordingly, the site must constantly post something new, update it in accordance with the wishes. For example, in South Korea they shave differently than in Russia, and they don’t like the smell of pine in the fragrance, but, for example, vanilla carrots.
Since the content of the site needs to change quickly, software development is changing a lot. Through the software, we must find out what the client wants. Previously, we learned this by some workarounds, for example, through business management. Then we designed, laid down the requirements in the IT system, and everything was cool. Now it’s different - software is designed by everyone who is involved in the process, including engineers, because they learn how the market works through technical specifications and also share their insights with business.
For example, at Qik, we suddenly found out that people really like to upload contact lists to the server, and they put us an application. Initially, we did not think about it. In a classic company, everyone would decide that this is a bug, since the spec does not say that this should work great and was generally implemented on the knee, they would turn off the feature and say: “Nobody needs this, the most important thing is that the main functionality works” . And the technology company sees an opportunity in this and starts changing the software in accordance with this.
In 1968, the visionary guy Melvin Conway formulated the following idea.
An organization that creates a system is constrained by a design that replicates the structure of communication in that organization.
In more detail, in order to produce systems of a different type, one must also have a communication structure within a company of a different type. If your communication structure is top hierarchical, then this will not allow you to make systems that can provide a very high Time-to-market.
Read can . It is important to understand the DevOps culture or philosophy, as the only thing that changes fundamentally in DevOps is the structure of communication between teams.
In terms of process, before DevOps, all stages: analytics, development, testing, operation, took place linearly.
In the case of DevOps, all these processes go on simultaneously.
Time-to-market is the only way to do it. For people who worked in the old process, it looks a bit cosmic, and generally so-so.
So why do we need DevOps?
For digital product development. If your company does not have a digital product, DevOps is not needed - this is very important.
DevOps overcomes the speed limits of serial software production. In it, all processes occur simultaneously.
Difficulty is increasing. When DevOps evangelists say that it will make it easier for you to release software, this is nonsense.
With DevOps, things will only get more difficult.
At the conference at the Avito booth, you could see what it means to deploy a Docker container - an unrealistic task. The complexity becomes prohibitive, you have to juggle many balls at the same time.
DevOps completely changes the process and organization in the company - more precisely, it is not DevOps that changes, but a digital product. To come to DevOps, you still need to completely change this process.
Questions for a specialist
What do you have? Questions you can ask yourself while working in a company and developing as a specialist.
Do you have a strategy for creating a digital product? If there is, it's good. This means that your company is moving towards DevOps.
Is your company already creating a digital product? This means that you can climb even one step higher, doing things more interesting - from the point of view of DevOps again. I only speak from this point of view.
Is your company one of the market leaders in the digital product niche? Spotify, Yandex, Uber are companies that are at the peak of technological progress now.
Ask yourself these questions, and if all the answers are no, then maybe you shouldn't be doing DevOps at this company. If you're really interested in DevOps, maybe… should you move to another company? If your company wants to go DevOps, but you answered “No” to all the questions, then it is like this beautiful rhinoceros who will never change.
Organization
As I said, according to Conway's law, the organization changes in the company. To begin with, what prevents DevOps from penetrating inside the company is from the point of view of the organization.
The problem of "wells"
The English word "Silo" is translated here into Russian as "well". The point of this problem is that there is no exchange of information between teams. Each team digs its expertise in depth, while not building a common map in which to navigate.
In some ways, this is reminiscent of a person who has just arrived in Moscow and still does not know how to navigate the metro map. Muscovites usually know their area very well, and throughout Moscow they are guided by the metro map. When you come to Moscow for the first time, you don't have this skill and you're just disoriented.
DevOps proposes to go through this moment of disorientation and build a common interaction map for all departments together.
Two factors hinder this.
Consequence of the corporate management system. It is built by separate hierarchical "wells". For example, there are certain KPIs in companies that support this system. On the other hand, the brains of a person who find it difficult to go beyond their expertise and navigate the entire system interfere. It's just uncomfortable. Imagine that you are at the Bangkok airport - you will not quickly find your bearings there. DevOps is hard to navigate too, which is why people say you have to find a guide to get there.
But the most important thing is that the problem of "wells" for an engineer who is imbued with the spirit of DevOps, read Fowler and a bunch of other books, is expressed in the fact that "wells" do not allow you to do "obvious" things. We often get together after DevOps Moscow, talk to each other, and people complain:
- We just wanted to launch CI, but it turned out that the management did not need it.
This happens precisely because CI и continuous delivery process are on the borderline of many expertise. Just without overcoming the problem of "wells" at the organizational level, you will not be able to move forward, no matter what you do and no matter how sad it is.
Each participant in the process in the company: backend and frontend developers, testing, DBA, operation, network, digs in their own direction, and no one has a common map, except for the manager, who somehow oversees them and manages the “divide and conquer” method.
People are fighting for some kind of stars or flags, everyone is digging their own expertise.
As a result, when the task arises to connect all this together and build a common pipeline, and there is no need to fight for asterisks and flags, the question arises - what to do in general? We need to negotiate somehow, but no one taught us at school how to do it. We have been taught since school: eighth grade - wow! compared to seventh grade! It's the same here.
Is it the same in your company?
To test this, you can ask yourself the following questions.
Do the teams use common tools and contribute to changes in these common tools?
How often do teams re-form - some specialists from one team move to another team? It is in a DevOps environment that this becomes normal, because sometimes a person simply cannot understand what another area of expertise is doing. He moves to another department, works there for two weeks to create for himself a map of orientation and interaction with this department.
Is it possible to form a change committee and change something? Or does this require a strong hand of the highest leadership and an order? Recently, I wrote on Facebook how one little-known bank is implementing tools through orders: we wrote an order, we are implementing it for a year, we look at what happens. This, of course, is long and sad.
How important is it for managers to receive personal achievements without taking into account the achievements of the company?
If you answer these questions for yourself, it will become clearer whether you have such a problem in the company.
Infrastructure as code
After this problem is passed, the first important practice, without which it is difficult to move forward in DevOps, is infrastructure as code.
Most often, infrastructure as code is perceived as follows:
- Let's automate everything in bash, cover ourselves with scripts so that admins have less manual work!
But it's not.
Infrastructure as code means that you describe the IT system you work with in the form of code in order to constantly understand its state.
Together with other teams, you create a map in the form of a code that is understandable to everyone, by which you can navigate and navigate. It doesn't matter what it's done on - Chef, Ansible, Salt, or using YAML files in Kubernetes - it doesn't matter.
At the conference, a colleague from 2GIS told how they made their own internal thing for Kubernetes, which describes the structure of individual systems. To describe 500 systems, they needed a separate tool that generates this description. When there is this description, everyone can check with each other, monitor changes, how to change and improve it, what is missing.
Agree, individual bash scripts usually do not give this understanding. In one of the companies where I worked, there was even a name “write only” - a script - when the script is written, but it is no longer possible to read it. I think this is familiar to you too.
Infrastructure as code is code that describes the current state of the infrastructure. Many product, infrastructure, and service teams work together on this code, and, most importantly, they all need to understand how this code generally works.
Code is maintained according to best coding practices: collaborative development, code reviews, XP programming, testing, pull requests, CI for code infrastructures - it's all good and usable.
Code becomes the common language for all engineers.
Changing the infrastructure in the code does not take much time. Yes, infrastructure code can also have technical debt. Usually, teams encounter it a year and a half after they started implementing “infrastructure as code” in the form of a bunch of scripts or even Ansible, which they write like spaghetti code, and they throw bash scripts into the top!
It's important: if you have not tried this rubbish, remember that Ansible is not bash! Carefully read the documentation, study what they generally write about it.
Infrastructure as code is the division of infrastructure code into separate layers.
In our company, we distinguish 3 basic layers, which are very clear and simple, but there can be more. You can look at your infrastructure code and tell if you have this condition or not. If no layers are selected, then you need to take the time and refactor a little.
base layer - this is how the OS is configured, backups and other low-level things, for example, how Kubernetes is deployed at a basic level.
Service layer - these are the services that you give to the developer: logging as a service, monitoring as a service, database as a service, balancer as a service, queue as a service, Continuous Delivery as a service - a bunch of services that individual teams can provide to development. All this needs to be described by separate modules in your configuration management system.
Layer where applications are made and describes how they will unfold on top of the previous two layers.
test questions
Do you have a shared infrastructure repository in your company? Do you control infrastructure technical debt? Do you use development practices in the infrastructure repository? Is your infrastructure sliced into layers? You can refer to the Base-service-APP schema. How difficult is it to make a change?
If you have experienced that it took a day and a half to make changes, this means that you have a technical debt and you need to work with it. You have just stumbled upon a rake of technical debt in infrastructure code. I remember many such stories, when in order to change some CCTL, you need to rewrite half of the infrastructure code, because creativity and the desire to automate everything led to the fact that everything is bugged everywhere, all handles are removed, and you need to refactor.
Continuous supply
Let's compare a debit with a credit. First comes the description of the infrastructure, which can be quite basic. It is not necessary to describe everything in detail, but some basic description is required so that you can work with it. Otherwise, it is not clear on top of what further continuous delivery should be done. All these practices unfold at the same time when you come to DevOps, but you need to start by understanding what you have and how to manage it. This is just the practice of infrastructure as code.
After it became clear that you have how to manage it, you begin to figure out how to send the developer's code to production as quickly as possible. I mean together with the developer - we remember the problem of "wells", that is, not individuals come up with this, but a team.
When we are with Vanya Evtukhovich saw the first book Jeza Humble and groups of authors "Continuous Delivery", which was released in 2009, we thought for a long time how to translate its name into Russian. They wanted to translate it as "Constantly deliver", but, unfortunately, they translated it as "Continuous supply". It seems to me that in our name there is something so Russian, with pressure.
Consistently delivering means
The code that is in the product repository can always be rolled out to production. He may not be pumped out, but he is always ready for it. Accordingly, you always write code with a hard-to-explain feeling of some anxiety under the coccyx. It often appears when you roll out infrastructure code. This feeling of some unease should be present - it triggers brain processes that allow you to write code a little differently. This should be fixed in the rules within the development.
To deliver consistently, you need an artifact format that traverses the infrastructure platform. If you throw “life waste” across an infrastructure platform of various formats, then it becomes unified, it is difficult to maintain, and the problem of technical debt arises. The format of the artifact needs to be aligned - this is also a collective task: we all need to get together, rustle our brains and come up with this format.
The artifact is continuously improved and changed to suit the production environment as it moves through the delivery pipeline. When an artifact moves along the pipeline, it constantly encounters some things that are inconvenient for it, which are similar to what an artifact that you put into production encounters. If in classical development this is done by a system administrator who rolls out, then in the DevOps process this happens all the time: here they twisted it with some tests, then they threw it into the Kubernetes cluster, which is more or less similar to production, then they suddenly launched load testing .
This is somewhat reminiscent of the game Pacman - the artifact goes through some kind of story. At the same time, it is important to control whether the code actually goes through history and whether it is somehow connected with your production. Stories from production can be dragged inside the Continuous Delivery process: it was like this when something fell, now let's just program this scenario inside the system. Each time the code will go through this scenario too, and you won't run into this problem next time. You will know about it much earlier than it goes to your client.
Different deployment strategies. For example, you use AB testing or canary deployments to “feel” the code differently on different clients, get information about how the code works, and much earlier than when it rolls out to 100 million users.
"Constantly deliver" looks like this.
The Dev, CI, Test, PreProd, Prod delivery process is not a separate environment, these are stages or stations with fireproof amounts through which your artifact passes.
If you have infrastructure code that is described as Base Service APP, then it helps don't forget all scripts, and write them down as code for this artifact, too, promote an artifact and change it as you go.
Self-test questions
Is the time from feature description to rollout to production less than a week in 95% of cases? Does the quality of the artifact increase at each stage of the pipeline? Is there a story that he follows? Do you use different deployment strategies?
If all the answers are yes, then you are unrealistically cool! Write the answers in the comments - I will be glad).
Contact Us
This is the hardest practice of all. At the DevOpsConf conference, a colleague from Infobip, talking about it, was a little confused in words, because this is really a very difficult practice about the fact that you need to monitor everything in general!
For example, a long time ago, when I worked at Qik and we realized that we need to monitor everything in general. We did it, and we have 150 items in Zabbix, which are constantly monitored. It was scary, the technical director twisted his finger at his temple:
- Guys, why are you raping a server with some unknown reason?
But then an incident occurred that showed that this is really a very cool strategy.
One of the services constantly started to fall. Initially, it did not crash, which is interesting, the code was not added there, because it was a basic broker in which there was practically no business functionality - it just sent messages between separate services. The service did not change for 4 months, and suddenly it began to crash with a “Segmentation fault” error.
We were shocked, opened our charts in Zabbix, and it turned out that, it turns out, a week and a half ago, the behavior of requests in the API service that this broker uses has changed a lot. Next, we saw that the frequency of sending a certain type of message has changed. After we found out that these are android clients. We asked:
- Guys, what happened to you a week and a half ago?
In response, they heard an interesting story that they redesigned the UI. It is unlikely that anyone will immediately say that they have changed the HTTP library. For android clients, it's like changing the soap in the bathroom - they just don't remember it. As a result, after 40 minutes of conversation, we found out that they did change the HTTP library, and its default timings changed. This led to the fact that the traffic behavior on the API server changed, which led to a situation that caused a race inside the broker, and it began to fall.
Without deep monitoring it is impossible to open it at all.. If the organization still has the problem of “wells”, when everyone throws at each other, this can live for years. You just restart the server because there is no way to fix the problem. When you monitor, track, track all the events that you have, and use monitoring as testing - write code and immediately indicate how to monitor it, also in the form of code (we already have the infrastructure as code), everything becomes clear as on the palm. Even such complex problems are easily tracked.
Collect all the information about what happens with the artifact at each stage of the delivery process - not in production.
Load monitoring on CI, and some basic things will already be visible there. Then you will see them in Test, and in PredProd, and in load testing. Collect information at all stages, and not only metrics, statistics, but also logs: how the application rolled out, anomalies - collect everything.
Otherwise, it will be difficult to understand. I have already said that DevOps is more complex. To cope with this complexity, you need to have normal analytics.
Questions for self-control
Is your monitoring and logging the development tool for you? Do your developers, including you, when they write code, think about how to monitor it?
Do you hear about problems from customers? Do you understand the client better from monitoring and logging? Do you understand the system better from monitoring and logging? Do you change the system simply because you saw that the trend in the system is growing and you understand that another 3 weeks and everything will die?
Once you have these three components in place, you can think about what kind of infrastructure platform you have in your company.
infrastructure platform
The point is not that this is a set of disparate tools that every company has.
The point of the infrastructure platform is that all teams use these tools and develop them together.
It is clear that there are separate teams that are responsible for the development of individual pieces of the infrastructure platform. But at the same time, each engineer is responsible for the development, performance, and promotion of the infrastructure platform. At the internal level, it becomes a common tool.
All teams develop the infrastructure platform, treat it with care as if it were their own IDE. In your IDE, you install different plugins to make everything nice and fast, and set up hotkeys. When you open Sublime, Atom or Visual Studio Code, you have code errors pouring in there and you realize that it’s impossible to work at all, you immediately feel sad and you run to fix your IDE.
Treat your infrastructure platform the same way. If you understand that something is wrong with it, leave a request if you cannot fix it yourself. If there is something simple - edit it yourself, send a pull request - the guys consider it, add it. This is a slightly different approach to the engineering toolkit in the developer's head.
The infrastructure platform ensures the transfer of the artifact from development to the client with continuous quality improvement. The UI is programmed with a set of stories that happen to code in production. Over the years of development, these stories become very numerous, some of them are unique and apply only to you - they cannot be googled.
At this point, the infrastructure platform becomes your competitive advantage., because it contains something that is not in the competitor's tool. The deeper your IP, the greater your competitive advantage in terms of Time-to-market. Appears here vendor lock problem: you can take someone else's platform, but using someone else's experience, you will not understand how relevant it is to you. Yes, not every company can build an Amazon-like platform. This is a difficult line where the company's experience is relevant to its position in the market, and vendor lock cannot be released there. This is also important to think about.
scheme
This is a basic blueprint for an infrastructure platform that will help you set up all the practices and processes in a DevOps company.
Let's see what it consists of.
Resource orchestration system, which provides CPU, memory, disk to applications and other services. On top of this - low level services: monitoring, logging, CI / CD Engine, artifact storage, infrastructure as a system code.
higher level services: database as a service, queues as a service, Load Balance as a service, image resizing as a service, Big Data factory as a service. On top of this - pipeline that delivers constantly modified code to your client.
You get information about how your software works for the client, change it, deliver this code again, get information - and so you constantly develop both the infrastructure platform and your software.
In the scheme, the delivery pipeline consists of many stages. But this is a schematic diagram, which is given as an example - you do not need to repeat it one by one. Stages interact with services as with services - each brick of the platform has its own story: how resources are allocated, how the application is launched, works with resources, is monitored, changes.
It is important to understand that each part of the platform carries a story, and ask yourself what story this brick carries, maybe it should be thrown away and replaced with a third-party service. For example, is it possible to put Okmeter instead of a brick? Perhaps the guys have already pumped this expertise much more than we do. But maybe not - maybe we have a unique expertise, we need to install Prometheus and develop it further.
Creation of the platform
This is a complex communication process. When you have base practices, you start communication between different engineers and specialists who develop requirements and standards, and constantly change them to different tools and approaches. The culture that DevOps has is important here.
With culture, everything is very simple - is cooperation and communication, that is, the desire to work in a common field with each other, the desire to own one instrument together. There is no rocket science here - everything is very simple, trite. For example, we all live in the entrance and keep it clean - such a level of culture.
What do you have?
Again, questions you can ask yourself.
Is there a dedicated infrastructure platform? Who is responsible for its development? Do you understand the competitive advantages of your infrastructure platform?
These are the questions you need to ask yourself all the time. If something can be moved to third-party services, you need to take it out; if a third-party service starts blocking your movement, then you need to build a system within yourself.
So DevOps...
... this is a complex system, it should have:
- digital product.
- Business modules that develop this digital product.
- Product teams that write the code.
- Continuous Delivery practices.
- Platforms as a service.
- Infrastructure as a service.
- Infrastructure as code.
- Separate reliability practices hardwired inside DevOps.
- Feedback practice that describes it all.
You can use this scheme by coloring in it what you already have in your company in some form: it has developed or still needs to be developed.
It'll be gone in a couple of weeks . as part of RIT++. Come to the conference, where you will find a lot of cool talks about continuous delivery, infrastructure as code and DevOps transformation. , last price deadline May 20
Source: habr.com