On December 7, the third conference was held , organized by the Moscow DevOps community with the support of Mail.ru Cloud Solutions. In addition to reports from leading DevOps practitioners, participants could attend short motivating Lightning Talks, workshops and chat in open spaces.
We collected important insights from six presentations and interviewed several speakers to find out what was left behind the presentations.
Inside:
- Baruch Sadogursky, JFrog: "Let software flow from vendor to user like a liquid"
- Pavel Selivanov, Southbridge: "Dev and Ops have one common task - to make a product that works"
- Vladimir Utratenko, X5 Retail Group: “DevOps in Enterprise is development without pain and fires”
- Sergey Puzyrev, Facebook: "Production Engineer cares about the service as a whole: so that both users and developers feel good"
- Mikhail Chinkov, AMBOSS: “One department cannot follow the DevOps path, the whole company must follow it”
- DevOps enthusiasts of Rosbank: "1000 days to implement DevOps in a bloody enterprise"
1. Baruch Sadogursky, JFrog: “Let software flow from vendor to user like a liquid”
Fails when updating software happen hourly and for everyone. Here is just one scary story from the speech: Knight Capital lost $440 million in an hour after an unsuccessful upgrade.
Baruch spoke about DevOps patterns of continuous updates that will help avoid failures and user hatred:
Local rollback - keep the previous version of the software on the device to roll back if something happens. This will protect you if things get so bad that you can't send the patch over the air.
Over the air updates - continuous is better. It will be different, as with the Jaguar developers: due to a bug in the brake system, which could not be updated over the air, the cars had to be withdrawn from sale. It was painful and expensive.
Continuous updates - update the software continuously as soon as a new feature is ready. With rare updates, features are grouped, a critical update can wait for non-critical ones. Like in Tesla - an update that was supposed to fix random braking was waiting for an update to the chess game.
Automated deployment - replace people with machines, as people are poorly able to perform routine actions.
Frequent updates - help to develop a habit and get rid of fear. Rare updates turn into an emergency event.
Knowing the state of the device - test updates, not installation from scratch. This is important because updates can behave differently depending on the state of the device.
Canary releases - roll out updates to a small number of users and observe. This reduces the damage if something goes wrong.
Updates without unavailability - let customers notice only new features, and not be left without service for several weeks while you roll out the update.
We talked with Baruch Sadogursky about how the view of DevOps differs in Russian and Western IT, whether Cloud will soon do everything for us, and whether all software services will slide into the aaS scheme - see the interview:
2. Pavel Selivanov, Southbridge: "Dev and Ops have one common goal - to make a product that works"
The introduction of Kubernetes will not help achieve DevOps, and on the contrary, it can break everything. Pavel explained why technology (even the coolest) cannot solve all your problems:
The complexity of the project has moved beyond the code. Previously, there was a complex application: interaction within the project and complex development, but a simple structure - the administrator deployed it, everything works. We switched to microservices: each service is a simple application, communication using standard protocols and fast development, but the project structure has become more complicated. The complexity of the project with microservice architecture has not gone away - it has moved beyond the code. Now a DevOps engineer is responsible for it.
Developers do not want changes after implementing DevOps. As a result, the workflow with Kubernetes continues to look like throwing tasks from Dev to Ops over a wall, only not a metaphorical one anymore - Git becomes such a wall. The developer puts the code there and works as before, and the admins have Kubernetes, CI / CD and everything else.
However, developers need to accept the changes. The situation when the developers do not know what the admins are doing, and the admins do not know what the developers are doing, creates problems.
If the developers have not changed anything, they do not realize that the operation of the application is their responsibility - various technical tricks will not work.
With the advent of DevOps and Kubernetes, a lot will change in development. Dev must be competent in Ops, and vice versa. These specialists have their own specific skills, but they must be aware of each other's work. Dev and Ops need to be friends before the introduction of Kubernetes, otherwise you will break what you have.
Pavel Selivanov talked about what will happen to Kubernetes in 5 years and what a modern startup should build a technology stack on - watch the interview:
3. Vladimir Utratenko, X5 Retail Group: “DevOps in Enterprise is development without pain and fires”
Companies come to DevOps transformation when they realize that development is too slow and does not correspond to realities, they have a desire to develop better and roll out faster.
Vladimir told how it happens, and what's the catch:
- First, companies hire a DevOps engineer. This is a Senior System Administrator, he is involved in deploying a release to production, standardizing the development environment, setting up the infrastructure, detecting and fixing various problems, automating processes and other technical tasks.
- Then one DevOps engineer is no longer enough, and the company hires a DevOps team. This is a center of competence that organizes the efforts of disparate engineers and allows them to focus in one direction.
- In fact, the DevOps engineer and DevOps teams are the anti-patterns of DevOps transformation. Since DevOps is about practices and culture, in addition, there are DevOps implementations in technology companies (SRE, Production Engineering).
What to do? Hire a temporary DevOps team that will help implement the DevOps transformation, will carry practices, grow development culture and technology culture.
When a business comes into play and invests in DevOps, there are several possible scenarios: everything will fall apart on takeoff; will remain as SRE/Production Engineering or Embedded Ops; will move to BizOps when processes are based on business metrics.
Vladimir Utratenko told us about how “bloody” DevOps really is in the enterprise and how practices are implemented within large retail — see the interview:
4. Sergey Puzyrev, Facebook: “Production Engineer cares about the service as a whole: so that both users and developers feel good”
Facebook is a huge company, with a lot of components, servers, people, data centers. Despite its huge size, it is very fast - developers can roll out services many times a day. Also, Facebook is growing rapidly, and it's not just the growth in the number of users and servers, the number of developers is also increasing, which complicates the processes.
Sergey told what a Production Engineer does on Facebook:
- A Production Engineer does a lot of coding, he must have system knowledge: operating systems, file systems, databases, networks and the like. Must have experience with distributed systems and Reliability Engineering, that is, in supporting the reliability of the product. It must be on-call, that is, available for a call at any time.
- Production Engineer differs from Software Engineer in advanced skills in operation, but, in fact, is a subspecies of Software Engineer. Software Engineers code more, they may have additional skills related, for example, to data processing. In Facebook, such specialists must also be on-call, which comes as an unpleasant surprise for many.
- The pyramid of needs for a Production Engineer in a company begins with monitoring servers and their life cycle, that is, obtaining new hardware, configuring it, putting it into operation. The next level is the same at the level of services: monitoring services and their life cycle. Then comes seamless scaling and advanced monitoring. They switch to autoscaling after the life cycle of the service is automated. And in the end, you need to deal with tuning, so that scaling is effective, the company saves money and resources.
5. Mikhail Chinkov, AMBOSS: “One department cannot follow the DevOps path, the whole company must follow it”
Mikhail believes that DevOps is a continuous development. You can not introduce some tools and stop there. What problems prevent companies from becoming DevOps and how to implement practices?
Understanding DevOps Difference. The canonical devops, as evangelists see it, rests on 5 pillars:
- culture — focus on people and collaboration;
- automation - delegation of routine to workflow;
- lean - emphasis on delivering value to the user;
- sharing - continuous exchange of knowledge;
- metrics and receiving feedback with their help.
Companies usually focus only on automation and delivering value to the user. But the culture, knowledge sharing, and DevOps metrics that you can use to track development are fading into the background.
DevOps Standardization Challenges. Product goals are different for all companies, they cannot be standardized. The state of DevOps in a company depends on the company itself, but many do not understand this and believe that it is enough to hire a DevOps engineer.
Why are we not DevOps yet? There are two key issues. In Enterprise - the slow development of the organization, the difficulty with changing the vector in the minds of thousands of employees. In startups, there is a lack of sources of knowledge, a problem with the allocation of resources for transformation.
Development stages of DevOps in the company:
- the first is the infrastructure in the cloud, but no one knows how it works, except for one or two admins;
- the second is that the infrastructure is transparent and understandable to all engineers, but the processes are not streamlined;
- the third - engineers independently launch and repair live services;
- fourth - engineers optionally contribute to the core infrastructure, transparent code in the cloud, deploy by button.
Ideal scheme - everyone has the same access to the infrastructure, all engineers get access to the product and understand what they are doing.
By closing all cultural and technical gestalts, the DevOps transformation of the company will take into account the feedback from business and platform metrics.
6. Rosbank DevOps enthusiasts: “1000 days to implement DevOps in a bloody enterprise”
Yuri Bulich, Dina Maltseva, Evgeny Pankov from Rosbank told how they came to DevOps in three years. There were two goals: to solve specific problems in specific teams and implement centralized tools.
Here are the results achieved:
Results for product teams: 30 times faster assembly, 6 times faster installation, up to 30% savings on a full cycle. We got the opportunity to roll into productivity by clicking the button
Results for platform teams: 10x faster build and install, 87% more installs, 46% autotest coverage. The integration team has ceased to be a bottleneck
So, how to implement DevOps practices in a bloody enterprise?
Implement a pilot project first: choose teams, decide how to implement the architecture, and choose tools. We chose tools with an open license, with the presence of an installation in the bank and expertise in working with them. Rosbank deployed a private cloud at the same time as the DevOps platform, and this helped at the start. In the cloud, it was possible to get the necessary resources in 15 minutes by clicking on a button; previously, such a process could take a week.
In banks and other enterprises, you need to work out the restrictions in advance with the information security team and find a solution that will allow you to implement changes.
After the pilot, a successful solution must be scaled.
- It is important to “straighten” the pipeline as much as possible, excluding unnecessary links from it, single out value providers, and remove the remaining components. Intermediates are antipatterns. For example, in Rosbank, a number of teams did not develop internal development, only external development remained. This led to the emergence of a dedicated DevOps team that ensured the transfer of code from external developers to internal ones. The problem was solved by integrating external development into CI / CD, so that they would not only transfer the code from themselves to the bank, but also be responsible for its success.
- The maturity model included elements of DevOps practices, listed tools, took into account the peculiarities of working with external vendors - in the future, this helped to quickly cut the backlog of tasks when implemented in new teams.
- We need Governance in the form of soft control and recommendations. A DevOps Handbook that works well is a set of organizational and instrumental specifications that help teams get the platform right.
- It is worth immediately paying attention to culture, then many changes will be faster and easier. Grow your inner community, hold meetups, technical workshops, trainings, fun activities. This is bearing fruit: people share practices, see who has done what, they know where to turn, there is hype and healthy competition within the company.
- It makes no sense to work with those who are not involved in the process, with teams that have not matured, it is better to invest in interested teams and loyal people.
- The chosen solution should be convenient for those engineers who use it.
- External development is not a blocker, you can also implement practices there, the main thing is that the team itself has a desire.
A little more benefit
List of books worth reading for those who are in DevOps, from Alexander Chistyakov, vdsina.ru:
- Irina Yakutenko "Will and self-control".
- Daniel Kahneman "Thinking, Fast and Slow".
- Barbara Oakley A Mind for Numbers.
- Maxim Dorofeev Jedi Techniques.
- Viktor Frankl Man's Search for Meaning.
Stay tuned
We love DevOps too. Stay tuned for episode announcements and @Kubernetes, as well as other Mail.ru Cloud Solutions events, in our Telegram channel:
Source: habr.com