The report will talk about some DevOps practices, but from a developer's point of view. Usually, all engineers who come to DevOps already have several years of administration experience behind them. But this does not mean that the developer does not have a place here. More often than not, developers are busy fixing "another urgent-critical bug of the day", and they do not have time to even take a look at the DevOps sphere with one eye. In the understanding of the author of DevOps, this is, firstly, common sense. Secondly, it is an opportunity to be more efficient. If you are a developer, have common sense and want to be more effective as a team player, this report is for you.
I will introduce myself, I fully admit that there are people in the hall who do not know me. My name is Anton Boyko, I am Microsoft Azure MVP. What is MVP? This is Model-View-Presenter. Model-View-Presenter is me.
In addition, I currently hold the position of solution architect at Ciklum. And just recently I bought myself such a beautiful domain, and my email, which I usually show at presentations, has been updated. You can write to me at: me [dog] byokoant.pro. You can email me with questions. I usually answer them. The only thing I would not like to receive questions in the mail, which relate to two topics: politics and religion. For everything else, you can email me. It will take some time, I will answer.
A few words about myself:
- I have been in this field for 10 years now.
- I worked at Microsoft.
- I am the founding father of the Ukrainian Azure community, which we founded around 2014. And we still have it and develop it.
- I am also the founding father of the Azure conference, which we have in Ukraine.
- I also help organize the Global Azure Bootcamp in Kyiv.
- As I said, I am Microsoft Azure MVP.
- I speak at conferences quite often. I love speaking at conferences. Last year I managed to perform about 40 times. If you run past Ukraine, Belarus, Poland, Bulgaria, Sweden, Denmark, the Netherlands, Spain, or give or take another country in Europe, then it is quite possible that when you go to a conference that has the topic of clouds in your stream, you can see me on the list of speakers.
- I'm also a Star Trek fan.
Let's talk a little about Agenda. Our Agenda is very simple:
- We'll talk about what DevOps is. Let's talk about why this is important. Previously, DevOps was a keyword that you wrote on your resume and received immediately +500 dollars to your salary. Now you need to write, for example, blockchain in your resume to get +500 dollars to your salary.
- And then, when we understand a little what it is, we will talk about what DevOps practices are. But not so much in the context of DevOps in general, but about those DevOps practices that could be of interest to developers. I will tell you why they might be of interest to you. I’ll tell you why you should do it at all and how it can help you experience less pain.
The traditional picture that many show. This is what happens in many projects. This is when we have development and operations departments that support our software. And these departments do not communicate with each other.
Perhaps, if you weren’t able to feel it so clearly in the DevOps and operations departments, you can draw an analogy with the Dev and QA departments. There are people who develop software and there are QA who are bad in terms of developers. For example, I commit my beautiful code to the repository, and some villain sits there, who returns this code to me and says that your code is bad.
This all happens because people do not communicate with each other. And they, through some kind of wall of misunderstanding, throw some packages, some kind of application to each other and try to do something with them.
It is precisely to destroy this wall that the DevOps culture is called upon, i.e. make people communicate with each other and at least understand what different people do in the project and why their work is important.
And when we talk about DevOps, then someone will tell you that DevOps is when there is continuous integration in the project; someone will say that DevOps is if the project implements the practice of "infrastructure as code"; someone will say that the first step to DevOps is feature branching, feature flags.
In fact, this is all true in its own way. But these are just the final practices that we have. Before moving on to these practices, I suggest looking at this slide, which shows the 3 stages of implementing a Dev-Ops methodology in your project, in your company.
This slide has a second unofficial title. You can search the web for what 3 Musketeers DevOps are. It is possible that you will find this article. Why 3 Musketeers? At the bottom it says: people, processes and products, i.e. PPP - Porthos, Porthos and Porthos. Here are the 3 Musketeers of DevOps. This article describes in more detail why this is important and what it entails.
When you start implementing a DevOps culture, it is very important that you implement it in the following order.
Initially, you need to talk to people. And you need to explain to people what it is and how they can get some benefits from it.
Our conference is called DotNet Fest. And as the organizers told me, we basically had an audience of developers invited here, so I hope that most people in the hall are engaged in development.
We'll talk about people, talk about what developers want to do every day. What do they want the most? They want to write some new code, use newfangled frameworks, cut new features. What do developers want the least? Fix old bugs. I hope you will agree with me. This is what developers want. They want to write new features, they don't want to fix bugs.
The number of bugs that a particular developer produces depends on how straight his arms are and how much they grow from his shoulders, and not from pockets on his butt. But nevertheless, when we have a large project, sometimes it happens that it is impossible to keep track of everything, so it would be nice for us to use some approaches that will help us write more stable and better code.
What do QA want most? I don't know if they are in the room. It's hard for me to say that I want QA, because I have never been one. And no offense to the guys will be told that I never will, I hope. But not for the reason that I consider their work meaningless and useless, but because I do not consider myself a person who could do this work with high quality, so I will not even try to do it. But from what I understand, QA most of all does not like to go to work in the morning, constantly run some kind of regression tests, step on the same bugs that they reported to the developers 3 sprints ago and say: “When are you, Monsieur D 'Artagnan, fix this bug.' And Monsieur D'Artagnan answers him: "Yes, yes, yes, I already fixed it." And how it happens that one bug was fixed, 5 were made along the way.
The people who maintain this solution in production want this solution to work without bugs, so they don't have to restart the server every Friday when all normal people go to the bar. The developers deployed on Friday, the admins sit until Saturday, trying to raise and fix this deployment.
And when you explain to people that they are aimed at solving the same problems, you can move on to formalizing processes. It is very important. Why? Because when we say “formalization”, it is important for you to describe how your processes take place at least somewhere on a napkin. You need to understand that if you, for example, deploy to a QA environment or a production environment, then it always happens in this order, at such stages we run, for example, automatic unit tests, UI tests. After the deployment, we check whether the deployment went well or badly. But you already have a clear list of actions that should be repeated over and over again when you deploy to production.
And only when your processes are formalized, you start choosing products that will help you automate these processes.
Unfortunately, I very often see this happening in reverse. As soon as someone hears the word “DevOps”, they immediately offer to install Jenkins, because they believe that as soon as they install Jenkins, they will have DevOps. They installed Jenkins, read the “How to” articles on the Jenkins website, tried to stuff the processes into these How to articles, and then they came to people and bent people down, saying that the book says that you need to do it like this, so we do it like this.
Not that Jenkins is a bad tool. By no means do I want to say that. But this is just one of the products. And which product you will use should be your last decision, and by no means your first. Your product should not drive the introduction of culture and approaches. This is very important to understand, which is why I'm spinning around on this slide for so long and explaining it all for so long.
Let's talk about DevOps practices in general. What are they? What is the difference? How to measure them? Why are they important?
The first practice you may have heard of is called Continuous Integration. Perhaps someone on the project has Continuous Integration (CI).
The biggest problem is that most often when I ask a person: “Do you have CI on the project?” and he says: “Yes”, then when I ask what he does, he describes to me absolutely the entire automation process. This is not entirely true.
In fact, the practice of CI is only aimed at the fact that the code that different people write is integrated into some kind of single code base. That's all.
Along with CI, there are usually other practices along the way - such as Continuous Deployment, Release Management, but we'll talk about this later.
CI itself tells us that different people write code and this code should be integrated continuously into a single code base.
What does it give us and why is it important? If we have DotNet, then that's good, it's a compiled language, we can compile our application. If it compiles, then that's a good sign. This doesn't mean anything yet, but it's the first good sign that we can at least compile.
Then we can run some tests, which is also a separate practice. The tests are all green - this is the second good sign. But again, that doesn't mean anything.
But why would you do it? All the practices that I will talk about today carry approximately the same value, i.e., approximately the same benefits and are also measured in approximately the same way.
First, it allows you to expedite delivery. How does this allow you to speed up delivery? When we have some new changes in our code base, we can immediately try to do something with this code. We don't wait until we have Thursday, because on Thursday we will release it on the QA Environment, we do it right here and right here.
I will tell you one sad story from my life. It was a long time ago when I was still young and handsome. Now I am young, handsome and smart, and modest. Some time ago I was in one project. We had a large team of about 30 developers. And we had a big-big Enterprise project that had been developing for about 10 years. And we had different branches. In the repository, we had a branch in which the developers walked. And there was a branch that displayed the version of the code that is on production.
The production branch was 3 months behind the branch that was available to developers. What does this mean? This means that as soon as I have a bug somewhere in production due to the fault of the developers, because they allowed it, and through the fault of QA, because they overlooked it, this means that if I receive a task on hotfix for production, then I have to roll back my code changes to 3 months ago. I have to remember what I had 3 months ago and try to fix it there.
If you haven't had this experience yet, you can try it on your home project. Most importantly, do not try commercial. Write a couple of lines of code, forget about them for half a year, and then come back and try to quickly explain what these lines of code are about and how you can fix or optimize them. It's a very, very exciting experience.
If we have the practice of Continuous Integration, then this allows us right here and right now, as soon as I wrote my code, to check it with a number of automatic tools. Let this not give me the full picture, but nevertheless, it will remove at least some of the risks. And if there is some potential bug, I will know about it right now, that is, literally in a couple of minutes. I won't need to roll back 3 months. I will need to roll back just 2 minutes ago. In 2 minutes, a good coffee machine won’t even have time to brew coffee, so it’s pretty cool.
It carries the value that it can be repeated over and over again on every project, i.e. not just the one you have it set up on. You can repeat both the practice itself, and CI itself will be repeated for each new change that you make to the project. This allows you to optimize resources because your team is more efficient. You will no longer have a situation when a bug arrives to you from the code that you worked with 3 months ago. You will no longer have a context switch when you sit and for the first two hours try to understand what happened then and enter the essence of the context before you start to correct something.
How can you measure the success or failure of this practice? If you report to the big boss what we implemented on the CI project, then he hears blah blah blah. Implemented, OK, why, what did it bring us, how will we measure it, how correctly or incorrectly are we implementing it?
The first is that due to CI we can deploy more and more often, and more often for the very reason that we have potentially more stable code. In the same way, we have less time to find an error and less time to fix this error for the very reason that we are getting a response from the system right here and right now, what is wrong with our code.
Another practice that we have is the practice of Automated Testing, which most often comes with the practice of CI. They go hand in hand.
What is important to understand here? It is important to understand that our tests are different. And each automated test is aimed at solving its own problems. We have, for example, unit tests that allow us to test a particular module separately, i.e. how it works in a vacuum. This is good.
We also have integration tests that allow us to understand how different modules integrate with each other. It is also good.
We can have UI automation tests that allow us to check how well the work with the UI complies with certain requirements that the customer has set, etc.
Which tests you run can affect how often they run. Unit tests are usually written short, small. And they can be run regularly.
If we are talking about UI automation tests, then it's good if your project is small. Your UI automation tests can take some adequate time. But usually a UI automation test is a thing that takes several hours on a large project. And it's good if a few hours. The only thing is that it makes no sense to run them for each build. It makes sense to run them at night. And when everyone came to work in the morning: both testers and developers, they received some kind of report that we ran a UI autotest at night and got these results. And here, an hour of server work, which will check that your product meets some requirements, will be much cheaper than an hour of work of the same QA engineer, even if it is a Junior QA engineer who works for food and thanks. Still, the hour of the car will be cheaper. That is why it makes sense to invest in it.
I have another project that I have been working on. We had two week sprints on this project. The project was big, important for the financial sector and it was impossible to make a mistake. And according to the two-week sprint, according to the development cycle, the testing process continued, which took another 4 weeks. Try to imagine the scale of the tragedy. We write the code for two weeks, then we do a la CodeFreeze, pack it into a new version of the application, and roll it out to testers. Testers are testing it for another 4 weeks, i.e. while they are testing it, we have time to prepare two more versions for them. This is a really sad case.
And we told them that if you want to work more productively, it makes sense for you to implement the practice of Automated Testing, because this is what hurts you right here and right now.
Practice Continuous Deployment. Great, you've done build. It's already good. Your code has compiled. Now it would be nice to deploy this build on some kind of environment. For example, in a developer environment.
Why is it important? First, you can see how successful your deployment process is. I met such projects, when I ask: “How do you deploy a new version of the application?”, the guys say to me: “We collect it and pack it into a zip archive. We send to the administrator by mail. The admin downloads this archive, expands it. And the whole office starts praying that the server will pick up the new version.”
Let's start with something simple. For example, they forgot to put CSS in the archive or forgot to change the hashtag in the name of the java-script file. And when we make a request to the server, the browser thinks that it already has a java-script file and decides not to download it. And there was an old version, something was missing. In general, there can be many problems. Therefore, the practice of Continuous Deployment allows you, at a minimum, to test what will happen if you take a clean reference image and upload it to a completely clean new environment. You can see where this leads.
In addition, when you integrate the code among themselves, i.e. between the command, this allows you to also see what it looks like on the UI.
One of the problems that occurs where a lot of vanilla java-script is used is that two developers have rashly declared a variable with the same name in the window object. And then, as luck would have it. Whose java-script file is pulled out second, he will overwrite the changes of the other. This is also very exciting. You come in: for one person, one thing works, for another, the other does not work. And it's "fine" when it all gets out on production.
The next practice that we have is the practice of Automatic Recovery, namely rollback to the previous version of the application.
Why is this important for developers? There are still those who remember the far, far 90s, when computers were big and programs were small. And the way to web development lay only through php. It's not that php is a bad language, although it is.
But the problem was different. When we deployed a new version of our php site, how did we deploy it? Most often we opened Far Manager or something else. And uploaded these files to FTP. And we suddenly realized that we had some kind of small, small bug, for example, we forgot to put a semicolon or forgot to change the password from the database, and there is the password from the database, which is on the local host. And we decide to quickly connect to FTP and edit the files right there. This is just fire! This is what was popular in the 90s.
But, if you haven't looked at the calendar, the 90s were almost 30 years ago. Now everything is happening a little differently. And try to imagine the scale of the tragedy when you are told: “We deployed to production, but something went wrong there. Here's your FTP login and password, connect to production and quickly fix it there. If you are Chuck Norris, then this will work. If not, you run the risk of fixing one bug, making 10 more. That's why this practice of reverting to the previous version allows you to achieve a lot.
Even if something somehow bad somehow crawled onto prod, then this is bad, but not fatal. You can roll back to the previous version you have. Call it a backup, if it's easier to perceive in such terminology. You can rollback to this previous version and users will still be able to work with your product, and you will have adequate headroom. You can calmly, without haste, take all this and locally test it, fix it, and then upload the new version. It really makes sense to do so.
Now let's try to combine somewhere somehow the two previous practices together. We will get the third one, which is called Release Management.
When we talk about Continuous Deployment in its classical form, we say that we should pull the code from some branch from the repository, compile and deploy it. It's good if we have one environment. If we have several environments, then this means that we must pull the code every time, even if it is from the same commit. We will pull it every time, we will build it every time and we will deploy it to a new environment. Firstly, this is the time, because on the build of the project, if you have a large one and come from the 90s, then it can take several hours.
In addition, there is another sadness. When you build even on the same machine, you will be building the same sources, you still have no guarantee that this machine is in the same state as it was during the last build.
Let's say someone came in and updated DotNet for you, or, conversely, someone decided to delete something. And then you have a cognitive dissonance that from this commit two weeks ago we built a build and everything was fine, but now it seems like the same machine, the same commit, the same code that we are trying to build, but it doesn’t work. You will deal with this for a long time and it is not a fact that you will figure it out. At the very least, spoil your nerves a lot.
Therefore, the practice of Release Management suggests introducing an additional abstraction, which is called an artifact repository or a gallery, or a library. You can call it whatever you like.
The main idea is that as soon as we have some kind of commit there, for example, in a branch that we are ready to deploy to our different environments, we collect applications from this commit and everything that we need for this application, we We pack it in a zip archive and save it in some kind of reliable storage. And from this storage we can get this zip archive at any time.
Then we take and automatically deploy it to the dev environment. We drive there, and if everything is fine, then we deploy it to the stage. If everything is fine, then we deploy the same archive to production, the same binaries compiled exactly once.
Also, when we have a gallery like this, it also helps us address the risks that we addressed on the last slide when we talked about rollback to the previous version. If you accidentally deployed something wrong, you can always take any other previous version from this gallery and deploy it to these environments in the same way. This allows you to easily roll back to the previous version in case of some kind of nix.
There is another great practice. We all understand that when we roll back our applications to a previous version, this may mean that we also need the infrastructure of the previous version.
When we talk about virtual infrastructure, many people think that this is something that admins set up. And if you need, say, to get a new server on which you want to test a new version of your application, then you must write a ticket to admins or devops. Devops will take 3 weeks for this. And after 3 weeks they will say that we have installed a virtual machine for you, where there is one core, two gigabytes of RAM and a Windows server without DotNet. You say: "And I wanted DotNet." Them: "Ok, come back in 3 weeks."
The idea is that with the Infrastructure as Code practice, you can treat your virtual infrastructure as just another resource.
Perhaps if any of you develop applications on DotNet, you may have heard of a library such as Entity Framework. And you may have even heard that the Entity Framework is one of the approaches that Microsoft is actively pushing. For working with a database, this is an approach called Code First. This is when you describe in code how you want your database to look like. And then you deploy the application. It connects to the database, it itself determines which tables are, which tables are not, and creates everything you need.
You can do the same with your infrastructure. There is no difference between whether you need a database for a project or a Windows server for a project. It's just a resource. And you can automate the creation of this resource, you can automate the configuration of this resource. Accordingly, every time you want to test some new concept, some new approach, you will not need to write a ticket to the devops, you can take and simply deploy an isolated infrastructure for yourself from ready-made templates, from ready-made scripts, and run an isolated infrastructure there all your experiments. You can delete it, get some results and report it further.
The next practice, which also exists and is also important, but which few people use, is Application Performance Monitoring.
I wanted to say only one thing about Application Performance Monitoring. What is most important in this practice? This is what Application Performance Monitoring is about the same as repairing an apartment. It's not a final state, it's a process. You need to do it regularly.
In a good way, it would be good to carry out Application Performance Monitoring for almost every build, although, as you understand, this is far from always possible. But, at least, for each release it needs to be carried out.
Why is it important? Because if you suddenly have a drawdown in performance, then you need to clearly understand why. If you have, for example, two-week sprints in your team, then at least once every two weeks you should deploy your application to some separate server, where you have a clearly fixed processor, RAM, disks, etc. And run those same performance tests. You get the result. See how it has changed from the previous sprint.
And if you find out that the drawdown has gone sharply somewhere down, then this will mean that it was just because of the changes that have taken place over the past two weeks. This will allow you to identify and fix this problem much faster. And again, these are the same approximate metrics by which you can measure how successfully you did it.
The next practice we have is the practice of Configuration Management. Very few people take it seriously. But trust me, this is actually a very serious thing.
There was a funny story recently. The guys came to me and said: "Help us conduct a security audit of our application." We looked at the code together for a long time, they told me about the application, drew diagrams. And plus or minus everything was logical, understandable, safe, but there was one BUT! They had configuration files in the source control, including those from production with the database IP, with logins and passwords for connecting to these databases, etc.
And I say: “Guys, okay, you closed your production environment with a firewall, but the fact that you have a login and password from the production database right in the source control and any developer can read it is already a huge security risk. And no matter how super-secure your application is in terms of code, if you keep it in source control, then you will never pass any audit anywhere. That's what I'm talking about.
Configuration management. On a different environment, we may have a different configuration. For example, we may have different logins and passwords for databases for QA, demo, production environments, etc.
This configuration can also be automated. It should always be separate from the application itself. Why? Because you built the application once, and then the application doesn’t care - you connect to the SQL server by such and such an IP or by such and such an IP, it should work the same way. Therefore, if suddenly one of you is still hardcoding the connection string in the code, then remember that I will find you and punish you if you find yourself on the same project with me. This is always taken out in a separate configuration, for example, in web.config.
And this configuration is already managed separately, that is, this is exactly the moment when a developer and an admin can come and sit in the same room. And the developer can say: “Look, here are the binaries of my application. They work. The application needs a database to work. Here is a file next to the binaries. In this file, this field is responsible for the login, this is for the password, this is for the IP. Deploy it anywhere." And the administrator is simple and clear. He can deploy it really anywhere, managing this configuration.
And the last practice that I would like to talk about is a practice that is very, very related to the clouds. And it brings the maximum effect if you work in the cloud. This is Automatic deletion of your environment.
I know that there are several people at this conference from the teams that I work with. And with all the teams that I work with, we use this practice.
Why? Of course, it would be great if there was a virtual machine for each developer that would work 24/7. But perhaps this is news to you, perhaps you did not pay attention, but the developer himself does not work 24/7. The developer usually works 8 hours a day. Let him come to work early, he has a big lunch, during which he goes to the gym. Let it be 12 hours a day when the developer actually uses these resources. According to our legislation, we have 5 out of 7 days in a week that are considered working.
Accordingly, on working days, this machine should not work 24 hours, but only 12, and on weekends, this machine should not work at all. It would seem that everything is very simple, but what is important to say here? By implementing this simple practice on such a basic schedule, this allows you to reduce the cost of maintaining these environments by 70%, i.e. you took the price of your dev, QA, demo, environment and divided by 3.
The question is what to do with the rest of the money? For example, buy ReSharper developers if you haven't bought it yet. Or have a cocktail party. If you used to have one environment where both dev and QA grazed, and that's it, now you can make 3 different ones, which will be isolated, and people will not interfere with each other.
As for the slide with constant performance measurement, how can we compare the performance if we had 1 records in the database in our project, two months later there are a million of them? How to understand why and what is the point in measuring performance?
This is a good question, because you should always measure performance on the same resources. That is, you roll out new code, you measure performance on new code. For example, you need to test different performance scenarios, let's say you want to test how an application performs on a light load, where there are 1 users and a database size of 000 gigabytes. You measured, got the numbers. Let's take another scenario. For example, 5 users, database size 5 terabyte. We got the results, remember.
What is important here? The important thing here is that depending on the scenario, on the amount of data, on the number of concurrent users, etc., you can run into certain limits. For example, to the limit of the network card or to the limit of the hard disk, or to the limit of the processor's capabilities. Here's what's important for you to understand. In different scenarios, you run into certain limits. And you need to understand the numbers when you run into them.
Here we are talking about measuring performance on a special test environment? That is, this is not production?
Yes, this is not production, this is a test environment, which is always the same so that you can compare it with previous measurements.
Understood thanks!
If there are no questions, I think we can finish. Thank you!
Source: habr.com