Our company is currently undergoing the onboarding process for the SRE team. I joined this effort from the development side. Throughout this process, I have gained thoughts and insights that I want to share with other developers. In this reflective article, I discuss what is happening, how it is happening, and how everyone can live with it moving forward.
Continuation of a series of articles inspired by presentations at our internal event :
2. Infrastructure as code. (You are here)
3. Generating TypeScript contracts from C# models. (In progressā¦)
4. Introduction to the Raft consensus algorithm. (In progressā¦)
ā¦
We decided to form an SRE team, embodying the ideas of . We recruited programmers from our own developers and sent them for training for several months.
The team faced the following learning tasks:
- Describe our infrastructure, mostly in Microsoft Azure, in the form of code (Terraform and everything around it).
- Teach developers to work with infrastructure.
- Prepare developers for on-call duties.
Introducing the concept of Infrastructure as code
In the conventional model (classical administration), knowledge about infrastructure exists in two places:
- Either as knowledge in the minds of experts.
- Or this information is stored on certain machines, part of which are known to experts. But it's not certain that a newcomer (in case our entire team suddenly passes away) will be able to understand how everything works. There can be a lot of information on the machine: access permissions, cron jobs, mounted (see ) disks, and simply an endless list of what might be happening. Itās hard to grasp what is really going on.
In both cases, we find ourselves trapped, becoming dependent:
- either on a person who is mortal, subject to illness, love, mood swings, and simply routine dismissals;
- or on a physically operating machine, which can also fail, get stolen, or present unexpected issues and inconveniences.
It clearly suggests a solution: ideally, everything should be converted into human-readable, maintainable, and well-written code.
Thus, Infrastructure as Code (IaC) is a description of all existing infrastructure in the form of code, along with the associated tools for working with it and creating real infrastructure from it.
Why translate everything into codePeople are not machines. They cannot remember everything. The reactions of humans and machines are different. Everything automated potentially works faster than anything done by humans. The most important thing is a single source of truth.
Where new SRE engineers come fromSo, we've decided to hire new SRE engineers, but where do we find them? The book with the right answers () tells us: from developers. After all, they work with code, and you achieve the ideal state.
We searched long and hard for them in the job market outside our company. But we have to admit that we didn't find anyone that met our requirements. We had to sift through our own.
Problems with Infrastructure as Code
Now, letās look at examples of how infrastructure can be encoded in code. The code is well-written, high quality, with comments and indentation.
Example code from Terraform.
Example code from Ansible.
Gentlemen, if only it were that simple! We are in the real world, and it is always ready to surprise you, presenting surprises and problems. This is no exception here.
1. The first problem is that in most cases, IaC is some kind of DSL.
A DSL, in turn, is a description of the structure. More precisely, it specifies what you should have: JSON, YAML, modifications from some large companies that invented their own DSL (HCL is used in Terraform).
The trouble is that it may easily lack some familiar elements such as:
- variables;
- conditions;
- sometimes comments are absent; for instance, in JSON, they are not provided by default;
- functions;
- and I'm not even mentioning high-level concepts like classes, inheritance, and all that.
2. The second problem with such code is that it is most often a heterogeneous environment. Usually, you sit and work with C#, meaning you are using one language, one stack, one ecosystem. But here you have a vast variety of technologies.
A quite realistic situation occurs when a bash script with Python launches a process that receives some JSON input. You analyze it, then some generator produces another 30 files. All of this is dependent on input variables from Azure Key Vault, fetched by a plugin for drone.io written in Go, and these variables pass through a YAML file generated from a jsonnet template. Itās quite challenging to maintain perfectly well-documented code in such a diverse environment.
Traditional development within a single task involves using one programming language. Here, however, we work with a multitude of languages.
3. The third problem is tooling.. We are used to powerful editors (Ms Visual Studio, Jetbrains Rider) that do everything for us, and even if we make a mistake, they will point it out. This seems normal and natural.
But nearby, there is VSCode, which has some plugins that may or may not be supported. New versions are released and might not be supported. A simple transition to implementing a function (even if it exists) becomes a complex and non-trivial problem. A straightforward variable rename is just a replace in a project across several files. Itās a stroke of luck if it replaces exactly whatās needed. Of course, thereās some syntax highlighting, auto-completion, and formatting available (though I couldn't get it to work in Terraform on Windows).
At the time of writing this article, had not yet been released to support version 0.12, although it has been out for 3 months.
It's time to forget about...
- Debugging.
- Refactoring tool.
- Auto completion.
- Error detection during compilation.
Itās funny, but this actually increases development time and the number of errors that inevitably occur.
The scariest part is that we have to focus not on how to design, organize files into folders, decompose, and create maintainable, readable code, but on how to correctly write this command, because I wrote it incorrectly.
As a beginner, you are trying to understand Terraform, and the IDE doesnāt help you at all. When thereās documentation, you can look it up. But if you were learning a new programming language, the IDE would suggest that such a type exists and that one does not, at least at the level of int or string. This is often very helpful.
But what about the tests?
You might ask: "What about the tests, fellow programmers?" Serious folks test everything in production, and it's tough. Here's an example of a unit test for a Terraform module from the site. .
They have good documentation. I've always liked Microsoft's approach to documentation and training. But you don't need to be Uncle Bob to realize that the code here isnāt perfect. Notice the validation pushed to the right.
The problem with the unit test is that we can check the correctness of the JSON output. I provided 5 parameters, and I received a JSON blob of 2000 lines. I can analyze whatās happening here, validate the test resultā¦
It's difficult to analyze JSON in Go. You should write in Go because Terraform in Go is a good practice, testing in the language you write in. The organization of the code itself is quite weak. Meanwhile, it's the best library for testing.
Microsoft itself writes its modules, testing them this way. Of course, it's Open Source. Everything I'm talking about you can come and fix. I could sit down and fix everything in a week, open source VS Code plugins, Terraform, create a plugin for Rider. Maybe write a couple of analyzers, attach linters, contribute to the testing library. I can do all that. But that's not what I should be doing.
Best practices for Infrastructure as Code
Let's move on. If there are no tests in IaC, if the IDE and tooling are poor, then there should at least be best practices. I simply went to Google Analytics and compared two search queries: Terraform best practices and C# best practices.
What do we see? Ruthless statistics are not in our favor. In terms of available material, itās the same. In C# development, we are just swimming in materials; we have top-notch practices, books written by experts, and also books written by other experts critiquing those books. A sea of official documentation, articles, training courses, and now even open source development.
As for the query on IaC: here you are trying to piece together information from high-load talks or HashiConf, from official documentation, and numerous issues on GitHub. How do you even distribute these modules, what to do with them? It seems like a real problem... There is a community, folks, where for any question youāll get 10 comments on GitHub. But thatās not certain.
Unfortunately, experts are just starting to emerge at the moment. They are still too few. And the community is still in its infancy.
Where is all this headed and what to do
You can abandon everything and go back to C#, to the world of the rider. But no. Why would you even get into this if not to find a solution? Here are my subjective conclusions. You can argue with me in the comments; it would be interesting.
Personally, I bet on a few things:
- Development in this field is progressing very rapidly. Here is a graph of requests related to DevOps.
It may be a trending topic, but the very fact that the field is growing instills some hope.If something is growing this quickly, wise people will definitely emerge to advise on what to do and what not to do. The rise in popularity may lead someone to finally find the time to finish a jsonnet plugin for vscode, allowing function implementation navigation instead of searching for it through ctrl+shift+f. As everything evolves, there is more material available. The recent release of Google's book on SRE is a great example.
- There are developed methodologies and practices in regular development that we can successfully apply here. Yes, there are nuances with testing and heterogeneous environments, insufficient tooling, but a huge number of practices have been accumulated that can be useful and helpful.
A simple example: collaborative work through pair programming. It helps a lot to understand. When you have a neighbor next to you who is also trying to understand something, together you'll grasp it better.
Understanding how refactoring is done helps even in such situations to carry it out. That is, you may not change everything at once, but change the naming first, then the layout, and then perhaps isolate a certain part. Oops, and here comments are missing.
Conclusion
Despite the fact that my reflections may seem pessimistic, I look to the future with hope and sincerely believe that we (and you) will succeed.
The second part of the article is coming up. In it, I will talk about how we tried applying agile development practices to improve our learning process and work with the infrastructure.
Source: habr.com
