🥇Infrastructure as Code: How to Overcome Issues Using XP

Hello, Habr! I used to complain about life in the Infrastructure as Code paradigm and proposed nothing to resolve the situation. Today, I'm back to share approaches and practices that can help pull us out of the abyss of despair and steer the situation in a positive direction.

In the previous article Infrastructure as Code: A First Introduction I shared my impressions of this field, tried to reflect on the current situation in this area, and even suggested that standard practices familiar to all developers might help. It may have seemed that there were many complaints about life, but no proposals for getting out of the situation.

Who we are, where we are, and what problems we have

We are currently in the SRE Onboarding Team, consisting of six programmers and three infrastructure engineers. We are all trying to write Infrastructure as Code (IaC). We do this because we are essentially capable of coding and, in the past, have been developers of above-average skill.

We have a set of advantages: a certain background, knowledge of practices, coding skills, and a desire to learn new things.
And there is a lagging part, which is a disadvantage: a lack of knowledge in the underlying infrastructure.

The technology stack we use in our IaC.

Terraform for resource creation.
Packer for building images. These are Windows and CentOS 7 images.
Jsonnet for powerful builds in drone.io, as well as for generating Packer JSON and our Terraform modules.
Azure.
Ansible for preparing images.
Python for auxiliary services and provisioning scripts.
And all of this in VSCode with plugins shared among team members.

The conclusion from my previous article was this: I was trying to instill (first and foremost in myself) optimism, wanting to say that we would try the approaches and practices we know to tackle the difficulties and challenges present in this area.

Now we are facing the following IaC challenges:

Imperfect tools and code development resources.
Slow deployment. Infrastructure is part of the real world, and it can be unhurried.
A lack of approaches and practices.
We are newcomers and have much to learn.

Extreme Programming (XP) rushes to the rescue.

Every developer is familiar with extreme programming (XP) and the practices behind it. Many of us have worked in this approach, and it has been successful. So why not leverage the principles and practices established there to tackle infrastructure challenges? We decided to apply this approach and see what comes of it.

Assessing the applicability of XP to your fieldHere’s a description of the environment for which XP is well-suited and how it relates to us:

1. Dynamically changing software requirements. We understood what the end goal was. But details can vary. We decide where we need to steer, which is why requirements change periodically (mostly by us). If we take the SRE team that automates on its own and restricts requirements and scope, this point fits well.

2. Risks caused by fixed-time projects using new technology. We may encounter risks when using some things we are not familiar with. This is 100% our case. Our entire project involves technologies we were not completely familiar with. This is an ongoing issue since many new technologies constantly emerge in the infrastructure sector.

3,4. Small, co-located extended development team. The technology you are using allows for automated unit and functional tests. These two points do not quite fit us. Firstly, we are not a co-located team; secondly, there are nine of us, which can be considered a large team. However, according to some definitions, a 'large' team is one with 14+ members.

Let’s look at some practices from XP and how they influence the speed and quality of feedback.

The feedback cycle principle in XP

In my understanding, feedback is the answer to the question, am I doing this right, are we heading in the right direction? XP has a divine model for this: the feedback cycle over time. The interesting part is that the lower we are in the process, the quicker we can receive feedback to address the necessary questions.

This is quite an interesting topic for discussion, as in our IT industry, it’s possible to get feedback rapidly. Imagine how painfully difficult it is to work on a project for six months and only then discover that an error was made at the very beginning. This happens in design and in any building of complex systems.

In our case, IaC helps us with feedback. I immediately make a small adjustment to the above schema: the release plan is not a monthly cycle but occurs several times a day. Some practices are tied to this cycle, which we will discuss in more detail.

It is important: feedback can be the solution to all the problems mentioned above. Combined with XP practices, it can pull you out of the abyss of despair.

How to pull yourself out of the abyss of despair: three practices

Tests

Tests are mentioned twice in the XP feedback cycle. This is not accidental. They are extremely important for the entire technique of extreme programming.

It is assumed that you have Unit and Acceptance tests. The former provide feedback in a few minutes, while the latter take longer to write and are run less frequently.

There is a classic testing pyramid that shows that there should be more of certain tests.

How is this schema applicable to us in the IaC project? In fact… not at all.

Despite the fact that there should be a lot of Unit tests, there cannot be too many. Either they test something very indirectly, or we can say that we don’t write them at all. However, here are a few applications for such tests that we have managed to implement:
1. Code testing in jsonnet. This is, for example, our build pipeline in drone, which is quite complex. Code in jsonnet is well covered by tests.
  We use this Unit testing framework for Jsonnet.
2. Tests for scripts that run when a resource starts. The scripts are in Python, which means we can write tests for them.
It is potentially possible to check the configuration in tests, but we do not do that. There is also the option to set up a check for resource configuration rules via tflint. However, Terraform has too basic checks, but many testing scenarios have been written for AWS. And we are on Azure, so this does not fit again.
Component integration tests: it depends on how you classify and where you place them. But they work in principle.
This is what integration tests look like.

This is an example during the image build in Drone CI. To get to them, you have to wait 30 minutes for the Packer image to build, then another 15 minutes for them to pass. But they exist!
Image verification algorithm
1. First, Packer must prepare the image completely.
2. Next to the test, there is a Terraform setup with a local state that we use to deploy this image.
3. During deployment, a small module located nearby is used to facilitate working with the image.
4. Once the VM is deployed from the image, verification can begin. Mostly, checks are performed on the machine. We verify how the scripts executed during startup and how the services are running. For this, we access the newly created machine via SSH or WinRM and check the configuration status or whether the services have started.
A similar situation exists with integration tests and modules for Terraform. Here is a brief table explaining the specifics of such tests.

Feedback on the pipeline takes around 40 minutes. Everything is happening very slowly. It can be used for regression, but it's practically impossible for new development. If you prepare a lot for this, set up running scripts, it can be reduced to 10 minutes. But it still doesn't compare to Unit tests, which run 100 in 5 seconds.

The absence of Unit tests when building images or Terraform modules leads to the need to shift the work to separate services that can simply be called via REST or Python scripts.

For example, we needed to ensure that when the virtual machine starts, it registers itself with the service ScaleFT, and when the VM is destroyed, it removes itself.

Since ScaleFT is a service for us, we are forced to interact with it via the API. A wrapper was created that can be used to say: 'Go in and delete this and that.' It stores all the necessary settings and access.

We can write proper tests for this since it's no different from regular software: mock an API, you call it, and we see what happens.

Test results: Unit testing, which should be completed by the OS in a minute, does not provide it. However, higher types of testing in the pyramid yield results, but address only some problems.

Pair programming

Tests are certainly useful. They can be numerous and come in different forms. They will operate at their own levels and provide us with feedback. However, the issue of poor unit tests that provide the fastest OS remains. Still, we desire a fast OS, as it is easy and pleasant to work with. Not to mention the quality of the solution received. Fortunately, there are techniques that allow for even faster feedback than unit tests. This is where pair programming comes into play.

When writing code, we want to receive feedback on its quality as quickly as possible. Yes, you can write everything in a feature branch (to avoid breaking anything for anyone), make a pull request on GitHub, assign it to someone whose opinion matters, and wait for a response.

But waiting can take a long time. People are all busy, and even if a response comes, it may not be of the highest quality. Let's assume the response arrived immediately, the reviewer instantly understood the entire concept, but the response still comes with a delay, in hindsight. And we want it sooner. This is precisely what pair programming aims for – to provide feedback right at the moment of writing.

Next, I will outline the styles of pair programming and their applicability to working on IaC:

1. Classic, Experienced + Experienced, timer-based role switching. Two roles – driver and navigator. Two people. They work on the same code and swap roles at a pre-designated time interval.

Let's consider how our problems align with the style:

Problem: Imperfections in tools and means for code development.
Negative impact: Longer development times, we slow down, disrupting the work pace/rhythm.
How we cope: We apply different tools, a shared IDE, and also learn shortcuts.
Problem: Slow deployment.
Negative impact: Increases the time to create a working piece of code. We get bored waiting, our hands reach for something else to do while we wait.
How we cope: We haven't managed to overcome this.
Problem: Lack of approaches and practices.
Negative impact: No knowledge of how to do things well versus poorly. It prolongs the feedback process.
How we cope: The exchange of opinions and practices in paired work almost resolves the problem.

The main problem with this style in IaC is the uneven pace of work. In traditional software development, you have a very steady progress. You can spend five minutes and write N. Spend ten minutes and write 2N, fifteen minutes – 3N. Here, you can spend five minutes and write N, then spend another thirty minutes and write one-tenth of N. You don’t know anything, you hit a wall, a block. Investigating takes time and distracts from actual programming.

Conclusion: in its pure form, it doesn't suit us.

2. Ping-pong. This approach assumes that one participant writes the test, while the other implements it. Considering that Unit tests are complicated, and you have to write a long integration test, all the ease of ping-pong disappears.

I can say that we tried dividing responsibilities between designing the test scenario and implementing the code for it. One participant came up with the scenario, in this part of the work he was responsible, he had the last word. The other was responsible for implementation. This worked well. The quality of the scenario increases with this approach.

Conclusion: unfortunately, the pace of work does not allow us to use ping-pong as a pair programming practice in IaC.

3. Strong Style. A complex practice. The idea is that one participant becomes the directive navigator, while the other takes on the role of the executing driver. The right to make decisions lies solely with the navigator. The driver just types and can influence what happens with words. The roles do not change for a long time.It is well-suited for training but requires strong soft skills. This is where we stumbled. The technique was challenging. And it’s not even about the infrastructure.

Conclusion: potentially applicable, we do not stop trying.

4. Mobbing, swarming, and all known but unmentioned styles are not considered, as we haven't tried them and cannot speak about them in the context of our work.

General conclusions about using pair programming: We have an uneven work pace that throws us off.

We have hit a wall due to insufficient soft skills. The subject area does not help us overcome these shortcomings.

Long tests and tool problems make pair programming sluggish.

Long tests and problems with tools make pair development sluggish.

Long tests, issues with tools make pair programming sluggish.

5. Despite this, there were successes. We came up with our own method called "Convergence – Divergence." I will briefly describe how it works.

We have regular partners for several days (less than a week). We tackle one task together. For a period, we sit together: one writes, and the other observes like support staff. Then we separate for some time, each doing independent tasks. After that, we come back together, synchronize quickly, accomplish something jointly, and separate again.

Planning and Communication

The last block of practices that address operating system problems is organizing work with the tasks themselves. This also includes sharing experiences that are outside of paired work. Let's consider three practices:

1. Tasks through goal trees. We organized project management through a tree that infinitely extends into the future. Technically, it’s managed in Miro. There is one task – it represents an interim goal. From it, smaller goals can branch out, or groups of tasks. The tasks themselves follow from those. All tasks are created and managed on this board.

This scheme also provides feedback, which occurs once a day when we synchronize in meetings. Having a common plan that is structured and completely open allows everyone to stay informed about what is happening and how far we have progressed.

Advantages of visualizing tasks:

Causality. Each task leads to a global goal. Tasks are grouped by smaller goals. The domain of infrastructure is quite technical by itself. It’s not always immediately clear how, for instance, writing a runbook for migration to another nginx impacts the business. Having a target card nearby makes this clearer.

Causality is an important property of tasks. It directly answers the question: "Am I doing the right thing?"
Parallelism. There are nine of us, and it's physically impossible for everyone to tackle one task. Tasks from one area may not always suffice either. We are compelled to parallelize work among small working groups. During this time, groups focus on their tasks, and they can be reinforced by others. People sometimes drop off from these working groups. Some go on vacation, some prepare reports for the DevOps conf conference, and some write articles for Habr. It becomes very important to know what goals and tasks can be worked on in parallel.

2. Rotating leads for morning meetings. We've encountered a problem during stand-ups: many tasks are being worked on in parallel. Sometimes the tasks are loosely related, and there's no understanding of who is doing what. The opinion of another team member is very important. This adds information that can change the course of decision-making. Of course, there's usually someone to pair with, but consultations and tips are always welcome.

To improve this situation, we implemented the technique of 'Rotating Stand-up Leader'. Now they rotate according to a specific list, and this has an effect. When your turn comes, you are forced to dive in and understand what’s happening in order to conduct a good scrum meeting.

3. Internal demo. Help in solving tasks through pair programming, visualizing on a task tree, and assistance during morning scrum meetings is good but not perfect. In a pair, you are limited to just your knowledge. The task tree helps to globally understand who is doing what. However, the lead and colleagues during the morning meeting may not dive deeply into your issues. They might miss something.

The solution was found in demonstrating the work done to each other and subsequent discussion. We meet once a week for an hour and show details of the solutions to the tasks we worked on in the past week.

During the demonstration, it's essential to reveal the details of the task and definitely showcase its functionality.

The presentation can be guided by a checklist.1. Provide context. Where did the task come from, and why was it needed?

2. How was the task approached previously? For example, was mass clicking required, or was it impossible to accomplish anything?

3. How are we improving this? For example: 'Look, now there’s a little script, here’s the readme.'

4. Show how it works. Ideally, demonstrate a user scenario directly. I want X, I do Y, I see Z (or perhaps Y). For example, deploying NGINX, checking the URL, getting 200 OK. If the action takes a long time, prepare it in advance so you can show it later. It’s best not to break anything fragile at least an hour before the demo.

5. Explain how well the problem has been addressed, what difficulties remain, what hasn't been completed, and what improvements might be possible in the future. For example, currently CLI, later there will be full automation in CI.

Ideally, each speaker should stick to 5-10 minutes. If your presentation is particularly important and will take more time, please coordinate this in the sre-takeover channel in advance.

After the in-person part, there must be a discussion in the thread. This is when the crucial feedback regarding our tasks appears.

In the end, a survey is conducted to determine the usefulness of the event. This serves as feedback on the presentation itself and the importance of the task.

Long conclusions and what’s next

It may seem that the tone of the article is somewhat pessimistic. This is not the case. Two lower-level methods of obtaining feedback, specifically testing and pair programming, work. Not as perfectly as in traditional development, but there is a positive effect.

Tests, in their current form, provide only partial code coverage. Many configuration functions remain untested. Their impact on immediate coding work is low. However, there is an effect from integration tests, which allows for confident refactoring. This is a significant achievement. Also, with the shift of focus to development in high-level languages (we use Python, Go), the problem diminishes. There aren’t many checks necessary for the 'glue'; overall integration suffices.

Working in pairs depends more on the specific people involved. There’s the task factor and our soft skills. With some it works very well, with others it’s less effective. There’s definitely a benefit from it. Clearly, even with insufficient adherence to pair programming rules, the mere fact of working together positively impacts the quality of the result. Personally, I find it easier and more enjoyable to work in pairs.

Higher-level ways to influence the OS—planning and task management—definitely yield effects: quality knowledge exchange and improvement in development quality.