Approach IaC (Infrastructure as Code) consists not only of code stored in a repository, but also involves the people and processes surrounding that code. Can we reuse approaches from software development in managing and describing infrastructure? It would be wise to keep this idea in mind while reading the article.

This is a transcript of my of the presentation to DevopsConf 2019-05-28.

Slides and videos

English version
Russian version
Dry run 2019-04-24 SpbLUG
Video(RU) from DevopsConf 2019-05-28
Video(RU) from DINS DevOps EVENING 2019-06-20
Slides

Infrastructure as bash history

Suppose you join a new project, and you are told: "we have Infrastructure as Code". In reality, it turns out, Infrastructure as bash history or for example Documentation as bash history. This is quite a real situation; for example, Denis Lysenko described a similar case in his talk How to replace the entire infrastructure and start sleeping peacefully, he explained how they created a streamlined infrastructure on the project from bash history.

With some desire, one could say that Infrastructure as bash history it's like code:

reproducibility: you can take bash history, execute commands from it, and you might actually get a working configuration out.
versioning: you know who logged in and what was done, though it's not guaranteed this will lead you to a working configuration in the end.
story: the history of who did what. However, you won't be able to use it if you lose the server.

What should you do?

Infrastructure as Code

Even such a strange case as Infrastructure as bash history can be tenuously connected to Infrastructure as Code, but when we try to do something more complex than a good old LAMP server, we come to the point where this code needs to be modified, changed, and improved. Next, we will draw parallels between Infrastructure as Code and software development.

D.R.Y.

In a project for developing a storage system, there was a subtask to periodically configure SDS: we release a new version — it needs to be deployed for further testing. The task is quite simple:

ssh in here and execute the command.
copy the file there.
edit the config here.
start the service there.
…
PROFIT!

For the described logic, bash is more than enough, especially in the early stages of the project when it is just starting. It’s not bad that you are using bash., but over time there arise requests to expand something similar, but slightly different. The first thing that comes to mind: copy-paste. And now we already have two very similar scripts that do almost the same thing. Over time, the number of scripts grew, and we encountered the fact that there is a certain business logic for deploying installations that needs to be synchronized among different scripts, which is quite complex.

It turns out there is a practice called D.R.Y. (Don't Repeat Yourself). The idea is to reuse existing code. Sounds simple, but we didn't arrive at this instantly. In our case, it was a banal idea: to separate configs from scripts. That is, the business logic for how installations are deployed separately, and the configs separately.

S.O.L.I.D. for CFM

Over time, the project grew and naturally the continuation was the emergence of Ansible. The main reason for its emergence was the expertise in the team and that bash is not designed for complex logic. Ansible also started to contain complex logic. To prevent this complexity from turning into chaos, software development principles exist for organizing code. S.O.L.I.D. For example, Grigory Petrov in his report 'Why IT specialists need a personal brand' touched on the issue that a person, by nature, finds it easier to operate with certain social entities, which in software development are objects. If we combine these two ideas and continue to develop them, we can notice that in describing infrastructure, we can also use S.O.L.I.D. to make it easier to maintain and modify this logic in the future.

The Single Responsibility Principle

Each class performs only one task.

Do not mix code and create monolithic divine spaghetti monsters. The infrastructure should consist of simple building blocks. It turns out that if you break down the Ansible playbook into small pieces, read Ansible roles, they are easier to maintain.

The Open Closed Principle

The Principle of Open/Closed.

Open for extension: means that the behavior of an entity can be extended by creating new types of entities.
Closed for modification: as a result of extending the behavior of an entity, changes should not be made to the code that uses these entities.

Initially, we deployed the test infrastructure on virtual machines, but because the deployment business logic was separate from the implementation, we easily added deployment on bare metal.

The Liskov Substitution Principle

The Liskov Substitution Principle states that objects in a program should be replaceable with instances of their subtypes without altering the correctness of the program's execution.

If we look at it more broadly, it’s not just a feature of a specific project; it can be applied elsewhere. S.O.L.I.D., this generally applies to CFM; for instance, in another project, it’s necessary to deploy a boxed Java application over various Java servers, application servers, databases, OS, etc. I will use this example to discuss further principles. S.O.L.I.D.

In our case, within the infrastructure team, there's an agreement that if we have installed the role imbjava or oraclejava, we have a binary executable file for Java. This is necessary because higher-level roles depend on this behavior; they expect Java to be present. At the same time, it allows us to replace one implementation/version of Java with another without changing the application's deployment logic.

The issue lies in the fact that in Ansible, such implementation cannot be realized, hence, certain agreements arise within the team.

The Interface Segregation Principle

The Interface Segregation Principle states that many interfaces specifically designed for clients are better than one general-purpose interface.

Initially, we tried to consolidate all the variability of application deployment into one Ansible playbook, but it was difficult to maintain. The approach where our interface is specified externally (the client expects port 443) allows us to compose infrastructure from individual components for a specific implementation.

The Dependency Inversion Principle

The Dependency Inversion Principle states that high-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend on details. Details should depend on abstractions.

Here, the example will be based on an anti-pattern.

One of the clients had a private cloud.
Within the cloud, we ordered virtual machines.
However, due to the cloud's specifics, the deployment of the application was tied to the hypervisor that the VM landed on.

That is, the high-level logic of application deployment and dependencies passed to the underlying levels of the hypervisor, which meant problems when reusing this logic. Don't do that.

Interaction

Infrastructure as code is not only about code, but also about the relationships between code and people, about the interactions among infrastructure developers.

Bus factor

Let's say you have Vasya on your project. Vasya knows everything about your infrastructure; what will happen if Vasya suddenly disappears? This is a very real situation, as he could be hit by a bus. Sometimes this happens. If the knowledge about the code, its structure, how it works, along with the logins and passwords, is not distributed within the team, then you may face a number of unpleasant situations. To minimize these risks and distribute knowledge within the team, various approaches can be utilized.

Pair DevOpsing

It’s not like in a joke, where admins drink beer, change passwords, and this is similar to pair programming. That is, two engineers sit at one computer, one keyboard, and start configuring your infrastructure together: setting up the server, writing Ansible roles, etc. It sounds great, but it didn't work for us. However, specific cases of this practice were successful. A new employee, alongside their mentor, takes on a real task and works together—passing on knowledge.

Another specific case is an incident call. During a problem, a group of duty officers and involved parties gathers, one lead is appointed, who shares their screen and articulates their thought process. Other participants follow the lead's reasoning, look for tricks from the console, check to see if any lines were missed in the logs, and learn something new about the system. This approach worked more often than not.

Code Review

Subjectively, the dissemination of knowledge about the infrastructure and how it is structured was more effectively achieved through code reviews:

The infrastructure is described in code within the repository.
Changes occur in a separate branch.
During the merge request, you can see the delta of changes in the infrastructure.

The highlight here was that reviewers were chosen in turn, according to a schedule, meaning there was a certain chance you'd dive into a new area of the infrastructure.

Code Style

Over time, disagreements began to arise during reviews, as reviewers had their own styles, and the rotation of reviewers stacked them with different preferences: 2 spaces or 4, camelCase or snake_case. Implementing this wasn't straightforward.

The first idea was to recommend using a linter, after all, engineers are all smart. But different editors and OS made it inconvenient.
This evolved into a bot that would message in Slack about problematic commits and attach the linter output. However, in most cases, more urgent matters arose, and the code remained uncorrected.

Green Build Master

As time passed, we reached the point where we couldn't allow commits into master that didn't pass certain tests. Voilà! We invented the Green Build Master, which has long been practiced in software development:

Development occurs in a separate branch.
Tests are run on this branch.
If tests fail, the code won't be merged into master.

The decision was quite painful, as it sparked many disputes, but it was worth it, as merge requests began to arrive without style disagreements, and over time, the number of problematic areas decreased.

IaC Testing

In addition to style checks, you can use other methods, such as verifying that your infrastructure can actually be deployed. Or checking that changes in infrastructure won't lead to financial loss. Why is this necessary? It's a complex and philosophical question, best answered with a story about an auto-scaler on PowerShell that didn't check edge cases => too many VMs were created => the client spent more money than planned. Not pleasant, but this mistake could have been caught much earlier.

One might ask, why make complex infrastructure even more complicated? Tests for infrastructure, just like for code, are not about simplification, but about understanding how your infrastructure should work.

IaC Testing Pyramid

IaC Testing: Static Analysis

If you immediately deploy the entire infrastructure and check that it works, it may take a considerable amount of time and require a lot of resources. Therefore, the foundation should be something that works quickly, is plentiful, and covers many primitive aspects.

Bash is tricky

Let's consider a simple example: selecting all files in the current directory and copying them to another location. The first thing that comes to mind is:

for i in * ; do 
    cp $i /some/path/$i.bak
done

But what if there's a space in the file name? Well, okay, we're smart enough to use quotes:

for i in * ; do cp "$i" "/some/path/$i.bak" ; done

Good job? No! What if there’s nothing in the directory, i.e., globbing doesn’t work.

find . -type f -exec mv -v {} dst/{}.bak ;

Now are we good? Nope… Forgot that the file name might contain n.

touch x
mv x "$(printf "foonbar")"
find . -type f -print0 | xargs -0 mv -t /path/to/target-dir

Static analysis tools

The problem from the previous step could have been caught when we forgot the quotes; there are many tools for this purpose. Shellcheck, in fact, there are many, and you will probably find a linter for your IDE suitable for your stack.

Language
Tool

bash
Shellcheck

Ruby
RuboCop

python
Pylint

ansible
Ansible Lint

IaC Testing: Unit Tests

As we have seen from the previous example, linters are not omnipotent and cannot point out all problem areas. By analogy with testing in software development, we can recall unit tests. Immediately, the following come to mind: shunit, junit, rspec, pytest. But what about ansible, chef, saltstack, and similar tools?

In the beginning, we talked about S.O.L.I.D. and the fact that our infrastructure should consist of small building blocks. Their time has come.

Infrastructure is broken down into small building blocks, for example, Ansible roles.
Some environment is deployed, whether it's docker or VMs.
We apply our Ansible role to this test environment.
We check that everything has worked as we expect (we run tests).
We decide if it's okay or not.

IaC Testing: Unit Testing tools

The question is, what are tests for CFM? One could simply run a script, but one can also use ready-made solutions for this:

CFM
Tool

Ansible
Testinfra

Chef
Inspec

Chef
Serverspec

saltstack
Goss

An example for testinfra, checking that users test1, test2 exist and belong to the group sshusers:

def test_default_users(host):
    users = ['test1', 'test2']
    for login in users:
        assert host.user(login).exists
        assert 'sshusers' in host.user(login).groups

What to choose? This is a difficult and ambiguous question; here’s an example of changes in projects on GitHub from 2018-2019:

IaC Testing frameworks

The question arises: how to put it all together and run it? One option is to take and do everything yourself with enough engineers available. Or you can use ready-made solutions, though there aren't many:

CFM
Tool

Ansible
Molecule

Chef
Test Kitchen

Terraform
Terratest

An example of changes in projects on GitHub from 2018-2019:

Molecule vs. Testkitchen

Initially, we tried to use testkitchen:

to create VMs in parallel.
Apply Ansible roles.
Run inspec.

For 25-35 roles, it took 40-70 minutes, which was long.

The next step was to switch to jenkins / docker / ansible / molecule. Ideologically, it's all the same.

Lint the playbooks.
Lint the roles.
Start the container.
Apply Ansible roles.
Run testinfra.
Check idempotency.

Linting for 40 roles and tests for a dozen took about 15 minutes.

What to choose depends on many factors, such as the stack used, team expertise, etc. Everyone decides for themselves how to address the issue of unit testing.

IaC Testing: Integration Tests

On the next level of the testing pyramid, integration tests will appear. They are similar to unit tests:

Infrastructure is broken down into small bricks, for example, Ansible roles.
Some environment is deployed, whether it's docker or VMs.
This testing environment is applied. a variety of Ansible roles.
We check that everything works as we expect (we run tests).
We decide if it's okay or not.

Roughly speaking, we do not check the functionality of a single system element as in unit tests; we check how the server is configured as a whole.

IaC Testing: End to End Tests

At the top of the pyramid, we encounter End to End tests. That is, we do not check the functionality of a single server, a single script, or a single brick of our infrastructure. We check if multiple servers, united together, our infrastructure works as we expect. Unfortunately, I have not seen any ready-made box solutions, probably because infrastructure is often unique and difficult to template and create a framework for testing. As a result, everyone creates their own solutions. There is demand, but there is no answer. Therefore, I will share what exists to inspire others with sound ideas or point me out that everything has long been invented before us.

A project with a rich history. Used in large organizations, and probably each of you has indirectly encountered it. The application supports multiple databases, integrations, etc. Knowing how the infrastructure can look is a multitude of docker-compose files, and knowing which tests to run in which environment is jenkins.

This scheme worked for quite a while until we tried to move it to Openshift. The containers remained the same, but the execution environment changed (hello D.R.Y. again). a study We didn't try to move this to Openshift. The containers remained the same, but the runtime environment has changed (hello D.R.Y. again).

The idea of the research went further, and in OpenShift, there was something called APB (Ansible Playbook Bundle), which allows you to package knowledge on how to deploy infrastructure into a container. That is, there is a reproducible, testable point of knowledge on how to deploy infrastructure.

All of this sounded good until we hit a heterogeneous infrastructure: we needed Windows for testing. In the end, the knowledge of what, where, and how to deploy and test resides in Jenkins.