Many people know and use Terraform in their daily work, but there are still no best practices for it. Each team has to invent its own approaches and methods.
Your infrastructure almost certainly starts out simple: a few resources + a few developers. Over time, it grows in all directions. Do you find ways to group resources into Terraform modules, organize code into folders, and what else can possibly go wrong? (famous last words)
Time passes and you feel like your infrastructure is your new pet, but why? You are worried about inexplicable changes in the infrastructure, you are afraid to touch the infrastructure and code - as a result, you delay new functionality or reduce quality ...
After three years of managing a collection of Terraform community modules for AWS on Github and long-term maintenance of Terraform in production, Anton Babenko is ready to share his experience: how to write TF modules so that it doesn’t hurt in the future.
By the end of the talk, attendees will be more familiar with Terraform resource management principles, Terraform module-related best practices, and some infrastructure management-related continuous integration principles.
Disclaimer: I note that this report is dated November 2018 - 2 years have passed. The version of Terraform 0.11 considered in the report is no longer supported. Over the past 2 years, 2 new releases have been released, in which a lot of innovations, improvements and changes have appeared. Please pay attention to this and check the documentation.
Links:
- +
- — auto-formatting of code and documentation
- - generator of Terraform-modules (WIP)
- - Terraform code generator from visual diagrams (WIP)
- www.terraform-best-practices.com
- (New posts are on the personal website www.antonbabenko.com/)
- @antonbabenko - Twitter, and a bunch of different Slacks
My name is Anton Babenko. Some of you must have used the code I wrote. I will now speak about this with more confidence than ever, because I have access to statistics.
I'm involved with Terraform and have been an active contributor and contributor to a large number of open source projects related to Terraform and Amazon since 2015.
Since then, I've written enough code to make this interesting. And I will try to talk about this now.
I will talk about the intricacies and specifics of working with Terraform. But in fact it is not a subject for HighLoad. And now you will understand why.
Over time, I started writing Terraform modules. Users wrote questions, I rewrote them. Then I wrote various code formatting utilities with pre-commit hooks, etc.
There were many interesting projects. I like doing code generation because I like the computer to do more and more work for me and for the programmer, so I'm currently working on a Terraform code generator from visual diagrams. Perhaps some of you have seen them. These are beautiful boxes with arrows. And I think it's great if you can click the "Export" button and get it all as code.
I'm from Ukraine. I have lived in Norway for many years.
Also, the information for this report was collected from people who know my name and find me on social networks. I almost always have the same nickname.
As I mentioned, I am the main maintainer of Terraform AWS modules, which is one of the largest GitHub repositories where we host modules for the most common tasks: VPC, Autoscaling, RDS.
And what you have heard now is the most basic. If you doubt that you understand what Terraform is, then it is better to spend your time somewhere else. There will be a lot of technical terms here. And I did not hesitate to declare the level of the report as the highest. This means that I can talk using all the possible terms without much explanation.
Terraform appeared in 2014 as a utility that allowed you to write, plan and manage infrastructure as code. The key concept here is "infrastructure as code".
All documentation, as I said, is written in . I hope most people know about this site and have read the documentation. If so, then you are at the right place.
This is how a regular Terraform configuration file looks like, where we first define some variables.
In this case, we define "aws_region".
Then we describe what resources we want to create.
We run some commands, in particular "terraform init" in order to load dependencies, providers.
And we run the “terraform apply” command in order to check whether the specified configuration matches the resources that we created. Since we have not created anything before, Terraform suggests that we create these resources.
We confirm this. This is how we create a bucket called seasnail.
There are also several similar utilities. Many of you who use Amazon know AWS CloudFormation or Google Cloud Deployment Manager or Azure Resource Manager. Each of them has some kind of implementation for managing resources within each of these public cloud providers. Terraform is especially useful because it allows you to manage over 100 providers. (More )
The goals that Terraform pursued from the very beginning:
- Terraform gives you a single view of resources.
- Allows you to support all modern platforms.
- And Terraform was conceived from the very beginning as a utility that allows you to change the infrastructure safely and predictably.
In 2014, the word “predictably” sounded very unusual in this context.
Terraform is a universal utility. If you have an API, then you can control absolutely everything:
- You can use over 120 providers to manage everything your heart desires.
- For example, you can use Terraform to describe access to GitHub repositories.
- You can even create and close bugs in Jira.
- You can manage New Relic metrics.
- You can even create files in dropbox if you really want to.
This is all achieved with the help of Terraform providers, which have an open API that can be described in Go.
Let's say we started using Terraform, read some documentation on the site, watched some video, started writing main.tf, as I showed on the previous slides.
And everything is cool with you, you have a file that creates a VPC.
If you want to create a VPC, then you specify approximately these 12 lines. Describe in which region you want to create, which cidr_block of IP addresses to use. And that's it.
Naturally, the project will gradually grow.
And you'll be adding a bunch of new stuff there: resources, data sources, you'll be integrating with new providers, all of a sudden you'll want to use Terraform to manage users in your GitHub account, etc. You might want to use different DNS providers , cross everything. Terraform makes this easy.
Consider the following example.
You are gradually adding internet_gateway because you want resources from your VPC to have access to the internet. This is a good idea.
The result is this main.tf:
This is the top of main.tf.
This is the bottom part of main.tf.
Then you add subnet. By the time you want to add NAT gateways, routes, routing tables and a bunch of other subnets, you will not have 38 lines, but about 200-300 lines.
That is, your main.tf file is gradually growing. And quite often people put everything in one file. 10-20 Kb appears in main.tf. Imagine that 10-20 Kb is text content. And everything is connected to everything. This is getting more and more difficult to work with. 10-20 Kb is a good user case, sometimes more. And people don't always think it's bad.
As in conventional programming, i.e. not infrastructure as code, we are used to using a bunch of different classes, packages, modules, groupings. Terraform allows you to do the same.
- The code is growing.
- Resource dependencies are also growing.
And we have a big, big need. We understand that we can no longer live like this. At us the code becomes immense. 10-20 Kb is, of course, not very vast, but we are talking only about the network stack, i.e. you have added only network resources. We are not talking about Application Load Balancer, deployment ES cluster, Kubernetes, etc., where 100 Kb can still be easily woven. If you write all this, then you will very soon find out that Terraform provides Terraform modules.
Terraform modules are self-contained Terraform configurations that are managed as a group. That's all you need to know about Terraform modules. They are not smart at all, they do not allow you to make any complex connections depending on something. It all falls on the shoulders of the developers. That is, it's just some kind of Terraform configuration that you've already written. And you can just call it as a group.
So we're trying to figure out how we're going to optimize our 10-20-30 Kb of code. We gradually understand that we need to use some modules.
The first type of modules encountered are resource modules. They do not understand what your infrastructure is about, what your business is about, where and what conditions. These are exactly the modules that I, together with the open source community, administer, and which we put forward as the very initial building blocks for your infrastructure.
An example of a resource module.
When we call a resource module, we specify from which path we should load its contents.
We specify which version we want to download.
We pass a bunch of arguments there. And that's it. That's all we need to know when we use this module.
Many people think that if you use the latest version, then everything will be stable. But no. The infrastructure must be versioned, we must clearly answer which version this or that component was deployed.
Here is the code that is inside this module. security-group module. Here the scroll goes up to the 640th line. Creating a resource security-croup in Amazon in all sorts of configurations is a very non-trivial task. It is not enough just to create a security-group and tell it what rules to pass to it. It would be very easy. Inside Amazon, there are a million different restrictions. For example, if you use VPC endpoint, prefix list, various APIs and tries to cross it all with everything, then Terraform does not allow you to do this. And the Amazon API doesn't allow it either. Therefore, we need to hide all this terrible logic in a module and give the user a code that looks just like this.
The user does not need to know how it is done inside.
The second type of modules, which consists of resource modules, already solves tasks that are more applicable to your business. Often this is a place that is an extension for Terraform and sets some hard values for tags, for company standards. You can also add functionality there that Terraform does not currently allow you to use. It's right now. Now version 0.11, which is about to become a thing of the past. But still, preprocessors, jsonnet, cookiecutter and a bunch of other things are the auxiliary mechanism that must be used for full-fledged work.
Next, I will show some examples of this.
The infrastructure module is called in exactly the same way.
Specifies the source from where to download the content.
A bunch of values are passed, which are passed to this module.
Further, inside this module, a bunch of resource modules are called to create a VPC or Application Load Balancer, or to create a security-group or an Elastic Container Service cluster.
There are two types of modules. This is important to understand, because most of the information that I have grouped in this report is not written in the documentation.
And the documentation in Terraform right now is quite problematic, because it just says that there are such features, you can use them. But she does not say how to use these features, why it is better to use them. Therefore, a very large number of people write something that you can’t live with later.
Let's see how to write these modules next. Then we'll see how to call them and how to work with the code.
Terraform Registry -
Tip #0 is not to write resource modules. Most of these modules are already written for you. As I said, they are open source, they do not contain any of your business logic, they do not have hardcoded values for IP addresses, passwords, etc. The module is very flexible. And it's probably already written. There are a lot of modules for resources from Amazon. About 650. And most of them are of good quality.
In this example, someone comes to you and says, “I want to be able to manage the database. Create a module so I can create a database." The person does not know the implementation details of either Amazon or Terraform. It just says "I want to manage MSSQL". That is, we mean that it will call our module, pass the engine type there, and indicate the time zone.
And a person should not know that we will create two different resources inside this module: one for MSSQL, the second for everything else, just because in Terraform 0.11 you cannot specify time-zone values as optional.
And at the output of this module, a person will be able to simply receive the address. He will not know from which database, from which resource we create all this inside. This is a very important element of hiding. And this is applicable not only for those modules that are public in open source, but also for those modules that you will write inside your projects, teams.
This is the second argument, which is pretty important if you've been using Terraform for a while. You have a repository where you store all your Terraform modules for your company. And it is quite normal that over time this project will grow to a size of one or two megabytes. This is fine.
But the problem is how Terraform calls these modules. For example, if you call a module to create each individual user, then Terraform will first download the entire repository, and then go to the folder where this particular module is located. This way you will download one megabyte each time. If you manage 100 or 200 users, then you will download 100 or 200 megabytes, and then go to that folder. So, naturally, you don't want to load a bunch of stuff every time you hit "Terraform init".
There are two solutions to this problem. The first is to use relative paths. This is how you specify in the code that the folder is local (./). And before you run anything, you do a Git clone of that repository locally. So you do it once.
There are, of course, a bunch of downsides. For example, that you can't use versioning. And sometimes it's hard to live with it.
Second solution. If you have a lot of submodules and you already have some kind of installed pipeline, then there is an MBT project that allows you to collect many different packages from a monorepository and upload them to S3. This is a very good way. Thus, the iam-user-1.0.0.zip file will be only 1 Kb in size, because the code for creating this resource is very small. And it will work much faster.
Let's talk about what can not be used in modules.
Why is it evil in modules? The scariest thing is assume user. Assume user is an authentication option for a provider that can be used by different people. For example, we will all assume a role. This means that Terraform will take on this role. And then with this role will perform other actions.
And the evil is that if Vasya likes to connect to Amazon in one way, for example, using a variable environment by default, and Petya likes to use his shared key, which he has in a secret place, then both cannot be specified in Terraform. And in order for them not to experience suffering, this block should not be specified in the module. This should be indicated at the top level. That is, we have a resource module, an infrastructure module, and a composition on top. And somewhere higher it should be indicated.
The second evil is the provisioner. The evil here is not so trivial, because if you write code and it works for you, then you might think that if it works, then why change it.
The evil lies in the fact that you do not always control this provisioner when it will be launched specifically, firstly. And secondly, you don't control what aws ec2 means, i.e. we are talking now about Linux or about Windows. Thus, you cannot write something that will work the same on different operating systems or for different user cases.
The most common example, which is also indicated in the official documentation, is that if you write aws_instance, specify a bunch of arguments, then there is nothing wrong with that if you also specify the provisioner "local-exec" there and run your ansible-playbook .
Actually, yes, there is nothing wrong with that. But literally soon you will realize that this local-exec thing does not exist, for example, in launch_configuration.
And when you use launch_configuration, and you want to create an autoscaling group from one instance, then there is no concept of “provisioner” in launch_configuration. There is the concept of "user data".
Therefore, a more universal solution is to use user data. And it will be launched either on the instance itself, when the instance is enabled, or in the same user data, when the autoscaling group will use this launch_configuration.
If, nevertheless, there is a desire to launch a provisioner, because it is a gluing component, when one resource is created and at this moment you need to launch your provisioner, your team. There are many such situations.
And the most correct resource for this is called null_resource. Null_resource is a dummy resource that is never actually created. It doesn't touch anything, no API, no autoscaling. But it allows you to control when to run the command. In this case, the command is run at creation time.
Link
There are several signs. I won't go into all the features in great detail. There is an article about it. But if you have worked with Terraform or used other people's modules, then you often noticed that many modules, like most of the code in open source, are written by people for their own needs. The man wrote it, solved his problem. I made it on GitHub, let it live. It will live, but if there is no documentation and examples, then no one will use it. And if there is no functionality that allows you to solve a little more than its specific task, then no one will use it either. There are so many ways to lose users.
If you want to write something for people to use, then I recommend following these signs.
It:
- Documentation and examples.
- Full functionality.
- Reasonable defaults.
- Clean code.
- Tests.
Tests are a separate situation because they are quite difficult to write. I believe more in documentation and in examples.
So we've seen how to write modules. There are two arguments. First, and most importantly, don't write if you can, because a bunch of people have already done these tasks before you. And the second, if you still decide, then try not to use providers in modules and provisioner.
This is the gray part of the documentation. Now you might be thinking, “Something is not clear. Not convinced." But we'll see in six months.
Now let's talk about how to call these modules.
We understand that our code grows over time. We no longer have one file, we already have 20 files. All of them are in one folder. Or maybe five folders. Maybe we are starting to somehow break them down by region, by some components. Then we understand that now we have some rudiments of synchronization, orchestration should arise. That is, we must understand what to do if we changed network resources, what to do with the rest of our resources, how to call these dependencies, etc.
There are two extremes. The first extreme is all in one. We have one master file. For the time being, this was the official best practice on the Terraform website.
But now it is written as deprecated and removed. Over time, the Terraform community realized that this is far from being the best practice, because people start using the project in different ways. And there are problems. For example, when we specify all dependencies in one place. There are situations when we click "Terraform plan" and while Terraform updates the states of all resources, a lot of time can pass.
A lot of time is, for example, 5 minutes. For some, this is a lot of time. I have seen cases where it took 15 minutes. 15 minutes AWS API twitched in order to understand what was happening with the state of each resource. This is a very large area.
And, of course, a related problem will appear when you want to change something in one place, then you wait 15 minutes, and it gives you a canvas of some changes. You spat, wrote "Yes", and something went wrong. This is a very real example. Terraform does not try to isolate you from problems. That is, write what you want. There will be problems - your problems. While Terraform 0.11 does not try to help you in any way. There are certain interesting places in 0.12 that allow you to say: "Vasya, do you really want this, can you change your mind?".
The second way is to reduce this area, i.e., calls from one place from another place can be less connected.
The only problem is that you need to write more code, i.e. you need to describe variables in a large number of files, update this. Someone doesn't like it. For me, this is normal. And some people think: “Why write this in different places, I’ll zakolbas it all in one place.” It is possible and so, but this is the second extreme.
Who has it all in one place? One, two, three people, that is, someone uses.
And who calls one particular component, one block or one infrastructure module? Five to seven people. This is cool.
The most common answer is somewhere in the middle. If the project is large, then you will often have a situation where neither solution is suitable and not everything works out there, so you end up with a mixture. There is nothing wrong with this, as long as you understand that there is an advantage to both.
If something has changed in the stack VPC and you want to apply these EC2 changes, i.e. you want to update the autoscaling group because you have a new subnet, then I call this kind of dependency orchestration. There are some solutions: who uses what?
I can suggest what solutions are. You can use Terraform to do the magic, or you can use makefiles to use Terraform. And look, if something has changed there, you can run it here.
How do you like this decision? Does anyone believe that this is a cool solution? I see a smile, I see doubts crept in.
Of course, do not repeat this at home. Terraform was never designed to run from within Terraform.
I was told at one report: “No, this will not work.” The thing is, it shouldn't work. Although it looks so impressive when you can launch Terraform from Terraform, and there is also Terraform, but you don’t have to do that. Terraform should always start very simply.
If you need to orchestrate calls when something has changed in one place, then there is Terragrunt.
Terragrunt is a utility, it is an add-on to Terraform, which allows you to coordinate and orchestrate calls to infrastructure modules.
A typical Terraform configuration file looks like this.
You specify which specific module you want to call.
Which module has dependencies.
And what arguments this module accepts. That's all there is to know about Terragrunt.
The documentation is there, 1 stars on GitHub are also there. But in most cases, this is what you need to know. And this is very easy to implement in companies that have just started working with Terraform.
So orchestration is Terragrunt. There are other options.
Now let's talk about how to work with the code.
If you have a need to add new features to the code in most cases it's easy. You write a new resource, everything is simple here.
If you have some kind of resource that you created in advance, for example, you learned about Terraform after you opened an AWS account and want to use the resources that you already have, then it would be appropriate to extend your module in this way, so that it supports the use of existing resources.
And supported the creation of new resources using the block resource.
On output, we always return an output id depending on what was used.
The second very significant problem in Terraform 0.11 is working with lists.
The difficulty lies in the fact that if we have such a list of users.
And when we create these users using the block resource, everything goes fine. We go through the entire list, create a file for each. Everything is fine. And then, for example, user3, which is in the middle, should be removed from here, then all the resources that were created after it, they will be recreated, because the index will change.
Working with lists in a stateful environment. What is a stateful environment? This is the situation where a new value occurs when the resource is created. For example, AWS Access Key or AWS Secret Key, i.e. when we create a user, we receive a new Access or Secret Key. And every time we delete a user, that user will have a new key. But this is not feng shui, because the user will not want to be friends with us if we create a new user for him every time someone leaves the team.
The solution is this. This is code written in Jsonnet. Jsonnet is a templating language from Google.
This command allows you to accept this template and at the output it returns a json file that is made according to your template.
The template looks like this.
Terraform allows you to work with both HCL and Json in the same way, so if you have the ability to generate Json, then you can slip it into Terraform. The .tf.json file will be successfully uploaded.
And then we work with it as usual: terraform init, terramorm apply. And we create two users.
Now we are not afraid if someone leaves the team. We will just edit the json file. Vasya Pupkin left, Petya Pyatochkin stayed. Petya Pyatochkin will not receive a new key.
Integrating Terraform with other tools is not, in fact, Terraform's job. Terraform was created as a resource creation platform and that's it. And everything that comes after that is not Terraform's concern. And you don't have to put it in there. There is Ansible, which does everything that is needed.
But there are situations when we want to complement Terraform and call some command after something has been completed.
First way. We create output where we write this command.
And then we call this command from shell terraform output and specify this value that we want. Thus, the command is executed with all the substituted values. It is very comfortable.
The second way. This is the use of null_resource depending on changes in our infrastructure. We can call the same local-exec as soon as the ID of some resource changes.
Naturally, this is all smooth on paper, because Amazon, like all other public providers, has a bunch of its own edge cases.
The most common edge cases are that when you open an AWS account, it matters what regions you use; whether this feature is included there; maybe you opened it after December 2013; maybe you are using default in VPC etc. There are many limitations. And Amazon scattered them all over the documentation.
There are a few things I recommend avoiding.
To begin with, avoid all non-secret arguments inside the Terraform plan or Terraform CLI. All this can be added either to a tfvars file or to an environment variable.
But you don't have to memorize this whole magic command. Terraform plan - var and away we go. The first variable is var, the second variable is var, the third, fourth. The most important infrastructure-as-code principle that I use the most is that just by looking at the code, I should be able to understand perfectly what is deployed there, in what state and with what values. And so I do not need to read the documentation or ask Vasya about what parameters he used to create our cluster. It is enough for me to open a file with the tfvars extension, which often matches the environment, and see everything there.
Also don't use target arguments to reduce the scope. It is much easier to use small infrastructure modules for this.
Also, do not limit and increase parallelism. If I have 150 resources and want to increase Amazon's parallelism from 10, which is the default, to 100, then something is likely to go wrong. Or it might go well now, but when Amazon says you're making too many calls, you're in trouble.
Terraform will try to restart most of these problems, but you will achieve almost nothing. Parallelism=1 is an important thing to use if you stumble over some bug inside the AWS API or inside the Terraform provider. And then you need to specify: parallelism=1 and wait until Terraform finishes one call, then the second, then the third. He will launch them one by one.
People often ask me: "Why do I think that Terraform workspaces are evil?". I believe the principle of infrastructure as code is to see what infrastructure has been created and with what values.
Workspaces was not created by users. This does not mean that users wrote in GitHub issues that they cannot live without Terraform workspaces. No not like this. Terraform Enterprise is a commercial solution. Terraform from HashiCorp decided that we needed workspaces, so we will file it down. I find it much easier to put it in a separate folder. Then there will be a little more files, but it will be clearer.
How to work with code? In fact, working with lists is the only pain. And take Terraform easier. It's not the kind of thing that will make you feel great. No need to shove everything that is written in the documentation there.
The topic of the report was written “for the future”. I will say this very briefly. For the future, this means that 0.12 will be released soon.
0.12 is a bunch of new stuff. If you come from ordinary programming, then you miss all sorts of dynamic blocks, loops, regular and conditional comparison operations, where the left and right parts are not calculated simultaneously, but depending on the situation. You miss it very much, so 0.12 will solve it for you.
But! If you write less and easier, using ready-made modules, third-party solutions, then you will not have to wait and hope that 0.12 will come and fix everything for you.
Thanks for the report! You spoke about infrastructure as code and literally said one word about tests. Are unit tests necessary? Whose responsibility is this? Do I have to write it myself or is it the responsibility of the modules?
Next year will be bombarded with reports that we have decided to test everything. What to test is the biggest question. There are a lot of dependencies, a lot of restrictions from different providers. When you and I are talking and you say: “I need tests”, then I ask: “What will you test?”. You say that you will test in your region. Then I say that in my region it does not work. That is, we cannot even agree on this with you. Not to mention that there are a lot of technical problems. That is, how to write these tests so that they are adequate.
I am actively researching this topic, i.e. how to automatically generate tests based on the infrastructure that you wrote. That is, if you wrote this code, then I need to run it, based on this I can create tests.
- this is one of the most frequently mentioned libraries that allows you to write integration tests for Terraform. This is one of the utilities. I prefer a DSL type like rspec.
Anton, thanks for the report! My name is Valery. Let me ask you a little philosophical question. There is, conditionally, provisioning, there is deployment. Provisioning creates my infrastructure, in deployment we fill it with something useful, for example, servers, applications, etc. And it sits in my head that Terraform is more for provisioning, and Ansible is more for deployment, because Ansible is also on the physical infrastructure allows you to install nginx, Postgres. But at the same time, Ansible seems to allow you to provision, for example, Amazon or Google resources. But Terraform also allows you to deploy some software with the help of its modules. From your point of view, is there any border between Terraform and Ansible, where and what is better to use? Or, for example, do you think that Ansible is already garbage, should you try to use Terraform for everything?
Good question, Valery. I believe that Terraform has not changed in terms of purpose since 2014. It was built for infrastructure and died for infrastructure. We still had and will need the configuration management of Ansible. The challenge is that there is user data inside launch_configuration. And there you pull Ansible, etc. This is the standard delimitation that I like the most.
If we are talking about in beautiful infrastructure, then there are utilities like Packer that build this image. And then Terraform uses the data source to find this image and update its launch_configuration. That is, in this way, the pipeline consists in the fact that we first pull Tracker, then pull Terraform. And if there was a build, then a new change occurs.
Hello! Thanks for the report! My name is Misha, RBS company. You can call Ansible through a provisioner when creating a resource. And also in Ansible there is such a topic as dynamic inventory. And you can first call Terraform, and then call Ansible, which will take resources from the state and execute it. What's better?
And that and that people use with identical success. It seems to me that dynamic inventory in Ansible is a convenient thing, if we are not talking about an autoscaling group. Because in the autoscaling group we already have our own toolkit called launch_configuration. We write in launch_configuration everything that needs to be launched when we create a new resource. Therefore, with Amazon, using dynamic inventory and reading the Terraform ts file is overkill in my opinion. And if you use other tools where there is no concept of “autoscaling group”, for example, you use DigitalOcean or some other provider where there is no autoscaling group, then there you will need to pull the API handles, find IP addresses, create a dynamic inventory file , and Ansible will already roam it. That is, for Amazon there is launch_configuration, and for everything else there is dynamic inventory.
Source: habr.com