Representing infrastructure as code in a repeatable text format is a best practice for systems that doesn't require complex maneuvering. This practice is known as , and currently, there are two popular tools for implementing it, especially in AWS: and .
I compare my experiences with Terraform and CloudFormation
Before joining (also known as ) I worked and for about three years I used Terraform. At my new job, I also employed Terraform extensively, but then the company pushed for a complete transition to all things Amazon, including CloudFormation. I diligently developed best practices for both, using both tools in very complex organizational workflows. Later, after thoughtfully weighing the implications of transitioning from Terraform to CloudFormation, I concluded that Terraform is probably the best choice for the organization.
Terraform is Awful
Software Beta
Terraform hasn’t even released version 1.0 yet, which is a solid reason not to use it. Since I first tried it, it has changed significantly, but back then it often broke after several updates or just after a couple of years of use. I would say, 'now things are different', but... everyone says that, right? There are changes incompatible with previous versions, although they make sense, and it even feels like the syntax and resource storage abstractions are now as they should be. The tool seems to have genuinely improved, but... :-0 terraform apply On the other hand, AWS has done well in maintaining compatibility with previous versions. This is probably because their services are thoroughly tested internally before being published under a new name. So saying they 'did well' is an understatement. Maintaining compatibility with previous API versions for a highly variable and complex system like AWS is incredibly challenging. Anyone who has had to support publicly available APIs, used as widely, must understand how difficult this is over the years. However, as far as I remember, CloudFormation's behavior has not changed once over the years.
Meet your foot... it’s a bullet
As far as I know, deleting a resource
of a third party stranger It's impossible to import a CloudFormation stack into your CF stack. The same situation applies to Terraform. It allows you to import existing resources into your stack. The feature is remarkable, but with great power comes great responsibility. You can only add a resource to the stack, and while you're working with your stack, you cannot remove or modify that resource. This came back to haunt us once. Someone on Twitch, not intending any harm, accidentally imported someone else's AWS security group into their own Terraform stack. They ran a few commands and… the security group (along with incoming traffic) disappeared.
Terraform the Great
Recovery from Incomplete States
Sometimes, CloudFormation cannot fully transition from one state to another. In this case, it will attempt to revert to the previous state. Unfortunately, this is not always possible. Debugging what results is sometimes daunting — you never know if CloudFormation will be pleased that it's being hacked — even for repairs. Whether it is possible to revert to the previous state is something it cannot determine well, and by default, it hangs for hours in anticipation of a miracle.
Terraform, on the other hand, tends to recover from failed transitions in a more graceful manner and offers an extended debugging toolkit.
Clearer Changes in Document States
"Alright, load balancer, you're changing. But how?"
— a worried engineer, ready to press the 'accept' button.
Sometimes I need to make some adjustments to the load balancer in the CloudFormation stack — for example, to add a port number or change the security group. CloudFormation does a poor job of displaying changes. I, like on pins and needles, double-check the YAML file about ten times to ensure that I haven't deleted anything necessary or added something unnecessary.
Terraform is much clearer in this regard. Sometimes it's even too clear (read: annoying). Fortunately, the latest version includes improved change visibility — now it's clear what is changing.
doesn’t stop at one thing, like Yandex.Metrica, but evolves and is used in an increasing number of different projects and industries. It can be expanded by adding new features to address new tasks. For instance, it is often believed that storing logs in a DB is outdated, which is why
Write software from the end.
Speaking frankly, the most important distinguishing feature of long-lived software is its ability to adapt to changes. Write any software from the end goal backwards. I often fell into the trap of taking a 'simple' service and then trying to stuff everything into a single stack of CloudFormation or Terraform. And of course, after months, it turned out that I had misunderstood everything, and the service was not actually simple! I needed to somehow break the large stack into smaller components. When working with CloudFormation, this is only possible by first recreating the existing stack, which I, with my databases, do not do. However, Terraform allowed me to dissect the stack and break it down into more manageable smaller parts.
Modules in Git
Sharing Terraform code between multiple stacks is much easier than sharing CloudFormation code. With Terraform, you can place the code in a Git repository and reference it using semantic version control. Anyone with access to this repository can reuse the shared code. The CloudFormation equivalent is S3, but it lacks the same benefits, and there is no reason for us to abandon Git in favor of S3.
The organization grew, and the ability to share common stacks reached a critical level. With Terraform, this happens easily and naturally, while CloudFormation will make you jump through hoops before you achieve something similar.
Operations as Code
"Let's script it and call it a day."
- an engineer three years before inventing the Terraform bicycle.
When it comes to software development, Go or a Java program is not just code.
Code as Code
There is still the infrastructure on which it operates.
Infrastructure as Code
But where does it come from? How do you monitor it? Where does your code reside? Do developers need permission to access it?
Operations as Code
Being a software developer is not just about writing code.
Not by AWS alone: you surely utilize services from other providers. SignalFx, PagerDuty, or GitHub. Perhaps you have an internal Jenkins server for CI/CD or an internal Grafana dashboard for monitoring. Infra as Code is chosen for various reasons, and each is equally important for everything related to software.
When I worked at Twitch, we accelerated services within mixed embedded systems and AWS Amazon's systems. We were stamping and maintaining numerous microservices, increasing operational costs. Discussions were roughly along these lines:
- I: Damn, that's a lot of movement to speed up a single microservice. I'll have to use this thing to create an AWS account (we were heading for 2 accounts on microservice), then that one for setting up notifications, this one for the code repository, and this one for the email list, and this one...
- Lead: Let's script it and that's it.
- I: Okay, but the script will change. We’ll need a way to check that all these embedded Amazon gadgets are in the current state.
- Lead: Sounds good. And we'll write a script for that.
- I: Great! The script will definitely need some parameters. Will it accept them?
- Lead: Of course it will, where else would it go!
- I: The process may change, losing backward compatibility. We'll need some kind of semantic version control.
- Lead: Great idea!
- I: Tools can be changed manually, in the user interface. We'll need a way to check and fix this.
...3 years later:
- Lead: And we ended up with terraform.
The moral of the story is: even if you are up to your ears in everything Amazon, you still use something not from AWS, and those services have a state that uses a language for configuration to sync that state.
CloudFormation lambda vs git modules terraform
lambda is CloudFormation's solution for custom logic issues. With lambda, you can or . This approach presents additional complexities that are not present in the semantic version control of git modules in Terraform. For me, the most pressing issue became managing permissions for all these custom lambdas (which involves dozens of AWS accounts). Another important issue was the chicken or egg problem related to lambda code. This function itself is both infrastructure and code, and it also needs monitoring and updates. The final nail in the coffin was the difficulty in semantically updating lambda code changes; it also needed to ensure that stack actions wouldn't change between runs without direct commands.
I remember wanting to create a canary deployment for an Elastic Beanstalk environment with a classic load balancer. It would have been easiest to set up a second deployment for EB alongside the production environment, taking one more step: linking the autoscaling group of the canary deployment with the LB deployment in the production environment. And since Terraform uses , this would require four extra lines of code in Terraform. When I inquired whether there was a comparable solution in CloudFormation, I was pointed to an entire Git repository with a deployment pipeline and other resources: and all this just for what could be accomplished with a measly four lines of Terraform code.
It better detects drift
Ensure reality aligns with expectations.
— is a very powerful operations as code feature because it helps ensure that reality aligns with expectations. It is available with both CloudFormation and Terraform. However, as the stack grows, drift detection in CloudFormation produced more and more false positives.
With Terraform, you have much more advanced lifecycle hooks for drift detection. For example, you can input the command right in the ECS task definition if you want to ignore changes to a specific task definition while not ignoring changes across the entire ECS deployment.
CDK and the future of CloudFormation
CloudFormation is difficult to manage at large, cross-infrastructure scales. Many of these challenges are acknowledged, and the tool needs things like , a framework for defining cloud infrastructure in code and passing it through AWS CloudFormation. It will be interesting to see what awaits aws-cdk in the future, but it will struggle to compete with the other advantages of Terraform; to catch up with CloudFormation will require major changes.
To avoid disappointment with Terraform
This is 'infrastructure as CODE', not 'as text'.
My first impression of Terraform was fairly negative. I think I just didn’t understand the approach. Almost all engineers at first involuntarily perceive it as a text format that needs to be transformed into the desired infrastructure. YOU DON'T HAVE TO THINK THAT WAY.
The fundamentals of good software development also apply to Terraform.
I've seen many best practices for writing good code being ignored in Terraform. You've spent years learning to become a good programmer. Don't throw away that experience just because you're working with Terraform. The basic principles of good software development apply to Terraform as well.
How can you not document your code?
I've come across huge Terraform stacks with absolutely no documentation. How can you write code spanning pages without any documentation? Add documentation explaining your Terraform code, why this section is so important, and what you are doing. code Terraform (emphasis on the word 'code'), why this section is so important, and what you are doing.
How can you deploy services that once were a single large main() function?
I have encountered very complex Terraform stacks represented as a single module. Why don’t we deploy software this way? Why do we break large functions into smaller ones? The same answers apply for Terraform. If your module is too large, you need to break it into smaller modules.
Doesn't your company use libraries?
I've seen engineers, kickstarting a new project using Terraform, mindlessly copy-and-pasting huge chunks from other projects into their own, and then tweaking them until they started working. Would you work this way with 'production' code in your company? We don't just use libraries for no reason. Yes, but where would we be without shared libraries in the first place?!
Don't you use PEP8 or gofmt?
Most languages have a standard formatting scheme. In Python, it's PEP8. In Go, it's gofmt. Terraform has its own: terraform fmt. Use it freely!
Would you use React without knowing JavaScript?
Terraform modules can simplify some part of the complex infrastructure you create, but that doesn't mean you can ignore it completely. Want to use Terraform correctly without understanding the resources? You're doomed: time will pass and you still won't master Terraform.
Are you coding singletons, or injecting dependencies?
Dependency injection is a recognized best practice for software development, preferred by singletons. How does this apply to Terraform? I've encountered Terraform modules that depend on remote state. Instead of writing modules that extract from remote state, create a module that accepts parameters. Then pass those parameters into the module.
Do your libraries do ten things well or one thing excellently?
Libraries work best when focused on a single task that they perform excellently. Rather than writing large Terraform modules that attempt to do everything at once, break them down into parts that do one thing well. Then combine them as needed.
How do you make changes in libraries without breaking backward compatibility?
A general Terraform module, much like a regular library, needs to communicate changes to users without breaking backward compatibility. When such changes occur in libraries, it can be frustrating, and the same frustration applies when breaking changes are made in Terraform modules. It's recommended to use git tags and semver when working with Terraform modules.
Is your production service running on your laptop or in a data center?
Hashicorp has tools like for running your terraform. These centralized services simplify the management, auditing, and approval of terraform changes.
Aren't you writing tests?
Engineers acknowledge the need for code testing, but often neglect checks when working with Terraform. This can lead to tricky situations in infrastructure. I recommend 'testing' or 'creating examples' of stacks using modules that can be properly deployed for verification during CI/CD.
Terraform and microservices
The life and death of microservice companies depend on the speed, updating, and decommissioning of new microservice working stacks.
The most common negative aspect associated with microservice architectures, which cannot be avoided, relates to work rather than code. If you view Terraform merely as a way to automate just the infrastructural side of microservice architecture, you deprive yourself of the true advantages of this system. Now, it’s all about .
Source: habr.com
