Pitfalls of Terraform

Let's highlight some pitfalls, including those related to loops, if statements, and deployment methodologies, as well as more general issues concerning Terraform in general:

• the count and for_each parameters have limitations;
• zero downtime deployment constraints;
• even a good plan can turn out unsuccessful;
• refactoring can have its drawbacks;
• eventual consistency aligns... with procrastination.

The count and for_each parameters have limitations

In the examples of this chapter, the count parameter and for_each expression are actively used in loops and conditional logic. They perform well, but there are two important limitations to be aware of.

• You cannot reference any resource output variables in count and for_each.
• count and for_each cannot be used in module configuration.

You cannot reference any resource output variables in count and for_each.

Imagine you need to deploy several EC2 servers, and for some reason, you don't want to use an ASG. Your code might look like this:

resource "aws_instance" "example_1" {
   count             = 3
   ami                = "ami-0c55b159cbfafe1f0"
   instance_type = "t2.micro"
}

Let's consider them one by one.

Since the count parameter is assigned a static value, this code will work without issues: when you run the apply command, it will create three EC2 servers. But what if you want to deploy one server in each Availability Zone (AZ) within the current AWS region? You can set your code to load the list of zones from the aws_availability_zones data source and then loop through each of them to create an EC2 server using the count parameter and index access to the array:

resource "aws_instance" "example_2" {
   count                   = length(data.aws_availability_zones.all.names)
   availability_zone   = data.aws_availability_zones.all.names[count.index]
   ami                     = "ami-0c55b159cbfafe1f0"
   instance_type       = "t2.micro"
}

data "aws_availability_zones" "all" {}

This code will also work perfectly since the count parameter can successfully reference data sources. But what happens if the number of servers you need to create depends on the output of some resource? To demonstrate this, it's easiest to take the random_integer resource, which, as you can guess by its name, returns a random integer:

resource "random_integer" "num_instances" {
  min = 1
  max = 3
}

This code generates a random number from 1 to 3. Let's see what happens if we try to use the output result of this resource in the count parameter of the aws_instance resource:

resource "aws_instance" "example_3" {
   count             = random_integer.num_instances.result
   ami                = "ami-0c55b159cbfafe1f0"
   instance_type = "t2.micro"
}

If you run terraform plan for this code, you will get the following error:

Error: Invalid count argument

   on main.tf line 30, in resource "aws_instance" "example_3":
   30: count = random_integer.num_instances.result

The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.

Terraform requires that count and for_each be computed at the planning stage, before creating or modifying any resources. This means that count and for_each can reference literals, variables, data sources, and even lists of resources (provided their length can be determined during planning), but not computed output variables of resources.

count and for_each cannot be used in module configuration

At some point, you may be tempted to add a count parameter to module configurations:

module "count_example" {
     source = "..\/..\/..\/..\/modules\/services\/webserver-cluster"

     count = 3

     cluster_name = "terraform-up-and-running-example"
     server_port = 8080
     instance_type = "t2.micro"
}

This code tries to use count within the module to create three copies of the webserver-cluster resource. Or perhaps you want to make the module attachment optional depending on some boolean condition by assigning its count parameter the value 0. Such code may seem quite reasonable, but as a result of running terraform plan, you will receive the following error:

Error: Reserved argument name in module block

   on main.tf line 13, in module "count_example":
   13: count = 3

The name "count" is reserved for use in a future version of Terraform.

Unfortunately, as of the release of Terraform 0.12.6, the use of count or for_each in a module resource is not supported. According to the release notes for Terraform 0.12 (http://bit.ly/3257bv4), HashiCorp plans to add this feature in the future, so depending on when you are reading this book, it may already be available. To find out for sure, check the Terraform changelog here.

Zero downtime deployment limitations

Using the create_before_destroy block in conjunction with ASG is an excellent solution for organizing zero-downtime deployments, with one caveat: scaling policies are not supported. In other words, it resets the ASG size back to min_size during every deployment, which can be an issue if you relied on scaling policies to increase the number of running servers.

For example, the webserver-cluster module contains a couple of aws_autoscaling_schedule resources that increase the number of servers in the cluster from two to ten at 9 AM. If a deployment is executed, say, at 11 AM, the new ASG group will start with two servers instead of ten and will remain in that state until 9 AM the next day.

This limitation can be circumvented in several ways.

• Change the recurrence parameter in aws_autoscaling_schedule from 0 9 * * * ("run at 9 AM") to something like 0-59 9-17 * * * ("run every minute from 9 AM to 5 PM"). If there are already ten servers in the ASG, re-execution of this scaling policy won't change anything, which is our goal. However, if the ASG group has just recently been deployed, this rule ensures that within a minute, the number of its servers will reach ten. It's not the most elegant approach, and large jumps from ten to two servers and back can also cause issues for users.
• Create a custom script that utilizes the AWS API to determine the number of active servers in the ASG, invoke it using an external data source (see "External Data Source" on p. 249), and set the desired_capacity parameter of the ASG group to the value returned by this script. This way, each new ASG instance will always start with the same capacity, complicating maintenance of your Terraform code.

Of course, ideally, Terraform should have built-in support for zero-downtime deployments, but as of May 2019, the HashiCorp team did not plan to add this functionality (details here).

The correct plan may be unsuccessfully implemented

Sometimes, when executing the plan command, a seemingly correct deployment plan is generated, but the apply command returns an error. For example, try adding the aws_iam_user resource with the same name you used for the IAM user you created earlier in Chapter 2:

resource "aws_iam_user" "existing_user" {
   # Enter the name of the existing IAM user here,
   # to practice using the terraform import command
   name = "yevgeniy.brikman"
}

Now, if you run the plan command, Terraform will output a deployment plan that seems reasonable at first glance:

Terraform will perform the following actions:

   # aws_iam_user.existing_user will be created
   + resource "aws_iam_user" "existing_user" {
         + arn                  = (known after apply)
         + force_destroy   = false
         + id                    = (known after apply)
         + name               = "yevgeniy.brikman"
         + path                 = "\/"
         + unique_id         = (known after apply)
      }

Plan: 1 to add, 0 to change, 0 to destroy.

If you execute the apply command, the following error will occur:

Error: Error creating IAM User yevgeniy.brikman: EntityAlreadyExists:
User with name yevgeniy.brikman already exists.

   on main.tf line 10, in resource "aws_iam_user" "existing_user":
   10: resource "aws_iam_user" "existing_user" {

The problem, of course, is that an IAM user with that name already exists. This can happen not only with IAM users but with almost any resource. Someone may have created this resource manually or via the command line, but in any case, the ID conflict leads to issues. This error has many variations that often catch new Terraform users off guard.

The key point is that the terraform plan command considers only the resources that are listed in the Terraform state file. If resources are created in some other way (for example, manually, by clicking in the AWS console), they will not be included in the state file and, therefore, Terraform will not account for them when executing the plan command. As a result, a plan that appears correct at first glance will ultimately fail.

From this, two lessons can be drawn.

• If you have already started working with Terraform, do not use anything else. If part of your infrastructure is managed with Terraform, you can no longer modify it manually. Otherwise, you not only risk encountering strange Terraform errors but also undermine many benefits of IaC since the code will no longer accurately represent your infrastructure.
• If you already have some infrastructure in place, use the import command. If you are starting to use Terraform with existing infrastructure, you can add it to the state file using the terraform import command. This way, Terraform will know which infrastructure to manage. The import command takes two arguments. The first is the resource address in your configuration files. Here, the same syntax is used as in resource references: _. (like aws_iam_user.existing_user). The second argument is the identifier of the resource you want to import. For example, for the aws_iam_user resource, the resource ID is the username (like yevgeniy.brikman), while the aws_instance resource ID would be the EC2 server identifier (like i-190e22e5). How to import a resource is usually specified in the documentation at the bottom of its page.

  Below is the import command that allows you to synchronize the aws_iam_user resource you added to your Terraform configuration along with the IAM user from Chapter 2 (naturally, replace yevgeniy.brikman with your username):

  $ terraform import aws_iam_user.existing_user yevgeniy.brikman

  Terraform will call the AWS API to find your IAM user and create a link in the state file between it and the aws_iam_user.existing_user resource in your Terraform configuration. From that point on, when you run the plan command, Terraform will know that the IAM user already exists and will not attempt to create it again.

  It is worth noting that if you have many resources you want to import into Terraform, manually writing the code and importing each of them one by one can be a tedious task. Therefore, it’s advisable to look at a tool like Terraforming (http://terraforming.dtan4.net/), which can automatically import code and state from your AWS account.

  Refactoring can have its pitfalls

  Refactoring is a common practice in programming, where you change the internal structure of the code while keeping its external behavior unchanged. This is necessary to make the code clearer, cleaner, and easier to maintain. Refactoring is an indispensable technique that should be applied regularly. However, when it comes to Terraform or any other IaC tool, one should be very cautious about what is meant by the 'external behavior' of a piece of code, otherwise unforeseen issues may arise.

  For example, a common type of refactoring is replacing variable or function names with more understandable ones. Many IDEs have built-in support for refactoring and can automatically rename variables and functions throughout the entire project. In general-purpose programming languages, this is a trivial procedure that can be overlooked; however, in Terraform, caution is crucial, or you may encounter operational disruptions.

  For instance, the webserver-cluster module has an input variable cluster_name:

  variable "cluster_name" {
     description = "The name to use for all the cluster resources"
     type          = string
  }

  Imagine you started using this module to deploy a microservice named foo. Later, you wanted to rename your service to bar. This change may seem trivial, but in reality, it can lead to operational issues.

  The thing is, the webserver-cluster module uses the variable cluster_name across a number of resources, including the name parameter of two security groups and the ALB:

  resource "aws_lb" "example" {
     name                    = var.cluster_name
     load_balancer_type = "application"
     subnets = data.aws_subnet_ids.default.ids
     security_groups      = [aws_security_group.alb.id]
  }

  If you change the name parameter in any resource, Terraform will delete the old version of that resource and create a new one in its place. But if that resource is an ALB, during the period between its deletion and the loading of the new version, you will have no mechanism to redirect traffic to your web server. Similarly, if a security group is deleted, your servers will start rejecting any network traffic until a new group is created.

  Another type of refactoring that might interest you is changing the Terraform identifier. Let's take the aws_security_group resource in the webserver-cluster module as an example:

  resource "aws_security_group" "instance" {
    # (...)
  }

  The identifier of this resource is called instance. Imagine that during refactoring, you decided to change it to a more understandable name, in your opinion, cluster_instance:

  resource "aws_security_group" "cluster_instance" {
     # (...)
  }

  What will ultimately happen? Correct: an operational disruption.

  Terraform associates the ID of each resource with the cloud provider's identifier. For example, iam_user is linked to the IAM user ID in AWS, while aws_instance is tied to the AWS EC2 server ID. If you change the resource identifier (say, from instance to cluster_instance, as in the case with aws_security_group), Terraform will see this as if you deleted the old resource and added a new one. If these changes are applied, Terraform will delete the old security group and create a new one, in the meantime, your servers will start rejecting any network traffic.

  Here are four key lessons you should take away from this discussion.

  • Always use the plan command. It can reveal all these issues. Carefully review its output and pay attention to situations where Terraform plans to delete resources that likely should not be removed.
  • Create before deleting. If you want to replace a resource, think carefully about whether you need to create the replacement before deleting the original. If the answer is yes, create_before_destroy can help. The same result can be achieved manually by performing two steps: first, add the new resource to the configuration and run the apply command, then remove the old resource from the configuration and use the apply command again.
  • Changing identifiers requires modifying the state. If you want to change the identifier associated with a resource (for example, renaming aws_security_group from instance to cluster_instance) without deleting the resource and creating a new version, you need to update the Terraform state file accordingly. Never do this manually—use the terraform state command instead. When renaming identifiers, you should execute the terraform state mv command, which has the following syntax:

    terraform state mv

    ORIGINAL_REFERENCE is the expression referencing the resource in its current form, while NEW_REFERENCE is the location where you want to move it. For example, when renaming the aws_security_group group from instance to cluster_instance, you need to run the following command:

    $ terraform state mv 
       aws_security_group.instance 
       aws_security_group.cluster_instance

    This informs Terraform that the state previously associated with aws_security_group.instance should now be linked to aws_security_group.cluster_instance. If after renaming and executing this command, terraform plan does not show any changes, then you have done everything correctly.

  • Certain parameters cannot be modified. Many resource parameters are immutable. If you try to change them, Terraform will delete the old resource and create a new one instead. The documentation usually specifies what happens when changing each parameter, so make sure to check the documentation. Always use the plan command and consider the feasibility of using the create_before_destroy strategy.

  Eventual consistency agrees... with latency

  The APIs of some cloud providers, such as AWS, are asynchronous and have eventual consistency. Asynchronicity means that the interface can return a response immediately without waiting for the requested action to complete. Eventual consistency means that it may take time for changes to propagate throughout the system; during this period, your responses may be inconsistent and depend on which data source replica is responding to your API calls.

  For example, imagine you make an API call to AWS requesting the creation of an EC2 server. The API will return a 'successful' response (201 Created) almost instantly without waiting for the server itself to be created. If you immediately try to connect to it, it will almost certainly fail because AWS is still initializing the resources, or the server hasn't loaded yet. Moreover, if you make another call to retrieve information about this server, you may receive an error (404 Not Found). The fact is that information about this EC2 server may still be propagating through AWS, and it may take a few seconds for it to become available everywhere.

  Each time you use an asynchronous API with eventual consistency, you need to periodically retry your request until the action is complete and propagated throughout the system. Unfortunately, the AWS SDK does not provide any good tools for this, and the Terraform project has previously suffered from many issues like 6813 (https://github.com/hashicorp/terraform/issues/6813):

  $ terraform apply
  aws_subnet.private-persistence.2: InvalidSubnetID.NotFound:
  The subnet ID 'subnet-xxxxxxx' does not exist

  In other words, you create a resource (like a subnet) and then try to retrieve some information about it (such as the ID of the just created subnet), but Terraform can't find it. Most of these errors (including 6813) have been fixed, but they still occasionally appear, especially when Terraform adds support for a new resource type. It's frustrating, but in most cases, it doesn't cause any harm. Running terraform apply again should work since at that point, the information will already be propagated through the system.

  This excerpt is taken from Eugene Brinkman's book “Terraform: Infrastructure as Code”.

Source: habr.com