🥇These 6 cloudformation lessons I learned for life

I started working with cloudformation 4 years ago. Since then, I have broken many infrastructures, even those that were already in production. But each time I broke something, I learned something new. Because of this experience, I will share some of the most important lessons I learned.

Lesson 1: Check changes before deploying them

I learned this lesson soon after I started working with cloudformation. I don't remember exactly what I broke back then, but I clearly remember using the command aws cloudformation update. This command simply applies the template without any checks on the changes that will be deployed. I don't think any explanation is needed on why it's important to check all changes before deploying them.

After that failure, I immediately changed my deployment pipeline, replacing the update command with the create-change-set

# OPERATION is either "UPDATE" or "CREATE"
changeset_id=$(aws cloudformation create-change-set 
    --change-set-name "$CHANGE_SET_NAME" 
    --stack-name "$STACK_NAME" 
    --template-body "$TPL_PATH" 
    --change-set-type "$OPERATION" 
    --parameters "$PARAMETERS" 
    --output text 
    --query Id)

aws cloudformation wait 
    change-set-create-complete --change-set-name "$changeset_id"

Once the change set is created, it does not affect the existing stack in any way. Unlike the update command, the change set approach does not trigger an actual deployment. Instead, it creates a list of changes that you can review before deployment. You can view the changes in the AWS console interface. But if you prefer to automate everything possible, check them in the CLI:

# this command is presented only for demonstrational purposes.
# the real command should take pagination into account
aws cloudformation describe-change-set 
    --change-set-name "$changeset_id" 
    --query 'Changes[*].ResourceChange.{Action:Action,Resource:ResourceType,ResourceId:LogicalResourceId,ReplacementNeeded:Replacement}' 
    --output table

This command should produce output similar to the following:

--------------------------------------------------------------------
|                         DescribeChangeSet                        |
+---------+--------------------+----------------------+------------+
| Action  | ReplacementNeeded  |      Resource        | ResourceId |
+---------+--------------------+----------------------+------------+
|  Modify | True               |  AWS::ECS::Cluster   |  MyCluster |
|  Replace| True               |  AWS::RDS::DBInstance|  MyDB      |
|  Add    | None               |  AWS::SNS::Topic     |  MyTopic   |
+---------+--------------------+----------------------+------------+

Pay special attention to changes where Action is Replace, Delete or where ReplacementNeeded is True. These are the most dangerous changes and usually lead to data loss.

Once the changes are reviewed, they can be deployed

aws cloudformation execute-change-set --change-set-name "$changeset_id"

operation_lowercase=$(echo "$OPERATION" | tr '[:upper:]' '[:lower:]')
aws cloudformation wait "stack-${operation_lowercase}-complete" 
    --stack-name "$STACK_NAME"

Lesson 2: Use stack policy to prevent replacing or deleting resources while maintaining state

Sometimes simply reviewing changes is not enough. We are all human and we all make mistakes. Shortly after we started using change sets, my teammate inadvertently executed a deployment that led to a database update. Nothing serious happened as it was a testing environment.

Even though our scripts displayed the change list and requested confirmation, the Replace change was overlooked because the change list was so extensive that it didn’t fit on the screen. And since this was a routine update in a testing environment, not much attention was paid to the changes.

There are resources that you would never want to replace or delete. These are stateful services like an RDS database instance or an Elasticsearch cluster, etc. It would be great if AWS automatically denied deployments that would require the deletion of such resources. Fortunately, CloudFormation has a built-in way to do this. It’s called stack policy, and you can learn more about it in the documentation:

STACK_NAME=$1
RESOURCE_ID=$2

POLICY_JSON=$(cat <<EOF
{
    "Statement" : [{
        "Effect" : "Deny",
        "Action" : [
            "Update:Replace",
            "Update:Delete"
        ],
        "Principal": "*",
        "Resource" : "LogicalResourceId/$RESOURCE_ID"
    }]
}
EOF
)

aws cloudformation set-stack-policy --stack-name "$STACK_NAME" 
    --stack-policy-body "$POLICY_JSON"

Lesson 3: Use UsePreviousValue when updating a stack with secret parameters

When you create an RDS MySQL entity, AWS requires you to provide MasterUsername and MasterUserPassword. Since it’s better not to store secrets in the source code, and I wanted to automate everything, I implemented a 'smart mechanism' where the credentials would be fetched from S3 before deployment, and if the credentials were not found, new credentials would be generated and stored in S3.

These credentials would then be passed as parameters to the cloudformation create-change-set command. During experiments with the script, it happened that the connection to S3 was lost, and my 'smart mechanism' interpreted this as a signal to generate new credentials.

If I started using this script in a production environment and the connection issue occurred again, it would update the stack with new credentials. In this particular case, nothing bad would happen. However, I avoided that approach and began using another method, providing the credentials only once — during stack creation. And later, when the stack requires an update, I would simply use UsePreviousValue=true:

aws cloudformation create-change-set 
    --change-set-name "$CHANGE_SET_NAME" 
    --stack-name "$STACK_NAME" 
    --template-body "$TPL_PATH" 
    --change-set-type "UPDATE" 
    --parameters "ParameterKey=MasterUserPassword,UsePreviousValue=true"

Lesson 4: use rollback configuration

Another command I worked with utilized a feature cloudformation, called rollback configuration. I hadn't encountered it before and quickly realized that it would make my stack deployments even better. Now I use it every time I deploy my code in Lambda or ECS with CloudFormation.

How it works: you specify CloudWatch alarm arn in the parameter —rollback-configuration, when you create a change set. Later, when you execute the change set, AWS monitors the alarm for at least one minute. It rolls back the deployment if, during that time, the alarm changes state to ALARM.

Below is an example snippet of the template cloudformation, where I create cloudwatch alarm, tracking a custom cloud metric that counts errors in cloud logs (the metric is created via MetricFilter):

Resources:
  # this metric tracks number of errors in the cloudwatch logs. In this
  # particular case it's assumed logs are in json format and the error logs are
  # identified by level "error". See FilterPattern
  ErrorMetricFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: !Ref LogGroup
      FilterPattern: !Sub '{$.level = "error"}'
      MetricTransformations:
      - MetricNamespace: !Sub "${AWS::StackName}-log-errors"
        MetricName: Errors
        MetricValue: 1
        DefaultValue: 0

  ErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${AWS::StackName}-errors"
      Namespace: !Sub "${AWS::StackName}-log-errors"
      MetricName: Errors
      Statistic: Maximum
      ComparisonOperator: GreaterThanThreshold
      Period: 1 # 1 minute
      EvaluationPeriods: 1
      Threshold: 0
      TreatMissingData: notBreaching
      ActionsEnabled: yes

Currently alarm can be used as a rollback trigger when executing the toolkit:

ALARM_ARN=$1

ROLLBACK_TRIGGER=$(cat <<EOF
{
  "RollbackTriggers": [
    {
      "Arn": "$ALARM_ARN",
      "Type": "AWS::CloudWatch::Alarm"
    }
  ],
  "MonitoringTimeInMinutes": 1
}
EOF
)

aws cloudformation create-change-set 
    --change-set-name "$CHANGE_SET_NAME" 
    --stack-name "$STACK_NAME" 
    --template-body "$TPL_PATH" 
    --change-set-type "UPDATE" 
    --rollback-configuration "$ROLLBACK_TRIGGER"

Lesson 5: Ensure that you are deploying the latest version of the template

It's easy to deploy a version of the cloudformation template that isn't the most recent, but this can cause significant damage. Once, a developer failed to push the latest changes from Git and unknowingly deployed a previous version of the stack. This resulted in downtime for the application using that stack.

Something simple, like adding a check to see if the branch is up to date before performing the deployment, will be beneficial (assuming git is your version control tool):

git fetch
HEADHASH=$(git rev-parse HEAD)
UPSTREAMHASH=$(git rev-parse master@{upstream})

if [[ "$HEADHASH" != "$UPSTREAMHASH" ]] ; then
   echo "Branch is not up to date with origin. Aborting"
   exit 1
fi

Lesson 6: Don't reinvent the wheel

It may seem that deploying with cloudformation — is easy. You just need a bunch of bash scripts running aws cli commands.

Four years ago, I started with simple scripts that called the aws cloudformation create-stack command. Soon, the script was no longer simple. Each lesson learned made the script more and more complex. It was not only complicated, but also had a lot of bugs.

Now I work in a small IT department. Experience shows that every team has its own way of deploying cloudformation stacks. This is problematic. It would be better if everyone used a unified approach. Fortunately, there are many tools that help deploy and manage cloudformation stacks.

These lessons will help you avoid mistakes.

Source: habr.com