Hello everyone. Today we share with you the final part of the article , translated specially for students of the course .
Deployment Testing
The testing style discussed is a powerful approach that allows us to perform white-box testing to verify the inner workings of our infrastructure code. However, it somewhat limits what we can check. Tests are executed based on an in-memory deployment plan created by Pulumi prior to the actual deployment, and therefore the deployment itself cannot be tested. For such cases, Pulumi has an integration testing framework. These two approaches work great together!
The Pulumi integration testing framework is written in Go, and it is how we test most of our internal code. While the previously discussed modular testing approach resembled white-box testing, integration testing is more like black-box testing. (There are also options for rigorous internal testing.) This framework was designed to take an entire Pulumi program and perform various lifecycle operations on it, such as deploying a new stack from scratch, updating it with variations, and removing it, possibly several times. We run them regularly (for example, at night) and as stress tests.
(We similar integration testing capabilities available in the native SDKs of languages. You can use the Go integration testing framework independently of the language your Pulumi program is written in).
By running a program using this framework, you can verify the following:
- Your project code is syntactically correct and runs without errors.
- Stack and secret configuration settings work and are interpreted correctly.
- Your project can be successfully deployed to the cloud provider of your choice.
- Your project can be successfully updated from the initial state to N other states.
- Your project can be successfully destroyed and removed from your cloud provider.
As we will soon see, this framework can also be used for runtime validation.
Simple Integration Test
To see this in action, we'll look at the repository pulumi/examples, as our Pulumi team and community use it to test their own pull requests, commits, and nightly builds.
Below is a simplified test of our :
example_test.go:
package test
import (
"os"
"path"
"testing"
"github.com/pulumi/pulumi/pkg/testing/integration"
)
func TestExamples(t *testing.T) {
awsRegion := os.Getenv("AWS_REGION")
if awsRegion == "" {
awsRegion = "us-west-1"
}
cwd, _ := os.Getwd()
integration.ProgramTest(t, &integration.ProgramTestOptions{
Quick: true,
SkipRefresh: true,
Dir: path.Join(cwd, "..", "..", "aws-js-s3-folder"),
Config: map[string]string{
"aws:region": awsRegion,
},
})
} This test goes through the basic lifecycle of creating, updating, and destroying a stack for the folder aws-js-s3-folder. It will take about a minute to report the passed test:
$ go test .
PASS
ok ... 43.993s There are many parameters to configure the behavior of these tests. The complete list of options can be found ProgramTestOptions. For example, you can configure the Jaeger endpoint for tracing (Tracing), state that you expect the test to fail in negative testing (ExpectFailure), apply a series of "edits" to the program for state transition (EditDirs), and much more. Let’s see how to use them for testing application deployment.
Checking Resource Properties
The integration mentioned earlier ensures that our program "works" — it does not crash. But what if we want to check the properties of the resulting stack? For instance, to verify that certain types of resources were (or were not) prepared and that they have specific attributes.
Parameter ExtraRuntimeValidation for ProgramTestOptions allows us to look at the state captured by Pulumi after deployment (post-deployment state), so we can perform additional checks. This includes a complete snapshot of the resulting stack's state including configuration, exported output values, all resources and their property values, as well as all dependencies between resources.
To see a basic example of this, let’s check that our program creates one S3 Bucket:
integration.ProgramTest(t, &integration.ProgramTestOptions{
// as before...
ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
var foundBuckets int
for _, res := range stack.Deployment.Resources {
if res.Type == "aws:s3/bucket:Bucket" {
foundBuckets++
}
}
assert.Equal(t, 1, foundBuckets, "Expected to find a single AWS S3 Bucket")
},
})Now, when we run go test, it will not only go through the lifecycle test suite, but also, after a successful stack deployment, perform an additional check on the resulting state.
Runtime tests
So far, all tests have been solely about deployment behavior and the Pulumi resource model. What if you want to verify that your provisioned infrastructure is actually working? For example, that a virtual machine is running, an S3 bucket contains what we expect, and so on.
You may have already guessed how to do this: the option ExtraRuntimeValidation for ProgramTestOptions — is a great opportunity for this. At this stage, you run an arbitrary Go test with access to the full state of your program's resources. This state includes information such as the IP addresses of virtual machines, URLs, and anything else needed for real interaction with the resulting cloud applications and infrastructure.
For instance, our test program exports the property webEndpoint of the bucket named websiteUrl, which represents the full URL where we can access the configured index document. While we could dig through the state file to find bucket and read this property directly, in many cases our stacks export useful properties like this that are convenient for checking:
integration.ProgramTest(t, &integration.ProgramTestOptions{
// as before ...
ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
url := "http://" + stack.Outputs["websiteUrl"].(string)
resp, err := http.Get(url)
if !assert.NoError(t, err) {
return
}
if !assert.Equal(t, 200, resp.StatusCode) {
return
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if !assert.NoError(t, err) {
return
}
assert.Contains(t, string(body), "Hello, Pulumi!")
},
}) Like our previous runtime checks, this check will be executed immediately after the stack has been raised, all in response to a simple call go test. And this is just the tip of the iceberg — all Go testing capabilities are available for you to implement in your code.
Continuous Infrastructure Integration
It's great to be able to run tests on a laptop when numerous changes are made to the infrastructure, to verify them before submission for code review. However, we and many of our clients also test the infrastructure at various stages of the development lifecycle:
- In every open pull request for testing before merging.
- In response to each commit, to double-check that the merge was completed correctly.
- Periodically, for instance, at night or weekly for additional testing.
- As part of performance testing or stress testing, which typically runs over an extended period and executes tests in parallel and/or deploys the same program multiple times.
For each of these, Pulumi supports integration with your favorite continuous integration system. With continuous integration, this gives you the same test coverage for your infrastructure as for your application software.
Pulumi supports popular CI systems. Here are a few of them:
For more details, refer to the documentation on .
Ephemeral Environments
A very powerful feature that emerges is the ability to deploy ephemeral environments solely for acceptance testing purposes. The concept of Pulumi is designed to easily deploy and tear down fully isolated and independent environments, all with just a few simple CLI commands or using the integration testing framework.
If you are using GitHub, Pulumi offers , which helps you connect acceptance testing to pull requests within your CI pipeline. Simply install the app in your GitHub repository, and Pulumi in your CI will add information about the infrastructure preview, updates, and test results to the pull requests:
By using Pulumi for your core acceptance tests, you will gain new automation capabilities that enhance team productivity and instill confidence in the quality of changes.
Summary
In this article, we found that using general-purpose programming languages opens up many software development methods that have been helpful in building our applications. These include unit testing, integration testing, and their interactions for extensive runtime testing. Tests can be easily executed on demand or within your CI system.
Pulumi — open-source software that is free to use and works with your favorite programming languages and clouds — !
→
Source: habr.com
