Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add documentation for JUnit output from 'terraform test' #36307

Merged
merged 6 commits into from
Jan 22, 2025
Merged
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions website/docs/cli/commands/test.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ The following options apply to the Terraform `terraform test` command:

* `-json` - Displays machine-readable JSON output for your testing results.

* `-junit-xml=<output file path>` - Saves a test report in JUnit XML format to the specified file. This is currently incompatible with remote test execution using the the `-cloud-run` option. The file path must be relative or absolute.

* `-test-directory=<relative directory>` - Overrides the directory that Terraform looks into for test files. Note that Terraform always loads testing files within the main configuration directory. The default testing directory is `tests`.

* `-verbose` - Prints out the plan or state for each `run` block within a test file, based on the `command` attribute of each `run` block.
Expand Down Expand Up @@ -134,3 +136,92 @@ The testing directory must be beneath the main configuration directory, but it c
> Note: Test files within the root configuration directory are always loaded, regardless of the `-test-directory` value.

We do not recommend changing the default test directory. The option for customization is included for configuration authors who may have included a `tests` submodule in their configuration before the `terraform test` command was released. In general, the default test directory of `tests` should always be used.


## Example: Test Output Format Options

Below is a contrived example of Terraform testing that makes assertions about the values of local variables `true` and `false`.
There are two test files: one contains a passing test, and one contains a failing test.

```hcl
# main.tf
locals {
true = "true"
false = "true" # incorrect, should be "false"!
}
```

The assertion that local.true == "true" in example_1.tftest.hcl will pass:

```hcl
# example_1.tftest.hcl
run "true_is_true" {
assert {
condition = local.true == "true"
error_message = "local.true did not match expected value"
}
}
```

The assertion that local.false == "false" in example_2.tftest.hcl will fail:

```hcl
# example_2.tftest.hcl
run "false_is_false" {
assert {
condition = local.false == "false"
error_message = "local.false did not match expected value"
}
}
```

### Test output in JUnit XML format, saved to file

Below is the output of the `terraform test -junit-xml=./output.xml` command using the example files above.
The test output is:

* Printed to the terminal in the default, human-readable format.
* Also saved in JUnit XML format in the file specified by the flag.

Below is the contents of the resulting `output.xml` file:

```xml
<?xml version="1.0" encoding="UTF-8"?><testsuites>
<testsuite name="example_1.tftest.hcl" tests="1" skipped="0" failures="0" errors="0">
<testcase name="true_is_true" classname="example_1.tftest.hcl" time="0.002295" timestamp="2025-01-13T19:23:16Z"></testcase>
</testsuite>
<testsuite name="example_2.tftest.hcl" tests="1" skipped="0" failures="1" errors="0">
<testcase name="false_is_false" classname="example_2.tftest.hcl" time="0.001468" timestamp="2025-01-13T19:23:16Z">
<failure message="local.false did not match expected value"><![CDATA[
Error: Test assertion failed

on example_2.tftest.hcl line 3, in run "false_is_false":
3: condition = local.false == "false"
├────────────────
│ local.false is "true"

local.false did not match expected value
]]></failure>
</testcase>
</testsuite>
</testsuites>
```

If a run block contains a failing assertion, the `<testcase>` element will contain a `<failure>` element that includes the error message and further details.

A `<testcase>` element could contain a `<skipped>` element that includes details about why a test was skipped. This could either be due to an error causing remaining run blocks to be skipped, or due to the command being interrupted.

#### Mapping Terraform test command concepts to JUnit XML format

The test report generated when using `-junit-xml` maps Terraform test command concepts to JUnit XML format according to the table below:

| Terraform test concept | Element in JUnit XML output |
| ----------------------------------| --------------------------------------------|
| Test directory | `<testsuites>` |
| Test file | `<testsuite>` |
| Run block | `<testcase>` |
| Run block assertion | None; details are included only on failure |
| Test failure | `<failure>` |
| Test was skipped | `<skipped>` |
| Test stopped due to error | `<error>` |
| Any unhandled warnings or errors | `<system-err>` |
Loading