Consider Persisting Each Plan File During Each Step

[bflad]

terraform-plugin-testing version

v1.2.0

Use cases

When using the TF_ACC_PERSIST_WORKING_DIR=1 environment variable, the directory is filled with some handy contents showing file artifacts during the testing (output assuming #114 implementation):

$ tree /Users/bflad/test/testing55    
/Users/bflad/test/testing55

0 directories, 0 files

$ TF_ACC_PERSIST_WORKING_DIR=1 go test -count=1 -v ./internal/provider
=== RUN   TestExampleDataSource_basic
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleDataSource_basic/step_1
--- PASS: TestExampleDataSource_basic (0.54s)
=== RUN   TestExampleResource_basic
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleResource_basic/step_1
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleResource_basic/step_2
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleResource_basic/step_3
--- PASS: TestExampleResource_basic (0.89s)
PASS
ok      github.com/bflad/terraform-provider-framework/internal/provider 1.717s

$ tree /Users/bflad/test/testing55
/Users/bflad/test/testing55
├── TestExampleDataSource_basic
│   └── step_1
│       ├── terraform.tfstate
│       ├── terraform_plugin_test.tf
│       └── tfplan
├── TestExampleResource_basic
│   ├── step_1
│   │   ├── terraform.tfstate
│   │   ├── terraform.tfstate.backup
│   │   ├── terraform_plugin_test.tf
│   │   └── tfplan
│   ├── step_2
│   │   ├── terraform.tfstate
│   │   ├── terraform.tfstate.backup
│   │   ├── terraform_plugin_test.tf
│   │   └── tfplan
│   └── step_3
│       ├── terraform.tfstate
│       ├── terraform.tfstate.backup
│       ├── terraform_plugin_test.tf
│       └── tfplan
├── work1058235289
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   ├── terraform_plugin_test.tf
│   └── tfplan
├── work4271933255
│   ├── terraform.tfstate
│   └── terraform_plugin_test.tf
└── work763334115
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform_plugin_test.tf
    └── tfplan

10 directories, 25 files

However, looking at the plan files, they always are the last plan to occur during the test step (which is usually a plan showing no operations to verify providers aren't introducing permanent drift). It would be great to see all the plans that ran, e.g. the plan that actually created the resources in the first step or the plan updating resources in a subsequent step.

Attempted solutions

Manually look through the TRACE logging to find each plan output.

Proposal

When executing a plan and when TF_ACC_PERSIST_WORKING_DIR is enabled, ensure a plan file is always saved and copy the saved plan with a friendly name. Importantly, these saved plans should not be copied/persisted between steps to prevent confusion.

Test Step Mode	Plan	Expected Persisted File
Config	Pre-Apply	config-pre-apply.tfplan
Config	Post-Apply Pre-Refresh	config-post-apply-pre-refresh.tfplan
Config	Post-Apply Post-Refresh	config-post-apply-post-refresh.tfplan
Import	Post-Import	import-post-import.tfplan

$ tree /Users/bflad/test/testing55    
/Users/bflad/test/testing55

0 directories, 0 files

$ TF_ACC_PERSIST_WORKING_DIR=1 go test -count=1 -v ./internal/provider
=== RUN   TestExampleDataSource_basic
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleDataSource_basic/step_1
--- PASS: TestExampleDataSource_basic (0.54s)
=== RUN   TestExampleResource_basic
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleResource_basic/step_1
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleResource_basic/step_2
    testing_new.go:478: Working directory and files have been copied to: /Users/bflad/test/testing55/TestExampleResource_basic/step_3
--- PASS: TestExampleResource_basic (0.89s)
PASS
ok      github.com/bflad/terraform-provider-framework/internal/provider 1.717s

$ tree /Users/bflad/test/testing55
/Users/bflad/test/testing55
├── TestExampleDataSource_basic
│   └── step_1
│       ├── config-post-apply-post-refresh.tfplan
│       ├── config-post-apply-pre-refresh.tfplan
│       ├── config-pre-apply.tfplan
│       ├── terraform.tfstate
│       └── terraform_plugin_test.tf
├── TestExampleResource_basic
│   ├── step_1
│   │   ├── config-post-apply-post-refresh.tfplan
│   │   ├── config-post-apply-pre-refresh.tfplan
│   │   ├── config-pre-apply.tfplan
│   │   ├── terraform.tfstate
│   │   ├── terraform.tfstate.backup
│   │   └── terraform_plugin_test.tf
│   ├── step_2
│   │   ├── import-post-import.tfplan
│   │   ├── terraform.tfstate
│   │   ├── terraform.tfstate.backup
│   │   └── terraform_plugin_test.tf
│   └── step_3
│       ├── config-post-apply-post-refresh.tfplan
│       ├── config-post-apply-pre-refresh.tfplan
│       ├── config-pre-apply.tfplan
│       ├── terraform.tfstate
│       ├── terraform.tfstate.backup
│       └── terraform_plugin_test.tf
├── work1058235289
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   ├── terraform_plugin_test.tf
│   └── tfplan
├── work4271933255
│   ├── terraform.tfstate
│   └── terraform_plugin_test.tf
└── work763334115
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform_plugin_test.tf
    └── tfplan

10 directories, 31 files

References

• Fix path used when persisting working directory #113
• Consider Friendlier Naming When Persisting Working Directories #114

[bflad]

Just to capture this somewhere, it seems like some provider developers have created "debug" plan checks to deal with situations where the Terraform human-readable plan output is hiding differences, but they do show up in the machine-readable plan: https://github.com/mongodb/terraform-provider-mongodbatlas/blob/22d8c0a5da17aa52b2fc532a6900951b9a239b31/internal/testutil/acc/plan_check.go

I think there are multiple ways to interpret that sort of plan check:

• One, as it relates to this particular feature request, is that the lack of ability to inspect all plan files from all steps means that workarounds to output the information during the particular test step. If this enhancement was in place, the troubleshooting recommendation could be to run the Terraform command to output the data from the desired plan file.
• Two, if differences are detected, consider logging the machine-readable plan.
• Three, consider natively offering this sort of plan check.

I feel like options two and three could be problematic in multiple senses, which is why I hesitate to suggest them as new feature requests, such as having too much irrelevant data and potentially having subjective opinions on the output formatting. Option one makes it so developers can inspect the data without needing to copy paste the output into a file (since it will already be a file) or a website like the jq playground (unless they really want to).

Related: hashicorp/terraform#31887