Skip to content

Introduce Go Fuzz Testing for Critical Code Paths #238

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
bflad opened this issue Nov 18, 2022 · 2 comments
Closed

Introduce Go Fuzz Testing for Critical Code Paths #238

bflad opened this issue Nov 18, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@bflad
Copy link
Contributor

bflad commented Nov 18, 2022

terraform-plugin-go version

v0.14.1

Use cases

Given this Go module underpins the majority of the Terraform Provider ecosystem written in Go, we should be extra sure that critical code paths do not panic or exhibit unexpected errors with inputs. One methodology for teasing out these sort of issues to implement fuzz testing, which will randomize inputs and build up a corpus of failing tests (if found). Go natively introduced a fuzz testing framework as part of Go 1.18.

Attempted solutions

Await bug reports.

Proposal

To be determined.

References
@bflad bflad added the enhancement New feature or request label Nov 18, 2022
bflad added a commit that referenced this issue Jan 24, 2024
@bflad
tftypes: Guarantee Type interface MarshalJSON error safety
c8af68d 
Reference: #238
Reference: #365

This change removes the necessity for protocol type conversion to handle JSON encoding errors, therefore greatly simplifying those packages. Type system conversions into the protocol should always be possible, albeit potentially not valid for usage in Terraform. The protocol conversion code is not the place to handle that consideration, rather if developers want to validate their schema/function/type definitions before its sent across the protocol with respect to the validation rules imposed by Terraform, separate validation functionality should be introduced. This is not conceptually new, does not introduce behavior changes, and uses fuzz testing to verify that a panic is not possible today in the one way that developers could potentially introduce a new panic.

The fuzz testing has been run for 60 seconds on a currently powerful workstation without generating any failure cases:

```console
$ go test -fuzz=Fuzz -fuzztime=60s ./tftypes
fuzz: elapsed: 0s, gathering baseline coverage: 0/138 completed
fuzz: elapsed: 0s, gathering baseline coverage: 138/138 completed, now fuzzing with 10 workers
fuzz: elapsed: 3s, execs: 406242 (135409/sec), new interesting: 6 (total: 144)
fuzz: elapsed: 6s, execs: 833268 (142312/sec), new interesting: 7 (total: 145)
fuzz: elapsed: 9s, execs: 1269960 (145562/sec), new interesting: 7 (total: 145)
fuzz: elapsed: 12s, execs: 1684371 (138143/sec), new interesting: 8 (total: 146)
fuzz: elapsed: 15s, execs: 2137768 (151160/sec), new interesting: 9 (total: 147)
fuzz: elapsed: 18s, execs: 2574729 (145641/sec), new interesting: 11 (total: 149)
fuzz: elapsed: 21s, execs: 3011973 (145722/sec), new interesting: 12 (total: 150)
fuzz: elapsed: 24s, execs: 3442147 (143418/sec), new interesting: 12 (total: 150)
fuzz: elapsed: 27s, execs: 3868833 (142210/sec), new interesting: 12 (total: 150)
fuzz: elapsed: 30s, execs: 4313780 (148320/sec), new interesting: 14 (total: 152)
fuzz: elapsed: 33s, execs: 4763813 (150034/sec), new interesting: 14 (total: 152)
fuzz: elapsed: 36s, execs: 5201686 (145936/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 39s, execs: 5613562 (137285/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 42s, execs: 6051164 (145875/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 45s, execs: 6485230 (144677/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 48s, execs: 6912045 (142294/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 51s, execs: 7349416 (145756/sec), new interesting: 16 (total: 154)
fuzz: elapsed: 54s, execs: 7792201 (147626/sec), new interesting: 16 (total: 154)
fuzz: elapsed: 57s, execs: 8225683 (144471/sec), new interesting: 18 (total: 156)
fuzz: elapsed: 1m0s, execs: 8637257 (137227/sec), new interesting: 18 (total: 156)
fuzz: elapsed: 1m0s, execs: 8637257 (0/sec), new interesting: 18 (total: 156)
PASS
ok      github.com/hashicorp/terraform-plugin-go/tftypes        60.883s
```
@bflad bflad mentioned this issue Jan 24, 2024
Merged
@bflad
Copy link
Contributor Author

bflad commented Jan 24, 2024

We can do this as the need arises. e.g. cases like #371

@bflad bflad closed this as not planned Won't fix, can't repro, duplicate, stale Jan 24, 2024
bflad added a commit that referenced this issue Jan 24, 2024
@bflad
tftypes: Guarantee Type interface MarshalJSON error safety (#371)
c9a4f80 
Reference: #238
Reference: #365

This change removes the necessity for protocol type conversion to handle JSON encoding errors, therefore greatly simplifying those packages. Type system conversions into the protocol should always be possible, albeit potentially not valid for usage in Terraform. The protocol conversion code is not the place to handle that consideration, rather if developers want to validate their schema/function/type definitions before its sent across the protocol with respect to the validation rules imposed by Terraform, separate validation functionality should be introduced. This is not conceptually new, does not introduce behavior changes, and uses fuzz testing to verify that a panic is not possible today in the one way that developers could potentially introduce a new panic.

The fuzz testing has been run for 60 seconds on a currently powerful workstation without generating any failure cases:

```console
$ go test -fuzz=Fuzz -fuzztime=60s ./tftypes
fuzz: elapsed: 0s, gathering baseline coverage: 0/138 completed
fuzz: elapsed: 0s, gathering baseline coverage: 138/138 completed, now fuzzing with 10 workers
fuzz: elapsed: 3s, execs: 406242 (135409/sec), new interesting: 6 (total: 144)
fuzz: elapsed: 6s, execs: 833268 (142312/sec), new interesting: 7 (total: 145)
fuzz: elapsed: 9s, execs: 1269960 (145562/sec), new interesting: 7 (total: 145)
fuzz: elapsed: 12s, execs: 1684371 (138143/sec), new interesting: 8 (total: 146)
fuzz: elapsed: 15s, execs: 2137768 (151160/sec), new interesting: 9 (total: 147)
fuzz: elapsed: 18s, execs: 2574729 (145641/sec), new interesting: 11 (total: 149)
fuzz: elapsed: 21s, execs: 3011973 (145722/sec), new interesting: 12 (total: 150)
fuzz: elapsed: 24s, execs: 3442147 (143418/sec), new interesting: 12 (total: 150)
fuzz: elapsed: 27s, execs: 3868833 (142210/sec), new interesting: 12 (total: 150)
fuzz: elapsed: 30s, execs: 4313780 (148320/sec), new interesting: 14 (total: 152)
fuzz: elapsed: 33s, execs: 4763813 (150034/sec), new interesting: 14 (total: 152)
fuzz: elapsed: 36s, execs: 5201686 (145936/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 39s, execs: 5613562 (137285/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 42s, execs: 6051164 (145875/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 45s, execs: 6485230 (144677/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 48s, execs: 6912045 (142294/sec), new interesting: 15 (total: 153)
fuzz: elapsed: 51s, execs: 7349416 (145756/sec), new interesting: 16 (total: 154)
fuzz: elapsed: 54s, execs: 7792201 (147626/sec), new interesting: 16 (total: 154)
fuzz: elapsed: 57s, execs: 8225683 (144471/sec), new interesting: 18 (total: 156)
fuzz: elapsed: 1m0s, execs: 8637257 (137227/sec), new interesting: 18 (total: 156)
fuzz: elapsed: 1m0s, execs: 8637257 (0/sec), new interesting: 18 (total: 156)
PASS
ok      github.com/hashicorp/terraform-plugin-go/tftypes        60.883s
```
@github-actions GitHub Actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 23, 2024
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bflad