Skip to content

Improve the nofuzz system to allow specific exclusions #2832

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 11 commits into
base: fuzz-assignment
Choose a base branch
from

Conversation

MichaelChirico
Copy link
Collaborator

Continues the chain from #2818 to #2829 (the last of which this is currently based against).

I wasn't satisfied with the approach to just comment out & ignore large swaths of the test suite needed to implement # nofuzz through now, especially after the comment_injection_fuzzer turned out to require such large # nofuzz regions, but I was stuck on how to proceed with specific exclusions.

I think the approach taken here is pretty satisfactory: as a preprocessing step, we convert the # nofuzz markup into new code in the suite that signals activated/deactivated fuzzers for each execution of expect_lint().

The approach could be improved -- it can be a bit of a pain to debug, e.g. when expect_lint(file = .) was erroring due to encoding issues, the error causes the test to exit and thus fails to run the activate_fuzzers() step, leading to cascading confusing errors it's hard to pin down. And there's no simple workaround here, as e.g. using a withr::defer() approach to ensure the activate_fuzzers() step is run is not straightforward.

But I think there's enough good progress here (and it turned up a few more true positive fixes!) that it is ready for review.

Copy link

codecov bot commented Mar 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.64%. Comparing base (ad85636) to head (357f56f).

Additional details and impacted files
@@               Coverage Diff                @@
##           fuzz-assignment    #2832   +/-   ##
================================================
  Coverage            99.64%   99.64%           
================================================
  Files                  126      126           
  Lines                 7042     7043    +1     
================================================
+ Hits                  7017     7018    +1     
  Misses                  25       25           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant