Add a mutation score threshold so that we can still pass without kill… #1052
Conversation
…ing every single mutant
|
Hi @mpownby, thank for the PR, this is a great idea and a great feature! You'd need to update some test cases which are failing, and add a new test case covering this specific feature. You can put the test case somewhere here tests-lit/tests, feel free to simply copy an existing test and modify as needed. |
removed extra logging because it breaks existing tests
Added integration test for mutation score threshold feature (mutation score should be 50%, we set our threshold to 49% and expect the test runner to still return success)
Ok, I think I finally got it. It was a real conundrum figuring out how to run the lit tests locally but I finally figured it out. I added an integration test that achieves a mutation score of 50% with a desired threshold of 49% so the runner is verified to still return success. |
Add a mutation score threshold so that we can still pass without killing every single mutant.
Motivation:
I want to use mull in my CI/CD pipeline to make sure incoming pull requests meet a minimum testing standard.
Currently, mull can be configured to pass (return exit code 0) if a) all mutants are killed or b) all surviving mutants are ignored. This is accomplished via the "--allow-surviving" command line option.
I'd like a more granular option. I might like to have (nearly) all mutations enabled, but killing every single mutant may not be worth it to me, especially if they are inside code that isn't critical like formatting log messages or rare catch blocks. I'd prefer to pass/fail based on a mutation score threshold.
Hence this pull request.