coverage starts too late with warning filter

[dopplershift]

Summary

When trying to get coverage on something like myproject, while also running a warning filter on a warning within that project, say myproject.MyWarning coverage is started too late. For example:

python -m pytest -W error::myproject.MyWarning --cov=myproject tests/

will result in myproject being imported (to get MyWarning), before pytest-cov starts coverage. Any imports (and code that runs at import time) as a results of importing myproject will be missed. You do get the warning: Coverage.py warning: Module myproject was previously imported, but not measured (module-not-measured)

I'm not sure if anything can be done about it (since pytest is the one controlling when warning filters are set up), but at the very least it would be great to put this as a gotcha in the docs somewhere.

[ionelmc]

I guess this section should be extended to outline your usecase as well (it's the same solution): https://pytest-cov.readthedocs.io/en/latest/plugins.html

[dopplershift]

Yeah, that might have helped. I definitely didn't think to look at that section.

[flying-sheep]

I wonder if there is a better solution possible.

• if pytest is interested in fixing this, we can use this test: flying-sheep/pytest@f538235

  An approach to fix it in pytest would need to avoid importing the package under coverage during plugin initialization or earlier. So we could add a delayed_imports = ['pkg1'] config setting or something like that which causes filterwarnings entries mentioning these packages to only be applied in later stages (e.g. in the pytest_runtest_protocol hook, but not in the pytest_load_initial_conftests hook)

  a setting like this would be kinda visible, but pretty niche.

• another solution would be to add a pytest_cov.warnings plugin or so that can be used instead of warnings (addopts = ['-pno:warnings', '-ppytest_cov.warnings']). When pytest-cov is configured to record coverage for a module, e.g. using --cov=myproj, that warning plugin would then remove the corresponding filters (*:myproj.*).

  the disadvantage would be of course that the user has to do that, and things continue to not work in a hard to diagnose fashion for them if they don’t.

• a combined solution would be for pytest to allow other plugins to hook into the filterwarnings processing of the warnings plugin, so pytest-cov can patch in that behavior.

  that would be less heavyweight than a whole new setting, and avoid having to tell the user to configure things.