Should integration tests be excluded from coverage ?

[Avasam]

Anytime something is changed in setuptools/tests/integration/, coverage tests will fail.

Even with coverage aggregation jaraco/skeleton#130, integration tests are not run in PRs. And the cov plugin is disabled when running integration tests.

Should then setuptools/tests/integration/ just be excluded from the coverage report ?

[abravalheri]

My personal opinion is that yes, they should.

A while ago we got a PR proposing to add coverage to them: #3643.
While reviewing this PR I noticed that the coverage report was not including all the code paths that were traversed. In fact, the report was missing the majority of the code paths that were definitely exercised by the integration tests.

In the integration tests we are running the full stack from the backend API to build_ext, but because they need to run in subprocess (via pip or build), they are not reported. I did try a series of approaches suggested in the coverage.py docs, as described in #3643 (comment); but all of them were unsuccessful so I gave up on those¹.

So my opinion is that coverage for the integration tests at the current state of the repository has so many false negatives that it is not even worth running. On top of that, there are all the argument that @Avasam already mentioned: for the sake of speed/resource utilisation integration tests are excluded from regular PRs - so coverage checks on them will inevitably fail.

Footnotes

1. I do welcome any member of the community that would like to give it another try. Also almost 2 years have passed since I last tried so coverage.py may also have evolved to make such things easier. ↩

[Avasam]

I've created #4589