Incorrect coverage report

[drasmuss]

This issue can be observed in this log https://travis-ci.org/nengo/nengo/jobs/510052898#L894. Sorry I couldn't come up with a smaller test case, but it seems to be a specific interaction with this repo somehow.

Specifically, running tests with

pytest --cov

reports 69% test coverage, while

coverage run -m pytest

reports 82% coverage (scroll farther down in the log to see this). My understanding is that those two should be equivalent.

This was previously reported in #117, but the fixes there didn't solve the problem (I figured since that one had been closed for two years it was better to open a new issue).

[ionelmc]

@drasmuss could you try this with the master of pytest-cov?

[blueyed]

(in case using master does not fix it)
Also add --cov-report=term-missing / coverage report -m to see differences.
Often this might be due to things being imported already, before pytest-cov kicks in.

[drasmuss]

Unfortunately master looks like it has the same behaviour. You can see an example of the missing lines here https://codecov.io/gh/nengo/nengo/src/template-ci/nengo/base.py (that file has 100% coverage when not running through pytest-cov). So you can see that it's missing all the function/class definitions, which does make me think that it's not counting some initial "import" pass through the code as @blueyed suggests.

Is there anything we can do to avoid that? I tried setting the COV_CORE_SOURCE... variables from https://pytest-cov.readthedocs.io/en/latest/plugins.html (our project isn't a plugin, but I thought it might help anyway), but it didn't seem to make a difference.

[ionelmc]

To clarify what's going on there ... xdist is being used. Actual command is pytest -v -n 2 --color=yes --durations 20 --cov=nengo.

So this test suite only runs on travis? It looks like you pushed a commit just to try the master of pytest-cov.

[drasmuss]

I'm testing it with and without xdist. You can see without xdist here:
https://travis-ci.org/nengo/nengo/jobs/510953456#L917
(69% coverage)

and without xdist, without pytest-cov here:
https://travis-ci.org/nengo/nengo/jobs/510953456#L1214
(82% coverage)

We do eventually want to run with xdist (that's why we're interested in pytest-cov), but I took xdist out of the equation for now to simplify the debugging.

And yes, this issue seems to be environment specific, when I run it on my local machine I don't see the issue. So I have to do everything through travis for now.

[ionelmc]

Well something really funny is going on there. Does the problem go away if you remove the nengo.backends entrypoint (from setup.py)?

[drasmuss]

No change after removing the entrypoint, unfortunately.

[ionelmc]

Well one way to really nail this down is to have an early failure in base.py (just make a syntax error I guess). Then you'll see whatever tried to import it too early.

[drasmuss]

That helped me track the issue down to this line https://github.com/nengo/nengo/blob/master/setup.cfg#L41, which I had forgotten we were setting. So that definitely makes sense that early-loading that file would mess up the import order.

Do you know if there is a simple way to make pytest-cov work in combination with that pytest -p option? I can also try to figure out a way to rework our test infrastructure so that we don't require that, just thought I'd check if you knew a simple solution off the top of your head.

[ionelmc]

https://pytest-cov.readthedocs.io/en/latest/plugins.html should solve it. If it doesn't show a failing build with the applied configuration.

[nicoddemus]

Do you know if there is a simple way to make pytest-cov work in combination with that pytest -p option?

Using -p for early loading plugins will work in pytest 4.4: https://docs.pytest.org/en/features/usage.html?highlight=pytest_cov#early-loading-plugins

[ionelmc]

Interesting, is this implemented in master already @nicoddemus ?

[nicoddemus]

No, only in features. 😉

master contains only bug fixes, so we can make patch releases (say 4.3.2 would be the next version).

features contains new features and bug fixes for the next minor release (4.4 will be the next one).

I think we will release 4.4 this week or the next, I believe. 👍

[drasmuss]

Manually starting the pytest-cov engine as in https://pytest-cov.readthedocs.io/en/latest/plugins.html solved the problem for me, thanks! I swear I tried that before, but I must have had some other changes happening at the same time that prevented it from working 🤷‍♂️ .

And if I understand things correctly, after pytest 4.4 we could accomplish a similar thing, early starting the pytest-cov engine, via

pytest -p pytest_cov -p some_other_plugin

?

[nicoddemus]

And if I understand things correctly, after pytest 4.4 we could accomplish a similar thing, early starting the pytest-cov engine, via

Yes, and you probably won't need -p some_other_plugin, starting pytest-cov early and let some_other_plugin load later when setuptools entry points kick in. 👍

[drasmuss]

Minor update: I found that manually starting the pytest-cov engine works, unless we use one of the options defined in the plugin that we're trying to get pytest-cov to load before. E.g.,

COV_CORE_SOURCE=... pytest --cov --cov-append

works, but

COV_CORE_SOURCE=... pytest --cov --cov-append --custom-plugin-arg

results in the same incorrect coverage report as before. My guess would be that when pytest has to execute the custom plugin code to evaluate that argument, that preempts what we're trying to do with the manual pytest-cov start.

In any case, it seems like this is something that isn't really in pytest-cov's power to resolve, it's just a consequence of the way plugins are loaded. So I'm OK if you want to close this. I haven't had a chance to try out the 4.4 features branch yet, but hopefully that provides a more permanent solution.

[blueyed]

My guess would be that when pytest has to execute the custom plugin code to evaluate that argument, that preempts what we're trying to do with the manual pytest-cov start.

Arguments (and the plugin) are loaded regardless of options are used. But likely the plugin itself then imported things already?!
You should be able to tell via the early failure in base.py I guess.

[lephuongbg]

Same problem here.

I have tried pytest -p pytest_cov (mine is pytest 4.5) but function declarations are still not in coverage.

[ionelmc]

@lephuongbg do you have an example project that reproduces the problem?

[lephuongbg]

@ionelmc I can't guarantee to have time to make an example project from our large code base but I have found the way to circumvent the problem:

OUR SOLUTION: At the top of our tests/__init__.py, we are currently importing some files from our non-test codes. Moving all of these imports into each fixture functions where they are needed resolve the problem.

The thing is, we didn't have to do that until now. Our code has changed but the tests/__init__.py itself didn't change, so it might be some complicated interactions in module importing.

[aecay]

I had a similar issue. What I found is that tox installs the package being tested into its virtualenv. However, coverage looks for the package in the working directory. Thus, it looks as though the files have 0% coverage -- and they do, because the coverage info has been recorded against the copy installed in the tox directory.

What I don't understand is if you pass --cov=mypkg to coverage, why it thinks that you mean only mypkg in the current directory, rather than mypkg wherever it is imported from.

In any case, I was able to fix this by adding the skipsdist = true option in tox.ini. This makes tox not install a copy of the package in its virtualenv, and the coverage report is correct again.

[facundopadilla]

I have the same issue

[themattmorris]

Thank you for this thread! I was running into the same problem. In my case I was able to get it working using -p flag the others have mentioned, along with --cov-append and using coverage run -m instead of pytest ... or python -m pytest ...:

[tool.hatch.envs.test.scripts]
pytest = "coverage run -m pytest {args}"

...

[tool.pytest.ini_options]
addopts = "-p pytest_cov --cov=src --cov-append"

It was important that I include the both the --cov-append flag and -p pytest_cov, otherwise I was still getting low coverage being reported.

[symonk]

I found some success in using ... pytest -p pytest-cov -p my_plugin (I assume my plugin was importing some stuff early and skewing the coverage a bit etc (not sure!) must force pytest-cov to do it's thing before my new plugin got rolling/imported by pluggy