Incorrect coverage report

[jgosmann]

Using py.test --cov-config .coveragerc --cov nengo -n 6 nengo a lot of lines that should be hit get reported as missed (like class and function definitions in a module). This might be related to #19 as the project has a conftest file importing other modules from the project.

Using coverage run --rcfile .coveragerc --source nengo -m py.test nengo instead a correct coverage report is generated, but this command does not support xdist.

[ionelmc]

It might be possible to measure it with a workaround like this: https://github.com/ionelmc/pytest-benchmark/blob/master/tox.ini#L32-L33 - it forces the engine used for subprocess measurements to load early.

[yunstanford]

Have this been fixed ? i am still getting this issue.

[ionelmc]

Not yet fixed. There is a workaroud tho.

[ionelmc]

Show an example and maybe we can figure it out.

[yunstanford]

you can checkout the last travis ci build of Sanic https://github.com/channelcat/sanic

[ionelmc]

Maybe your local checkout is dirty (have any .coverage/.coverage.* files?). Can we narrow it down to something smaller?

[ionelmc]

Ah yes, it's a pytest plugin, so plugin loading is currently unpredictable. See pytest-dev/pytest#935

[yunstanford]

yeah, you're right. i had .coverage files, after git clean -xfd, i can also reproduce locally.

[yunstanford]

How can we fix that ? should we load pytest-cov first ? what step matters actually when running pytest, pytest_configure ?

[ionelmc]

Well ... this being a problem in pytest, we can only talk about workarounds here.

[yunstanford]

yeah, any suggestion ? it works fine for me when i use uranium another python build system (https://github.com/yunstanford/pytest-sanic/blob/master/ubuild.py#L16). but it doesn't work with tox.

[ionelmc]

Afaik it's either the workaround I use in pytest-benchmark (forceload pytest-cov subprocess engine) or ronny's plugin trick (haven't tried it).

[yunstanford]

https://github.com/ionelmc/pytest-benchmark/blob/master/tox.ini#L52-L53 ?

[ionelmc]

Ah ... now I get the confusion, line numbers changed: https://github.com/ionelmc/pytest-benchmark/blob/51b3bf7ebf85bfd73445446e167a9a89171582d6/tox.ini#L25-L27

[yunstanford]

hmm.. it doesn't work for me by just setting these env.. am i missing anything ? and it shows

None
  Module already imported so can not be re-written: pytest_cov

[ionelmc]

What happens if you run

coverage combine --append
coverage report

[yunstanford]

Then looks correct.

[ionelmc]

Well there you go, just run coverage combine --append before publishing coverage results.

[yunstanford]

yeah, but pytest always prints out the incorrect one first. is there any way to disable that ?

[yunstanford]

nvm, --cov-report=

[ionelmc]

This is fixed right? Also, #177 will clarify the solutions.