Performance regression in 0.18.1 – 10x slowdown in pytest collection

[davidparsson]

We're using pytest-asyncio in a repo with 20000+ tests. With 0.18.0 I get a pytest collection time of about 25 seconds, running pytest . --collect-only. Updating to 0.18.1 increases the collection time to almost 300 seconds.

0.18.1 only contains one logical change, so that's the cause, but I don't (yet) have a clue about a resolution.

[davidparsson]

We are by the way using the strict asyncio mode.

Could it be an option to not allow async fixtures to be used in sync tests for strict mode?

[seifertm]

Thanks for pointing out this issue. I don't think pytest-asyncio has enough awareness of its performance, especially in large code bases.

This could be a duplicate of #720.

There has been recent work in pytest to raise a warning if a sync test requests an async fixture (see pytest-dev/pytest#10839). Specifically, pytest-dev/pytest#10839 (comment) points out that pytest-asyncio doesn't do anything if a sync test requests an async fixture. A more specific warning was added to pytest-asyncio in #979 (unreleased at the time of writing). I don't think pytest-asyncio should immediately raise an error just yet. We should at least release the update that emits the warning and see what issues are reported by the users. Then we can completely deprecate it and raise an error.

That said, I think the first step would be to add some kind of performance test to the pytest-asyncio tests. We could use this as a baseline for optimizing the collection time.