Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

rerun_filter called twice per test #167

Open
ryan-woo opened this issue Aug 11, 2020 · 3 comments
Open

rerun_filter called twice per test #167

ryan-woo opened this issue Aug 11, 2020 · 3 comments

Comments

@ryan-woo
Copy link

I am finding that the rerun_filter function is called twice per test.

Version: 3.6.1
Pytest version: 5.4.3

Here is some sample code to recreate the problem:

def my_rerun_filter(*args):
    log.debug("ran filter")
    return True

runs = 0
class TestFlaky:

    @flaky(max_runs=10, rerun_filter=my_rerun_filter)
    def test_4_fails(self):
        global runs
        runs += 1
        log.debug("ran test {}".format(runs))
        if runs < 5:
            raise ValueError()

This logs ran filter 8 times even though the test only runs 4 times. A stack trace shows that the flaky_pytest_plugin.py will call self._should_rerun_test twice: once at line 92 and once at line 101.

Relevant stack traces:

File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 92, in pytest_runtest_protocol
    self.runner.pytest_runtest_protocol(item, nextitem)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/_pytest/runner.py", line 85, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/_pytest/runner.py", line 100, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 141, in call_and_report
    if self._will_handle_test_error_or_failure(item, name, err):
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/_flaky_plugin.py", line 152, in _will_handle_test_error_or_failure
    return self._should_handle_test_error_or_failure(test) and self._should_rerun_test(test, name, err)

File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 101, in pytest_runtest_protocol
    should_rerun = not skipped and self.add_failure(item, call_info.excinfo)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 327, in add_failure
    return self._handle_test_error_or_failure(item, error)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/_flaky_plugin.py", line 190, in _handle_test_error_or_failure
    if self._should_rerun_test(test, name, err):
@ns-ppawaskar
Copy link

Hello,
Did you find any resolution to this? Even I am facing similar issue.

@nicki-krizek
Copy link

I ran into this as well. It might be useful to at least document this behaviour near the delay_rerun example in README.rst.

@cygnus2048
Copy link

Just ran into this also. Any status on the fix?

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants