Incorrect outcome for flaky tests [Edge Case]

[pawelpasterz]

Describe the bug

A clear and concise description of what the bug is.

Sometimes flank's outcome is success when there are flaky tests

To Reproduce

Steps to reproduce the behavior:

So far no reliable way to reproduce it (with 10/10 rate) found. One that gave the same result is to run a test annotated with @Ignore in SavedMatrixTest

Expected behavior

A clear and concise description of what you expected to happen.

If there is at least one flaky test outcome should be flaky as well. The ignored test mentioned earlier must pass.

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

Post the output of flank --version.

Flank v20.07.0

Additional context

Add any other context about the problem here.

It is suspected that in the case of summary=null for both, step and execution, flank is unable to deduce outcome correctly.

This description should be updated in case of any new information.

[Sloox]

@pawelpasterz looking at this task, can we confirm it is still relevant? The SavedMatrix classes received a large refactor recently and it might have solved the issue.

[pawelpasterz]

@Sloox can't say. It might be as you say. But this bug was really hard to reproduce and non-deterministic even before refactor. We can have some time-box for this issue (like 4h or sth) and if no luck with reproducing, consider as done.

cc @adamfilipow92

[pawelpasterz]

@adamfilipow92 please remove the ignored test from SavedMatrixTest suite