Only rerun tests that failed timeouts

[plocket]

Edit: I've made this feature request in the cucumberjs lib for retrying tests as it seems there's no good way to do it at the moment. I've been told we shouldn't try to manipulate the below values.

In the cucumberjs API documentation for After(), you can pass an option to a callback called willBeRetried. I think that would let us control whether a test will be retried.

That might be a better way to:

• handle retrying tests that fail with a timeout
• avoid rerunning failed random tests.

[plocket]

Edit: I've been told we shouldn't try to manipulate these values.

This may be relevant, as willBeRetried might move. Also, though, we have to look into whether this is a property we should avoid touching.

Edit: Maybe also something in modifying test results issue at the bottom or linked in the new functionality (though I think that new functionality may actually not be capable of this).

[plocket]

Instead of avoiding rerunning random tests, we can use a seed to make random tests replicable. I'm not sure how we can then pass that value to the retry, but we can research that.

[plocket]

I think we can identify repeating tests! In Before(), the scenario argument (the first argument) looks like this between two runs of the same failing test:

First attempt

{
  "other stuff...": "things",
  "pickle": {
    "id": "a02b6fc8-40d4-48e5-a489-af98eef19710",
    "other pickle stuff": "stuff"
  },
  "testCaseStartedId": "c7acedf8-7ea7-42b3-8960-e06e716e870f"
}

Second attempt

{
  "other stuff...": "things",
  "pickle": {
    "id": "a02b6fc8-40d4-48e5-a489-af98eef19710",
    "other pickle stuff": "stuff"
  },
  "testCaseStartedId": "08d8ba5f-ab69-4af0-aaae-ce635ddd7573"
}

The pickle.id remains the same between the two runs. Even better, the testCaseStartedId changes.

We can possibly use this to take different action during a re-try and to offer authors different options, like always retrying vs. only retrying when the server was reloading. Exciting!