Skip to content

Reliablility report is inaccurate #647

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
MoLow opened this issue Sep 29, 2022 · 2 comments
Open

Reliablility report is inaccurate #647

MoLow opened this issue Sep 29, 2022 · 2 comments

Comments

@MoLow
Copy link
Member

MoLow commented Sep 29, 2022

I started investigating nodejs/node#44805
when taking a look at nodejs/reliability#388:

Reason sequential/test-watch-mode-inspect
Type JS_TEST_FAILURE
Failed PR 9 (nodejs/node#44787, nodejs/node#44741, nodejs/node#44767, nodejs/node#44770, nodejs/node#44376, nodejs/node#44693, nodejs/node#44686, nodejs/node#44771, nodejs/node#44711)
Appeared test-nearform-macos10.15-x64-2, test-nearform-macos10.15-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46819/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46857/
Example
not ok 3789 sequential/test-watch-mode-inspect
  ---
  duration_ms: 120.67
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    TAP version 13
    (node:37761) ExperimentalWarning: The test runner is an experimental feature. This feature could change at any time
    (Use `node --trace-warnings ...` to show where the warning was created)
  ...


the first PR in the Failed PR row has two CI runs prior to this report: 46819, 46808

the first CI run reports a failure for this based on the previous osx1015 run https://ci.nodejs.org/job/node-test-commit-osx/47578/ that did not actually run in this build (due to java.net.ConnectException: Connection timed out) but Jenkins still considers the child of a previous run to be the child of this run.

this obviously happened in other PRS/CI runs as well, wich makes the report links and count very hard to rely on when investigating flaky test issues

@richardlau
Copy link
Member

The reports here are generated using the ncu-ci walk ... command from node-core-utils.

@MoLow
Copy link
Member Author

MoLow commented Sep 30, 2022

So this issue should be transferred, but I don't have write accees on the ncu repository so I cannot transfer it

@richardlau richardlau transferred this issue from nodejs/reliability Sep 30, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants