Skip to content

investigate flaky test-vm-timeout-escape-promise-module-2 on freebsd #36627

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
Trott opened this issue Dec 25, 2020 · 4 comments
Closed

investigate flaky test-vm-timeout-escape-promise-module-2 on freebsd #36627

Trott opened this issue Dec 25, 2020 · 4 comments
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. freebsd Issues and PRs related to the FreeBSD platform.

Comments

@Trott
Copy link
Member

Trott commented Dec 25, 2020

  • Test: parallel/test-vm-timeout-escape-promise-module-2
  • Platform: freebsd
  • Console Output:
20:55:36 not ok 2591 parallel/test-vm-timeout-escape-promise-module-2
20:55:36   ---
20:55:36   duration_ms: 0.397
20:55:36   severity: fail
20:55:36   exitcode: 1
20:55:36   stack: |-
20:55:36     (node:79375) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
20:55:36     (Use `node --trace-warnings ...` to show where the warning was created)
20:55:36     /usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22
20:55:36           throw new Error(
20:55:36                 ^
20:55:36     
20:55:36     Error: escaped timeout at 173 milliseconds!
20:55:36         at loop (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22:13)
20:55:36         at vm:module(0):1:30
20:55:36         at SourceTextModule.evaluate (node:internal/vm/module:229:23)
20:55:36         at assert.rejects.code (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:38:16)
20:55:36         at processTicksAndRejections (node:internal/process/task_queues:93:5)
20:55:36   ...
@Trott Trott added freebsd Issues and PRs related to the FreeBSD platform. flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels Dec 25, 2020
@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

https://ci.nodejs.org/job/node-test-commit-freebsd/38718/nodes=freebsd11-x64/console

00:20:50 not ok 2672 parallel/test-vm-timeout-escape-promise-module-2
00:20:51   ---
00:20:51   duration_ms: 0.440
00:20:51   severity: fail
00:20:51   exitcode: 1
00:20:51   stack: |-
00:20:51     (node:46479) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
00:20:51     (Use `node --trace-warnings ...` to show where the warning was created)
00:20:51     /usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22
00:20:51           throw new Error(
00:20:51                 ^
00:20:51     
00:20:51     Error: escaped timeout at 179 milliseconds!
00:20:51         at loop (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22:13)
00:20:51         at vm:module(0):1:30
00:20:51         at SourceTextModule.evaluate (node:internal/vm/module:224:23)
00:20:51         at assert.rejects.code (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:38:16)
00:20:51         at processTicksAndRejections (node:internal/process/task_queues:94:5)
00:20:51   ...

@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

My local results suggest that parallelism may help rather than hurt this test but I want to confirm that on CI, because that's counterintuitive, especially when FreeBSD in CI has a history of being our most parallel-sensitive platform (probably having to do with the way the CI host counts processors).

Stress test with no parallelism: https://ci.nodejs.org/job/node-stress-single-test/231/

Stress test with parallelism: https://ci.nodejs.org/job/node-stress-single-test/232/

@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

Stress test with no parallelism: https://ci.nodejs.org/job/node-stress-single-test/231/

Stress test with parallelism: https://ci.nodejs.org/job/node-stress-single-test/232/

Results sort of show it's flaky with parallelism and reliable without it, but I imagine the p value is small. 0 failures in 1000 runs in the first case, and 2 failures in 1000 runs in the second place.

Let's try them both again with 10000 runs to get a little more confidence:

Not parallel: https://ci.nodejs.org/job/node-stress-single-test/233/
Parallel: https://ci.nodejs.org/job/node-stress-single-test/235/

Trott added a commit to Trott/io.js that referenced this issue Mar 20, 2021
Test is reliable in sequential and not in parallel, so move it there.

Fixes: nodejs#36627
@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

Optimistically opened a PR moving it to sequential. #37842

ruyadorno pushed a commit that referenced this issue Mar 24, 2021
Test is reliable in sequential and not in parallel, so move it there.

Fixes: #36627

PR-URL: #37842
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this issue May 1, 2021
Test is reliable in sequential and not in parallel, so move it there.

Fixes: #36627

PR-URL: #37842
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. freebsd Issues and PRs related to the FreeBSD platform.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant