Skip to content

CI Reliability 2022-08-01 #334

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
11 tasks
github-actions bot opened this issue Aug 1, 2022 · 0 comments
Open
11 tasks

CI Reliability 2022-08-01 #334

github-actions bot opened this issue Aug 1, 2022 · 0 comments

Comments

@github-actions
Copy link

github-actions bot commented Aug 1, 2022

Failures in node-test-pull-request/45660 to node-test-pull-request/45758 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2022-08-01 00:16 0 6 41 1 52 6.06%

Jenkins Failure

Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 10 (nodejs/node#43967, nodejs/node#43979, nodejs/node#43977, nodejs/node#43921, nodejs/node#43990, nodejs/node#43993, nodejs/node#44001, nodejs/node#44013, nodejs/node#43942, nodejs/node#44064)
Appeared test-digitalocean-freebsd12-x64-2, test-digitalocean-freebsd12-x64-1, test-equinix-ubuntu2004_container-armv7l-2
First CI https://ci.nodejs.org/job/node-test-pull-request/45660/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45747/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 107.170.28.213/107.170.28.213:34572' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:216)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	at com.sun.proxy.$Proxy79.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1213)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1205)

Build Failure

Reason ERROR: Build step failed with exception
Type BUILD_FAILURE
Failed PR 3 (nodejs/node#43492, nodejs/node#43521, nodejs/node#43999)
Appeared test-equinix-ubuntu2004_container-armv7l-2
First CI https://ci.nodejs.org/job/node-test-pull-request/45688/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45704/
Example
ERROR: Build step failed with exception
java.lang.NullPointerException: no workspace from node hudson.slaves.DumbSlave[test-equinix-ubuntu2004_container-armv7l-2] which is computer hudson.slaves.SlaveComputer@2ad6001 and has channel null
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:113)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
	at org.jenkinsci.plugins.postbuildscript.processor.Processor.processBuildSteps(Processor.java:180)
	at org.jenkinsci.plugins.postbuildscript.processor.Processor.processScripts(Processor.java:91)

Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 2 (nodejs/node#43629, nodejs/node#44032)
Appeared test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu1804_container-arm64-2, test-equinix-ubuntu2004_container-arm64-2, test-equinix-ubuntu1804_sharedlibs_container-arm64-3, test-equinix-debian10_container-armv7l-2, test-equinix-ubuntu2004_container-armv7l-1, test-equinix-debian10_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2
First CI https://ci.nodejs.org/job/node-test-pull-request/45681/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45723/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

undefined

Reason Unknown
Type undefined
Failed PR 21 (nodejs/node#43967, nodejs/node#43976, nodejs/node#43921, nodejs/node#43760, nodejs/node#43990, nodejs/node#43993, nodejs/node#43629, nodejs/node#43977, nodejs/node#43492, nodejs/node#44001, nodejs/node#43984, nodejs/node#43999, nodejs/node#44013, nodejs/node#43521, nodejs/node#44018, nodejs/node#43784, nodejs/node#44048, nodejs/node#44032, nodejs/node#43942, nodejs/node#43904, nodejs/node#44056)
Appeared test-equinix-debian10_container-armv7l-1, test-nearform_arm-win10_vs2019-arm64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/45660/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45755/
Example
Unknown

JSTest Failure

Reason parallel/test-worker-heap-snapshot
Type JS_TEST_FAILURE
Failed PR 6 (nodejs/node#43629, nodejs/node#43784, nodejs/node#43984, nodejs/node#43521, nodejs/node#44032, nodejs/node#43904)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-8, test-digitalocean-ubuntu1804_sharedlibs_container-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-9, test-rackspace-win2012r2_vs2019-x64-4, test-rackspace-win2012r2_vs2015-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4, test-rackspace-win2012r2_vs2019-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/45685/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45749/
Example
not ok 3036 parallel/test-worker-heap-snapshot
  ---
  duration_ms: 1.7
  severity: crashed
  exitcode: -11
  stack: |-
  ...


Reason parallel/test-worker-fshandles-error-on-termination
Type JS_TEST_FAILURE
Failed PR 4 (nodejs/node#43902, nodejs/node#43904, nodejs/node#43784, nodejs/node#44070)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-10, test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-softlayer-ubuntu1804_sharedlibs_container-x64-2, test-digitalocean-alpine312_container-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/45673/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45756/
Example
not ok 2967 parallel/test-worker-fshandles-error-on-termination
  ---
  duration_ms: 120.111
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-worker-fshandles-open-close-on-termination
Type JS_TEST_FAILURE
Failed PR 4 (nodejs/node#43902, nodejs/node#43904, nodejs/node#43784, nodejs/node#44070)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-10, test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-softlayer-ubuntu1804_sharedlibs_container-x64-2, test-digitalocean-alpine312_container-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/45673/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45756/
Example
not ok 2969 parallel/test-worker-fshandles-open-close-on-termination
  ---
  duration_ms: 120.130
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-net-connect-memleak
Type JS_TEST_FAILURE
Failed PR 3 (nodejs/node#43760, nodejs/node#43993, nodejs/node#43629)
Appeared test-azure_msft-win2016_vs2017-x64-5, test-rackspace-win2012r2_vs2015-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/45675/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45685/
Example
not ok 436 parallel/test-net-connect-memleak
  ---
  duration_ms: 0.260
  severity: fail
  exitcode: 1
  stack: |-
    Mismatched noop function calls. Expected exactly 1, actual 0.
        at Proxy.mustCall (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:338:10)
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-net-connect-memleak.js:33:40)
        at Module._compile (internal/modules/cjs/loader.js:1085:14)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
        at Module.load (internal/modules/cjs/loader.js:950:32)
        at Function.Module._load (internal/modules/cjs/loader.js:790:12)
        at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
        at internal/main/run_main_module.js:17:47
  ...


Reason abort/test-addon-uv-handle-leak
Type JS_TEST_FAILURE
Failed PR 2 (nodejs/node#43990, nodejs/node#44018)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/45676/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45729/
Example
not ok 3720 abort/test-addon-uv-handle-leak
  ---
  duration_ms: 300.188
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-vm-break-on-sigint
Type JS_TEST_FAILURE
Failed PR 2 (nodejs/node#43902, nodejs/node#44070)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-10, test-digitalocean-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6
First CI https://ci.nodejs.org/job/node-test-pull-request/45684/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45756/
Example
not ok 2966 parallel/test-vm-break-on-sigint
  ---
  duration_ms: 120.140
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason sequential/test-worker-prof
Type JS_TEST_FAILURE
Failed PR 2 (nodejs/node#43760, nodejs/node#43904)
Appeared test-rackspace-win2012r2_vs2019-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/45675/
Last CI https://ci.nodejs.org/job/node-test-pull-request/45750/
Example
not ok 861 sequential/test-worker-prof
  ---
  duration_ms: 1.950
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:123
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: child exited with non-zero status:                      {
      status: 3221225477,
      signal: null,
      output: [ null, '', '' ],
      pid: 5728,
      stdout: '',
      stderr: ''
    }
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\sequential\test-worker-prof.js:61:10)
        at Module._compile (node:internal/modules/cjs/loader:1113:14)
        at Object.Module._extensions..js (node:internal/modules/cjs/loader:1167:10)
        at Module.load (node:internal/modules/cjs/loader:991:32)
        at Function.Module._load (node:internal/modules/cjs/loader:832:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
        at node:internal/main/run_main_module:17:47 {
      generatedMessage: false,
      code: 'ER...

Progress

  • Backing channel 'JNLP4-connect connection from ... is disconnected. (10)
  • ERROR: Build step failed with exception (3)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (2)
  • Unknown (21)
  • parallel/test-worker-heap-snapshot (6)
  • parallel/test-worker-fshandles-error-on-termination (4)
  • parallel/test-worker-fshandles-open-close-on-termination (4)
  • parallel/test-net-connect-memleak (3)
  • abort/test-addon-uv-handle-leak (2)
  • parallel/test-vm-break-on-sigint (2)
  • sequential/test-worker-prof (2)
kvakil added a commit to kvakil/node that referenced this issue Aug 2, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.
kvakil added a commit to kvakil/node that referenced this issue Aug 2, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: nodejs#43499
Refs: nodejs#43084
nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Aug 5, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: #43499
Refs: #43084
PR-URL: #44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
danielleadams pushed a commit to nodejs/node that referenced this issue Aug 16, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: #43499
Refs: #43084
PR-URL: #44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
ruyadorno pushed a commit to nodejs/node that referenced this issue Aug 23, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: #43499
Refs: #43084
PR-URL: #44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
targos pushed a commit to nodejs/node that referenced this issue Sep 5, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: #43499
Refs: #43084
PR-URL: #44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Fyko pushed a commit to Fyko/node that referenced this issue Sep 15, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: nodejs#43499
Refs: nodejs#43084
PR-URL: nodejs#44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
juanarbol pushed a commit to nodejs/node that referenced this issue Oct 10, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: #43499
Refs: #43084
PR-URL: #44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
juanarbol pushed a commit to nodejs/node that referenced this issue Oct 11, 2022
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: #43499
Refs: #43084
PR-URL: #44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
guangwong pushed a commit to noslate-project/node that referenced this issue Jan 3, 2023
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: nodejs/node#43499
Refs: nodejs/node#43084
PR-URL: nodejs/node#44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
guangwong pushed a commit to noslate-project/node that referenced this issue Jan 3, 2023
These tests seem to timeout quite often. I don't know why, but one
possible reason is that they are starting a lot of threads. It seems
that tests in `test/parallel` are assumed to only start one thread each,
so having 11 threads running at a time feels like a lot.

It also seems that these tests fail in a correlated fashion: take a look
at [this reliability report][]. The failures all occur on the same build
machines on the same PRs. This suggests to me some sort of CPU
contention.

[this reliability report]: nodejs/reliability#334

On my Linux machine decreasing the parallelism & iterations here reduce
the `user` time from ~11.5 seconds to ~2 seconds, depending on the test.
I have seen these tests take 30-60 seconds on CI (Alpine in particular).

I went back to the diffs that introduced that introduced these changes
and verified that they failed at least 90% of the time with the reduced
iteration count, which feels sufficient.

Refs: nodejs/node#43499
Refs: nodejs/node#43084
PR-URL: nodejs/node#44090
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

0 participants