Parallel cypress test run failures now much more often #24514

glomotion · 2022-11-03T03:11:31Z

Current behavior

We have configured our cypress/react tests setup to run parallel in github actions across 6 different machines.

Just recently (after upgrading to 10.10.0) - we are now often seeing our test runs in cicd fail with the following response (> 50% of the time):

You passed the --parallel flag, but we do not parallelize tests across different environments.

This machine is sending different environment parameters than the first machine that started this parallel run.

The existing run is: https://dashboard.cypress.io/projects/[REDACTED]/runs

In order to run in parallel mode each machine must send identical environment parameters such as:

 - specs
 - osName
 - osVersion
 - browserName
 - browserVersion (major)

This machine sent the following parameters:

{
  "osName": "linux",
  "osVersion": "Ubuntu - 20.04",
  "browserName": "Chrome",
  "browserVersion": "107.0.[53](https://github.com/immutable/imx-marketplace-next/actions/runs/3382557523/jobs/5617596861#step:5:54)04.87",
  "specs": [
      REDACTED
  ]
}

https://on.cypress.io/parallel-group-params-mismatch
Test run failed, code 1
More information might be available above
Cypress module has returned the following error message:
Could not find Cypress test run results
Error: Could not find Cypress test run results

Until 10.10 - all of these tests were passing 100% in parallel in cicd. We are also seeing the same thing happen in 10.11.0

Meanwhile, we if view this above cicd test run in cypress dashboard, all tests are shown as passing! 🤪

Desired behavior

All tests should run and pass if they are valid, in parallel in cicd

Test code to reproduce

n/a

Cypress Version

10.11.0

Node version

16.18.0

Operating System

"ubuntu-latest"

Debug Logs

No response

Other

No response

The text was updated successfully, but these errors were encountered:

gofr · 2022-11-03T10:38:40Z

I've run into this issue before.

Your failures are caused by this change: actions/runner-images@0c8b93e

GitHub periodically updates Chrome to the latest version on their action runners. The runners aren't all updated at the same time. They are upgraded over a period of a few days. You can't control which action runner versions your workflow uses.

If you're running Cypress tests in parallel on multiple runners during such an upgrade phase, you can end up with different runners using different Chrome versions. This is the error you get when that happens.

However, when Cypress fails like this, the tests simply do not get run on that runner. What happens instead is that they remain in the queue and end up being picked up later by another runner. That's why your dashboard is still green.

glomotion · 2022-11-03T10:53:52Z

I see! That's very good to know. Thank you kindly @gofr.

nagash77 · 2022-11-03T18:20:45Z

This seems to be an outside issue causing your problems. I am going to close this issue. If new information comes to light that suggests something on the Cypress end feel free to comment on this issue and we can reopen.

vm · 2022-11-04T06:40:00Z

Is there a way to force a specific version of Chrome to be run? I think this is a very hard issue for cypress users to debug as is. I googled for a while before finding this thread.

cgraham-rs · 2022-11-04T13:23:29Z

Some additional debug in case anyone hits this issue in the future.

As @gofr mentioned the Github hosted runners apparently do not get updated at the same time. I hit the failed parallelization error logged here and it took me a lot of digging and logging support requests to figure out why.

My setup:

Cypress 10.10.0
Cypress Dashboard (love it)
CI runs everything on Github hosted runners using runs-on: ubuntu-latest
CI runs all tests on the very latest browsers using browser-actions/setup-chrome@latest

In my case all of my runners are definitely using the same browser version as we're literally installing the latest and there's plenty of debug for that workflow step to confirm that.

Run browser-actions/setup-chrome@latest
  with:
    chrome-version: latest
  env:
    AWS_DEFAULT_REGION: us-east-2
    AWS_REGION: us-east-2
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_SESSION_TOKEN: ***
Setup chromium latest
Attempting to download latest...
Acquiring 1066434 from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Linux_x64%2F1066434%2Fchrome-linux.zip?alt=media
Installing chromium...
/usr/bin/unzip -o -q /home/runner/work/_temp/a0a2f9cb-714f-4b1d-8432-4f2147df18d3
Successfully Installed chromium to /opt/hostedtoolcache/chromium/latest/x64
Successfully installed chromium to /opt/hostedtoolcache/chromium/latest/x64/chrome
Successfully setup chromium version latest
/opt/hostedtoolcache/chromium/latest/x64/chrome --version
Chromium 109.0.5397.0

However, while the runners are all using the same reported version of Linux the runner versions are definitely not the same. It's super hard to miss, and the Cypress debug does not current provide enough detail to figure this out on your own.

Runner that failed to execute Cypress parallelized:

You passed the --parallel flag, but we do not parallelize tests across different environments.

This machine is sending different environment parameters than the first machine that started this parallel run.

The existing run is: https://dashboard.cypress.io/projects/dhybrn/runs

In order to run in parallel mode each machine must send identical environment parameters such as:

 - specs
 - osName
 - osVersion
 - browserName
 - browserVersion (major)

This machine sent the following parameters:

{
  "osName": "linux",
  "osVersion": "Ubuntu - 20.04",
  "browserName": "Chrome",
  "browserVersion": "106.0.5249.119",
  "specs": [
    "cypress/e2e/auth.spec.js",
    "cypress/e2e/dashboard.spec.js",
    etc. etc.
  ]
}

Runner that could successfully execute Cypress parallelized.

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        10.10.0                                                                        │
  │ Browser:        Chrome 107 (headless)                                                          │
  │ Node Version:   v16.18.0 (/opt/hostedtoolcache/node/16.18.0/x64/bin/node)                      │
  │ Specs:          22 found (auth.spec.js, dashboard.spec.js, home.spec.js, search.spec.js, sylla │
  │                 bus.spec.js, termsAndConditions.spec.js, assessment/assessment_logic_flow.spec │
  │                 .js, assessment/general_assessment.spec.js, assessment/group_report_panel.spec │
  │                 .js, exercise_bank/data...)                                                    │
  │ Searched:       cypress/e2e/**/*.{js,jsx,ts,tsx}                                               │
  │ Params:         Tag: false, Group: chrome, Parallel: true                                      │
  │ Run URL:        https://dashboard.cypress.io/projects/dhybrn/runs/1584                         │
  │ Experiments:    experimentalFetchPolyfill=true,experimentalSessionAndOrigin=true               │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

Looking into the runners more:

Runner that failed to execute Cypress parallelized:

Current runner version: '2.298.2'
Operating System
  Ubuntu
  20.04.5
  LTS
Runner Image
  Image: ubuntu-20.04
  Version: 20221024.1
  Included Software: https://github.com/actions/runner-images/blob/ubuntu20/20221024.1/images/linux/Ubuntu2004-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu20%2F20221024.1
Runner Image Provisioner
  2.0.91.1
GITHUB_TOKEN Permissions
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'actions/checkout@v3' (SHA:93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8)
Download action repository 'actions/setup-node@v3' (SHA:8c91899e586c5b171469028077307d293428b516)
Download action repository 'aws-actions/configure-aws-credentials@v1' (SHA:67fbcbb121271f7775d2e7715933280b06314838)
Download action repository 'browser-actions/setup-chrome@latest' (SHA:7fad76f55ddc71d600884f213f8cc18bd7931b3e)
Download action repository 'browser-actions/setup-firefox@latest' (SHA:c990ef23a9bedbbe657b163aaf0f3b7c608ea9f0)
Download action repository 'cypress-io/github-action@v4' (SHA:30008f1458a5a2c97054bfe118fe33d75976c482)

Runner that could successfully execute Cypress parallelized.

Current runner version: '2.298.2'
Operating System
  Ubuntu
  20.04.5
  LTS
Runner Image
  Image: ubuntu-20.04
  Version: 20221027.1
  Included Software: https://github.com/actions/runner-images/blob/ubuntu20/20221027.1/images/linux/Ubuntu2004-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu20%2F20221027.1
Runner Image Provisioner
  2.0.91.1
GITHUB_TOKEN Permissions
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'actions/checkout@v3' (SHA:93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8)
Download action repository 'actions/setup-node@v3' (SHA:8c91899e586c5b171469028077307d293428b516)
Download action repository 'aws-actions/configure-aws-credentials@v1' (SHA:67fbcbb121271f7775d2e7715933280b06314838)
Download action repository 'browser-actions/setup-chrome@latest' (SHA:7fad76f55ddc71d600884f213f8cc18bd7931b3e)
Download action repository 'browser-actions/setup-firefox@latest' (SHA:c990ef23a9bedbbe657b163aaf0f3b7c608ea9f0)
Download action repository 'cypress-io/github-action@v4' (SHA:30008f1458a5a2c97054bfe118fe33d75976c482)

Note the runner images are different: Version: 20221027.1 vs Version: 20221024.1 However, the image is the same, and version of Ubuntu is the same.

Eventually the parallelization failure stoped on its own. As best I can tell it was fixed once all of the runners started using the same Runner Image Version: 20221027.1

gofr · 2022-11-04T17:11:18Z

@cgraham-rs You may definitely be installing the latest version of Chromium (109 there), but are you sure Cypress is actually using it? Because your Cypress logs still show your tests using Chrome 106 and 107. Don't you need to specify your browser to be chromium or the full path to the browser you installed?

One way of working around this problem is to not let your CI depend on the results of your individual parallel runners. Aggregate the results of the parallel runners in a separate step and pass/fail based on that. Individual runners can still fail, but Cypress will eventually run all your tests.

At worst, you end up in a situation where the first runner that started is the only runner on the old/new Chrome and it ends up having to run all your tests because none of the other runner environments match.

cgraham-rs · 2022-11-04T18:27:55Z

@gofr Thank you for QA'ing my workflow. 🥇 Off to fix stuff...

nagash77 closed this as not planned Won't fix, can't repro, duplicate, stale Nov 3, 2022

cgraham-rs mentioned this issue Nov 4, 2022

Provide more detailed debug when parallelization fails #24540

Closed

baptou12 mentioned this issue Sep 20, 2023

Fix Cypress fail aléatoire betagouv/aides-jeunes#3991

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel cypress test run failures now much more often #24514

Parallel cypress test run failures now much more often #24514

glomotion commented Nov 3, 2022 •

edited

Loading

gofr commented Nov 3, 2022 •

edited

Loading

glomotion commented Nov 3, 2022

nagash77 commented Nov 3, 2022

vm commented Nov 4, 2022

cgraham-rs commented Nov 4, 2022

gofr commented Nov 4, 2022

cgraham-rs commented Nov 4, 2022

Parallel cypress test run failures now much more often #24514

Parallel cypress test run failures now much more often #24514

Comments

glomotion commented Nov 3, 2022 • edited Loading

Current behavior

Desired behavior

Test code to reproduce

Cypress Version

Node version

Operating System

Debug Logs

Other

gofr commented Nov 3, 2022 • edited Loading

glomotion commented Nov 3, 2022

nagash77 commented Nov 3, 2022

vm commented Nov 4, 2022

cgraham-rs commented Nov 4, 2022

gofr commented Nov 4, 2022

cgraham-rs commented Nov 4, 2022

glomotion commented Nov 3, 2022 •

edited

Loading

gofr commented Nov 3, 2022 •

edited

Loading