Run selected subset of files from Cypress interface (multiple tests)

[mattmazzola]

Current behavior:

As far as I know cypress only allows you to run in 3 types of ways.

• UI
  • Run all specs located in the tests/integration folder
  • Run an individual test by clicking it
• CI / Command line
  • Run all specs
  • Run a folder of test using --spec option and specifying folder

Desired behavior:

It would be nice if we could run tests in more flexible combinations. Below is picture of what I'm imagining:

I assume this feature could be hidden by default so it doesn't overwhelm users, but maybe enabled with a setting such as "Advanced Test Running" or something to opt-in to.

Each test and folder would have a check box. Modifying check of folder affects all children folders/tests.

There is button / warning to indicate when your selection deviates from normal and option to clear it in case people have checkbox selected but scroll down and might forget about it. In worse case I think they would see it trying to run test that had accidentally remained checked, stop the tests, uncheck it and continue.

Related option would be that the "Run All Specs" option only runs the tests that are satisfying the search text filter.

Versions

Cypress: Beta Version 3.0.3
Windows 10: 10.0.17763 Build 17763
Chrome: Version 72.0.3626.119 (Official Build) (64-bit)

[jennifer-shehane]

You should be able to do this via the command log using the --spec option. Like: cypress run --spec 'cypress/integration/#/**/*'

Also, make sure to be aware the issues with the 'Run all tests' implementation here #1586

The code to update the specs list, to add checkboxes, etc can be found here: https://github.com/cypress-io/cypress/blob/develop/packages/desktop-gui/src/specs/specs-list.jsx

We would however have to update how the specs are executed per the url when it is then opened in the Test Runner on click.

[mattmazzola]

You should be able to do this via the command log using the --spec option. Like: cypress run --spec 'cypress/integration/#/**/*'

Oh, my understanding of the spec option is that this line would only run tests in the login folder, but what if we want to run tests outside that folder as well?

On that issue: #1586 it's mentioned:

Enable clicking folders to automatically run a subset of specs

This would help a lot and be inline with checking the box for the folder and clicking run selected. Also this also seems very similar to using the --spec option and specifying the folder path.

My understanding of the issues you were referring to in (#1586) is that when running a lot of tests in interactive mode it will take up too much memory and it's advised to limit the set of tests run.

If we had check boxes and only ran those checked or only those in a folder this would also be limiting the number of tests so it seems also to help with that issue.

Can you explain a bit more about the url issue, I didn't quite understand?

I know when you click a single test that file name is put in the url like:
http://localhost:58054/__/#/tests/integration\CreateModels\WaitVsNoWaitActions.js
and my understanding of your suggesting is that if we did this change the url would need to be updated as the interactive mode switched form testA to testB, etc

When you click "Run all specs" it seems to show:
http://localhost:58054/__/#/tests/__all

So I thought we could so something like:
http://localhost:58054/__/#/tests/__filtered

Which would prevent having to deal with updating the url

[jennifer-shehane]

Oh, my understanding of the spec option is that this line would only run tests in the login folder, but what if we want to run tests outside that folder as well?

You should be able to comma separate globs as well so you can run multiple separate directories or files, like:

cypress run --spec 'cypress/integration/examples/**/*,cypress/integration/other-folder/**/*,cypress/integration/other-file.js'

Can you explain a bit more about the url issue, I didn't quite understand?

Yeah, what you guessed is on the right train of thought. Cypress currently will look for an exact match to 1 file (like when http://localhost:58054/__/#/tests/integration\CreateModels\WaitVsNoWaitActions.js it will look for a single file matching CreateModels\WaitVsNoWaitActions.js within your integrationFolder) or will look for __all and run all of the tests within your integrationFolder. We'd have to have some logic here for it to know how to run a subset of specs from a url.

[yagudaev]

Would love to see this added as soon as possible. This is a simple enough feature that will have a huge positive impact on developer experience.

Scenarios like "I only changed the customer app, so I don't need to run the admin specs" would be easy. It will also make it easier to debug leaky tests that fail when they run together.

[jennifer-shehane]

Related issue with how we want to change the behavior of 'run all specs' in general #1586

[atopper]

I'd like to see annotations on the test level. So you could add a test to multiple test groups (smoke, daily, full, etc.) and then run Cypress specifying the group.

[jennifer-shehane]

From @bahmutov suggestion in #6581

Our users are asking us sometimes to run some specs when using the interactive mode. I think the simplest way we could make them happy is to use the search pattern when the user clicks "Run All Specs" button.

[jennifer-shehane]

We've released the ability to run a selected subset of files based on the search filter in 4.12.0 - closing #6581.

Full instructions here: https://on.cypress.io/writing-and-organizing-tests#Run-filtered-specs

I don't believe this exactly covers the original criteria of this feature request, so this issue will remain open, but I thought it may be helpful to some people following this issue.

[OldSchooled]

We've released the ability to run a selected subset of files based on the search filter in 4.12.0 - closing #6581.

Full instructions here: https://on.cypress.io/writing-and-organizing-tests#Configuration

I don't believe this exactly covers the original criteria of this feature request, so this issue will remain open, but I thought it may be helpful to some people following this issue.

This is awesome, thank you. It would be more awesome, however, if I could enter multiple, separate search terms and capture the results for all of them.

e.g.
Searching (login; add-todo; delete-todo) returns all tests matching login, all tests matching add-todo and all tests matching delete-todo

[NevenaSymphony]

I would agree @atopper, adding annotations on test level would be a massive improvement, especially in the context of CI/CD: we could enable/disable cherry-picked tests to run on certain environments, create specified test suits for different deployments... possibilities are endless.

[feliperaul]

@jennifer-shehane Jennifer, is there any estimate for this feature?

Not having to constantly write .only and .skip in tests (and not forgetting to remove before a git commit) is something that would hugely improve the user experience.

This guy created a plugin here (https://github.com/bahmutov/cypress-skip-and-only-ui), but it broke after a Cypress update; his UI was great. Maybe it could get some love from the official developers?

[Zombobot1]

We've released the ability to run a selected subset of files based on the search filter in 4.12.0 - closing #6581.

Full instructions here: on.cypress.io/writing-and-organizing-tests#Run-filtered-specs

I don't believe this exactly covers the original criteria of this feature request, so this issue will remain open, but I thought it may be helpful to some people following this issue.

Is it possible to do in component testing? Its UI is quite limited

[BlueWinds]

Is it possible to do in component testing? Its UI is quite limited

We are reworking and unifying the UI for e2e and component testing at the moment, which should help with missing features. Currently expecting to release with 10.0.0, mid-January or so.

[AStoker]

Our use case is one where we have our e2e tests which live in the integration folder (totally fine there), but unit tests that test specific files are alongside the actual file that's being tested. This allows for modules to be moved around and test files go with them and not screw up references. I would love to be able to say, "Find all test files in 'Integrations' folder as well as any .spec.js file inside the 'src' folder. Are the new changes going to support that kind of use case?

[hodapp512]

Is there any progress on this issue in the new UI?