Control the sequence of test files

[sapjax]

Like the situation in #754.
The E2E test is not like the unit test, Many tests in our app depend on other tests run before.

01-test.e2e.js
02-test.e2e.js
...

but the solution above is not good enough:

1. look at the file name I can't recognize what is the test inside it.

2. What If I want to run some specific test files with reuse?

3. If I want to change the sequence, I must change the file name, it will cause the git think I made a lot of changes.

We need an option to config the sequence in an array, like:

[
  `a.e2e.js`,
  `b.e2e.js`,
  `c.e2e.js`,
  `d.e2e.js`,
]

If I want to change the sequence of b & c, I can just set it as:

[
  `a.e2e.js`,
  `c.e2e.js`,
  `b.e2e.js`,
  `d.e2e.js`,
]

[noomorph]

I honestly don't understand why you demand this from Detox.

Detox just runs on top of your preferred test runner as a handful of helper methods on steroids. How do you imagine adding this feature if Detox is not a test runner and has no direct control over the loading sequence?

In other words, if there's a feature to specify exact test suite order in Mocha and Jest, and you clearly see that Detox CLI stands in its way (e.g. ignores/overrides CLI arguments or configs), then let's talk about that specific issue. Otherwise, this is something beyond of Detox's reach and there's not much to do about it in the scope of Detox's codebase.

[noomorph]

@sapjax , as you see there are open issues like this one: jestjs/jest#6194

[sapjax]

@noomorph
I just throw the question, I think this is a common situation in an E2E test.
About the test runner (Mocha/Jest), they are just unit test runner, so they no need to care about the sequence, most of the unit tests not depend on each other.

If the test runners can't meet the demand of E2E test.
Maybe we should found or implement a test runner which can run in a specific sequence.

Thanks for the related issue in Jest project, but the solution(just as i'm using) is still not good.
If we require all test files in an index.spec.js, some options will not works, like http://jestjs.io/docs/en/configuration.html#bail-boolean.

[noomorph]

@sapjax , sure, when a test runner is too much oriented towards unit tests, sometimes it fails to deliver features normally expected for E2E test suites. For example, currently, there is an unsolved issue about Jest and Ctrl+C (extremely abrupt exit without a chance to clean up resources). So, your issue falls into this unlucky category.

Anyway, I'm trying to say that I don't see ways to implement this in Detox as long as test runners don't provide the capabilities you seek.

UPD: Added label invalid/wontfix. Don't perceive too seriously the invalid part, but in any case, as I mentioned above, I don't see acceptable ways to address this from Detox source code.

If you have a prototype which shows how to uniformly leverage both Mocha and Jest API to configure a specific sequence of test files, please demonstrate.

[vonovak]

having your tests depend on order in which they are executed is, imho, a really bad idea that one day will turn against you. Test should be able to run independently. You should not need to run test A to run test B. Test B should just be able to run by itself - without running A first. Apart from better isolation this brings the possibility to run tests in parallel. just my two cents.

[sapjax]

@vonovak
Suppose I have a wallet management program, the first step is to create a wallet account. The second step is to transfer money.
So there are two test files： createAccount.test.js and transfer.test.js.

If I haven't created an account yet. How can I transfer money?

[vonovak]

@sapjax the money transfer test suite will need to have a beforeEach section where the account is created

[sapjax]

@vonovak
Think about a bitcoin wallet.
We need to create a new bitcoin address first, we want to test the real process, so the address is random.
the private key of this address must store in the App after created, when transfer, we need the private key to sign the transaction.

If we haven't created the bitcoin address, we don't know the wallet address and private key.

This is just an example, there are many random processes in the real-world app, you can't decide which state you put in beforeEach section.

And even if some data in beforeEach make the transfer.test.js passed. it's no relationship with createAccount.test.js, so you get all your test files passed, you still can't ensure the whole process will work as expected in the real world.

[LeoNatan]

If you are unable to perform this logic in beforeEach, you should look into mocking the functionality. Especially in those "many random processes in the real-world app", you do not want to be dependent on such randomness.

You should test the real account creation in one test, then mock accounts in all others.

This is not planned at this point.