Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Rider uses an older version of tests when running unit tests from the IDE #831

Closed
j-h-a opened this issue Oct 7, 2018 · 1 comment
Closed
Assignees
Milestone

Comments

@j-h-a
Copy link

j-h-a commented Oct 7, 2018

Rider Version: 2018.2.3
Plugin Build: 182.4231.496
Unity Version: 2018.2.8f1

When doing TDD and running tests regularly after making changes, I noticed that test execution allows Unity to use an older version of the tests and/or code than what is currently saved.

Expected behaviour:

When running unit tests from Rider, the results reflect the current state of the saved code and tests.

Actual behaviour:

When running unit tests from Rider, the results can temporarily reflect an older state of the code and tests, or permanently reflect an older state in the case where compilation of code or tests fails.

To reproduce:

  1. Start with a test-suite with some passing tests in it.
  2. Run the tests from Rider, all should pass.
  3. Introduce a new test:
[Test]
public void FakeTest()
{
	Assert.Fail();
}
  1. Save all, and immediately run the unit tests (⌘S, ⌘;, R)
  2. Rider immediately recognises the new test, but Unity doesn't, this presents as an inconclusive/not run test:
    28 Tests with one inconclusive
  3. Wait a few seconds and without making any changes run the unit tests again. Unity has now caught up and runs the failing test:
    28 Tests with one failing
  4. Make the test pass by removing the Assert.Fail(); line, save, and immediately run the tests again. The test which should now pass, still fails:
    28 Tests with one failing
  5. Once again, wait a few seconds and without making any changes run the unit tests again. Unity has now caught up and all tests pass.
  6. Introduce a compile error into the tests:
[Test]
public void FakeTest()
{
	please fail to compile
}
  1. Running the tests immediately has the same effect as above, they still all show passing. But now, even waiting a few seconds won't work because Unity never catches up - it continues to use the previous working version of the code/tests because it can't compile the new version.

Speculation on cause and proposed fix:

Cause: Rider triggers Unity to run unit tests without triggering a re-compile, or waiting for any ongoing re-compile to finish.
Proposed fix: Delay triggering a test run until Unity has compiled any saved changes.

Cause: Rider allows Unity to run old compiled code and tests when the current code/tests don't compile.
Proposed fix: Report back to Rider a total failure to run any tests in the case where Unity can't compile the latest saved changes.

@van800 van800 self-assigned this Oct 8, 2018
@van800 van800 added this to the Rider 2018.3 milestone Oct 8, 2018
van800 added a commit that referenced this issue Nov 23, 2018
van800 added a commit that referenced this issue Nov 26, 2018
 refresh Unity before running tests #831
@van800 van800 closed this as completed Nov 26, 2018
@van800 van800 reopened this Nov 26, 2018
@van800
Copy link
Member

van800 commented Nov 28, 2018

Fixed with #929

@van800 van800 closed this as completed Nov 28, 2018
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants