StopOnFail doesn't seem to be supported in net core test runner

[ErikApption]

I am not sure if this is a limitation of dotnet test but I cannot get xunit.execution.StopOnFail to work with the dotnet test command line. I tried --stoponfail=true and - xunit.execution.StopOnFail=true but none of these are actually stopping the execution after the first failed test.

[Mayron]

I also want to see this added. Struggled for 2 hours and giving up now.

[tig]

In addition, I can't get this to work either:

{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "parallelizeTestCollections": false,
  "parallelizeAssembly": false,
  "stopOnFail": true
}

The schema does not inlcude stopOnFail:

https://xunit.net/schema/current/xunit.runner.schema.json

[bradwilson]

That is correct, the current schema (as of today) does not support stop-on-fail.

[tig]

@bradwilson, I upgraded to 2.5.0 and have this xunit.runner.json in my project:

{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "parallelizeTestCollections": false,
  "parallelizeAssembly": false,
  "stopOnFail": true
}

Test runner is not stopping on the first failure. What could I be missing?

[bradwilson]

@tig I'll see if I can come up with a repro.

[bradwilson]

@tig Here's my first repro attempt.

I created three test classes. Each has 5 failing tests and 20 passing tests (due to randomization we don't know in what exact order things will be run, so I wanted to have a good baseline for "some test somewhere in the middle should fail"). I used your xunit.runner.json settings, and then did "run everything" from Test Explorer.

Here's the result:

In the first class it chose to run, it managed to run 7 passing tests before hitting a failing test. It then stopped running all tests. So we get 7 passes, 1 fail, and 67 not run.

This is precisely the behavior I would've expected.

It looks like you're using the Resharper runner. I cannot account for which features they do and do not support. I can guess that support for stopOnFail is probably not yet integrated into their runner, because they would've had to pick up a recent build of xunit.runner.utility as well as integrating in a bit of code to ensure that the stop signal gets propagated to all test assemblies (not exactly the situation you've describe, but it does need to be done if they are trying to run assemblies in parallel, as illustrated by our changes to the VSTest adapter).

Are you using Visual Studio? If so, does the behavior differ if you use Test Explorer instead of the Resharper test runner?

[bradwilson]

It's worth mentioning that I can get similar behavior from dotnet test, albeit with less helpful UI to illustrate it. 😄

[tig]

Thanks for looking into this Brad.

Just to ensure it wasn't Resharper I did the following on a machine that does not have Reshaper installed:

git clone tig:gui-cs\Terminal.Gui
cd Terminal.Gui

Added "stopOnFail": true to ./UnitTests/xunit.runner.json.

Updated xunit in ./UnitTests to

    <PackageReference Include="xunit" Version="2.5.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.5.0">

Changed a unit test to Assert.Fail():

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using Xunit;
using Xunit.Abstractions;

// Alias Console to MockConsole so we don't accidentally use Console
using Console = Terminal.Gui.FakeConsole;

namespace Terminal.Gui.DriverTests {
	public class ConsoleDriverTests {
		readonly ITestOutputHelper output;

		public ConsoleDriverTests (ITestOutputHelper output)
		{
			this.output = output;
		}

		[Theory]
		[InlineData (typeof (FakeDriver))]
		//[InlineData (typeof (NetDriver))]
		//[InlineData (typeof (CursesDriver))]
		//[InlineData (typeof (WindowsDriver))]
		public void Init_Inits (Type driverType)
		{
			Assert.Fail ("testing xunit stoponfail");
		}

dotnet test -v Detailed

Got this:

I also tried same with Test Explorer and got same result; all tests were run even after the one with Assert.Fail ran...

[tig]

Note we have a bug in v2_develop that causes the test process to not close. It's fixed in another branch and just in case that was causing this, I back-ported it. It cleans up the error spew, but the tests still don't stop:

...

[bradwilson]

@tig You're missing step 2 of the configuration file instructions: https://xunit.net/docs/configuration-files#file

This means that xunit.runner.json is not getting copied to the output folder, and therefore is not being used (so any configuration settings, including stopOnFail, are being ignored).

[tig]

@tig You're missing step 2 of the configuration file instructions: https://xunit.net/docs/configuration-files#file

This means that xunit.runner.json is not getting copied to the output folder, and therefore is not being used (so any configuration settings, including stopOnFail, are being ignored).

OFFS!

This has been this way for years! Egads, that's embarrassing.

[tig]

@bradwilson Do you know if there's a way for us to set stopOnFail to true as part of our GItHub CI/CD actions?

We definitely want it there, but not necessarily when doing local dev work.

[bradwilson]

At the moment, there isn't a way short of swapping in a different xunit.runner.json file. I'm planning on adding support for all configuration items in RunSettings, which would allow you to either pass a runsettings file with your CI settings using dotnet test --settings filename or via individual values via dotnet test -- name=value [name=value...]. More information on the dotnet test command line arguments is available here: https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test

[tig]

‘sed -i 's/"stopOnFail": false/"stopOnFail": true/g' ./UnitTests/xunit.runner.json‘

[bradwilson]

@tig The next prerelease build of xunit.runner.visualstudio 2.5.1 will support passing command-line based options, per #378. If you want to try a build now, MyGet has 2.5.1-pre.5 which has the feature in it. Docs page is available here: https://xunit.net/docs/runsettings