./build.sh --test fails to find 'xunit.console.dll'

[tmds]

I'm working on making our internal CI build .NET 9 (that is: main) branches.

Our runtime jobs fail while executing tests with this message (for each test project):

 The application to execute does not exist: 'xunit.console.dll'

I think this stopped working since 8726d6b. Maybe it has to do with the changes to xunit.props in that commit.

In our builds, we're building the clr+libs+host+packs+libs.tests subsets.
And we're running tests using ./build.sh --subset libs.tests --test.

@ViktorHofer do you have some thoughts on what is the cause, and how it can be fixed?

@ericstj I saw your PR here: #97454 and thought it might be related. I gave your branch to our CI, but unfortunately it did not solve the issue.

cc @omajid

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I'm working on making our internal CI build .NET 9 (that is: main) branches.

Our runtime jobs fail while executing tests with this message:

 The application to execute does not exist: 'xunit.console.dll'

I think this stopped working since 8726d6b. Maybe it has to do with the changes to xunit.props in that commit.

In our builds, we're building the clr+libs+host+packs+libs.tests subsets.
And we're running tests using ./build.sh --subset libs.tests --test.

@ViktorHofer do you have some thoughts on what is the cause, and how it can be fixed?

@ericstj I saw your PR here: #97454 and thought it might be related. I ran gave your branch to our CI, but unfortunately it did not solve the issue.

cc @omajid

Author:	tmds
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[hoyosjs]

@ViktorHofer was removing the unit packref intentional?

[ericstj]

I'd say this is definitely caused by #96826 which my PR is reverting. In that PR we intentionally try to avoid copying Xunit.Console to the output unless we're sending tests to helix. Copying to the output was interfering with test discovery by dotnet test. We found later that multiple parts of the build are expecting xunit.console to be in the output and those need to be cleaned up.

It's possible that the arcade update PR is also causing this, but I don't see how.

[ViktorHofer]

@ViktorHofer was removing the unit packref intentional?

Yes. We don't reference the meta package anymore as we need to control the xunit analyzer version separately. Arcade has been doing this for a while now: https://github.com/dotnet/arcade/blob/72f32d7d6880a97b457553ff601c383aaf69c902/src/Microsoft.DotNet.Arcade.Sdk/tools/XUnit/XUnit.targets#L9-L13

[hoyosjs]

Yes, but then the runner targets bring in xunit.runner.console

[tmds]

I'd say this is definitely caused by #96826 which my PR is reverting.

This was also my first suspect, but after doing some builds I suspect the Arcade PR instead.

It's possible that the arcade update PR is also causing this, but I don't see how.

The Arcade PR went in before your PR, and this what is in my build log for it:

...
++ git rev-parse HEAD
https://github.com/dotnet/runtime is at 8726d6b940bb0cfecbcade46159a1634137d7c18
...
  The application to execute does not exist: 'xunit.console.dll'
...
  The application to execute does not exist: 'xunit.console.dll'
...

[ViktorHofer]

@tmds do you have a binlog that I could look at?

[tmds]

I tried to reproduce the issue on my machine, and found out it only happens when DotNetBuildFromSource is set to true.

I use the following command to reproduce it:

./build.sh --subset libs.tests+clr+libs+host --test /p:NoPgoOptimize=true /p:DotNetBuildFromSource=true --portablebuild false -rc Release -bl

do you have a binlog that I could look at?

I'll send you two via mail (one with and one without DotNetBuildFromSource) built from 8726d6b.

[ViktorHofer]

Tom and I synced offline and we found the reason. I submitted dotnet/arcade#14413 to fix this. See my detailed commit message in there.

Additionally I submitted dotnet/arcade#14414 to make it possible to disable the entire ExcludeFromBuild Arcade infrastructure as in runtime we use a different (better) mechanism to filter out projects. After this PR is in, we want to set DisableArcadeExcludeFromBuildSupport=true in our Versions.props file.