Exceptions in Roslyn.Builder

[AndreyAkinshin]

I have some troubles with local run of our unit tests. I have about 82 failed tests, here is a typical example (for BenchmarksThatUseTypeThatRequiresDifferentRuntime with latest changes from the master branch):

BenchmarkDotNet.IntegrationTests.Classic.ReferencesTests.BenchmarksThatUseTypeThatRequiresDifferentRuntimeAreSupported

Xunit.Sdk.TrueException
There are no available measurements
Expected: True
Actual:   False
   at BenchmarkDotNet.IntegrationTests.Classic.BenchmarkTestRunner.CanCompileAndRun[TBenchmark](ITestOutputHelper output) in W:\Work\BenchmarkDotNet\tests\BenchmarkDotNet.IntegrationTests.Classic\BenchmarkTestRunner.cs:line 20
   at BenchmarkDotNet.IntegrationTests.Classic.ReferencesTests.BenchmarksThatUseTypeThatRequiresDifferentRuntimeAreSupported() in W:\Work\BenchmarkDotNet\tests\BenchmarkDotNet.IntegrationTests.Classic\ReferencesTests.cs:line 29



// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   BenchmarksThatUseTypeThatRequiresDifferentRuntime.Benchmark: Dry(LaunchCount=1, RunStrategy=ColdStart, TargetCount=1, UnrollFactor=1, WarmupCount=1)

// Validating benchmarks:
// **************************
// Benchmark: BenchmarksThatUseTypeThatRequiresDifferentRuntime.Benchmark: Dry(LaunchCount=1, RunStrategy=ColdStart, TargetCount=1, UnrollFactor=1, WarmupCount=1)
// *** Generate *** 
// Result = Success
// BinariesDirectoryPath = W:\Work\BenchmarkDotNet\tests\BenchmarkDotNet.IntegrationTests.Classic\bin\Release

// *** Build ***
BuildScript: W:\Work\BenchmarkDotNet\tests\BenchmarkDotNet.IntegrationTests.Classic\bin\Release\BDN.Generated.bat
// Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.IO.FileSystem, Version=4.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.IO.FileSystem, Version=4.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at Roslyn.Utilities.FileUtilities.OpenFileStream(String path)
   at Microsoft.CodeAnalysis.AssemblyMetadata.CreateFromFile(String path)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at BenchmarkDotNet.Toolchains.Roslyn.Builder.Build(GenerateResult generateResult, ILogger logger, Benchmark benchmark, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Build(ILogger logger, IToolchain toolchain, GenerateResult generateResult, Benchmark benchmark, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

It seems that we have a problem here:

BenchmarkDotNet/src/BenchmarkDotNet.Toolchains.Roslyn/Builder.cs

Line 34 in f5d88e2

.Select(assembly => AssemblyMetadata.CreateFromFile(assembly.Location))

@adamsitnik, could you take a look? Also, I think that we need better logging here because it's hard to understand what's going on here.

[adamsitnik]

@AndreyAkinshin It's most probably due to the fact that I don't copy all required dlls when running runClassicTests.bat. I will take a look into it during the weekend

[adamsitnik]

Ok @AndreyAkinshin it wasn't very pleasant to do, but I have managed to get it working again ;)

I decided to move all tests to .IntegrationTests. I have ported old csprojs to the new format. Now we have only modern projects, except F# and VB which are not supported by the new VS 2017 yet ;(

I removed .DifferentRuntime tests because they don't make any sense today. When we were supporting .NET 4.5 it was good to check if things work for .NET 4.6. Now we are 4.6 only.
I had to comment F# and VB tests because when I reference them from new csproj I can build the solution in VS, but not from console. And we run tests from console, so they were failing to build. As soon as VS fixes this I am going to bring these 2 tests back to live (and our F# .NET Core project too)

[AndreyAkinshin]

@adamsitnik, thank you so much for doing it, it's awesome!
However, I still have failed tests. Could you take a look at my logs?
runClassicTests.txt
runCoreTests.txt

[adamsitnik]

@AndreyAkinshin hmm it's strange because all your projects output is _out folder.

For .NET Core I can see that build succeeds, but the dll is not there. I guess that it's in some runtime-specific folder.

Could you apply [KeepBenchmarkFiles] to one of the tests, let's say BenchmarkDotNet.IntegrationTests.ProcessPropertiesTests.HighPriorityIsSet and run it from console via dotnet test tests\BenchmarkDotNet.IntegrationTests\BenchmarkDotNet.IntegrationTests.csproj -f netcoreapp1.1 -c Release --filter "FullyQualifiedName=BenchmarkDotNet.IntegrationTests.ProcessPropertiesTests.HighPriorityIsSet" and tell me what is in your output folder?

[AndreyAkinshin]

HighPriorityIsSet.txt

[adamsitnik]

@AndreyAkinshin it looks like you did not use [KeepBenchmarkFiles]

[AndreyAkinshin]

Hooray, I solved the problem! My dotnet --version was 1.0.0, but I had a lot of preview versions in C:\Program Files\dotnet\sdk\. I removed it, now all my tests are passed. It's so hard to support bleeding edge technologies. =(
@adamsitnik, do you have any issues that you want to fix in 0.10.4? I want to carefully check current version in all my environments and (if everything is fine) publish the new version.

[adamsitnik]

@AndreyAkinshin Yesterday I fixed all the little problems I knew. My last TODO are the docs for Hardware Counters ;)