tests fail intermittently when run in parallell

[krijohan]

Description

When running dotnet test in a project with [assembly: Parallelize(Workers = 4, Scope = ExecutionScope.ClassLevel)], you will occasionally get unwarranted test failures from InvalidOperationException / "Collection was modified; enumeration operation may not execute". This makes the feature of running tests in parallell not very useful, unless you are willing to accept flaky tests.

While not clear from the stack trace in the resulting test failure, debugging indicates that the exception originates from https://github.com/microsoft/testfx/blob/rel/2.2.8/src/Adapter/MSTest.CoreAdapter/Execution/StackTraceHelper.cs#L239, where there is a foreach on a static List<string> member.

Steps to reproduce

1. Clone https://github.com/krijohan/BugRepro1/ (2 test classes, with 1 always passing and 1 always skipped test in each).
2. Run dotnet test over and over again until it fails, e.g. by running the included script repro.ps1 in powershell. On my laptop this takes about 5 minutes.

Expected behavior

dotnet test never fails (as the included test cases are always passed or skipped)

Actual behavior

Eventually, one of the skipped test cases will fail with

  Failed AlwaysSkip
  Error Message:
   Exception thrown while executing test. If using extension of TestMethodAttribute then please contact vendor. Error message: Collection was modified; enumeration operation may not execute., Stack trace:    at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.ExecuteInternal(Object[] arguments)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.Invoke(Object[] arguments)
   at Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute.Execute(ITestMethod testMethod)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.ExecuteTest(TestMethodInfo testMethodInfo)

The error message hides the real issue. Attaching a debugger to dotnet test and adding a breakpoint on InvalidOperationException , the below stack trace suggests the exception originates from https://github.com/microsoft/testfx/blob/rel/2.2.8/src/Adapter/MSTest.CoreAdapter/Execution/StackTraceHelper.cs#L239, where there is a foreach on a static List<string> member.

Debugger stack trace:

        System.Private.CoreLib.dll!System.ThrowHelper.ThrowInvalidOperationException_InvalidOperation_EnumFailedVersion() Line 432    C#
>       Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.StackTraceHelper.HasReferenceToUTF(string stackFrame) Line 142     C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.StackTraceHelper.TrimStackTrace(string stackTrace) Line 81 C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.StackTraceHelper.CreateStackTraceInformation(System.Exception ex, bool checkInnerExceptions, string stackTraceString) Line 127     C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Extensions.ExceptionExtensions.TryGetStackTraceInformation(System.Exception exception) Line 35       C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Extensions.ExceptionExtensions.TryGetUnitTestAssertException(System.Exception exception, out Microsoft.VisualStudio.TestTools.UnitTesting.UnitTestOutcome outcome, out string exceptionMessage, out Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.StackTraceInformation exceptionStackTrace) Line 44  C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.HandleMethodException(System.Exception ex, string className, string methodName) Line 344    C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.ExecuteInternal(object[] arguments) Line 265        C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo.Invoke(object[] arguments) Line 133 C#
        Microsoft.VisualStudio.TestPlatform.TestFramework.dll!Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute.Execute(Microsoft.VisualStudio.TestTools.UnitTesting.ITestMethod testMethod) Line 46    C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.ExecuteTest(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo testMethodInfo) Line 321    C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.RunTestMethod() Line 124  C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.Execute() Line 66 C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner.RunSingleTest(Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.TestMethod testMethod, System.Collections.Generic.IDictionary<string, object> testContextProperties) Line 116     C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTestsWithTestRunner(System.Collections.Generic.IEnumerable<Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase> tests, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutionRecorder testExecutionRecorder, string source, System.Collections.Generic.IDictionary<string, object> sourceLevelParameters, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner testRunner) Line 349 C#
        Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTestsInSource.AnonymousMethod__6() Line 285    C#
        System.Private.CoreLib.dll!System.Threading.Tasks.Task.InnerInvoke() Line 2397  C#
        System.Private.CoreLib.dll!System.Threading.Tasks.Task..cctor.AnonymousMethod__272_0(object obj) Line 2376      C#
        System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 183 C#
        System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot, System.Threading.Thread threadPoolThread) Line 2333     C#
        System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread threadPoolThread) Line 2277 C#
        System.Private.CoreLib.dll!System.Threading.Tasks.ThreadPoolTaskScheduler..cctor.AnonymousMethod__10_0(object s) Line 36      C#
        System.Private.CoreLib.dll!System.Threading.Thread.StartCallback() Line 105     C#

Environment

> dotnet --version
6.0.201

https://github.com/krijohan/BugRepro1/blob/main/BugRepro1/BugRepro1.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <IsPackable>false</IsPackable>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.8" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.8" />
  </ItemGroup>
</Project>

My laptop runs Windows 11 Pro Version 21H2.

[krijohan]

FWIW the same behaviour can be reproduced on a 2.2.9 preview version:

<PackageReference Include="MSTest.TestAdapter" Version="2.2.9-preview-20220210-07" />  
<PackageReference Include="MSTest.TestFramework" Version="2.2.9-preview-20220210-07" />

[krijohan]

A work-around for the issue seems to be to force the initialization of the unprotected collection before tests start running in parallell: https://github.com/krijohan/BugRepro1/pull/1/files#diff-1fb3d50c7b48fae266c651acc227133823af0bf272f14ab3a601dd68121dd84aR12