System.Text.Json.SourceGeneration.Tests.MetadataWithPerTypeAttributeContextTests.RoundTripCollectionsDictionary failing with Actual: 0001-01-01T00:00:00.0000000+00:00

[radical]

Build for #67944:

    System.Text.Json.SourceGeneration.Tests.MetadataWithPerTypeAttributeContextTests.RoundTripCollectionsDictionary [FAIL]
      Assert.Equal() Failure
      Expected: 2019-08-01T00:00:00.0000000-07:00
      Actual:   0001-01-01T00:00:00.0000000+00:00
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs(530,0): at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.VerifyWeatherForecastWithPOCOs(WeatherForecastWithPOCOs expected, WeatherForecastWithPOCOs obj)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs(89,0): at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.RoundTripCollectionsDictionary()
        /_/src/mono/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.Mono.cs(397,0): at System.Reflection.RuntimeMethodInfo.InvokeNonEmitUnsafe(Object obj, IntPtr* byrefParameters, Span`1 argsForTemporaryMonoSupport, BindingFlags invokeAttr)

Runfo Tracking Issue: system.text.json.sourcegeneration.tests.mixedmodecontexttests.roundtripcollectionsdictionary

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1738148	runtime	Rolling	net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3	console.log			runclient.py

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build for #67944:

    System.Text.Json.SourceGeneration.Tests.MetadataWithPerTypeAttributeContextTests.RoundTripCollectionsDictionary [FAIL]
      Assert.Equal() Failure
      Expected: 2019-08-01T00:00:00.0000000-07:00
      Actual:   0001-01-01T00:00:00.0000000+00:00
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs(530,0): at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.VerifyWeatherForecastWithPOCOs(WeatherForecastWithPOCOs expected, WeatherForecastWithPOCOs obj)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs(89,0): at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.RoundTripCollectionsDictionary()
        /_/src/mono/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.Mono.cs(397,0): at System.Reflection.RuntimeMethodInfo.InvokeNonEmitUnsafe(Object obj, IntPtr* byrefParameters, Span`1 argsForTemporaryMonoSupport, BindingFlags invokeAttr)

Author:	radical
Assignees:	-
Labels:	area-System.Text.Json, blocking-clean-ci
Milestone:	-

[danmoseley]

Not Mono specific, not OS specific, failing on 1x per day.

TestResults
| join kind=inner WorkItems on WorkItemId
| join kind=inner Jobs on JobId
| where Type == "System.Text.Json.SourceGeneration.Tests.MetadataWithPerTypeAttributeContextTests" and Method == "RoundTripCollectionsDictionary"
| where Result == "Fail"
| where Message contains "2019-08"
| project //Type, 
Method,
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  Pipeline_Configuration = tostring(parse_json(Properties).configuration),
  OS = QueueName,
  Container = tostring(parse_json(Properties).operatingSystem), 
  Arch = tostring(parse_json(Properties).architecture),
 Finished,
 Build
  //Duration,
 // Method,
 // Build = tostring(parse_json(Properties).BuildNumber),
,Message//,StackTrace

Method	Pipeline	Pipeline_Configuration	OS	Container	Arch	Finished	Build	Message
RoundTripCollectionsDictionary	runtime-extra-platforms	Release	windows.10.arm64.open	Windows.10.Arm64.Open	arm64	2022-04-10 10:36:11.0780000		Assert.Equal() Failure Expected: 2019-08-01T00:00:00.0000000-07:00 Actual: 0001-01-01T00:00:00.0000000+00:00
RoundTripCollectionsDictionary	runtime	Debug	windows.10.amd64.open.rt	Windows.10.Amd64.Open	x64	2022-04-11 22:03:59.8120000		Assert.Equal() Failure Expected: 2019-08-01T07:00:00.0000000+00:00 Actual: 0001-01-01T00:00:00.0000000+00:00
RoundTripCollectionsDictionary	runtime-extra-platforms	Release	windows.10.arm64.open	Windows.10.Arm64.Open	arm64	2022-04-12 00:08:41.2910000		Assert.Equal() Failure Expected: 2019-08-01T00:00:00.0000000-07:00 Actual: 0001-01-01T00:00:00.0000000+00:00
RoundTripCollectionsDictionary	runtime-extra-platforms	Release	windows.10.arm64.open	Windows.10.Arm64.Open	arm64	2022-04-12 11:16:05.8730000		Assert.Equal() Failure Expected: 2019-08-01T00:00:00.0000000-07:00 Actual: 0001-01-01T00:00:00.0000000+00:00
RoundTripCollectionsDictionary	runtime	Debug	osx.1200.amd64.open	OSX.1200.Amd64.Open	x64	2022-04-13 03:13:48.9760000		Assert.Equal() Failure Expected: 2019-08-01T00:00:00.0000000-07:00 Actual: 0001-01-01T00:00:00.0000000+00:00
RoundTripCollectionsDictionary	runtime	Debug	osx.1200.amd64.open	OSX.1200.Amd64.Open	x64	2022-04-14 14:30:14.1650000		Assert.Equal() Failure Expected: 2019-08-01T00:00:00.0000000-07:00 Actual: 0001-01-01T00:00:00.0000000+00:00
RoundTripCollectionsDictionary	runtime-extra-platforms	Release	windows.10.arm64.open	Windows.10.Arm64.Open	arm64	2022-04-15 19:29:50.5140000		Assert.Equal() Failure Expected: 2019-08-01T00:00:00.0000000-07:00 Actual: 0001-01-01T00:00:00.0000000+00:00
RoundTripCollectionsDictionary	runtime-extra-platforms	Release	windows.10.arm64.open	Windows.10.Arm64.Open	arm64	2022-04-18 10:47:09.4710000		Assert.Equal() Failure Expected: 2019-08-01T00:00:00.0000000-07:00 Actual: 0001-01-01T00:00:00.0000000+00:00

[VincentBu]

Failed again in: runtime-coreclr libraries-jitstress2-jitstressregs 20220423.1

Failed test:

net7.0-windows-Release-arm64-CoreCLR_checked-jitstress2_jitstressregs2-Windows.10.Arm64v8.Open

- System.Text.Json.SourceGeneration.Tests.MetadataWithPerTypeAttributeContextTests.RoundTripCollectionsDictionary

Error message:

Assert.Equal() Failure
Expected: 2019-08-01T00:00:00.0000000-07:00
Actual:   0001-01-01T00:00:00.0000000+00:00


Stack trace
   at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.VerifyWeatherForecastWithPOCOs(WeatherForecastWithPOCOs expected, WeatherForecastWithPOCOs obj) in /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs:line 564
   at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.RoundTripCollectionsDictionary() in /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs:line 89
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.RuntimeMethodInfo.InvokeNonEmitUnsafe(Object obj, IntPtr* arguments, Span`1 argsForTemporaryMonoSupport, BindingFlags invokeAttr) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs:line 383

[eiriktsarpalis]

@layomia could you take a look?

[BruceForstall]

In case it's related, there's another test in this assembly that has failed recently exactly once in the CI, on win-x86, in a jitstress job, but I can't repro the failure locally:

https://dev.azure.com/dnceng/public/_build/results?buildId=1735202&view=ms.vss-test-web.build-test-results-tab&runId=46974758&resultId=178905&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

C:\h\w\BD700A6B\w\A4B20982\e>set COMPlus 
COMPlus_JitStressRegs=1
COMPlus_TieredCompilation=0

C:\h\w\BD700A6B\w\A4B20982\e>call RunTests.cmd --runtime-path C:\h\w\BD700A6B\p 
----- start Sun 04/24/2022 10:14:53.03 ===============  To repro directly: ===================================================== 
pushd C:\h\w\BD700A6B\w\A4B20982\e\
"C:\h\w\BD700A6B\p\dotnet.exe" exec --runtimeconfig System.Text.Json.SourceGeneration.Roslyn3.11.Tests.runtimeconfig.json --depsfile System.Text.Json.SourceGeneration.Roslyn3.11.Tests.deps.json xunit.console.dll System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\BD700A6B\w\A4B20982\e>"C:\h\w\BD700A6B\p\dotnet.exe" exec --runtimeconfig System.Text.Json.SourceGeneration.Roslyn3.11.Tests.runtimeconfig.json --depsfile System.Text.Json.SourceGeneration.Roslyn3.11.Tests.deps.json xunit.console.dll System.Text.Json.SourceGeneration.Roslyn3.11.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Text.Json.SourceGeneration.Roslyn3.11.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.SourceGeneration.Roslyn3.11.Tests (found 1845 of 1972 test cases)
  Starting:    System.Text.Json.SourceGeneration.Roslyn3.11.Tests (parallel test collections = on, max threads = 4)
    System.Text.Json.SourceGeneration.Tests.SerializationWithPerTypeAttributeContextTests.RoundTripLocation [FAIL]
      Assert.Equal() Failure
      Expected: abc-12
      Actual:   (null)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs(395,0): at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.VerifyLocation(Location expected, Location obj)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/SerializationContextTests.cs(211,0): at System.Text.Json.SourceGeneration.Tests.SerializationContextTests.RoundTripLocation()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs(383,0): at System.Reflection.RuntimeMethodInfo.InvokeNonEmitUnsafe(Object obj, IntPtr* arguments, Span`1 argsForTemporaryMonoSupport, BindingFlags invokeAttr)
  Finished:    System.Text.Json.SourceGeneration.Roslyn3.11.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.Json.SourceGeneration.Roslyn3.11.Tests  Total: 3070, Errors: 0, Failed: 1, Skipped: 0, Time: 133.725s
----- end Sun 04/24/2022 10:17:26.98 ----- exit code 1 ----------------------------------------------------------

[mangod9]

We hit another similar failure in rolling build today: https://dev.azure.com/dnceng/public/_build/results?buildId=1738148&view=logs&j=ee1d0ee6-ff1e-56e7-4a8c-87d1cd9cc235&t=6efd574a-0ddf-56af-d302-e49d8731e67b

      System.ArgumentNullException : Value cannot be null. (Parameter 'source')
      Stack Trace:
        /_/src/libraries/System.Linq/src/System/Linq/ThrowHelper.cs(12,0): at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
        /_/src/libraries/System.Linq/src/System/Linq/OrderedEnumerable.cs(121,0): at System.Linq.OrderedEnumerable`2..ctor(IEnumerable`1 source, Func`2 keySelector, IComparer`1 comparer, Boolean descending, OrderedEnumerable`1 parent)
        /_/src/libraries/System.Linq/src/System/Linq/OrderBy.cs(11,0): at System.Linq.Enumerable.OrderBy[TSource,TKey](IEnumerable`1 source, Func`2 keySelector)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/RealWorldContextTests.cs(573,0): at System.Text.Json.SourceGeneration.Tests.RealWorldContextTests.VerifyWeatherForecastWithPOCOs(WeatherForecastWithPOCOs expected, WeatherForecastWithPOCOs obj)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/MixedModeContextTests.cs(142,0): at System.Text.Json.SourceGeneration.Tests.MixedModeContextTests.RoundTripCollectionsDictionary()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs(387,0): at System.Reflection.RuntimeMethodInfo.InvokeNonEmitUnsafe(Object obj, IntPtr* arguments, Span`1 argsForTemporaryMonoSupport, BindingFlags invokeAttr)

[krwq]

I'm curious if we've seen any recent instances as this might be already fixed. I've made a fix on April 28 (7654bf2) which fixed many concurrency issues in System.Text.Json. The last reported hit is on April 27 and it seems to be silent since then...

[krwq]

Actually considering monthly hits shows as zero seems the answer is this might be already fixed. I'll close this issue. Please re-open if you see any recent instances.