[wasm] v8 crashing at the end with "Reached heap limit" - for AOT builds

[radical]

Rolling build, and log:

[21:57:26] info: Using random seed for test cases: 1670696355
[21:57:26] info: Using random seed for collections: 1670696355
[21:57:26] info: Starting:    System.Collections.Immutable.Tests.dll

...

[21:58:11] info: [STRT] System.Collections.Frozen.Tests.FrozenSet_Generic_Tests_string_NonDefault.IEnumerable_Generic_Enumerator_MoveNextFalseAfterEndOfCollection
[21:58:11] info: Finished:    System.Collections.Immutable.Tests.dll
[21:58:11] info: 
[21:58:11] info: === TEST EXECUTION SUMMARY ===
[21:58:11] info: Total: 20385, Errors: 0, Failed: 0, Skipped: 50, Time: 44.875336s
[21:58:11] info: 
[21:58:16] info: 
[21:58:16] info: <--- Last few GCs --->
[21:58:16] info: 
[21:58:16] info: [417:0x5634905f12f0]    51715 ms: Mark-Compact 2023.1 (2064.6) -> 2023.1 (2064.6) MB, 763.97 / 0.00 ms  (average mu = 0.356, current mu = 0.089) allocation failure; scavenge might not succeed
[21:58:16] info: [417:0x5634905f12f0]    52663 ms: Mark-Compact 2030.8 (2064.6) -> 2030.8 (2080.3) MB, 942.13 / 0.00 ms  (average mu = 0.189, current mu = 0.007) allocation failure; scavenge might not succeed
[21:58:16] info: 
[21:58:16] info: 
[21:58:16] info: <--- JS stacktrace --->
[21:58:16] info: 
[21:58:16] info: 
[21:58:16] info: #
[21:58:16] info: # Fatal JavaScript out of memory: Reached heap limit
[21:58:16] info: #
[21:58:16] info: 
[21:58:43] info: /home/helixbot/.jsvu/bin/v8: line 2:   417 Trace/breakpoint trap   (core dumped) "/home/helixbot/.jsvu/engines/v8/v8" --snapshot_blob="/home/helixbot/.jsvu/engines/v8/snapshot_blob.bin" "$@"
[21:58:43] info: Process v8 exited with 133

Another instance - build, and log.

This was hit twice in three rolling build runs. Changes from the last passing build before the first hit: 76a8f4f...18321c6 .

cc @vargaz

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorPattern": "Fatal JavaScript out of memory: Reached heap limit",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=331212
Error message validated: Fatal JavaScript out of memory: Reached heap limit
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/6/2023 10:39:28 PM UTC

Report

Build	Definition	Test	Pull Request
334566	dotnet/runtime	normal-System.Text.RegularExpressions.Tests.WorkItemExecution	#88279
331212	dotnet/runtime	normal-System.Collections.Immutable.Tests.WorkItemExecution
329962	dotnet/runtime	normal-System.Collections.Immutable.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	3

[ghost]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Rolling build, and log:

[21:57:26] info: Using random seed for test cases: 1670696355
[21:57:26] info: Using random seed for collections: 1670696355
[21:57:26] info: Starting:    System.Collections.Immutable.Tests.dll

...

[21:58:11] info: [STRT] System.Collections.Frozen.Tests.FrozenSet_Generic_Tests_string_NonDefault.IEnumerable_Generic_Enumerator_MoveNextFalseAfterEndOfCollection
[21:58:11] info: Finished:    System.Collections.Immutable.Tests.dll
[21:58:11] info: 
[21:58:11] info: === TEST EXECUTION SUMMARY ===
[21:58:11] info: Total: 20385, Errors: 0, Failed: 0, Skipped: 50, Time: 44.875336s
[21:58:11] info: 
[21:58:16] info: 
[21:58:16] info: <--- Last few GCs --->
[21:58:16] info: 
[21:58:16] info: [417:0x5634905f12f0]    51715 ms: Mark-Compact 2023.1 (2064.6) -> 2023.1 (2064.6) MB, 763.97 / 0.00 ms  (average mu = 0.356, current mu = 0.089) allocation failure; scavenge might not succeed
[21:58:16] info: [417:0x5634905f12f0]    52663 ms: Mark-Compact 2030.8 (2064.6) -> 2030.8 (2080.3) MB, 942.13 / 0.00 ms  (average mu = 0.189, current mu = 0.007) allocation failure; scavenge might not succeed
[21:58:16] info: 
[21:58:16] info: 
[21:58:16] info: <--- JS stacktrace --->
[21:58:16] info: 
[21:58:16] info: 
[21:58:16] info: #
[21:58:16] info: # Fatal JavaScript out of memory: Reached heap limit
[21:58:16] info: #
[21:58:16] info: 
[21:58:43] info: /home/helixbot/.jsvu/bin/v8: line 2:   417 Trace/breakpoint trap   (core dumped) "/home/helixbot/.jsvu/engines/v8/v8" --snapshot_blob="/home/helixbot/.jsvu/engines/v8/snapshot_blob.bin" "$@"
[21:58:43] info: Process v8 exited with 133

Another instance - build, and log.

This was hit twice in three rolling build runs. Changes from the last passing build before the first hit: 76a8f4f...18321c6 .

cc @vargaz

Author:	radical
Assignees:	-
Labels:	arch-wasm, blocking-clean-ci, area-VM-meta-mono
Milestone:	-

[vargaz]

Can't reproduce with the same seed. Tried setting --max-heap-size= as well, it crashes with 512, but works with 612, so the app uses about 600mb of heap.

[lewing]

Did we disable this test or did it stop happening?

[vargaz]

This might be the same as:
#87078

[radical]

I think these run close to the limit, and sometimes end up exceeding the heap limit, so we don't see them all the time.

[lewing]

I think these run close to the limit, and sometimes end up exceeding the heap limit, so we don't see them all the time.

right but @kg fixed an issue that may have resolved this

[vargaz]

Does this still happen after that fix ?

[radical]

Does this still happen after that fix ?

I have not seen this recently.

[radical]

Closing this. It can be re-opened if it is hit again.