OOME new in Deno 1.44

[nleck]

Version: Deno 1.44.0

Applications that have been running on 1.43.x now crashes consistently with OOME.

Code to reproduce

<--- Last few GCs --->

[2262:0x140040000] 2931 ms: Mark-Compact (reduce) 1398.8 (1400.6) -> 1398.8 (1401.1) MB, pooled: 0 MB, 71.67 / 0.00 ms (+ 0.5 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 96 ms) (average mu = 0.772, curre[2262:0x140040000] 3040 ms: Mark-Compact 1399.8 (1401.1) -> 1399.8 (1405.1) MB, pooled: 0 MB, 106.62 / 0.00 ms (average mu = 0.533, current mu = 0.020) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

Fatal JavaScript out of memory: Reached heap limit

==== C stack trace ===============================

0   deno                                0x0000000103efdf88 v8::base::debug::StackTrace::StackTrace() + 24
1   deno                                0x0000000103f037d0 v8::platform::(anonymous namespace)::PrintStackTrace() + 24
2   deno                                0x0000000103efb070 v8::base::FatalOOM(v8::base::OOMType, char const*) + 68
3   deno                                0x0000000103f49964 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) + 628
4   deno                                0x000000010410b404 v8::internal::Heap::stack() + 0
5   deno                                0x0000000104109928 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) + 920
6   deno                                0x00000001040d4448 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 1888
7   deno                                0x00000001040d4cf4 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 52
8   deno                                0x00000001040bad70 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) + 440
9   deno                                0x00000001040b0b68 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArray(int, v8::internal::AllocationType) + 72
10  deno                                0x00000001040c70a0 v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllocationMode, v8::internal::AllocationType) + 332
11  deno                                0x00000001042017a8 v8::internal::JsonParser<unsigned short>::BuildJsonArray(unsigned long) + 164
12  deno                                0x0000000104200468 v8::internal::JsonParser<unsigned short>::ParseJsonArray() + 1932
13  deno                                0x00000001041ff8dc v8::internal::JsonParser<unsigned short>::ParseJsonObject(v8::internal::Handle<v8::internal::Map>) + 908
14  deno                                0x00000001041ff884 v8::internal::JsonParser<unsigned short>::ParseJsonObject(v8::internal::Handle<v8::internal::Map>) + 820
15  deno                                0x00000001041ff884 v8::internal::JsonParser<unsigned short>::ParseJsonObject(v8::internal::Handle<v8::internal::Map>) + 820
16  deno                                0x00000001041fc9a8 v8::internal::JsonParser<unsigned short>::ParseJson(v8::internal::Handle<v8::internal::Object>) + 536
17  deno                                0x00000001041fc684 v8::internal::JsonParser<unsigned short>::Parse(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::Object>) + 124
18  deno                                0x0000000103f7c050 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) + 288
19  deno                                0x0000000105199754 Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit + 84
20  deno                                0x00000001050fda90 Builtins_InterpreterEntryTrampoline + 304
21  deno                                0x000000010513cb04 Builtins_AsyncFunctionAwaitResolveClosure + 68
22  deno                                0x0000000105210738 Builtins_PromiseFulfillReactionJob + 56
23  deno                                0x000000010512b894 Builtins_RunMicrotasks + 628
24  deno                                0x00000001050fb514 Builtins_JSRunMicrotasksEntry + 180
25  deno                                0x000000010404b038 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 1764
26  deno                                0x000000010404b8f8 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 88
27  deno                                0x000000010404ba44 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) + 60
28  deno                                0x0000000104091b74 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) + 360
29  deno                                0x00000001040919e0 v8::internal::MicrotaskQueue::PerformCheckpointInternal(v8::Isolate*) + 88
30  deno                                0x0000000102dc8670 deno_core::ops_builtin_v8::op_run_microtasks::op_run_microtasks::v8_fn_ptr::h883981ff19d737f7 + 40
31  deno                                0x00000001050ffc3c Builtins_CallApiCallbackGeneric + 188
32  ???                                 0x000000016ed00fa4 0x0 + 6154096548
33  deno                                0x00000001050fb648 Builtins_JSEntryTrampoline + 168
34  deno                                0x00000001050fb294 Builtins_JSEntry + 180
35  deno                                0x000000010404b078 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 1828
36  deno                                0x000000010404a924 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 140
37  deno                                0x0000000103f57b70 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 532
38  deno                                0x0000000102df71bc deno_core::runtime::jsruntime::JsRuntime::do_js_event_loop_tick_realm::h8e60c313da60fd1e + 8852
39  deno                                0x0000000102df1a8c deno_core::runtime::jsruntime::JsRuntime::poll_event_loop::heafc8bf4f70f6828 + 1492
40  deno                                0x000000010298f084 _ZN9deno_core7runtime9jsruntime9JsRuntime14run_event_loop28_$u7b$$u7b$closure$u7d$$u7d$17hac84394b34a5e36fE + 100
41  deno                                0x000000010280fddc _ZN12deno_runtime6worker10MainWorker14run_event_loop28_$u7b$$u7b$closure$u7d$$u7d$17ha2fb5faf2fb36603E + 80
42  deno                                0x0000000102b1fa28 _ZN4deno6worker13CliMainWorker33evaluate_module_possibly_with_npm28_$u7b$$u7b$closure$u7d$$u7d$17h0e713b5376988de0E + 476
43  deno                                0x0000000102b1ebd8 _ZN4deno6worker13CliMainWorker37execute_main_module_possibly_with_npm28_$u7b$$u7b$closure$u7d$$u7d$17h8fca0b24ac70f98aE + 368
44  deno                                0x0000000102b1e178 _ZN4deno6worker13CliMainWorker3run28_$u7b$$u7b$closure$u7d$$u7d$17h9dbcee303be3cd75E + 1160
45  deno                                0x0000000102aeccb8 _ZN4deno5tools3run10run_script28_$u7b$$u7b$closure$u7d$$u7d$17h7fa9c49ceed51745E + 4964
46  deno                                0x0000000102b324cc _ZN4deno16spawn_subcommand28_$u7b$$u7b$closure$u7d$$u7d$17hbada03c742427b80E + 524
47  deno                                0x00000001029759ec _ZN93_$LT$deno_unsync..task..MaskFutureAsSend$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$4poll17h89f96388045c74d2E + 40
48  deno                                0x00000001028e6230 tokio::runtime::task::raw::poll::hc17266e3a0356e1a + 84
49  deno                                0x0000000102b3f31c deno::main::hf77d18a9b576d6fd + 4636
50  deno                                0x000000010284089c std::sys_common::backtrace::__rust_begin_short_backtrace::h4e9d3f0b60c8b04c + 12
51  deno                                0x0000000102ba3dac main + 668
52  dyld                                0x000000019b7d20e0 start + 2360

[lucacasonato]

Can you share the program that is crashing?

[nleck]

Sure, here is the bash script and program. This is runs on MacOS M2 with 64BG. Has been running well for all versions up to 1.44

OOME.zip

[nleck]

Can you share the crashing program?

There is a lot of trading data that I can't share. The problem arises when there is heavy usage on MacOS.

[lucacasonato]

Without a reproduction we can run, we can unfortunately not fix this or determine what the root cause is.

When running in versions prior to 1.44, was this program consuming more than 4GB of RAM?

[nleck]

Without a reproduction we can run, we can unfortunately not fix this or determine what the root cause is.

When running in versions prior to 1.44, was this program consuming more than 4GB of RAM?

Yes, it's configured to use 8gb and have 24 workers.

I'll see if I can make a shareable example. The OOME occurs very quickly, hopefully I can create a simple example

[nleck]

Can we have a flag to prevent point compression? Large JSON files in workers no longer work since v1.44 ( I'm stuck on 1.43.6)

[nleck]

Deno version v1.45.5 fails with OOME, and all versions before 1.44 seem to work OK. Please see attached

[Uploading bug-24057v2.zip…]()


#
# Fatal JavaScript out of memory: MarkCompactCollector: young object promotion failed
#
==== C stack trace ===============================


<--- Last few GCs --->

[42428:0x140120000]      286 ms: Scavenge 208.4 (219.9) -> 208.4 (219.9) MB, pooled: 0 MB, 0.62 / 0.00 ms  (average mu = 1.000, current mu = 1.000) task; 
[42428:0x140120000]      300 ms: Scavenge 208.4 (219.9) -> 208.4 (222.9) MB, pooled: 0 MB, 13.12 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure; 
[42428:0x140120000]      313 ms: Scavenge 208.4 (222.9) -> 208.4 (222.9) MB, pooled: 0 MB, 12.83 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure; 


<--- JS stacktrace --->



#
# Fatal JavaScript out of memory: MarkCompactCollector: young object promotion failed
#
==== C stack trace ===============================

    0   deno                                0x0000000105967ecc v8::base::debug::StackTrace::StackTrace() + 24    
0   deno                                0x0000000105967ecc v8::base::debug::StackTrace::StackTrace() + 24
            1   deno                                0x000000010596d714 v8::platform::(anonymous namespace)::PrintStackTrace() + 240   deno                                0x0000000105967ecc v8::base::debug::StackTrace::StackTrace() + 24

1   deno                                0x000000010596d714 v8::platform::(anonymous namespace)::PrintStackTrace() + 24    2   deno                                0x0000000105964f84 v8::base::FatalOOM(v8::base::OOMType, char const*) + 68

    2   deno                                0x0000000105964f84 v8::base::FatalOOM(v8::base::OOMType, char const*) + 68
    3   deno                                0x00000001059b3a40 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) + 628    3   deno                                0x00000001059b3a40 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) + 628    

1   deno                                0x000000010596d714 v8::platform::(anonymous namespace)::PrintStackTrace() + 24
        4   deno                                0x0000000105b7a284 v8::internal::Heap::stack() + 04   deno                                0x0000000105b7a284 v8::internal::Heap::stack() + 0    

2   deno                                0x0000000105964f84 v8::base::FatalOOM(v8::base::OOMType, char const*) + 68
    3   deno                                0x00000001059b3a40 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) + 628        
5   deno                                0x0000000105bab374 void v8::internal::LiveObjectVisitor::VisitMarkedObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor>(v8::internal::PageMetadata*, v8::internal::EvacuateNewSpaceVisitor*) + 10485   deno                                0x0000000105bab374 void v8::internal::LiveObjectVisitor::VisitMarkedObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor>(v8::internal::PageMetadata*, v8::internal::EvacuateNewSpaceVisitor*) + 1048

        4   deno                                0x0000000105b7a284 v8::internal::Heap::stack() + 0
    6   deno                                0x0000000105baac98 v8::internal::Evacuator::RawEvacuatePage(v8::internal::MutablePageMetadata*) + 384
    7   deno                                0x0000000105baa9f0 v8::internal::Evacuator::EvacuatePage(v8::internal::MutablePageMetadata*) + 328    5   deno                                0x0000000105bab374 void v8::internal::LiveObjectVisitor::VisitMarkedObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor>(v8::internal::PageMetadata*, v8::internal::EvacuateNewSpaceVisitor*) + 10486   deno                                0x0000000105baac98 v8::internal::Evacuator::RawEvacuatePage(v8::internal::MutablePageMetadata*) + 384

        6   deno                                0x0000000105baac98 v8::internal::Evacuator::RawEvacuatePage(v8::internal::MutablePageMetadata*) + 3847   deno                                0x0000000105baa9f0 v8::internal::Evacuator::EvacuatePage(v8::internal::MutablePageMetadata*) + 328


    8   deno                                0x0000000105bb9fb8 v8::internal::PageEvacuationJob::Run(v8::JobDelegate*) + 844    
8   deno                                0x0000000105bb9fb8 v8::internal::PageEvacuationJob::Run(v8::JobDelegate*) + 844        
7   deno                                0x0000000105baa9f0 v8::internal::Evacuator::EvacuatePage(v8::internal::MutablePageMetadata*) + 3289   deno                                0x0000000105969634 v8::platform::DefaultJobWorker::Run() + 216
    
9   deno                                0x0000000105969634 v8::platform::DefaultJobWorker::Run() + 216    
    8   deno                                0x0000000105bb9fb8 v8::internal::PageEvacuationJob::Run(v8::JobDelegate*) + 84410  deno                                0x000000010596ebfc v8::platform::DefaultWorkerThreadsTaskRunner::WorkerThread::Run() + 160

        9   deno                                0x0000000105969634 v8::platform::DefaultJobWorker::Run() + 21611  deno                                0x0000000105967d6c v8::base::ThreadEntry(void*) + 160

        12  libsystem_pthread.dylib             0x0000000187d15f94 _pthread_start + 13610  deno                                0x000000010596ebfc v8::platform::DefaultWorkerThreadsTaskRunner::WorkerThread::Run() + 160

    13  libsystem_pthread.dylib             0x0000000187d10d34 thread_start + 8
        10  deno                                0x000000010596ebfc v8::platform::DefaultWorkerThreadsTaskRunner::WorkerThread::Run() + 16011  deno                                0x0000000105967d6c v8::base::ThreadEntry(void*) + 160

    12  libsystem_pthread.dylib             0x0000000187d15f94 _pthread_start + 136
        11  deno                                0x0000000105967d6c v8::base::ThreadEntry(void*) + 16013  libsystem_pthread.dylib             0x0000000187d10d34 thread_start + 8

    12  libsystem_pthread.dylib             0x0000000187d15f94 _pthread_start + 136
    13  libsystem_pthread.dylib             0x0000000187d10d34 thread_start + 8
run.sh: line 9: 42428 Trace/BPT trap: 5       deno run --v8-flags=--max-old-space-size=8192 --allow-read --allow-write --allow-net src/Process.ts

[bartlomieju]

Can we have a flag to prevent point compression? Large JSON files in workers no longer work since v1.44 ( I'm stuck on 1.43.6)

No, pointer compressions is a build option, it can't be configured during runtime. Can you provide a reproduction code that shows the problem? It's very strange that V8 would crash with only ~230Mb heap, are you running in a memory constrained environment? Have you tried tweaking the heap size using --v8-flags=--max_old_space_size flag?

[nleck]

Can we have a flag to prevent point compression? Large JSON files in workers no longer work since v1.44 ( I'm stuck on 1.43.6)

No, pointer compressions is a build option, it can't be configured during runtime. Can you provide a reproduction code that shows the problem? It's very strange that V8 would crash with only ~230Mb heap, are you running in a memory constrained environment? Have you tried tweaking the heap size using --v8-flags=--max_old_space_size flag?

An example of the crashing code was uploaded in the previous comment. It's a cut down of the production code and simulates the generation of the market data.

In the run.sh file in the above zip file you can see that I've set the max heap to 8gb ( the maximum that I could set )

See the
"Uploading bug-24057v2.zip…" link

[nleck]

Are the worker processors not getting the v8 flags anymore? I'm getting a lot of OOME issues wherever I've used workers too many to work around so I've locked the version of demo to the last working version.

[nleck]

@lucacasonato > Code to reproduce
I've created a cut-down version to reproduce the issue. Code to reproduce. Can we remove the label "needs info"?

[marvinhagemeister]

Can confirm, with the provided reproduction I get the described OOM in >= 1.45.5 and none in 1.43.6.

[bartlomieju]

@devsnek is currently investigating this issue, we might have to reach out to V8 team to better understand the limitations here.

[devsnek]

@nleck in the short term if you want to, could you try building deno from source with these env vars and see if it fixes your issue? it will probably take 20-40m to build.

V8_FROM_SOURCE=1
GN_ARGS="v8_enable_pointer_compression_shared_cage=false"

[nleck]

FYI I Just tried v1.46.1, and it gets the same error.

[nleck]

The problem seems to have been fixed in 1.46.2 :-)

Thank you so much for fixing this; I appreciate it.