Getting heap allocation error for a large entry size

[azarmadr]

I was trying to create an ArrayKeyMap of size 7,66,000 with each key of type
[int, string, 9 to 29 additional integers]

<--- Last few GCs --->

[31048:0000017180A75FD0]    14792 ms: Scavenge 2016.7 (2056.4) -> 2016.2 (2067.1) MB, 6.3 / 0.0 ms  (average mu = 0.161, current mu = 0.112) a
llocation failure
[31048:0000017180A75FD0]    14805 ms: Scavenge 2023.2 (2067.1) -> 2022.6 (2067.4) MB, 7.4 / 0.0 ms  (average mu = 0.161, current mu = 0.112) a
llocation failure
[31048:0000017180A75FD0]    14822 ms: Scavenge 2024.0 (2067.4) -> 2023.1 (2089.1) MB, 14.9 / 0.0 ms  (average mu = 0.161, current mu = 0.112)
allocation failure


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF60F11E3EF v8::internal::CodeObjectRegistry::~CodeObjectRegistry+111951
 2: 00007FF60F0ADA36 v8::internal::WebSnapshotDeserializer::context_count+65446
 3: 00007FF60F0AE8ED node::OnFatalError+301
 4: 00007FF60F9EA78E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF60F9D540D v8::SharedArrayBuffer::Externalize+781
 6: 00007FF60F857F0C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468
 7: 00007FF60F864AAF v8::internal::Heap::PublishPendingAllocations+1103
 8: 00007FF60F861ABA v8::internal::Heap::PageFlagsAreConsistent+2842
 9: 00007FF60F854B2F v8::internal::Heap::CollectGarbage+1967
10: 00007FF60F85D05B v8::internal::Heap::GlobalSizeOfObjects+315
11: 00007FF60F8A0A8B v8::internal::StackGuard::HandleInterrupts+891
12: 00007FF60F5A72E6 v8::internal::DateCache::Weekday+6902
13: 00007FF60FA72B11 v8::internal::SetupIsolateDelegate::SetupHeap+472849
14: 00007FF60FA4E7E0 v8::internal::SetupIsolateDelegate::SetupHeap+324576
15: 000001718268B110

[anko]

Allocation failed - JavaScript heap out of memory

Hard to say without seeing your code whether this library is responsible for the memory use amplification. Maps seem to be quite memory-hungry though, and this library uses a lot of them, especially if the key paths have lots of long and unique paths, because a new Map is created for every unique path element. That's what you might be seeing here.

It may be possible to improve memory usage by internally splitting path elements onto separate Maps only where they actually fork, and keeping paths in array form wherever they don't. It wouldn't require an API change, but it would complicate the implementation. I may get around to this eventually; I'll keep this issue open as a reminder.

For now, a workaround may be to increase the heap limit, e.g. with node --max-old-space-size=8192 script.js to set it to 8GiB. The default limit depends on Node.js version and processor bitness. On 64-bit Node 16.x it's 4GiB.