getLogs accumulates all results in memory, causing OOM on large block ranges

[Jean-Grimal]

Problem

When calling getLogs over a large block range, viem internally subdivides the range into smaller RPC requests (which is great), but accumulates all parsed results into a single in-memory array before returning. On high-throughput chains like Arbitrum, this causes a JavaScript heap out of memory crash during JSON.parse of the response, even with --max-old-space-size=8192.

For context, indexing Morpho Blue events on Arbitrum (~140M blocks) consistently OOMs at ~66% progress, with the V8 heap at ~4GB.

Expected behavior

A way to process logs incrementally (e.g. via a callback, async iterator, or built-in chunking) so that results from previous sub-requests can be garbage collected before the next batch is parsed.

Workaround

Manually chunking the block range in the caller and calling getLogs per chunk, which allows GC between iterations. This works but shifts the responsibility to every consumer.

[Jean-Grimal]

We dug deeper into the memory issue and found that the root cause is in ShardedCache, rather than the getLogs accumulation itself.

How bins are stored

ShardedCache groups all bins for a given filter (address + topics) under a single key.

On Arbitrum with ~100M blocks and binSize: 10_000, a single group accumulates 10,000+ bins.

Observed sizes after syncing:

┌───────────────┬───────────────────┬───────────────────────┬────────┬─────────────┐
│    Filter     │ Compressed (disk) │ Decompressed (memory) │  Bins  │ Cached logs │
├───────────────┼───────────────────┼───────────────────────┼────────┼─────────────┤
│ Morpho events │ 25 MB             │ 268 MB                │ 10,032 │ 375,850     │
├───────────────┼───────────────────┼───────────────────────┼────────┼─────────────┤
│ IRM events    │ 15 MB             │ 128 MB                │ 10,150 │ 208,519     │
└───────────────┴───────────────────┴───────────────────────┴────────┴─────────────┘

Why this causes OOM

Every time a new bin completes, setGroup() performs a read-modify-write on the entire group:

async setGroup(group, items) {
    const mapRaw = await this.store.get(group);             // Read ALL bins (~268MB JSON)
    const map = mapRaw !== null ? this.parse(mapRaw) : {};  // Parse into ~500MB+ JS objects

    for (const item of items) {
        map[item.subkey] = item.value;                      // Add 1 new bin
    }

    await this.store.set(group, this.stringify(map));       // Re-serialize everything
}

Each call transiently allocates ~1 GB+ (old JSON + parsed objects + new JSON).
With multiple filters updating concurrently, peak memory reaches 8 GB, and the process crashes.

MemoryStore is required but makes things worse

The MemoryStore in the HierarchicalStore is needed to provide a read-your-writes guarantee for setGroup's read-modify-write cycle.

Without it, DebouncedStore.get() reads stale data from disk, and newly added bins are silently lost when a subsequent setGroup overwrites them.

However, the MemoryStore holds the full uncompressed JSON strings (~268MB+), further contributing to the OOM.

Additional issue: stack overflow on push(...spread)

handleGetLogs in logs-cache/handlers.ts uses spread syntax:

allLogs.push(...logs.flat());
allLogs.push(...validLogs);

When a large uncached range returns 100K+ logs, push(...spread) exceeds V8's argument limit and throws:

RangeError: Maximum call stack size exceeded

Suggested fix

Store each bin as its own key instead of grouping all bins under one key. This way:

• setGroup only reads/writes the affected bin (KB-sized), not the entire history (hundreds of MB)
• MemoryStore only needs to hold recently written bins
• the push(...spread) issue goes away if results are smaller
• disk writes are small and independent

[haydenshively]

MemoryStore is deprecated in favor of LruStore, which at least partially addresses this. I also have a pending task to eliminate the ShardedCache entirely, so we'll just have to make sure that that refactor keeps this issue in mind.