Standardize memory leak detection for fuzzing campaigns

[anishnaik]

Problem

Memory leaks in medusa are currently discovered reactively — users report OOM kills on long campaigns, then we profile with pprof against real-world targets (e.g., aave-v4-scfuzzbench) to identify the source. This approach has three problems:

1. Leaks ship to users before detection. There's no automated check that catches regressions.
2. Simple test contracts can't reproduce the leaks. The three leaks fixed in Fix memory leaks in corpus pruner, trieDB, and RPC pool #812 required a large corpus with long call sequences and active pruning. A unit test with generate_all_types.sol (5 corpus entries, 130 branches) showed no measurable difference with or without the fix, even over 10-minute runs with 57M calls and 9 pruner invocations.
3. No baseline to compare against. Without a known-good memory profile, it's hard to tell whether a change introduced a regression or just shifted allocation patterns.

What we tried

We experimented with a TestHeapProfile test that:

• Ran a 10-minute fuzzing campaign with PruneFrequency=1 to exercise the pruner
• Captured heap snapshots every 5 seconds via runtime.ReadMemStats + pprof.Lookup("heap")
• Computed growth rate (MB/min) and checked for monotonic heap increase
• Logged pprof diff commands for manual analysis

Result: Both with-fix and without-fix runs showed flat 5 MB HeapAlloc, -0.1 MB/min growth, identical behavior. The test contract is too simple to generate the corpus size and sequence complexity needed to surface the leak.

What would actually work

The leaks only manifest with:

• Large corpora (hundreds of entries, not 5)
• Long call sequences (50-100 calls, not trivial single-call sequences)
• Active pruning (pruner replaying hundreds of sequences, not 5)
• Complex contracts (thousands of branches generating sustained corpus growth)

Options to explore

1. Dedicated benchmark target. Create or adopt a multi-contract Solidity target specifically designed to generate high corpus churn — many branches, many methods, diverse state transitions. This would be checked into testdata/ and used only for memory profiling (behind a build tag). The key challenge is keeping it maintainable and representative.

2. Nightly profiling against scfuzzbench targets. Run medusa against real-world targets (aave-v4, compound-v3, etc.) in a scheduled CI job. Capture heap profiles at start and end, compute growth rate, and alert if it exceeds a threshold (e.g., >5 MB/min). This is the most realistic but requires CI infrastructure for Solidity compilation and longer runtimes.

3. Synthetic corpus injection. Instead of waiting for the fuzzer to build a corpus organically, inject a pre-built corpus with hundreds of long sequences before starting the profiling run. This would exercise the pruner and coverage paths with realistic data without needing a complex contract. The corpus would need periodic regeneration as the fuzzer's serialization format evolves.

4. Integration-level memory budget test. Set GOMEMLIMIT or monitor runtime.MemStats.Sys during a campaign and fail if it exceeds a fixed budget (e.g., 200 MB for a 5-minute run). Coarse but catches catastrophic leaks without needing pprof analysis.

Context

• PR Fix memory leaks in corpus pruner, trieDB, and RPC pool #812 fixed three leaks totaling ~10 MB/min (coverage maps in pruner, trieDB hashdb, fork-mode RPC pool)
• Issue Unify coverage cleanup: extract from tracer instead of AdditionalResults #811 proposes unifying coverage cleanup to prevent the pruner leak class
• The fundamental tension: unit tests are fast but too simple to surface real leaks; real-world tests are slow but catch real problems