Memory leak with Julia 1.11's GC (discovered in SymbolicRegression.jl)

[MilesCranmer]

We're seeing memory leaks in PySR/SymbolicRegression.jl that appear related to Julia 1.11's parallel GC. The user (@GoldenGoldy) tried various solutions including heap size hints and other parameter adjustments, but memory usage would steadily climb until OOM crashes occurred after 8-11 hours. The issue vanishes completely when switching to Julia 1.10 - no other changes needed. While we don't yet have a minimal working example in pure Julia, I wanted to raise this as it's causing OOM crashes in production workloads.

Full reproduction steps and details in: MilesCranmer/PySR#764, including detailed diagnostics on the memory usage `​​​​

[vchuravy]

Does this reproduce without Python in the loop?

[MilesCranmer]

Just checked and, yes, it seems to continually to increase in memory, albeit slowly. It's slow enough that the OOM error only happens after 11 hours of production runtime.

using SymbolicRegression

X = randn(Float32, 5, 10_000)
y = 2 * cos.(X[4, :]) + X[1, :] .^ 2 .- 2

options = SymbolicRegression.Options(;
    binary_operators=[+, *, /, -],
    unary_operators=[cos, exp],
)

hall_of_fame = equation_search(X, y; niterations=1000000000, options=options, parallelism=:multiprocessing)

To clarify, Python is only used to call a Julia method that runs for 12 hours without passing control back to Python. I don't expect Python to factor into things. The new Julia processes are spawned from Julia and do not interact with Python.

Here are some plots @GoldenGoldy made, with memory usage per Julia process over the course of the 12-hour run:

Julia 1.11:

and Julia 1.10:

Note that I automatically set a conservative --heap-size-hint for spawned Julia processes which is why these hover at around 0.6 GiB.

[gbaraldi]

If possible can you take heap snapshots at the start and after a while to see what is being allocated?

[MilesCranmer]

What's the best way to do that on macOS?

(I think @GoldenGoldy is on Linux though FYI. I'm not sure if the memory leak is OS dependent or not)

[gbaraldi]

Profile.take_heap_snapshot it creates a file on pwd if I recall correctly. Then you can open it using the chrome dev tools

[MilesCranmer]

Ok. I made some heap snapshots. Do you want the full files? (250 MB each). Not sure what I am supposed to look for though.

[gbaraldi]

Are the start one and the end one the same size? 🤔 But yeah, could you zip them and send them to me?

[MilesCranmer]

Maybe it's hard to see much a difference in ~5 minutes (it might even go down - see plot above), it's only increased slightly over that time.

Also, does take_heap_snapshot record other Julia processes too, or just the main process?

One more question. Would the heap snapshot even record the memory leak? Or do you just want to see if it's a real leak or actual allocations somewhere?

[gbaraldi]

The heap snapshot will see everything that is live. I want to see if it's an actual leak i.e a pointer we forgot to call free on or extra live objects around for some reason. take_heap_snapshot records only the current process.

[MilesCranmer]

So the heap snapshots are about the same size, even over a 10 minute interval. And I am looking at the memory breakdown and they seem quite similar in where memory is allocated too. But despite this, the process's allocated memory continues to increase.

So does this mean it's a real leak?

[d-netto]

Might be useful to check whether the memory increase/possible leak is stemming from an increase in pool allocated pages or an increase in mallocd memory.

I.e. #55794 (comment).

Also, does this only reproduce when using multiple GC threads or also reproduces with a single GC thread?