Memory usage (VSZ) keeps growing during long runs

[drichg]

When running a long computation, I observed that the virtual memory size (VSZ) keeps increasing over time, even though the resident set size (RSS) shows GC activity. Eventually, the VSZ reaches several hundred GB. Is this expected behavior due to Julia’s memory allocation strategy?

[Gertian]

I had issues with this as well on an older version of Julia.

In my case the issue was reduced by calling Julia with the --heap-size-hint= ... GB argument. You might want to try this too.

[drichg]

I had issues with this as well on an older version of Julia.

In my case the issue was reduced by calling Julia with the --heap-size-hint= ... GB argument. You might want to try this too.

I've already tried using the --heap-size-hint= ... GB argument for this, but the issue persists. My julia version is 1.11.2

[Gertian]

I'm afraid that's all I could say. I suspect one of the devs will have more insightful suggestions soon :)

[lkdvos]

Can you share a bit more about what kind of computations you are running? If this is something where the virtual spaces keep changing sectors it might be the case that what you are seeing is the tensorkit caches filling up.
Alternatively it might be informative to take a heap snapshot to see where everything is going?

[drichg]

Can you share a bit more about what kind of computations you are running? If this is something where the virtual spaces keep changing sectors it might be the case that what you are seeing is the tensorkit caches filling up. Alternatively it might be informative to take a heap snapshot to see where everything is going?

I’m computing the ground state of the 2D J1-J2 Heisenberg model. Here's a simplified version of my code:

J = [0.1 0.2]
Threads.@threads for J in J_list
    Dcut = 100
    L = 8
    W = 4
    lattice = InfiniteCylinder(L, L * W)
    P = Rep[U₁](1 // 2 => 1, -1 // 2 => 1)
    V1 = Rep[U₁](-1 // 2 => Dcut, 1 // 2 => Dcut, 3 // 2 => Dcut ÷ 1.5, -3 // 2 => Dcut ÷ 1.5,
        5 // 2 => Dcut ÷ 2, -5 // 2 => Dcut ÷ 2, 7 // 2 => Dcut ÷ 4, -7 // 2 => Dcut ÷ 4)
    V2 = Rep[U₁](0 => Dcut, 1 => Dcut ÷ 1.5, -1 => Dcut ÷ 1.5, 2 => Dcut ÷ 2, -2 => Dcut ÷ 2,
        3 => Dcut ÷ 4, -3 => Dcut ÷ 4, 4 => Dcut ÷ 5, -4 => Dcut ÷ 5)
    state = InfiniteMPS(repeat([P, P], L * W ÷ 2), repeat([V1, V2], L * W ÷ 2))
    H = hamiltonian(J, lattice; symmetry=U1Irrep)
    groundstate, envs, delta = find_groundstate(state, H,
        IDMRG2(; trscheme=truncdim(6 * Dcut), maxiter=20,
            tol=1e-12))
end

Thanks for your help!

[lkdvos]

Sorry for taking a bit of time to get back to you, I was out of office for the week.

Looking a bit more into this, the VSZ is only showing you the virtual address space that is being used, not the actual amount of memory taken up.
In other words, the plot you are showing means that the amount of memory used is not actually growing, but it is being spread out over a larger and larger virtual address space.
My best guess would be that what is going on here is that the address space is becoming more and more fragmented, which is slightly expected since Julia's GC does not actually compact the memory, so keeping long lived objects around would necessarily lead to this kind of behavior.

Note also that many processes don't actually return the memory to the OS after a GC run, since virtual address space on 64 bit systems is quite cheap, and as mentioned before, this is not actually being backed by physical memory usage.

I would think the caches in TensorKit are the culprits here, as they are filling up over time with moderately-sized objects that are stored in non-contiguous memory patterns, therefore filling up the address space.
While this is not necessarily a problem, it might still be a good idea to flush these caches every now and then, especially when doing many dynamic bond-dimension simulations.
In any case, currently we are being over-eager and storing way too many entries for the precomputed symmetry-data.
You can check this by having a look at the following global variables in TensorKit:

# at the tensor level
TensorKit.treepermutercache
TensorKit.treebraidercache
TensorKit.treetransposercache

# at the fusion tree level
TensorKit.transposecache
TensorKit.braidcache

These can be safely emptied, for example after every run, since they will simply get repopulated when needed (since our caches are backed by LRUCache.jl's LRU, this is thread-safe too)
There will be a slight performance decrease right after emptying the cache, since the new entries have to be recomputed, but overall this shouldn't be that bad.
Alternatively/Additionally, it can be useful to simply reduce the maximal size of these caches.
I think the sizes are currently too high, storing 10^5 different entries. You could simply have a look at the typical number required for a single IDMRG2 iteration for your usecase, and change the size to be in that ballpark. (keeping in mind that if you are multithreading you might want to increase this a bit, since the caches are shared by all threads).

In summary: the VSZ might not necessarily be a problem, but it might still be a good idea to sporadically clear the caches and reduce their sizes.
We are currently working on making a slightly better interface for this as well: QuantumKitHub/TensorKit.jl#244

[drichg]

Sorry for taking a bit of time to get back to you, I was out of office for the week.

Looking a bit more into this, the VSZ is only showing you the virtual address space that is being used, not the actual amount of memory taken up. In other words, the plot you are showing means that the amount of memory used is not actually growing, but it is being spread out over a larger and larger virtual address space. My best guess would be that what is going on here is that the address space is becoming more and more fragmented, which is slightly expected since Julia's GC does not actually compact the memory, so keeping long lived objects around would necessarily lead to this kind of behavior.

Note also that many processes don't actually return the memory to the OS after a GC run, since virtual address space on 64 bit systems is quite cheap, and as mentioned before, this is not actually being backed by physical memory usage.

I would think the caches in TensorKit are the culprits here, as they are filling up over time with moderately-sized objects that are stored in non-contiguous memory patterns, therefore filling up the address space. While this is not necessarily a problem, it might still be a good idea to flush these caches every now and then, especially when doing many dynamic bond-dimension simulations. In any case, currently we are being over-eager and storing way too many entries for the precomputed symmetry-data. You can check this by having a look at the following global variables in TensorKit:

at the tensor level

TensorKit.treepermutercache
TensorKit.treebraidercache
TensorKit.treetransposercache

at the fusion tree level

TensorKit.transposecache
TensorKit.braidcache
These can be safely emptied, for example after every run, since they will simply get repopulated when needed (since our caches are backed by LRUCache.jl's LRU, this is thread-safe too) There will be a slight performance decrease right after emptying the cache, since the new entries have to be recomputed, but overall this shouldn't be that bad. Alternatively/Additionally, it can be useful to simply reduce the maximal size of these caches. I think the sizes are currently too high, storing 10^5 different entries. You could simply have a look at the typical number required for a single IDMRG2 iteration for your usecase, and change the size to be in that ballpark. (keeping in mind that if you are multithreading you might want to increase this a bit, since the caches are shared by all threads).

In summary: the VSZ might not necessarily be a problem, but it might still be a good idea to sporadically clear the caches and reduce their sizes. We are currently working on making a slightly better interface for this as well: Jutho/TensorKit.jl#244

Thank you very much for the detailed explanation!

That makes a lot of sense. I’ll definitely try flushing the TensorKit caches after each run, and I’ll also look into tuning down the maximum cache sizes as you suggested. Thanks again for the helpful insights!

[lkdvos]

I'll close this for now, but feel free to reopen if an issue persists/reappears