Memory usage when calculating expectation value on two far away sites on the same row/column

[Yue-Zhengyuan]

When calculating the expectation value <A(i) B(j)> on two sites i = (0, 0), j = (r, 0) or (0, r) using expectation_value, I notice that the memory usage increases as r increases, which can reach over 100GB when D = 12, χ = 64 and r = 7. But when r = 1 or 2 it only uses ~30GB of memory.

The memory usage should be more or less the same for any r when the two sites are on the same row (or same column) of the iPEPS. I think either the internal logic of expectation_value can be improved, or some cache clearing should be added.

[Yue-Zhengyuan]

I realized where the problem is...

What I was trying to measure is <c+_up(i) c_up(j)> in the t-J model. I cannot construct each c_up separately since it has odd parity, so I must create c+_up(i) c_up(j) first as a whole. This make the contraction highly inefficient. The memory will not increase with r only when A, B can be construct separately as a one-site operator.

In general, the issue will arise whenever A, B by themselves are not symmetric / have even fermion parity, but the tensor product A ⊗ B is symmetric / has even fermion parity. We may still need to think of a way to calculate such correlation functions efficiently.

[lkdvos]

It's definitely possible that this makes the problem slightly worse, but that does not really explain why it scales with r: in principle this is simply a constant factor d^2.

I'm assuming this is not in the context of AD right? Because there we would expect that you need to store all temporary objects, so that does increase with r.

If you have some MWE I can definitely try and look into it a bit more, but if I had to guess I would say that this might just be one of those cases where the Julia GC is not aggressive enough.
Large tensor contractions allocate quite a few temporary objects that are then immediately discarded, and sometimes Julia just does not trigger the GC in time. There is a --heap-size-hint CLI flag you can pass it to be more aggressive and see if that helps anything, and if so we can also experiment with some allocation strategies that are supported by TensorOperations.

[Yue-Zhengyuan]

Thanks for the suggestion! No AD is currently involved; just plain measurement. I tried adding --heap-size-hint=60GB when running my code, and found that it doesn't work.

Below is an MWE.

using TensorKit
using MPSKitModels
using PEPSKit
using PEPSKit: LocalOperator
using Random
using BenchmarkTools

Random.seed!(0)

function meas_corX(
    peps::InfinitePEPS,
    env::CTMRGEnv,
    op::AbstractTensorMap{T,S,2,2},
    site0::CartesianIndex{2},
    sep::Int,
) where {T<:Number,S<:ElementarySpace}
    @assert sep >= 1
    lattice = collect(space(t, 1) for t in peps.A)
    site1 = site0 + CartesianIndex(sep, 0)
    O = LocalOperator(lattice, (site0, site1) => op)
    return expectation_value(peps, O, env)
end

Nr, Nc = 2, 2
D, chi = 6, 10
peps = InfinitePEPS(2, D; unitcell=(Nr, Nc))
env = CTMRGEnv(peps, chi)
op = real(S_exchange())
println(meas_corX(peps, env, op, CartesianIndex(0, 0), 1))
for sep in 1:5
    @btime meas_corX(peps, env, op, CartesianIndex(0, 0), $sep)
end

Benchmark output: (I think the "Memory estimate" (which grows linearly with sep) is the amount of memory allocated throughout the program, not the maximal memory usage occurred? Previously I was simply reading the memory usage from htop)

• sep = 1

BenchmarkTools.Trial: 205 samples with 1 evaluation per sample.
 Range (min … max):  14.241 ms … 114.560 ms  ┊ GC (min … max): 10.32% … 86.93%
 Time  (median):     18.608 ms               ┊ GC (median):    16.11%
 Time  (mean ± σ):   24.402 ms ±  21.194 ms  ┊ GC (mean ± σ):  34.88% ± 20.15%

  ▂█▆▆▅▁                                                        
  ██████▇▄▇▇▁▄▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▆▇▄▁▄▁▄▆ ▆
  14.2 ms       Histogram: log(frequency) by time       114 ms <

 Memory estimate: 89.39 MiB, allocs estimate: 684.

• sep = 2

BenchmarkTools.Trial: 98 samples with 1 evaluation per sample.
 Range (min … max):  30.284 ms … 152.221 ms  ┊ GC (min … max):  3.92% … 77.26%
 Time  (median):     38.158 ms               ┊ GC (median):    15.12%
 Time  (mean ± σ):   51.125 ms ±  35.573 ms  ┊ GC (mean ± σ):  36.21% ± 22.20%

  ▆█▁█▅▁ ▁                                                      
  ██████▄█▄▃▁▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▃▁▁▁▁▃▁▃▄▃▄▃▄ ▁
  30.3 ms         Histogram: frequency by time          150 ms <

 Memory estimate: 168.76 MiB, allocs estimate: 828.

• sep = 4

BenchmarkTools.Trial: 52 samples with 1 evaluation per sample.
 Range (min … max):  56.570 ms … 188.707 ms  ┊ GC (min … max):  2.68% … 68.34%
 Time  (median):     71.026 ms               ┊ GC (median):    13.09%
 Time  (mean ± σ):   96.672 ms ±  47.339 ms  ┊ GC (mean ± σ):  37.69% ± 24.62%

  ▅ ███▅     ▂                                         ▂        
  ██████▅██▅▅██▁▁▁▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▁▁▁▁▅▅█▁█▅█▅▁▁▅ ▁
  56.6 ms         Histogram: frequency by time          187 ms <

 Memory estimate: 328.43 MiB, allocs estimate: 1246.

• sep = 6

BenchmarkTools.Trial: 33 samples with 1 evaluation per sample.
 Range (min … max):   95.055 ms … 233.757 ms  ┊ GC (min … max):  2.53% … 53.45%
 Time  (median):     139.882 ms               ┊ GC (median):    19.32%
 Time  (mean ± σ):   155.716 ms ±  50.728 ms  ┊ GC (mean ± σ):  37.51% ± 21.91%

  █ ▃   █▃           ▃                          █          ▃     
  █▇█▁▁▁██▇▁▇▇▁▁▇▇▁▁▁█▁▁▁▇▁▁▁▁▁▁▁▁▁▁▁▁▁▁▇▇▁▁▁▁▁▁█▁▁▇▇▁▁▇▇▇▇█▁▁▇ ▁
  95.1 ms          Histogram: frequency by time          234 ms <

 Memory estimate: 488.28 MiB, allocs estimate: 1671.

[lkdvos]

Yes indeed, the memory estimate is the total amount used, so that indeed should be growing linearly.
In that sense, these results look more or less reasonable. I'll try and investigate a bit more next week.

[Yue-Zhengyuan]

I can now verify that the memory usage of correlator (#210) is under control. I close this issue now.