Need advice on weird memory usage pattern while building IndexIVFPQ iteratively

[FlorianIragne]

Hi,

I'm trying to build an IndexIVFPQ (with an IndexHNSW quantizer, if that matters) and am observing a weird memory usage pattern.
Each iteration of build will add roughly 6B vectors to the index. The size on disk for 1 chunk is ~ 76GB, which is fine and expected, and is around 80GB once loaded (again, expected). The expected size in the end is ~ 300GB.

However, while building each chunk, i can see that the memory usage is increasing a lot : for example, at the end of the first chunk computation, memory usage was around 130GB (really used memory, not commited), although the on-disk size after write is 76GB.

Using this first chunk index as base and adding another chunk of 6B vectors leads to an increase of memory usage to ~ 130GB just after index read, and rapidly growing to ~ 190GB. It then seems stable near the end of the chunk computation, but increases to more than 300GB just before the end and gets OOM killed.

Looking at the code, i thought it might be due to the ArrayInvertedList using std::vectors and std::vectors reallocation pattern (doubles its size when capacity is reached). So i made a subclass of ArrayInvertedList where i limit the reallocation factor. It leads to a bit lower performances, but should in theory limit over-allocation. I read the index and then swap the 2 std::vectors of its ArrayInvertedList with the std::vectors of my implementation, move the invlist pointer to my implementation and delete the original ArrayInvertedList.

However, this doesn't work, at all. I observe almost the same memory usage pattern. I've read again the code and was unable to find another data structure or data manipulation that could explain this behavior.

Thanks to anyone that'd have advice about this situation

Using faiss 1.14.1 release, compiled with gcc 14, either with avx2 or avx512 options, with glibc malloc or tcmalloc.

[FlorianIragne]

So, my analysis of the issue was the right one. The reallocation factor of std::vector was the culprit. My implementation of the fix wasn't good enough to coerce the over-allocation pattern.
I've fixed my implementation and the memory over-usage is now very limited and i'm able to build my indexes and let them grow dynamically over time. I've also implemented a method that calls shrink_to_fit on the std::vectors when i commit the index to disk. Thus, if there's no update for a long time, the memory usage goes down.