Memory (not VRAM) leak on long runs...

[valden80]

I'm not sure that it's really SD or Automatic's bug and should go to bag reports, so I'm just asking here first:

I'm run Web UI on Manjaro Linux, my card is GTX 1650, 4GB.

I'm running UI with options --lowvram and --no-half (my card affected with black square issue if used fp16 mode, and on full precision even 512x512 cause CUDA error.) Also using xformers.

With settings above, it's work normally, I can generate somewhat between 10-30 images, but after it main SD python process (launch.py) eat up all my system RAM (32Gb), go in heavy swapping, and eventually become killed with OOM.
It's not sudden eating - it's constantly leaking through generation of every image. I try without xformers too - no difference.

Do someone experiencing the same issues? Any idea - what can be done?
I think, it's directly related with --lowvram mode (intensive swapping between RAM and VRAM) but I do not have technical ability to test even with --medvram mode by myself...

[ataa]

What's the output of cat config.json | grep cache and cat config.json | grep models_in_memory?

[valden80]

This:

$cat config.json | grep cache 
    "sd_checkpoint_cache": 0,
    "ldsr_cached": false,
    "sd_vae_checkpoint_cache": 0,
    "upscaling_max_images_in_cache": 5,

$cat config.json | grep models_in_memory
    "interrogate_keep_models_in_memory": false,

[valden80]

So, no one else experience similar symptoms?
This is really important - if it's my own only issue, then it's only me, who need to guess what wrong with my system or hw...

[FurretUber]

I am facing the same problem, using a GTX 1650 Max-Q. What I have noticed is that the memory increase doesn't happen on every image being generated, it seems to increase every time the generation is started (so every time the button "Generate" is clicked, for example). While generating images, I can notice fluctuations, but no consistent increase.

I can, for example, generate 100 images with no problem if I set the batch count to 20 and batch size to 5. However, if I try to generate 30 pictures one by one, then yes, there will be a significant memory increase every time the generation is started until OOM is triggered. The leak seem to be more severe using img2img or inpaint, and with resolutions larger than 512.

I could use --medvram in the past and I didn't notice this memory leak back then. Using CPU, I also don't notice any memory leak.

[davidastgtciv]

I get the same behavior as FurretUber. I run with the options: "--precision=full --no-half --medvram --xformers --use-cpu interrogate"

[westmancurtis]

Yeah, it's been an issue for a long while. Search for "Killed" and you'll find a very old thread with no real solutions beyond raising the swap size (which isn't really a fix to be honest).

[lsaa]

there is an issue for this... I've had good results running without --medvram but I cant generate big enough images without it

[valden80]

Glen-O found the solution of this problem. Here: #6722
Need to say great thanks to him.