[FIX]: split attention V1 fixes vram OOM issues

[2blackbar]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

--opt-split-attention-v1 --xformers
This solved it and that v1 split attention should be used by default imo
I can do 2048x2048 on 3090 like before or even 4048x640, probably even higher cause it reserves about 8gb only
Use it and forget about all the OOM crap
Wow 3090 can even do 4048x4048 but it is quite slow, doesnt OOM tho so it works !

Steps to reproduce the problem

.

What should have happened?

.

Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commits/master

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--opt-split-attention-v1 --xformers

List of extensions

.

Console logs

.

Additional information

No response

[FNSpd]

Do xformers do anything here? I think that only first optimization is applied

[2blackbar]

Do xformers do anything here? I think that only first optimization is applied

yeh they speedup the generation from 15 seconds to 6 seconds but they dont affect VRAM cause i was able to do same sizes without them

[elen07zz]

why --opt-split-attention-v1? and not --opt-split-attention

[2blackbar]

cause vram issues even on 24GB 3090 , v1 is the best and should be default one

[FNSpd]

V1 uses less memory, iirc

[EfourC]

Are you sure you're measuring the VRAM usage accurately? I thought the 3090 had 24GB of vram, but you mentioned 8GB of usage using --opt-split-attention-v1, and out-of-memory problems without it, leaving 16GB on the table. I probably misunderstand what you meant by 'reserves' 8GB.

I do believe it helped memory problems, though! I'm jealous, my max limit on 2070super 8GB seems to be 1280x1280 (but without anything that takes extra memory like ControlNet, which drops the max resolution a bit).
[Edit: not counting --medvram option]

I recommend using MSI Afterburner's hardware monitor graphs. It's pretty interesting to see the fluctuations with 100ms resolution.

Here's an example where I just did 15steps at 1280x1280:

[fractal-fumbler]

it's should be either --opt-split-attention-v1 or --xformers, because there is elif in code for attentions

stable-diffusion-webui/modules/sd_hijack.py

Line 40 in 0cc0ee1

if cmd_opts.force_enable_xformers or (cmd_opts.xformers and shared.xformers_available and torch.version.cuda and (6, 0) <= torch.cuda.get_device_capability(shared.device) <= (9, 0)):

[hananbeer]

it's should be either --opt-split-attention-v1 or --xformers, because there is elif in code for attentions

stable-diffusion-webui/modules/sd_hijack.py

Line 40 in 0cc0ee1

if cmd_opts.force_enable_xformers or (cmd_opts.xformers and shared.xformers_available and torch.version.cuda and (6, 0) <= torch.cuda.get_device_capability(shared.device) <= (9, 0)):

thanks for clarification. none of this was mentioned so I modified it a bit for more user friendly behavior:
6f0fe65

waiting for PR :)

[Sakura-Luna]

Your change is a bit redundant, just output a line of warning when multiple attention parameters are activated.

[hananbeer]

that is what I'm doing. can you elaborate what you think is redundant?

[Xyem]

Thanks for posting this. It helps with issue #8431 that I raised where unsplit prompts seem to demand ridiculous amounts of VRAM when a Lora is used. I think this will really help narrow down the cause!

[2blackbar]

It works , xformers help with speed, im gettin almost twice as fast speed iwth xformers , with 24gb ai can do 4048x4048 but it takes a while, still it is possible, so without v1 i was able to do like 1300x800 at max on 3090 with 24GB which was very weird, IMO default doggetx should be booted from the repo and v1 used instead

[Xyem]

I don't use xformers as I prefer deterministic results (which made my discovery that batch size affects the generated images to be extremely annoying. That caused a lot of confusion recently.)

[fractal-fumbler]

@Xyem check for --xformers-flash-attention command-line option

[Xyem]

That's really useful, thanks for sharing!

[Mich-666]

IMO default doggetx should be booted from the repo and v1 used instead

Ok, just for sake of testing, I ran this on 1050Ti 4GB VRAM, trying to generete picture with hiresfix on, resize: from 512x768 to 768x1152.

-opt-split-attention-v1 --lowram, result:
RuntimeError: CUDA out of memory. Tried to allocate 1.42 GiB (GPU 0; 4.00 GiB total capacity; 831.42 MiB already allocated; 882.44 MiB free; 2.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
getting big memory spike in the end and OOM error. IIRC those spikes were actually the reason why this method wasn't used as default.

--lowram with default doggettx, result:
normally generates the picture.

So there is still usecase for doggettx cross attention optimalization. The only difference between --opt-split-attention-v1 and dogettx is that --opt-split-attention-v1 doesn't actually crash your driver like dogettx when running out of memory due to image size being too large. But doggettx doesn't have any memory spikes in the end and allows you generate the image.

It would be probably the best if we could set the maximum amount of memory PyTorch can use in the settings or if doggettx implementation had its memory utilization fixed to prevent any driver crashes.

[Xyem]

It's very unclear what, if even only a singular problem, is happening. With Doggetx, I was only able to generate batches of 3-4 with HR to 600x960 depending on if I have a Lora and the prompt length. With v1, I can do batch sizes of 12 regardless (on an RTX 2060 12GB).