Multi-GPU setup

[stopandgo-44]

Good morning everyone,

I'm trying to run SD.Next on multiple AMD GPUs. My understanding is that you cannot actually split work between two GPUs for generating the one image. But you can use both GPUs in general to queue up different images and therefore you can double your (unique) image generation rate.

I'm using two Radeon GPUs which work just fine one at a time, but I do not know how to use them both at once. Should I create a second install of SD.next and try to run that at the same time? Surely there might be a better way?

Tried openVINO but my GPUs are not visible to it, and I believe it's because openVINO only supports Intel architecture.

Any help is appreciated, thanks!

[brknsoul]

Depending on your AMD GPUs, you can use ZLUDA (faster, less support for older GPUs) or DirectML (slower, supports pretty much any GPU).

You'd use the --device-id # and --port #### args to run using different GPU IDs and ports (each instance of SD.Next needs its own port number)

I don't think there's a need to install 2 copies on SD.Next, you should be able to run the one copy twice.

[brknsoul]

Oh, sorry. I assumed you were using Windows. ZLUDA (and DirectML) is a Windows-only platform. ROCm in Linux is superior.

[stopandgo-44]

That's interesting, zluda is available for Linux. I've used it to compare how HIP performs in blender vs zluda with this specific GPU (Radeon VII) and I found them to be basically identical. When running SD.Next with zluda (no errors, seems to run fine?) I also see no improvements or noticeable degradations.

[stopandgo-44]

Okay, some interesting notes.

I've tried this with multiple SD applications. EasyDiffusion for example allows you to directly select multiple GPUs in the settings. For SD.Next, I also was able to spawn 2 instances, one for each GPU. The end results are the same for now...

Here's the problem:

I'm only using 32 GB of RAM. So, while using 1 GPU at a time works fine because it does take up some RAM to run (not just VRAM, even though my VRAM is not maxed out), running two instances AGAIN causes the RAM to be filled up a second time.

Meaning I instantly go out of RAM as soon as the second instance begins trying to form an image.

Questions:

1. Is generation supposed to be using any system RAM at all? For text gen in LLMs, I don't think I've ever noticed the RAM being significantly utilized as long as the model can fit in the VRAM.

2. How can you avoid having to double-up on RAM if you're generating the same things across two different GPUs? I mean identical prompts, models, settings etc everything. How can you somehow tell the system to share the same RAM load to two instances?

3. What settings do I need to flip to keep my RAM usage as low as possible?

[vladmandic]

first, yes, using two instances of sdnext is currently the only way to use multiple gpus. it would be possible to add it in sdnext itself, but very few users have multiple gpus and doing it right is significant effort, so its not considered a priority at the moment.

regarding ram usage, sdnext typically loads model into ram and moves parts to vram as needed as its far more common for users to have low vram than low ram.
you can enable two things in settings which should pretty much eliminate ram usage if your models fully fit into vram:

• disable offloading completely (default is balanced offload enabled)
• enable load direct-to-gpu (default is disabled)

[mr44er]

My understanding is that you cannot actually split work between two GPUs for generating the one image.

Yes, the problem is this: two women can't produce one baby in 4,5 months, it needs 9.

When running SD.Next with zluda (no errors, seems to run fine?)

Yeah, ZLUDA runs fine on my side here on Arch.

I'm only using 32 GB of RAM.

I know spending money is no fun but in the long run you really should consider upgrading. Doing things in parallel needs "workspace" and air to breath and models won't shrink in the future, it will be the opposite.

[indrora]

You can mask what GPUs are seen by an instance of ROCM as well; set ROCM_VISIBLE_DEVICES=1 and HIP_VISIBLE_DEVICES=1 (or =0, or whatever) to mask which device is seen by ROCm/zluda.

This is necessary if you're on an all-AMD system, where AMD integrated graphics get seen but passing --device=1 will cause an AMD GPU to be misidentified and the pipeline won't work right.

ollama/ollama#11975