Accelerate doesn't use multi GPU with automatic1111

[PiotrRaszkowski]

Hi,

I am trying to setup multiple GPU on my generative AI dedicated server.
I installed 'accelerate' and configured it to use both GPUs (multi) I have.

The problem is that automatic1111 always starts processes on same GPU, I was unable to make it work on both.
Does anyone know how to fix it? Is there any method to run automatic1111 on both GPU? I suspect that it runs two instances of automatic1111 but both on same GPU, probably I should somehow pass CUDA_VISIBLE_DEVICES to each instance respectively but I don't know how to do it.

I have Ubuntu Ubuntu 22.04.3 LTS server without xserver.

accelerate test results

stdout: **Initialization**
stdout: Testing, testing. 1, 2, 3.
stdout: Distributed environment: MULTI_GPU  Backend: nccl
stdout: Num processes: 2
stdout: Process index: 0
stdout: Local process index: 0
stdout: Device: cuda:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: Distributed environment: MULTI_GPU  Backend: nccl
stdout: Num processes: 2
stdout: Process index: 1
stdout: Local process index: 1
stdout: Device: cuda:1
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: 
stdout: **Test process execution**
stdout: 
stdout: **Test split between processes as a list**
stdout: 
stdout: **Test split between processes as a dict**
stdout: 
stdout: **Test split between processes as a tensor**
stdout: 
stdout: **Test random number generator synchronization**
stdout: All rng are properly synched.
stdout: 
stdout: **DataLoader integration test**
stdout: 0 1 tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
stdout:         18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
stdout:         36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
stdout:         54, 55, 56, 57, 58, 59, 60, 61, 62, 63], device='cuda:0') <class 'accelerate.data_loader.DataLoaderShard'>
stdout: tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
stdout:         18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
stdout:         36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
stdout:         54, 55, 56, 57, 58, 59, 60, 61, 62, 63], device='cuda:1') <class 'accelerate.data_loader.DataLoaderShard'>
stdout: Non-shuffled dataloader passing.
stdout: Shuffled dataloader passing.
stdout: Non-shuffled central dataloader passing.
stdout: Shuffled central dataloader passing.
stdout: 
stdout: **Training integration test**
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: 
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Training yielded the same results on one CPU or distributed setup with no batch split.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: 
stdout: FP16 training check.Training yielded the same results on one CPU or distributes setup with batch split.
stdout: 
stdout: FP16 training check.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: 
stdout: Keep fp32 wrapper check.
stdout: Keep fp32 wrapper check.
stdout: 
stdout: **Breakpoint trigger test**
Test is a success! You are ready for your distributed training!

It is correct, two processes started, each GPU unit used.

automatic1111 run

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on piotr user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
python venv already activate or run without venv: /home/piotr/workspaces/stable-diffusion-webui/venv
################################################################

################################################################
Accelerating launch.py...
################################################################
launch.pyUsing TCMalloc: libtcmalloc_minimal.so.4
Python 3.10.13 (main, Nov 10 2023, 19:48:34) [GCC 11.4.0]
Version: v1.6.0-2-g4afaaf8a
Commit hash: 4afaaf8a020c1df457bcf7250cb1c7f609699fa7
Python 3.10.13 (main, Nov 10 2023, 19:48:34) [GCC 11.4.0]
Version: v1.6.0-2-g4afaaf8a
Commit hash: 4afaaf8a020c1df457bcf7250cb1c7f609699fa7
Launching Web UI with arguments: --no-half-vae --enable-insecure-extension-access --listen --medvram --no-half-vae --xformers
Launching Web UI with arguments: --no-half-vae --enable-insecure-extension-access --listen --medvram --no-half-vae --xformers
CivitAI Browser+: Aria2 RPC started
Loading weights [1fe6c7ec54] from /home/piotr/workspaces/stable-diffusion-webui/models/Stable-diffusion/juggernautXL_version6Rundiffusion.safetensors
CivitAI Browser+: Aria2 RPC restarted
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 8.7s (prepare environment: 1.7s, import torch: 2.9s, import gradio: 0.7s, setup paths: 1.2s, initialize shared: 0.1s, other imports: 0.6s, load scripts: 0.6s, create ui: 0.6s, gradio launch: 0.2s).
Creating model from config: /home/piotr/workspaces/stable-diffusion-webui/repositories/generative-models/configs/inference/sd_xl_base.yaml
CivitAI Browser+: Aria2 RPC restarted
Loading weights [1fe6c7ec54] from /home/piotr/workspaces/stable-diffusion-webui/models/Stable-diffusion/juggernautXL_version6Rundiffusion.safetensors
Running on local URL:  http://0.0.0.0:7861

To create a public link, set `share=True` in `launch()`.
Startup time: 11.1s (prepare environment: 1.7s, import torch: 2.9s, import gradio: 0.7s, setup paths: 1.2s, initialize shared: 0.1s, other imports: 0.6s, load scripts: 2.6s, create ui: 0.8s, gradio launch: 0.3s, app_started_callback: 0.1s).
Creating model from config: /home/piotr/workspaces/stable-diffusion-webui/repositories/generative-models/configs/inference/sd_xl_base.yaml
Applying attention optimization: xformers... done.
Model loaded in 7.3s (load weights from disk: 1.3s, create model: 0.6s, apply weights to model: 4.6s, calculate empty prompt: 0.7s).
Applying attention optimization: xformers... done.
Model loaded in 7.2s (load weights from disk: 1.5s, create model: 0.5s, apply weights to model: 4.5s, calculate empty prompt: 0.6s).

[w-e-w]

multi gpu is not currently supposed
but there are ways to chain muiltpel instances of webui like
https://github.com/papuSpartan/stable-diffusion-webui-distributed.git

[PiotrRaszkowski]

Yes, thank you, I already found this plugin and also I checked the SD source code figuring out that probably SD doesn't support multi-gpu.