Vulkan: Select device with env variable, and skip initialize for unused devices.

[stduhpf]

Introduces the SD_VK_DEVICE env variable that can be used to chose the device to run inference on.

Also backend was initialized on every device for no apparent reason.

I know it was already possible to hide some devices from the Vulkan backend by setting the GGML_VK_VISIBLE_DEVICES env variable, but this isn't ideal, as it affects every program that uses Vulkan GGML.

(Maybe the same should be done for CUDA and SYCL backends)

[wbruna]

Works for me... kind of 🙂 I don't think SD_VK_DEVICE can be a replacement for GGML_VK_VISIBLE_DEVICES:

SD_VK_DEVICE=0 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

SD_VK_DEVICE=1 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[WARN ] stable-diffusion.cpp:191  - Cannot find targeted vulkan device (1). Falling back to device 0.
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

GGML_VK_VISIBLE_DEVICES=0 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Vega 11 Graphics (RADV RAVEN) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

GGML_VK_VISIBLE_DEVICES=1 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

But it's still useful when used together:

GGML_VK_VISIBLE_DEVICES=0,1 SD_VK_DEVICE=1 ./sd (...)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Vega 11 Graphics (RADV RAVEN) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 1

As I understand it, ggml excludes the iGPU by default, so it doesn't even show up on the device list unless it's explicitly enabled.

(aside: I really wish there was a way to specify the devices by a stable ID. Notice how the '0' device points to different GPUs depending on the context...)

[stduhpf]

Ok I finally compiled sdcpp with Rocm, and at least it's selecting device 0 by default (and it's running faster than I thought compared to Vulkan), but It makes me think having an option to select backend device when initializing context could be a nice idea for all backends.

I'm thining maybe passing --device ID via cli, and also a --list-devices argument to know which id corresponds to what device?

It could also be worth it to find a way to select device for model parts, like end text encoders and VAE to device 1 and diffusion model to device 0 for example?