Multi GPU support

[stevelittlefish]

I have 4 GPUs in my server, but if the first one has no VRAM requests will start failing with CUDA errors, even if the other GPUs have loads of VRAM available. Would it be easy to add multiple GPU support?

[virdb]

On my server with 2 GPU, running docker compose.cuda.yml with count: all, it causes one GPU fall off the bus.