Multiple-GPU usage when running inside docker.

[vjethava]

Hi,

I've a custom docker that is using AsyncEngineArray to generate embeddings. It works fine on single-GPU but when running using multiple GPUs, infinity still uses only one GPU (seen using nvidia-smi). Is there some special flag that needs to be set for it?

docker --runtime=nvidia --gpus all <DOCKER>

Best
Vinay

[michaelfeil]

How would you want to have your GPUs used?
A model currently can’t be split across GPUs. Are you expecting to run multiple models?