Linux command line does not use all GPUs

[8192K]

Running whisper on the command line in linux, it will only use my first GPU to the max, but not the second GPU at all. nvtop shows both GPUs but only one working (95% with 5.6GB of VRAM used). Both are 3060s with 12GB of RAM and I am running the medium model. I installed whisper with pip. The command line is 'whisper "file" --model medium --language xx'.

[phineas-pta]

that's normal/default behavior

if u want multiple gpu u have to modify source code

but since checkpoint fit to 1 gpu there's no advantage to use multiple

[SametDulger]

Why does this happen?
Single-GPU default: The Whisper CLI, installed via pip, runs inference on a single GPU by default. It does not include built-in multi-GPU support.

Model size fits one GPU: Whisper models (like medium) typically fit comfortably on one GPU, so the code doesn't split workload across multiple GPUs automatically.

No multi-GPU code: The official Whisper repo and CLI don't implement multi-GPU parallelism (like data parallelism or model parallelism) out of the box.

How to utilize multiple GPUs?
To use both GPUs, you'd need to:

Modify source code:

Implement multi-GPU inference by modifying the PyTorch code to distribute batches across GPUs (DataParallel or DistributedDataParallel).

This is non-trivial because Whisper inference usually runs on one input at a time, limiting batch-based parallelism.

Run multiple processes:

Run multiple independent Whisper inference processes each bound to a different GPU (CUDA_VISIBLE_DEVICES=0 and CUDA_VISIBLE_DEVICES=1), then load-balance your audio files manually or via a script.

Use batch processing:

If you batch many audio files together, you can try to split batches across GPUs manually.

Summary
Default Whisper CLI uses only one GPU.

Multi-GPU usage requires custom code or running multiple parallel processes.

Since the model fits on a single GPU, multi-GPU inference might not give much speed-up unless you're processing many files concurrently.

[8192K]

Run multiple independent Whisper inference processes each bound to a different GPU (CUDA_VISIBLE_DEVICES=0 and CUDA_VISIBLE_DEVICES=1), then load-balance your audio files manually or via a script.

That's what I ended up doing. I had folder full of video files that could easily have been processed in parallel.

[phineas-pta]

in that case your should write a python script to load 1 whisper process for each gpu, then for each process do batch inference