I am trying to use the gpu. According to Windows, whisper only uses cpu.

[ErfolgreichCharismatisch]

This is my code:

import whisper
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
model = whisper.load_model("small")
result = model.transcribe(r'c:\...\somefile.mp3')
print(result["text"])

What do I need to add?

[iamdroppy]

Same issue here.

[cetiny]

I have this in my code.

DEVICE = 'cuda'
model = whisper.load_model(MODEL, device = DEVICE)

[xBLOCK-HEADx]

I have this problem too.

import torch import whisper torch.cuda.is_available() true model = whisper.load_model("tiny", device = "cuda" result = model.transcribe("path/to/file.mp3")

but still the CPU is used not GPU

[jongwook]

It could be the case that the CPU is spending the majority of the time decoding the MP3 and moving bytes around. Can you try the medium or large model with a long audio file so that it's less CPU-bound?

[xBLOCK-HEADx]

when I try to use medium model, I get:

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.46 GiB already allocated; 0 bytes free; 3.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I tried running whisper from command line instead of python with "--model tiny" and "device --cuda", and I could see about 10% activity on GPU:0 which according to my system monitor is the on-board Iris processor.

It's a bit confusing because in the error code above it says GPU:0 4.00GB total capacity, but my "GPU:0" doesn't have 4GB, according to system that would be GPU:1, my GTX1650 gpu

[mahi80]

for windows
set "PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512"
for linux
export "PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512"

above export / set command from terminal for current session , you can also change 512 to 128 for mini batch size allocation and run as below

import torch  
import whisper 
devices = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 
model = whisper.load_model("medium" , device =devices)

[mahi80]

below code should work flawlessly , just need to see capacity of GPU what model it can load in memory my AWS AMI image g4dx large was Tesla T4, only handled models from tiny, small, and medium(5GB model weight), large model 10GB was out of memory

import torch  
import whisper 
devices = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 
model = whisper.load_model("medium" , device =devices)

[cryptonautics]

Random (slightly adjusted) ChatGPT (GPT v4) advice that helped me. Might help others as well, YMMV:

"""
To resolve this issue, you should modify the instantiation of the ctranslate2._ext.Whisper object in your code to correctly match the expected constructor arguments. Assuming you want to use your GPU, the device should likely be 'cuda', and you may need to adjust the compute_type and other parameters according to your specific requirements and the capabilities of the ctranslate2 library.

For example, the corrected line might look something like this:

...
model = ctranslate2.models.Whisper(
    model_path="C:\\Users\\Kyle\\.cache\\huggingface\\hub\\models--Systran--faster-whisper-medium\\snapshots\\08e178d48790749d25932bbc082711ddcfdfbc4f",
    device='cuda', 
    device_index=0, 
    compute_type='float16', 
    intra_threads=4, 
    inter_threads=1
)
...

Make sure to adjust the model_path and other parameters according to your specific setup and requirements.
"""