Issue: Whisper ASR Model Not Loading on AMD MI-210 (ROCm 6.1)

[Sajid-Hameed63]

Whisper ASR model fails to load on AMD MI-210 GPU (ROCm 6.1), but works fine on Nvidia RTX 4060 (CUDA 12.4).

System Details

• Nvidia Setup (Works): RTX 4060, CUDA 12.4, Python 3.10.12, Torch 2.6.0, Ubuntu 22.04.5 LTS
• AMD Setup (Fails): MI-210, ROCm 6.1, Python 3.10.12, Torch 2.6.0+rocm6.1, Ubuntu 24.04.1 LTS

Error Logs

$ python3 whisper_load_test.py 
DEBUG: Using device - cuda
DEBUG: Loading Whisper model...
DEBUG: Error loading Whisper model - Error(s) in loading state_dict for Whisper:
	While copying the parameter named "encoder.blocks.0.attn.query.weight", whose dimensions in the model are torch.Size([384, 384]) and whose dimensions in the checkpoint are torch.Size([384, 384]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
DEBUG: Retrying on [CPU...]
DEBUG: Failed to load Whisper model on CPU - Error(s) in loading state_dict for Whisper:
	While copying the parameter named "encoder.blocks.0.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).

Python Script

import torch
import whisper

# Debugging: Check GPU and ROCm availability
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"DEBUG: Using device - {device}")

try:
    print("DEBUG: Loading Whisper model...")

    # Load Whisper model
    model = whisper.load_model("tiny.en")

    # Move model to the correct device
    model = model.to(device)
    
    print("DEBUG: Whisper model loaded successfully.")

except Exception as e:
    print(f"DEBUG: Error loading Whisper model - {e}")
    print("DEBUG: Retrying on CPU...")

    try:
        model = whisper.load_model("turbo").to("cpu")
        print("DEBUG: Whisper model loaded successfully on CPU.")
        
    except Exception as e:
        print(f"DEBUG: Failed to load Whisper model on CPU - {e}")

Key Observations

• Model fails to load on both GPU and CPU in ROCm setup
• Error occurs while copying parameters in encoder.blocks.0.attn.query.weight

Request for Assistance

• Is this a compatibility issue with ROCm and Whisper?
• Any known fixes or workarounds?
• Has anyone successfully run Whisper on AMD GPUs with ROCm?

Any help would be greatly appreciated. Thank you!

[glangford]

See the code for load_model here:

whisper/whisper/__init__.py

Line 130 in 517a43e

if device is None:

Note that load_model already preferentially loads using cuda and falls back to cpu so you don't need to move the model with model.to(device).

Similarly, for this code:

        model = whisper.load_model("turbo").to("cpu")

instead specify that you want to use device cpu in the second parameter, like this

        model = whisper.load_model("turbo", "cpu")

I don't use ROCm myself, but see this past conversation here, FYI:

• ROCm support #55 (comment)

[Sajid-Hameed63]

Thanks for the reply glangford. I got your point regarding the Nvidia GPU, but the major issue here is still unresolved: how can I resolve this on an AMD GPU.

[lukedupin]

If your issue is a poorly supported gfx version GPU, IE like mine gfx1103 - 790m APU. You can solve this issue by forcing a specific, better supported GFX version:

HSA_OVERRIDE_GFX_VERSION=11.0.2 python my_script.py