Inference slows down to CPU levels when using token suppression with Pipelines

[hypnaceae]

Python 3.10.12
Transformers 4.41.0.dev0
Whisper 20231117

Getting token IDs for suppression:

tokenizer = WhisperTokenizer.from_pretrained(model_path, language="Polish", task="transcribe")
suppress = [token for c in "0123456789" for token in tokenizer.encode(c)]

Pipeline setup:

whisper_asr = pipeline(task="automatic-speech-recognition",
                       model=model_path,
                       device="cuda",
                       tokenizer=tokenizer)

Inference (slow, roughly CPU speed):

for result in whisper_asr(KeyDataset(dataset, "audio"),
                                return_timestamps="word",
                                chunk_length_s=30,
                                generate_kwargs={'language': 'polish', 'suppress_tokens': [-1] + suppress}):

Inference (fast as expected on GPU)

for result in whisper_asr(KeyDataset(dataset, "audio"),
                                return_timestamps="word",
                                chunk_length_s=30,
                                generate_kwargs={'language': 'polish', 'suppress_tokens': [-1]}):

Has anyone else encountered this issue?

I've tried Faster-Whisper - there, token suppression makes no difference in speed of inference - but I prefer pipelines for my project.