Audio doesn't get properly translated when it has some words in another language

[chamini2]

I have 2 example recordings in spanish. They say

1. Quiero demostrarles cómo funciona Whisper.

   I want to show you how Whisper works
   Quiero demostrarles cómo funciona Whisper.

2. Vamos a ver que Whisper sí funciona.

   Let's see that Wisper does work.
   Let's see that Wisper does work.

When translating both of these to target en, the recording 1 is not translated at all and it just outputs the whole thing in spanish. Recording 2 does get translated correctly.

My theory is that since the last word was in english, it somehow thought the whole thing was english? Not sure if that makes sense, but I have seen it happen with other examples too.

It could also be related to Whisper being treated as a name rather than a word?

Is this a known limitation (spanglish is common, but maybe not considered for the model) or something that the team is willing to look into?

Examples (link expires in 7 days): https://fromsmash.com/whisper-spanglish

[phineas-pta]

cannot reproduce what u did

[chamini2]

Very interesting! My code is using transformers...

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "openai/whisper-large-v3"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
)
model.to("cuda:0")

processor = AutoProcessor.from_pretrained(model_id)

# Based on insanely-fast-whisper
pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=30,
    batch_size=24,
    torch_dtype=torch.float16,
    device="cuda:0",
    model_kwargs={"use_flash_attention_2": True},
)

res = pipe(
    url,
    return_timestamps=True,
    generate_kwargs={"language": "en"},
)

[blackpolarz]

I think there is a slight mistake in generate_kwargs={"language": "en"}.
It should be generate_kwargs={"language": "es"} as the language defined should take the language of the audio input and there should be an additional parameter within generate_kwargs, "task":"translate" to translate the audio to English.
Try modifying it and see if that works?