Understanding the --language Flag in Multilingual Version of Whisper

[Suryavf]

Hi!,

I've been using Whisper and have come across the multilingual version. I understand that this allows Whisper to work with multiple languages simultaneously. However, I'm having trouble understanding why it's necessary to specify the language using the --language flag.

As far as I know, there is only one pre-trained mono-language model on the official website and it is in English. Therefore, I have a couple of questions:

• Does indicating the language increase the model's accuracy, especially for non-English languages?
• Does the use of the --language flag affect the model's precision under normal audio conditions?

Thank you in advance for any insights or clarification on this topic.

[phineas-pta]

there is only one pre-trained model on the official website and it is in English

i'm not sure which website you have visited, there're 4 english-only models and 5 multi lingual models

for multilingual models, if you don't tell --language whisper uses the first 30s of audio to detect language, which isn't always reliable

[Suryavf]

I did not express myself well in the original post. I meant that the only mono-language model is English.
To your other comment, assuming that the language detection is perfect, would it make no difference whether or not to use the flag?
Thanks for the answer

[ryanheise]

There are two differences mentioned in the previous answer (paraphrasing):

1. If you leave off the --language option, it will take (very slightly) longer to transcribe, because it will spend some time to preview the first 30 seconds to try to guess the language.
2. If you leave off the --language option, and it therefore tries to guess the language from the first 30 seconds of audio, its guess will be less reliable than if you had explicitly told Whisper (via the --language option) which language to use.

So assuming that (2) is ruled out as a difference, that only leaves (1). That is, the only remaining difference is that It will take very slightly longer.

[Tex2002ans]

assuming that the language detection is perfect, would it make no difference whether or not to use the flag?

Yes, the only difference would be the initial "detect the language in the first 30 seconds" stage.

Like ryanheise said:

• Using --language lets Whisper start transcribing right away.
• Leaving --language out makes Whisper take slightly longer in the beginning.
  • It grabs the first 30 seconds, then has to guess which language the audio is.

Before you submit the audio...

If you know which language the audio is:

• it's probably a good idea to specify --language.

If you don't know what language the audio is:

• Whisper can make a good educated guess.
  • It might guess correctly MOST of the time, but may guess wrong too. (See Side Note.)

---

I'm having trouble understanding why it's necessary to specify the language using the --language flag.

Here's a different way of explaining it:

Let's say you have a Spanish audio you want to transcribe.

You could:

Method A. Specify nothing:

• whisper SpanishAudio.mp3 --model large

Method B. Specify language:

• whisper SpanishAudio.mp3 --model large --language Spanish

Method A will go a little slower, because Whisper is taking the MP3's first 30 seconds and trying to guess what language the audio is.

Method A can then:

• Pick Spanish correctly.
  • There will be no difference between Method A or B after the initial 30s.
• Guess language wrongly.
  • Maybe Whisper thinks it's a "Portuguese" MP3.
  • It will begin transcribing using the completely wrong language.

Method B will already be told it's Spanish, so Whisper can just begin transcribing right away.

---

Side Note: On accuracy of auto-guessing an audio's language... there can be a lot of wrenches thrown into the mix.

For example, you may have:

• Intro Music
  • An English podcast, but Whisper thinks intro is "Spanish".
• "Code-Switching"
  • Speakers who flip between multiple languages while speaking.
    • Like a mostly Ukrainian meeting, but with English introduction
• A less popular language
  • (Like Tatar.)
    • Whisper will more likely guess wrongly with Method A.
• Low Quality audio
  • English accidentally detected as Nynorsk

Or, if you take a look at Whisper's breakdown of language WER graph, the further your language is down the list, the more likely auto-guessing Method A might guess wrong.

For example:

• Hindi <-> Urdu confusion
  • See this post by jongwook which links to the Indian government fine-tuning/enhancing Whisper's Hindi transcription.

The better Whisper's accuracy becomes in a given language, the better job it'll do at detecting the exact language for you.

But... it'll never be as good as human who just says:

• "Hey! You see this MP3? This is definitely Spanish!"

[Tex2002ans]

I was rereading the Whisper paper (PDF, page 11), and saw this relevant image:

which may also help explain some of the accuracy of Whisper's auto-"language identification".

As you can see, across 102 languages, Whisper's models can go from:

• ~45% correct (tiny)
• ~65% correct (large-v2)

So, if you know your audio's language—and manually specify it—you'll be good.

But if you don't know your audio's language—and leave it up to Whisper—it may be guessing incorrectly quite often.

---

Side Note: In my experience, with English/Spanish/German audio, Whisper detects near-100% accurate though. Very rarely do I get a file where it was wrong—maybe only a few dozen times across ~15,000 MP3s.

(Usually it was just a freak accident or a strange thing with that specific file. When I reran Whisper on the file again, it usually got the language correct the second time around.)