Whisper-v3 works better on long audio than several short audios

[junyongyou]

Dear all, I am a newbies to ASR, and only tried Whisper-v3 for a couple of times. I have found that it performs good on a long speech. But the results become quite worse if I split the long speech to multiple short segments. Is that a common issue? Thanks a lot.

[phineas-pta]

may also depends on languages

[Purfview]

But the results become quite worse if I split the long speech to multiple short segments. Is that a common issue?

That's expected, because with splitting you lose context of the previous segments.

[junyongyou]

Are there others encountered the same problem?

[gaspardpetit]

I have experienced this, and it can be greatly improved if you provide the previous text in prompt. When calling model.transcribe, you want to provide the previous text in initial_prompt. If your segments are short, you can also consider concatenating several of them in a single prompt. This will guide the transcription into being more consistent.

In addition to using the prompt, I can provide two other advices with regards to short segments of audio:

1. Avoid splitting audio in the middle of words - use voice detection or diarization to find pauses and ideal places to split your audio;
2. Avoid extremely short segments (less than 1 second). If you provide 2ms of audio, Whisper will make something out of it. 9/10 it will generate something absurd, such as "Don't forget to subscribe" or "Subtitles by Some Random Guy" - I am guessing this comes from training sets which had these kind of text at at the end of audio-video clips without any associated speech, and now they pop when trying to transcribe silence or short nonsense audio.

[junyongyou]

Great, thanks a lot for the relevant answer and valuable answers. What I have done is to aggregate the audio input from the beginning to make a longer speech. It will sacrifice the transcription in the beginning but get better and better then.

[stri8ed]

@junyongyou What duration do you consider "long" audio vs short?

[yagudaev]

The original paper referenced that internally whisper breaks down audio into 30-second chunks (see: https://openai.com/index/whisper/).

Therefore, this should be a good start.

I'm working on a feature with live transcription and later stored for a longer summary. When live, a 2-second to 5-second chunk is better to show a few words of text on the screen.