Whisper leaves out chunks of speech in longer transcript

[ChristianNSchmitz]

Dear community,

we want to transcribe speech of up to 1 hour. However, if we use the whisper model larger-v2 with the limitation of 25 mb (so we chunk the files), multiple parts of the speech with a length of 10 to 40 seconds each are missing. We noticed, that the length of these missing pieces also depends on the prompt which is necessarily rather long.
Do you have a solution for that problem?

Best,
Christian

[phineas-pta]

if u run locally, there's no limit

if u using openai api, your question should be asked at: https://community.openai.com/tag/whisper

[ChristianNSchmitz]

Thanks for your reply! Even though the prompt may not be the reason for the bug, would you have a recommendation for a solution?

[ChristianNSchmitz]

I already run whisper locally, tried it with gpu and cpu. I had silent gaps with both approaches

[phineas-pta]

u have gaps even running entire file no cutting no chunking ?

[ChristianNSchmitz]

Yes, unfortunately

[phineas-pta]

can u check sound quality of those gaps

[ChristianNSchmitz]

The quality is actually pretty good, even better than some other parts of the speech

[ChristianNSchmitz]

By testing earlier version of whisper (such as 20230124), I noticed that these chunks are not left out in the transcript. As we are depending on the word-timestamps of the newer versions of whisper, is there any way to get an equally permissive behavior in newer versions of whisper?

[phineas-pta]

@ryanheise any idea ?

[ryanheise]

I think to investigate this you would sort of need all of the resources to reproduce it (the audio file, maybe the chunking script, the exact options, etc.) Or better, it should be possible to provide only one chunk with whatever prompt the chunker passed into it, and that alone should be enough to minimally reproduce the issue.

The testing of the old 20230124 version would have been without the word_timestamps option, so it may also well be that newer versions of Whisper do work, also, when the word_timestamps option is comparably omitted. So I think this is less to do with the version and more to do with activating the word_timestamps option which triggers a different algorithm to handle the moving window, and as mentioned above, looking into what's going on there would require the resources to reproduce it.

[ChristianNSchmitz]

Dear @ryanheise, thanks for your response!
It would be possible to provide all these resources. Would you be interested in having a look at this? After searching for this issue on GitHub and the OpenAI forum it appears, that several other people have similar issues.
If chunks would be ignored because of audio quality, is there an option to make whisper more permissive at the cost to produce maybe more non-sense or hallucinations?

[ryanheise]

You can try cleaning up the quality of the audio before passing it into Whisper. There are commercial products to clean up audio (e.g. lalal.ai - you can Google for others). Maybe Audacity has some filters. Programmatically, maybe some ffmpeg filters could help. Also, people have found Demucs useful in removing both background music and background noise to isolate just the vocals. As for hallucinations, there are many other discussions here about that which you might search for, including various tools and built-in options that can be used to automatically detect and skip over silence (which is one of the main triggers of hallucinations, although there are other triggers.)

[ChristianNSchmitz]

Great, I will try these tools and might get back to you in the next days if there are further questions. Thanks!