Is whisper production-ready?

[ururk]

The main issues I see with it are hallucinations during silence and word timings being off. Other than those two things, for ASR, it's pretty impressive when compared to AWS Transcribe.

Is there any effort to improve either of the above issues internally with whisper? Or are these baked into how it works, and any solutions for either of them will require using custom post-processing or third-party tools (ie, whisperX, stable-ts)?

[mayeaux]

They're both being worked on, I think hallucinations is the biggest issue at the moment but I think pretty soon it will be solved to a large extent. I'm pretty optimistic about this PR for example: #1155

[ururk]

Yes - word-level timestamps are not perfect, but it's an issue I could live with. They aren't off so much as to ruin context, and the higher quality of transcription offsets any issues. I mean, it properly transcribed eigenvalues, and other complex terms that AWS hilariously gets wrong. I'll give that PR a try.

[ururk]

Not sure if that PR solves the issue, at least not for this scenario - a recording with "silence" at the beginning:

whisper 1487020083-855-O-c79-12.mp4 --model small.en --word_timestamps True --append_punctuations True --device cpu

[00:00.000 --> 00:25.740]  You
[00:30.000 --> 00:55.740]  You
[01:00.000 --> 01:25.740]  You
[01:30.000 --> 01:55.740]  You
[02:00.000 --> 02:24.640]  You
[02:30.000 --> 02:46.580]  You
[02:57.020 --> 02:57.500]  You
[02:57.500 --> 02:58.440]  Happy Monday everybody

Both with/without the PR, my machine transcribed the beginning as You.

And medium.en with/without the PR:

[00:00.000 --> 00:29.580]  Thank you.
[00:30.000 --> 00:59.580]  Thank you.
[01:00.000 --> 01:29.580]  Thank you.
[01:30.000 --> 01:59.580]  Thank you.
[02:00.000 --> 02:29.980]  Thank you.
[02:30.000 --> 02:59.980]  Thank you.

It omits "Happy Monday everybody". Note the time overlap - "Happy" is spoken right around 2:57.

Adding --condition_on_previous_text False "restores" the missing phrase:

[00:00.000 --> 00:29.580]  Thank you.
[00:30.000 --> 00:59.580]  Thank you.
[01:00.000 --> 01:29.580]  Thank you.
[01:30.000 --> 01:59.580]  Thank you.
[02:00.000 --> 02:29.980]  Thank you.
[02:56.900 --> 02:59.000]  Happy Monday, everybody.