Whisper-Zero

[RaulKite]

Today Gladia has announced this service that claim to have 0 hallucinations.

https://www.gladia.io/whisper-zero

A complete rework of Whisper ASR that eliminates hallucinations and drastically improves accuracy. Built using over 1.5 million hours of audio, including phone and noisy data.

Do you think hallucinations will be avoid in nexts whisper releases?

[ejentos]

Today Gladia has announced this service that claim to have 0 hallucinations.

A complete rework of Whisper ASR that eliminates hallucinations and drastically improves accuracy. Built using over 1.5 million hours of audio, including phone and noisy data.

@RaulKite , Where can I find any tests, comparison or proves?

[RaulKite]

I'm not part of gladia. Only saw the tweet and would like to know how they are doing that.

I think the only way is to pay for the service and test it. I'm quite confortable with my whisper results but of course, a improvement is always welcome.

[ryanheise]

From their marketing pages, it looks like they are using techniques borrowed from existing open source projects like #397 , #435 , #937 and others. These work by doing a preprocessing step to clean up the audio to remove silence, filter out background noises and so on. There are other approaches to dealing with hallucinations as well such as #29 , #679 , #1253 and #1838 .

As an aside, the marketing is a bit misleading in some respects. E.g.

Gladia	OpenAI
Word-level timestamps	—
Any audio or video file type. Large files of up to 500MB and 135 mins, extendable on demand	Limited to the most common audio input formats. Files of 25MB and 30 sec max.
JSON, plain, SRT and VTT (for subtitles)	JSON

If Gladia is based on Whisper, then it could inherit all of these things rather than completely reworking it, since they are already implemented in Whisper. If Gladia is based on Whisper, then it also has a "30 sec max" in exactly the same sense that Whisper does.

[dgoryeo]

I just did a quick test with a 4min audio. I'm afraid the result from gladia contained many occurances of hallucination and repetition. I tested it with the free developer account, so perhaps their paid services perform better?

[sboudouk]

Hey @dgoryeo,

Gladia member here, we had some issues right after the release announcement, it should be fixed by now. Could you share the audio you had hallucinations with so we can test on our side ? Here is our support mail:

[email protected]

Thanks ! :)

[ejentos]

There is one more open-source competitor https://ai.meta.com/blog/seamless-m4t/
I haven't been able to get punctuation from transcription, but works pretty well.

[gvijqb]

But this is with non-commercial licence

[SanaArbisoft]

hi @ejentos
I am using whisper-V3 for transcription in my ongoing project and facing transcription related issues (missing words, modified words, and extra/hallucinated words) i need to improve it accuracy how can i do that, guidance would be highly appreciated. I have compared transcription with whisper-V3 and whisper-zero, whisper-V3 comparatively contains more word error than whisper-zero, but i need an open source solution.