Using Whisper to generate timestamps for existing transcriptions

[fredbru]

Hi all! I'm interested in the possibility of using Whisper for aligning text to audio, in a situation where I have accurate transcriptions for a number of audio files, but no timestamps on these transcriptions. I would like to use Whisper just for generating timestamps for the existing text. These timestamps can be in the standard Whisper phrase-level format.

Short of a manual merge process using complete Whisper transcriptions and the existing transcriptions, is there any built-in way of doing this in Whisper itself? If not, I'd be interested if anyone had any suggestions on how to approach this. Thanks!

[EtienneAb3d]

Provide me with a pair of file example (with/without timestamps), and I will find it funny to write such a sync tool for you.
:)

[lostmsu]

@EtienneAb3d how do you do line splitting?

[EtienneAb3d]

@lostmsu

1. Whisper is deciding how to put the timestamps in the SRT file. WhisperTimeSync then makes a simple projection of these timestamps over the proper text TXT/SRT file.
2. if the proper TXT/SRT file already contains new line chars, they are kept in the resulting projection (inside each timestamps text).

[lostmsu]

I find it interesting that Whisper appears to properly capitalize the first letter of new lines, but does not provide new line tokens.

[BladianMC]

@EtienneAb3d would it somehow be possible to use this in collaboration with whisper_timestamped's existing feature of giving word by word timestamps rather than sentence by sentence?

[EtienneAb3d]

@BladianMC
It works if you have real word timestamps, like WhisperHallu does = one timestamp for each word over a proper text.
By default, Whisper is repeating each line several times to visually highlight each word one by one. WhisperTimeSync has to be adapted to be able to process such repeated lines. But, yes, it's possible...

[aeciorc]

@EtienneAb3d I'm wondering if one could also hack whisper's prompt/prefix for this purpose? Maybe by feeding the groud-truth there, we can improve accuracy to a point where aligning the transcription with the script is not necessary. No idea how this would work for long content tho

[EtienneAb3d]

@aeciorc, prompt will only accept (very) short texts, and could bring you with unwanted effects. It will have a real effect only on the first 30s window of the file to be processed. One of the unwanted effect could be that Whisper will consider the first part of the spoken text as a repetition of the prompt part, thus to be removed (not produced in the output).
As far as I understand it, prefix will enforce Whisper to consider that the content of the prefix part was already processed in the sound to be analysed, thus will try to transcript what's next in the file.

[Evermaple]

Stable-ts meets your needs now.

[BladianMC]

Which section specifically allows for this?

And do you think it'll work for 1000 word texts?

[Evermaple]

Yeah,it did work for 1000 words in Japanese. I have tried.
Use the align() function like "result = model.align('audio.mp3', text, language='en')"
The repo link is https://github.com/jianfch/stable-ts.
Good luck to you.

[BladianMC]

Worked really well thanks for the help

[aehlke]

Is there a solution I can run in iOS

[Evermaple]

这是一封自动回复邮件。已经收到您的来信，我会尽快回复。