Automatic caption correction

[anton-codes]

I'm using whisper to generate .vtt files for the .wav files that are generated by a TTS engine. I have the actual text that's being spoken by the TTS, I just need to use the timings to create the subtitles.

Challenge: Identify incorrect transcription and fix the mismatches with original text while maintaining time-code alignment.

Am I in the completely uncharted territory here? Anyone know any helpful NLP libraries or perhaps there is something within whisper?

example original.txt
well, they will be gone soon

example post.vtt
WEBVTT

00:00.000 --> 00:00.620
well, they are

00:00.620 --> 00:01.020
be gone sun

[EtienneAb3d]

@anton-codes
What you are asking for is here:
https://github.com/EtienneAb3d/WhisperTimeSync
;-)

[anton-codes]

Thanks, this was easy to set up and did exactly what I needed.