Automatic dubbing system based on Whisper

[jordimas]

Hello! Open dubbing is an AI dubbing system which uses machine learning models to automatically translate and synchronize audio dialogue into different languages. It uses Whisper for the transcription task.

Details: https://github.com/jordimas/open-dubbing

[toanhn]

It is so great, thank you so much.

[edsu]

Have you done any analysis of how accurate it is?

[jordimas]

Hello! To give a bit of background there several models involved including stt (Whisper), voice separation, gender classification, translation, text to speech. Each one of these systems introduces errors stt (WER), translation (BLUE), etc. There is no single metric that can answer this question and also this changes a lot from language to language. What I am doing is human evaluation with the goal that the target dubbing helps to understand the content and humans gives thumps up on the small sets that I have tested. As summary, if you use case is dubbing for understanding it should work. By no means for example is a replacement for professional dubbers.
Hopefully this answers your question.

[edsu]

Yes, it means you haven't done any analysis of how accurate it is :-)