Adding voiceovers to Manim videos directly in Python using Whisper

[osolmaz]

Manim is an open source Python math animation library, created by 3blue1brown, with multiple forks, one of them being https://manim.community/. In Manim, you create math animations by writing Python code.

Creating a full-fledged video is a tedious process, one of the most time consuming parts being recording the voiceover and aligning it with video in a video editor.

I had previously created a Manim plugin called Manim Voiceover with let you add AI voices. But it was missing a crucial feature—being able to record an actual non-AI voiceover in Python. This was hard to accomplish, because one often needs to trigger animations at specific words in the voiceover, and you needed a good STT engine for that.

The only good ones were behind cloud services like Azure, and it is a pain to make every user of the library set up an account with a credit card. Simply not viable for an open source project.

Whisper came to the rescue, because it allows me to perform high quality STT locally. Word timestamps are apparently a bit janky, but found another project that improved that a lot, and I had the ability to record voiceovers the way I wanted in no time.

Thanks to Whisper, I now don't need to use a video editor to create voiceovered math animations, and can develop videos 100% in Python. The productivity increase is substantial: at least ~2x faster video production:

https://www.youtube.com/watch?v=HawoHpR45pc