Subsai: A subtitles' generation tool powered by Whisper and its variants

[absadiki]

Hello everyone 👋 ,

A bit late to the party but this is a small tool to generate subtitles based on the original Whisper and some other implementations from the community.

I decided to wrap them all in one package to easily switch between them.

As of now, the supported Models are:

• openai/whisper

  • Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identification.

• linto-ai/whisper-timestamped

  • Multilingual Automatic Speech Recognition with word-level timestamps and confidence

• ggerganov/whisper.cpp (using abdeladim-s/pywhispercpp)

  • High-performance inference of OpenAI's Whisper automatic speech recognition (ASR) model

    • Plain C/C++ implementation without dependencies
    • Runs on the CPU

The package also comes with other features:

• A simple Web UI
  • Fully offline, no third party services
  • Works on Linux, Mac and Windows
  • Lightweight and easy to use
  • Supports subtitle modification
  • Integrated tools:
    • Translation using xhluca/dl-translate:
      • Supported models:
        • facebook/m2m100_418M
        • facebook/m2m100_1.2B
        • facebook/mbart-large-50-many-to-many-mmt
    • Auto-sync using smacke/ffsubsync
• Command Line Interface
  • For simple or batch processing
• Python package
  • In case you want to develop your own scripts
• Supports different subtitle formats thanks to tkarabela/pysubs2
  • SubRip
  • WebVTT
  • substation alpha
  • MicroDVD
  • MPL2
  • TMP
• Supports audio and video files

Check out the Github repo for installation instructions and more details.

[dgoryeo]

Great thinking @abdeladim-s !!!
Have you considered to add WhisperX, and stable-ts to the package? They both are good compliments to the other variants.

[dgoryeo]

Good call @abdeladim-s. What are your thoughts on pairing your frameowrk with a VAD for long audio/videos? At the moment in my workflow I use a VAD ahead of the transcriber --calling transciber on VAD produced segments/chunks. I am wondering what would be the best way to use your framework with a VAD. I'm very interested to hear your thoughts on that. Thanks.

[absadiki]

If I understand your point well @dgoryeo, you are using a VAD to cut your long files into small pieces before the transcription ?
If that's the case you can create a text containing the paths to your files, and run subsai from the command line with that file as input. It will transcribe the files one by one. (that's the batch processing I described here).
The problem in your case is when you will want to merge them again into one file.
how do you deal with that in your current workflow ?

[dgoryeo]

Hi @abdeladim-s , at the moment I am using 2 different technics for VAD (cut, merge, retime) that I have found quite robust for long audios:

• (1) First one which was developed by @octimot : transcribe only certain time intervals . I pass the VAD chunks (times) as a list to be used as intervals. I have tested this with SileroVAD as well as with inaSpeechSegmenter. I like the simplicity of inaSpeechSegmenter but the model has only been trained for French language I understand.

• (2) The second method which was developed by @anon_entity: WhisperWithVAD .

cheers

[absadiki]

Yeah interesting workflow @dgoryeo 👍

I had actually an idea where you can create a script using subsai and silero-vad to automate all of that: instead of cut, merge and re-sync,
you can cut, keep track of the start time of the chunk, then when you transcribe the chunk, just add that time to the segment and the transcription will stay synced.
You can also save the progress along the way so you can resume from where you left.

I made a simple script to illustrate this idea.

Hope it will help.

[dgoryeo]

Thanks @abdeladim-s , the colab is very helpful. I also like the idea to add the timestamps to the filename of saved VAD chunks. That way one can keep the saved chunks for multiple post-processing scenarios --and to use the filename to adjust the timings for each chunk. Will try it later today, will let you know.

[mimbimbo]

@abdeladim-s any idea on how to install this on windows? Thanks!

[absadiki]

@mimbimbo You can install it on Windows following the installation instructions here: https://github.com/abdeladim-s/subsai#installation
If you find any problems during the installation, feel free to open an issue and I will try to help you.