Whsiper Causing problem with Long Audios

[Praj-17]

Hi, I am working on a video translation tool where given a video it generates translated subtitles.

Following is the code I am using whiper for.

class TranscribeSRT():
  def __init__(self) -> None:
    model_size = os.getenv("model_size")
    self.device = "cuda" if torch.cuda.is_available() else "cpu"
    self.model = whisper.load_model(model_size, device= self.device)

  def mp3_to_translated_srt(self, mp3_file,destination_language = "es", source_langauge = "en"):

    '''
    Function that transcribes an MP3 audio file and then translates the string into
    a specified language. The function outputs the Python string as an SRT file.

    Module Requirements
    ----------
    GitHub Repo: https://github.com/guillaumekln/faster-whisper
    from faster_whisper import WhisperModel

    PyPI Link: https://pypi.org/project/deep-translator/
    from deep_translator import GoogleTranslator

    Parameters
    ----------
    audio_file: mp3 file that will be transcribed into a Python string using the
    fast-whisper module.

    destination_language: Language that the string will be translated into using
    the deep-translator model and Google Translate. To check all available
    languages run the following function:
    GoogleTranslator().get_supported_languages(as_dict=True)

    Outputs
    ----------
    Original SRT File: writes an SRT file containing the text in its original
    language and timestamps.

    Translated SRT File: writes an SRT file containing the translated text and
    timestamps.
    '''

    #use the WhisperModel to transcribe the mp3 file
  
    if os.path.exists(mp3_file):
      print("This is the mp3 file", mp3_file, "running on ", self.device)
      audio = whisper.load_audio(mp3_file)
    else:
      raise ValueError(f"The Audio file {mp3_file} not found")
    # results = self.model.transcribe(mp3_file, beam_size=5)
    results = whisper.transcribe(self.model, audio, language=source_langauge)

    original_subs = ''

    for idx,segment in enumerate(results['segments']):
      original_subs += f'{idx+1}\n'
      original_subs += str(datetime.timedelta(seconds=segment['start']))
      original_subs += ' --> '
      original_subs += f'{str(datetime.timedelta(seconds=segment['end']))}'

      #write the text in its original language
      original_subs += f'\n{segment['text'][1:]}\n\n'

    

    #translate the text
    #text length is limited to 5000 characters
    translated_subs = ''
    for line in original_subs.split('\n'):
      translated_subs += f"{GoogleTranslator(source='auto', target=destination_language).translate(line)}\n"

    with open(os.path.join(os.path.dirname(mp3_file), os.getenv("default_text_save_file_name") ), mode="w", encoding = "utf-8") as txt_output:
      txt_output.write(results['text'])
    with open(os.path.join( os.path.dirname(mp3_file), os.getenv("defualt_original_subtitles_name")),mode='w', encoding = "utf-8") as sub_output:
      sub_output.write(original_subs)

    with open(os.path.join( os.path.dirname(mp3_file), os.getenv("default_srt_file_name")),mode='w',  encoding = "utf-8") as sub_output:
      sub_output.write(translated_subs)

    print('Translation Complete')
    
    return os.path.join( os.path.dirname(mp3_file), os.getenv("default_srt_file_name"))

I am using the Large Model.

But the problem is it generates a transcript like this

1
0:00:00,00 --> 0:00:02,100000
Introducción a la IA.

2
0:00:02,100000 --> 0:00:09,600000
Introducción a la IA.

3
0:00:09,600000 --> 0:00:15,600000
Introducción a la IA.

4
0:00:15,600000 --> 0:00:23,500000
Introducción a la IA.

5
0:00:23,500000 --> 0:00:28,300000
Introducción a la IA.

Any Suggestions? Or might this be a model hallucination? However, I retired thrice, all having the same results

Here is the audio file I am using
The audio file (extracted from the video) is around 75Mbs and around 1Hr 25 min in length.

Note - It works perfectly for smaller videos of around 2 to 3 minutes.

Thanks in Advance for any help.

[danik9711]

I have the same problem

[glangford]

Your code says

    from faster_whisper import WhisperModel

This discussion board is for openai-whisper which is a different project. So in general you should post questions about faster-whisper over there.

That said, it certainly looks like a hallucination. You might want to install this whisper and run it from the command line. Test if you get the same results with the large-v2 model on the first minute or so.