How to use whisper.utils.get_writer with a pretrained model

[itailulu]

Is it possible to use the get_writer util with a pretrained model?
I have this code which outputs the transcription as text. I want to get VTT or SRT but couldn't find how to use the get_writer util with it:

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

SAMPLING_RATE = 16000
has_cuda = torch.cuda.is_available()
model_path = './pretrained-model-path'

model = WhisperForConditionalGeneration.from_pretrained(model_path)
if has_cuda:
    model.to('cuda:0')

processor = WhisperProcessor.from_pretrained(model_path)
audio_file_path = 'audio.mp3'
audio_array, original_sampling_rate = librosa.load(audio_file_path, sr=None)
audio_resample = librosa.resample(audio_array, orig_sr=original_sampling_rate, target_sr=SAMPLING_RATE)
input_features = processor(audio_resample, sampling_rate=SAMPLING_RATE, return_tensors="pt").input_features
if has_cuda:
  input_features = input_features.to('cuda:0')
predicted_ids = model.generate(input_features, language='he', num_beams=5)
transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True)

print(f'Transcript: {transcript[0]}')

[glangford]

Have you tried
transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True, decode_with_timestamps=True)

[glangford]

Generate VTT file from speech to text
https://datascience.stackexchange.com/questions/128471/generate-vtt-file-from-speech-to-text

[itailulu]

Have you tried transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True, decode_with_timestamps=True)

It makes no change. The result is a list with one element - the transcription (string) and then and error in the last line:

word_options = {
    "highlight_words": True,
    "max_line_count": 1,
    "max_line_width": 42
}
vtt_writer = get_writer(output_format='vtt', output_dir='./')
vtt_writer(transcript, audio_file_path, word_options)

Traceback (most recent call last):
  File ".venv/lib/python3.9/site-packages/whisper/utils.py", line 87, in __call__
    self.write_result(result, file=f, options=options, **kwargs)
  File ".venv/lib/python3.9/site-packages/whisper/utils.py", line 231, in write_result
    for start, end, text in self.iterate_result(result, options, **kwargs):
  File ".venv/lib/python3.9/site-packages/whisper/utils.py", line 182, in iterate_result
    if len(result["segments"]) > 0 and "words" in result["segments"][0]:
TypeError: list indices must be integers or slices, not str

[itailulu]

Generate VTT file from speech to text https://datascience.stackexchange.com/questions/128471/generate-vtt-file-from-speech-to-text

Doesn't work
I get ...is not a valid model_id
in
transcriber = pipeline(model=model)

[glangford]

Ok. I don't use this approach (I just use whisper directly) but I thought I would try a couple of suggestions since no one else has responded yet.

[itailulu]

Thanks :) Much appreciated

…

On Sat, 25 May 2024 at 23:57 Glenn Langford ***@***.***> wrote: Ok. I don't use this approach (I just use whisper directly) but I thought I would try a couple of suggestions since no one else has responded yet. — Reply to this email directly, view it on GitHub <#2188 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGB7C2ICBRLE73G5ONKRL3ZED3LDAVCNFSM6AAAAABIIHCJMOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNJYGEYTO> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[phineas-pta]

no, u have to write your own function to export subtitle