Are there any examples on how to use the Prefix?

[rokaHakor]

I want to do real-time audio transcription/translation and have tested doing so with 5-10 second chunks. However the quality is much worse doing it in chunks compared to recording the full audio then running it through whisper. Even using the prompt with the previously transcribed text was still noticeably worse.

It seems like using the Prefix would help with this however it does not seem to be working properly or I am using it incorrectly.
I am transcribing the first 5 second chunk then using that transcription as the prefix for the next input while adding the new 5 second chunk to the previous chunk for a 10 second input, and so on until it hits the 30 second limit. However doing that only results in empty output or repetitions of the initial 5 second transcription with nothing new.

Are there any examples on how to use the Prefix option properly?

[phineas-pta]

i don't know anything about prefix but based on README i think it'd be like:

import whisper
model = whisper.load_model("base")
audio = whisper.load_audio("audio.mp3")
mel = whisper.log_mel_spectrogram(audio).to(model.device)
options = whisper.DecodingOptions(prefix="<previous chunk transcription>")
result = whisper.decode(model, mel, options)
print(result.text)

the problem is how to feed previous chunk into prefix, a loop maybe ?

[rokaHakor]

I'm trying this in a loop and for whatever reason it only decodes once or rather it only decodes once properly. I've got a 30 second sound bite that I've divided up so the first one is the first 5 seconds, the second is the first 10 seconds, the third is the first 15 seconds, etc. However it only ends up translating once or twice.

def main(model="medium"):
    print("Loading model...")
    model = whisper.load_model(model, device="cuda")

    text = ""

    for i in range(1,7):
        print(i)
        audio = whisper.load_audio("asoundbit{}.wav".format(i))
        audio2 = whisper.pad_or_trim(audio)
        mel = whisper.log_mel_spectrogram(audio2, device=model.device)

        options = whisper.DecodingOptions(task="translate", prefix=text.lstrip())
        result = whisper.decode(model, mel, options)
        
        decoded_text = result.text

        if decoded_text:
            text = decoded_text
            print(f'{datetime.now().strftime("%H:%M:%S")} {decoded_text}')


if __name__ == "__main__":
    main()
    pass

[phineas-pta]

maybe try specify language

[rokaHakor]

Even setting the language it still has the same issue.

[phineas-pta]

maybe problem with translate ? did u try transcribe only

[rokaHakor]

It had the same issue with transcribe

[rokaHakor]

If I only process the full 30 second sound bite then it translates the entire thing just fine.

[casic]

The whisper frame decoding window is 30 sec by default 1, Try with 2 , 3 minutes and see result 2. why audio = whisper.load_audio("asoundbit{}.wav".format(i)) is in the loop ??? На сб, 29.07.2023 г. в 7:43 ч. Irmuun ***@***.***> написа:

…

Even setting the language it still has the same issue. — Reply to this email directly, view it on GitHub <#1329 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYTSUMLOJFP2SWADFXPQRDXSSIGPANCNFSM6AAAAAAX2IGY2U> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rokaHakor]

I'm trying to use the prefix to real-time translate audio while outputting the transcription/translation in ~5 second chunks.
It should be possible based on this
#117

[phineas-pta]

it seem we dont need low level DecodingOptions

import whisper
model = whisper.load_model("base")
result = model.transcribe("audio.mp3", task="transcribe", prefix="<previous chunk transcription>")
print(result["text"])