Setting prefix results in empty transcription

[SinanAkkoyun]

Hi! When providing the prefix decoding option, the model spits out an empty transcription.
I supply the prefix as a string. Without the prefix, it transcribes as usual.

When I supply a whitespace, transcription quality decreases but outputs most often exist. When I supply a full sentence with "." at the end, the models output is empty. Same for sentences or words without "."

How can I fix that? Thank you

[SinanAkkoyun]

@jongwook Must the prefix contain the currently processed audio sample? Or can the prefix be arbitrary?

[jongwook]

With a "." in the prefix, the model assumes the segment starts with a period mark, and my guess is that the model hasn't seen many transcripts in the training data that starts a line with a ".". Even when it does, it's more likely a line containing "." only, hence an empty prediction. The same logic goes for the whitespace prefix, although the distribution may be slightly different.

I think you might be looking for prompt instead of prefix (cf. #117 (comment))

[SinanAkkoyun]

@jongwook
Thank you for your reply. I am trying to get "real-time" small chunk transcription to work. As you said, this is what your prefix was intended for.
Could you please help me on how to use the prefix option to supply the previously transcribed text (mid-sentence)?
Thank you very much

[SinanAkkoyun]

This is how I understand the prefix:
Given a 3s audio input clip: Whisper transcribes it.
Now, you can add additional 2s of audio on top of the 3s (have to provide everything), give it the prefix of the previous 3s and let it transcribe the rest. This would make live inference much faster.
There are two shortcomings:
1: when the new audio is only half a word, whisper will hallucinate and actually stick with the hallucination, because one would pass the prefix
2: You must provide the whole audio clip, attention is over the whole clip + positional embeddings still need to match, thus after 30s one must restart the streaming (which really is not all that bad)

Is that true and did I understand the prefix param? What ideas did you have for live transcription?

Please correct me if I'm wrong, I am just super eager to get real time transcription to work :)

[zw123han]

#1358 If I understood your issue correctly, there was a small bug when using default DecodingOptions enabling timestamp generation. Let me know if this fixes it.