Finetune whisper with prompts

[AvivSham]

Hi All,
How are you?
I’m trying to finetune Whisper by resuming its pre-training task and adding initial prompts as part of the model’s forward pass. I saw this amazing tutorial, however, it does not contain a section about using prompts as part of the fine-tuning dataset.
My motivation is we witness that Whisper is not acting as expected when transcribing with prompts. Sometimes the output is blank text and on other occasions the output text contains reoccurrence. We want to solve such behaviors by fine-tuning Whisper with prompts.
Any help will be appreciated!

[phineas-pta]

i dont have any exp with finetune but here my thoughts

suppose the transcript isABC and prompt is XYZ

in the section "Load WhisperTokenizer" of the finetune guide, the transcript must be tokenized into something like

<|startoftranscript|><|transcribe|>ABC<|endoftext|>

after digging into source files whisper/tokenizer.py and decoding.py, i think the tokenized prompt must prepend transcript, something like

<|startofprev|>XYZ<|startoftranscript|><|transcribe|>ABC<|endoftext|>

so i guess u have to find a way with the tokenizer to make this

[AvivNavon]

@phineas-pta Thanks for your response. What should be the labels in this example (in terms of masking etc.)? IIUC the length of labels must match the decoder_input_ids.

[phineas-pta]

what's decoder_input_ids ?