Finetuning Whisper ASR Model for Translation Tasks

[kikozi2000]

Hi there,

I'm interested in finetuning the Whisper ASR model to perform translation tasks. Has anyone here tried this before? If so, could you provide some guidance on the required dataset format and the process of training the model for this purpose? Any insights or experiences shared would be greatly appreciated.

Thanks in advance!

[phineas-pta]

i haven't done any fine-tuning, but i think you can just follow the official guide, when you load the tokenizer, you change to task="translate"

[WassayS]

@kikozi2000 Hey, did you try fine tuning it for translation tasks?
If yes, please share your experience like did it work

[ExtReMLapin]

At this point, just use NLLB-200 for translation, don't use whisper