the question about finetune whisper

[LithiumZhou]

Hi Yuan,

I'm very sorry to disturb you again. I really want to know how to fine-tune Whisper for Audioset and ESC-50.

[YuanGongND]

Our code freeze the backbone and tune a few layers on top of its intermediate representations.

In the paper. we did try to finetune the entire model, which leads to slightly better result. The tradeoff is the finetuned Whisper would lose its ASR ability.

-Yuan