Learning Rate for whisper-large v3 fine-tuning

[omarabb315]

I am fine-tuning whisper-large v3 and I wonder what was the learning rate used for pre-training this version of the model to help me choose the right value of LR in fine-tuning process, however this is my setup, I have about 3300 hours of training data and here are some experiments:

Train/Loss curve Using LR=5e-6 (with linear schedular) and batch-size=128 for first 0.6 epoch:

Train/Loss curve Using LR=4e-6 (with linear schedular) and batch-size=256 for first 1.0 epoch:

which setup do you think is better?

[80boys]

No one is helping?

[yuriyvnv]

Hello, I hope you are doing well. In this case, as you have plenty of data to train, I will suggest a smaller learning rate like 1e-6 or even 1e-7, however, with more epochs of training. Also, the train loss curves will always be decreasing; what you actually want is the curves from validation, which are the ones stating how well your model will perform on unseen data.
Usually, if you have a computer for a large batch size as yours, I would recommend a learning rate of 1e-7, 8 epochs, and control the validation if the curve is not increasing. If you have a steady decrease in validation loss, I would increase the number of epochs.
You can always refer to this repository https://github.com/vasistalodagala/whisper-finetune?utm_source=chatgpt.com where they state the maximum learning rate used for pre-training was 0.0002 so yours should be at leas 40 times lower.
Hope this helped.
Cheers