Fine Tuning a Fine Tuned Model

[yilmazay74]

Hi Everyone,
I wonder whether it is a good idea to fine tune a fine tuned whisper model?
Because of hardware constraints, we cannot use all the data for finetuning train at once.
Thus, I thought, training by utilizing our train data in batches, and generating intermediate models could be a solution for our problem.
Does this solution have any side effects or drawbacks?
I appreciate your recommendations on this issue?

[yilmazay74]

Just to make my question more clear:
Lets say I have a dataset, I divide this into 3 and then I use the first batch for finetuning whisper-base, and I get ft_base_model1.
And then I repeat finetuning by using second batch of data, this time by using ft_model1 as the base model, and generate ft_model2 and so on. This way, can I expect my final fine tuned model to have the same performance as if I had fine tuned with my all data at once?

[phineas-pta]

what kind of hardware constraints ?

u can fine tune with complete dataset, just adjust batch size according to hardware

[yilmazay74]

We have just one GPU, and 32 GB RAM. When I gave a dataset of about 50K, in the beginning of the training process while loading the data, the memory increases rapidly and after exceeding 32 GB RAM it starts using swap, which slowes down the process drastically, finally when it exceeds the max amount of swap it crashes. To aviod this I dropped down the batch size to 1, but it did not help, though the crash happens a little bit later.
Secondly; I thought with fine tuning, we could just train the model with a enough amount of data, lets say, 20K, and we don't need to retrain with all data, am I wrong? Isn't it the real logic behind fine-tuning ?

[phineas-pta]

the batch size to 1, but it did not help

meaning the model size too big for your hardware, cutting dataset wont help

train the model with a enough amount of data

good luck determine such threshold