There were missing keys in the checkpoint model loaded: ['proj_out.weight'].

[dmehta-huro]

I am trying to fine tune the model using the technique shown in this tutorial: https://huggingface.co/blog/fine-tune-whisper

My dataset is much larger and my PC cannot handle it all at once so I am training it in batches. I initialized the training_args and the trained as shown in the code. I removed the train_dataset and eval_dataset dataset from the trainer and add that later since the batches of data keep changing.

Here is my training procedure:

for batch in batches:
    # Update the trainer with the new datasets
    trainer.train_dataset = batch["train"]
    trainer.eval_dataset = batch["test"]
    trainer.train()

The training works fine for the first batch but for every batch after that, I get a warning message saying There were missing keys in the checkpoint model loaded: ['proj_out.weight'].

This is not an error and does not stop the training but I still wanted to confirm that the model is being trained correctly or not. This warning message also led me to wonder that is the model being trained on the 2nd batch using the updated weights after training on the first batch, or is it just using a fresh set of default weights?

Thanks

[marziye-A]

Hi,
i face the same exact problem. did you find any solution?

[dmehta-huro]

No I did not. Hopefully someone from OpenAI can answer it. @jongwook @sanchit-gandhi would appreciate it if you can help.

[dmehta-huro]

I added save_safetensors=False to the training args and it seems to have solved the issue. I am not sure if it is the right way though.

[ryanheise]

As it looks transformers related, you might try posting an issue here:

https://github.com/huggingface/transformers/issues

It also might be worth trying this workaround:

https://discuss.huggingface.co/t/unable-to-load-checkpoint-after-finetuning/50693

[huangruizhe]

I got the same exact issue as well. I am not able to find model.proj_out in this file: checkpoint-xxx/model.safetensors.index.json

[huangruizhe]

This may be related: https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/convert_openai_to_hf.py#L237
The proj_out layer is sharing weights with the decoder's embedding layer

[dmehta-huro]

I added save_safetensors=False to the training args and it seems to have solved the issue. I am not sure if it is the right way though.