Difference between finetuning and restart training

[matr1x-1]

My question is about the difference between finetuning and restarting. As far as I have understood, it seems that

• Finetuning: Here we can use an already trained model and train both networks on the new data, i.e. finetune the model for this or we can fix part of the model, e.g. the descriptor, and change only the rest.
• Restarting: This is where I am a bit stuck. It seems that if I restart from a checkpoint but have added new data, it's not able to learn the new data. My assumption is that this is meant to be just for training a bit further if the training has crashed.

Is this correct and if I want to train on new data, do I really have to use finetuning instead of restarting? Or could there be another reason why the model does not seem to learn the newly added data?

Thank you for your support!

[iProzd]

@matr1x-1 Yes, you are correct about the fine-tuning procedure.

As for restarting, it is often used when one wants to continue training if the training has crashed. If you add new data before restarting (such as new systems added in the input.json file), the restarting process will use the modified input script and only reload the model weights, starting training from the crashed steps (provided the model parameters are consistent). This means that the newly added data will be used.

You can even replace all the old data with new data systems to confirm this. Additionally, there's another approach called 'init-model' that allows you to only reload the model weights and start a new training session, as described here.

[matr1x-1]

@iProzd Thank you for your reply! It seems that the 'init-model' approach works a lot better for me for some reason. Do you have any idea why that could be? I haven't changed anything except for adding some more data.

[matr1x-1]

But even with the 'init-model' approach, once I continue training the model, the RMSEs are actually quite a bit higher than what they were in the previous run (not as high as without any training but still). Here an example:

1. Training from checkpoint with init:

2. Training again with model from step 1:

Shouldn't the start in 2. be relatively close the end of 1. in terms of RMSE if no changes in the data were made?