Finetuning Models/Multi-round Training Best Practices

[ardywibowo]

Hello,
I am just beginning to have a look at using Pytorch Lightning for my next project. I was thinking about some cases where I may need to finetune my model or use my model differently between each round of training.

For example, I may want to finetune on a different dataset after training on an initial dataset, I may want to enable quantization aware training after initially training in full-precision, or I may want to add additional modules to my model after an initial training round.

I typically do this by creating a new script for the different cases, but I was wondering if there are best practices that Pytorch Lightning enables that would make these things easier.

The main issue I see with these is in organization. It's sometimes hard to keep track which models you feed in to which script (and which version of the script).

Thanks so much!