Interleaving training of several models

[turian]

I am a long-time lightning fan, on the other hand in the past I have gotten bitten by how opinionated it is about training loops. I have a new project in pure torch, and I'm debating whether I can easily migrate it to lightning. I have a related question here: #14228

I have a pretraining model which trains only training data to minimize a self-supervised loss.
I have a downstream model which uses the pretrained model (either frozen or finetuning it), and trains on dev data and evaluates on test data.

I want to explore multiple training strategies, including pretrain then downstream (simple) and interleaved pretraining and downstream training. What are best practices for using lightning in this setting?