load_from_checkpoint giving different validation results

[keitht226]

I'm creating a classifier that first trains a VAE then passes it into a convolutional network. The psudo code below kind of describes it:

class VAE(pl.LightningModule):
# ...

class ConvNetwork(pl.LightningModule):
    def __init__(self, vae):
        # Trying both ways: pass in entire model vs loading checkpoint
        # self.vae = vae
        # self.vae = VAE.load_from_checkpoint(vae)
        freeze_training(self.vae) # sets all params to requries_grad=False

        self.sub_network = nn.Sequential(
            # Mix of convolutional layers, ReLU activations, and Batch Normalization
        )

    def forward(self, data):
         vae_decoded_results = self.vae(data)
         results_that_differ_wildly = self.sub_network(vae_decoded_results)
        

If I train the VAE and pass in the entire model before training the convolutional network, I get good training/validation results. What I would prefer, however, is to train the VAE in a separate script, save off checkpoints, then pass the path of the checkpoint into the convolutional network. Then in the convolutional network's init I load the vae network, freeze training on it, and proceed to train the convolutional network. When I do this, my training results seem okay, but my validation results are all over the place. Some things I've checked:

• After loading the VAE from a checkpoint, I verified the model parameters perfectly match the VAE that produced the checkpoint.
• In my forward function for the convolutional network I call the VAE's forward function. The results at this step differ by less than 1% between loading a checkpoint and passing in an entire model.
• After passing the VAE forward() results into the first stage of my Convolution network (consists of some convolution layers, ReLU activations, and batch normalization) I get very different results.

I can't for the life of me figure out why using the results from a loaded model would so wildly differ from the results of a model I train and pass in all in one script, especially when the parameters and vae output appear to match. I'm sure I'm just missing something stupid.

[kielnino]

Just a wild guess, but maybe the model is in train-mode after loading from a checkpoint. Have you tried model.eval() in addition to setting the requires_grad? I'm thinking about BN layers and so on, where this is important (see here).

[tomerperetz]

Hi. I have the same problem exactly.
I'm loading the model using load_from_checkpoint(), with adding model.freeze(), model.eval(), but results are different for identical input.
any ideas?

[Chrixtar]

Same problem here.
@jakub-h, @tomerperetz , or anyone else with a solution by now?

[tomerperetz]

hi @Chrixtar, in my case the problem was rooted in my data loader. I used a certain function that changed my DataFrame sorting, hence results were different (I'm working on time series).
Once I figured it out, model results were consistent.

[lanlanlan3]

i have the same problem. #13482

[ruiyan1995]

i have the same problem.