Compiled model, self.log() returns inconsistent values ? #17165
Unanswered
w2ex
asked this question in
Lightning Trainer API: Trainer, LightningModule, LightningDataModule
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I am using a lightning pipeline to train a CV model (classification task). I am logging the accuracy on my validation set, as well as the cross-entropy loss to monitor the evolution during training.
I have been trying to switch to pytorch2, lightning 2.0 and compiled model. I am now seeing values of validation accuracy = 1.0, cross_entropy = 1e-8, where I have an accuracy of 0.9995, and a cross_entropy of 5e-3 in the exact same configuration without compiling the model.
It is not that the performances are simply better : when loading the checkpoint of the compiled model in an uncompiled one, and freezing all of the model, the performances appeared to be in fact the same as the ones logged by the uncompiled model.
So it has to do with the values logged. Can it be the compilation mess up the accuracy/cross_entropy computation ? I do not know if I need to dig on pytorch or lightning side.
What is really weird is that the values of loss, accuracy and all remain consistent with each other...
Has anyone here noticed a difference between the performances logged by compiled or not models ?
Thanks :)
Beta Was this translation helpful? Give feedback.
All reactions