Do you need to have more than one accuracy metric for different splits?

[nateraw]

Howdy! 🤠 I came across something in the PytorchVideo repository that I wasn't so sure about. They're using 2 accuracy metric classes in their LightningModule for train and val. I've been using 1 for everything and its making me wonder if I was wrong for some reason?

Haven't dove into that code in a while...are the metrics being accumulated separately across the different loops, or are they accumulated together?

Here's the snippet in question...

 class VideoClassificationLightningModule(pytorch_lightning.LightningModule): 
     def __init__(self, args): 
         """ 
         This LightningModule implementation constructs a PyTorchVideo ResNet, 
         defines the train and val loss to be trained with (cross_entropy), and 
         configures the optimizer. 
         """ 
         self.args = args 
         super().__init__() 
         self.train_accuracy = pytorch_lightning.metrics.Accuracy() 
         self.val_accuracy = pytorch_lightning.metrics.Accuracy() 

I would normally just have self.accuracy and use that in both the train and val steps. Is that incorrect?

[ananthsub]

I would normally just have self.accuracy and use that in both the train and val steps. Is that incorrect?

I would say this is incorrect. You don't want to share the same metric state across training and validation. Otherwise you're mixing data, which will give misleading results compared to having dedicated train and val metric instances. Additionally, depending on how you log the metric value, the same instance could get reset at the end of the training and validation epoch. if you're running validation more frequently (e.g. using val_check_interval) this will be another problem.

cc @SkafteNicki @Borda @edenafek should we add this to the documentation for metrics in lightning? are there ways we can make this clearer in the API itself?

[nateraw]

imho, I think it makes sense for the Accuracy metric to know I'm in the train, val, or test steps and manage the data separately for each. If its not already doing this (as you've eluded to above), then I've been doing this super wrong for a while!! 😅 Yikes!

I'll wait for the folks above to weigh in before marking this as the answer, if it is the way you say it is 😄 Thank you!!

[nateraw]

It may be misleading to have the behavior I describe above, though, since its supposed to work without lightning. Might lead to confusion for those not using it with lightning.

[ananthsub]

imho, I think it makes sense for the Accuracy metric to know I'm in the train, val, or test steps and manage the data separately for each

I don't think the torchmetric definitions / implementations should know what context these classes are being used in. This keeps them generic and focused solely on the metric definition as opposed to training use cases.

I think it's the responsibility of the application that's using these metrics to handle creating different instances and ensuring that the right data is being fed to them.

[nateraw]

I think you're right! I'm always on the side of making things more 'dumb easy', but in this case I don't think its appropriate given torchmetric is a standalone thing. Closing this.

[SkafteNicki]

I think @ananthsub have covered both the why (to not mix metric states) and the main reasons for why we do not want this to happen automatically in lightning:

• Torchmetrics should work in both lightning and native torch
• Hard to know when to correctly reset to not mix states
  The first note on this page https://torchmetrics.readthedocs.io/en/latest/pages/overview.html mentions that users should reinitialize per mode.