How to accumulate metrics for multiple validation dataloaders

[potipot]

❓ Questions and Help

What is your question?

How to accumulate metrics for multiple validation dataloaders separately? Currently the metrics are accumulated for all dataloaders simultaneously.

Code

The validation step accepts dataset_idx parameter when running validation with multiple dataloaders.

def validation_step(self, batch, batch_idx, dataset_idx: Optional[int] = None):

However I'm not sure how to update the metrics separately for each dataloader. Would I have to create separate metrics, one for dataset A and second for B? Or maybe my metric could accept the dataset_idx parameter to know for which ds it should log given output.

This however wouldn't work with pl factory metrics like average precision, since they are dataset agnostic?

def update(self, preds: torch.Tensor, target: torch.Tensor):

Not sure how to approach this.

[SkafteNicki]

You would have to create seperate metrics per validation dataloader (similar to how you need seperate metrics for train/val/test). Something like this could maybe work for you

def __init__(self, ...)
    ...
    self.val_metrics = nn.ModuleList([pl.metrics.Accuracy() for _ in range(n_val_dataloaders)])

def validation_step(self, batch, batch_idx, dataset_idx):
    ...
    self.val_metrics[dataset_idx].update(preds, target)

[potipot]

Maybe it would be a good idea to add it to the documentation or as an example?

[SkafteNicki]

So there already is this note in the documentation:

@potipot do you think it is enough to just extend this note to also include multi-val/multi-test case?

[potipot]

I was trying to find some information about this in the docs, but phrases "multiple dataloaders" or "metrics for multiple dataloaders" didn't get me what I was looking for.
Maybe it would be beneficial to add your response with some extension like:

NOTE: if you want to separately collect metrics for multiple dataloaders you have to create seperate metrics for each validation dataloader (similar to how you need seperate metrics for train/val/test).
By default pytorch-lightning will aggregate metrics across multiple dataloaders into one.

def __init__(self, ...)
    ...
    self.val_metrics = nn.ModuleList([pl.metrics.Accuracy() for _ in range(n_val_dataloaders)])

def validation_step(self, batch, batch_idx, dataset_idx):
    ...
    self.val_metrics[dataset_idx].update(preds, target)

I have not yet tested your example but this is just something that I was hoping to find in the documentation :).

[SkafteNicki]

I think it is a great idea to add a second note. Would you be up for submitting a PR @potipot ?

[potipot]

I'm still not sure though if aggregation is the default behavior with the new metrics API. I'm trying to verify this or maybe you can confirm?

[SkafteNicki]

If you have a metric object for each individual dataloader, then the metric should accumulate separately.

[potipot]

But if we just do

def __init__(self, ...)
    ...
    self.val_metrics = pl.metrics.Accuracy()

def validation_step(self, batch, batch_idx, dataset_idx):
    ...
    self.val_metrics.update(preds, target)

they would be evaluated across all dataloaders and computed only at the end?

[SkafteNicki]

But if we just do

def __init__(self, ...)
    ...
    self.val_metrics = pl.metrics.Accuracy()

def validation_step(self, batch, batch_idx, dataset_idx):
    ...
    self.val_metrics.update(preds, target)

they would be evaluated across all dataloaders and computed only at the end?

Yes exactly :]

[zmurez]

It would be nice if metrics had a flag similar to LightningModule.log(add_dataloader_idx) that would automate this for you. Or better yet, if log(add_dataloader_idx=True) would automatically propagate to metrics that are logged. Is there a reason this wouldn't work? Any chance it is coming soon? Thanks!

[SimJeg]

Hello,

Is there any update on this issue ?

I have multiple validation loaders corresponding to different datasets (A, B, C...) and I would like to log the individual metrics (metric_A, metric_B, metric_C...) as well the average across datasets (metric_average). It is possible to create a metric_average object, and call the update method several times, but no to log this metric several times in the validation_step.

What would be the best solution ? and I would also like to do it with the loss ^^
Thank you