Low MulticlassAccuracy with average "macro" and unseen classes might be unexpected

[phisad]

Version: torchmetrics 0.10.3

I recently used the "macro" average because I had a class that had a majority of cases in the dataset. Now, I also have a class that never occurs in the dataset. I was able to fix the issue with the majority class. Still, the accuracy scores were rather low.

Interestingly this comes from the fact, that the measure calculates a 0% accuracy for classes that are never seen during training. I think this might confuse people. I would rather (a) ignore that class for the computation or (b) assume a 100% accuracy for that class.

from torchmetrics.classification import MulticlassAccuracy

m = MulticlassAccuracy(num_classes=4,
                       multidim_average='global',
                       average="macro")
gt = torch.Tensor([[1],
                   [2],
                   [3],
                   [1],
                   [2]]).long().squeeze()
preds = torch.Tensor([[0,1,0,0],
                   [0,0,1,0],
                   [0,0,0,1],
                   [0,1,0,0],
                   [0,0,1,0]]).float()
m(preds, gt)
tensor(0.7500)

With micro we get the expected result.

from torchmetrics.classification import MulticlassAccuracy

m = MulticlassAccuracy(num_classes=4,
                       multidim_average='global',
                       average="micro")
gt = torch.Tensor([[1],
                   [2],
                   [3],
                   [1],
                   [2]]).long().squeeze()
preds = torch.Tensor([[0,1,0,0],
                   [0,0,1,0],
                   [0,0,0,1],
                   [0,1,0,0],
                   [0,0,1,0]]).float()
m(preds, gt)
tensor(1.)