Question about the intended behavior of Accuracy with average="none" and nan results

[phiyodr]

Hi,

I'm using your Accuracy metric without averaging and have difficulties with nan results, as I want to process the tensor further.

Example with 4 binary labels (cat, dog, horse, cow) and 3 samples:

from torchmetrics import Accuracy # version '0.5.0'
accs = Accuracy(average="none", num_classes=4)

probs  = torch.tensor([[.9, 0.1, 0.9, 0.1], 
                       [.8, 0.2, 0.9, 0.2], 
                       [.7, 0.1, 0.8 , 0.2]])

targets = torch.tensor([[1, 1, 0, 0], 
                        [1, 1, 0, 0], 
                        [1, 1, 0, 0]])

accs(probs, targets)
#> tensor([1., 0., 0., nan])

On your Docs you write that this is intended: "If 'none' and a given class doesn’t occur in the preds or target, the value for the class will be nan."

But technically the last class has an Accuracy of 1.. Statistics like min or max do not work on tensors with nan. My real model has some thousand labels and imputing nan with 1. doesn't feel that good ;)

Internally you treat the last label with an Accuracy of 1.:

acc = Accuracy(num_classes=4)
acc(probs, targets)
#> tensor(0.5000)

To some extend I understand your thoughts, but I want to ask if this is really the intended result or if an argument like suppress_nan is possible.
Thanks for your module!

Best
Philipp

[SkafteNicki]

Hi @phiyodr,
Thanks for raising this issue. Looking at our code it may very well be that there is some kind of bug here.
I am pretty sure the comment "If 'none' and a given class doesn’t occur in the preds or target, the value for the class will be nan." is only meant to apply to multi-class problems, and not multi-label problems as you are showing.

[phiyodr]

Thank you! I will make it an issue.