Why do some metrics require num_classes=1 for binary classification?

[kapsner]

❓ Why do some metrics require num_classes=1 for binary classification?

What is your question?

Why do some metrics require the argument num_classes=1 for binary classification (and some don't) to give the correct results?

I find it rather unintuitively to calculate Recall/Precision/F1 with the argument num_classes=1 for a binary classification, whereas e.g. ConfusionMatrix requires num_classes=2 in the same situation.

Furthermore, using Recall/Precision/F1 with num_classes=2 for a binary classification gives wrong results - so this also might be considered a bug-report.

It took me quite some time to figure out, why calculated metrics are different from what I calculated by hand from the confusion matrix.

Code

import torch
from pytorch_lightning import metrics

# example data
preds = [0] * 200 + [1] * 30 + [0] * 10 + [1] * 20
targets = [0] * 200 + [1] * 30 + [1] * 10 + [0] * 20

preds = torch.tensor(preds)
targets = torch.tensor(targets)

# define method for printing metrics


def _print_some_metrics(preds, targets, num_classes):
    precision = metrics.classification.Precision(num_classes=num_classes)
    recall = metrics.classification.Recall(num_classes=num_classes)
    f1 = metrics.classification.F1(num_classes=num_classes)

    accuracy = metrics.classification.Accuracy()
    avg_precision = metrics.classification.AveragePrecision(
        num_classes=1)
    confusion_matrix = metrics.ConfusionMatrix(num_classes=2)

    # print results
    print("Precision:\n{}\n".format(precision(preds, targets)))
    print("Recall:\n{}\n".format(recall(preds, targets)))
    print("F1:\n{}\n".format(f1(preds, targets)))

    print("AVG Precision:\n{}\n".format(avg_precision(preds, targets)))
    print("Accuracy:\n{}\n".format(accuracy(preds, targets)))
    print("ConfMat:\n{}\n".format(confusion_matrix(preds, targets)))


_print_some_metrics(preds, targets, num_classes=1)
_print_some_metrics(preds, targets, num_classes=2)

Results in

$ _print_some_metrics(preds, targets, num_classes=1)
Precision:
0.6000000238418579

Recall:
0.75

F1:
0.6666666865348816

AVG Precision:
0.48846155405044556

Accuracy:
0.8846153616905212

ConfMat:
tensor([[200., 20.],
[ 10., 30.]])

$ _print_some_metrics(preds, targets, num_classes=2)
Precision:
0.8846153616905212

Recall:
0.8846153616905212

F1:
0.8846153616905212

AVG Precision:
0.48846155405044556

Accuracy:
0.8846153616905212

ConfMat:
tensor([[200., 20.],
[ 10., 30.]])

As one can see, Precision/Recall/F1 give different (wrong) results when setting num_classes=2 in a binary classification.
AveragePrecision doesn't even work with the binary usecase when setting num_classes=2 whereas ConfusionMatrix doesn't work when setting num_classes=1.

I wonder if there is a specific reason why one would set num_classes=1 in a binary classification (where actually 2 classes exist).

Wouldn't it be more straightforward to set num_classes=2 for binary classification for all metrics?

[SkafteNicki]

So I can try to clarify this a bit:

• we are aware of this, and many of these metrics have already been changed on the release/1.2-dev branch. For example will your example with precision = metrics.classification.Precision(num_classes=2) give an error because input tensors are clearly binary but the class was initialized with num_classes=2.
• confusion matrix needs num_classes=2 for binary problems, because we to know the size of the tensor that we should allocate
• the difference between specifying num_classes=1 or num_classes=2 really comes down to if you want to calculate the score on only the positive class (this is probably what you want) or both classes (which really does not make sense for binary problems, because many of the scores reduce to the same then).

[kapsner]

@SkafteNicki thx for your answer and explanation

the difference between specifying num_classes=1 or num_classes=2 really comes down to if you want to calculate the score on only the positive class (this is probably what you want) or both classes (which really does not make sense for binary problems, because many of the scores reduce to the same then).

I agree with you, that in the binary case one would like to calculate the score on the positive class.

I wonder, if there are any use cases, where the calculation using your defined num_classes=2 is really wanted, or if any case with 2 classes (= binary classification) would actually only use num_classes=1 and the next valid application would be a 3-class classification using num_classes=3.

• 2 classes: num_classes=1
• 3 classes: num_classes=3
• 4 classes: num_classes=4
• ...

[SkafteNicki]

so the usecase could be that the user wants set average=None to get the score for each class. Again in that case we kind of need to now the size of the tensor that we should allocate (the metric states).

[kapsner]

Ok, thx for your explanations.

[rchen19]

I feel this should be mentioned in the documentation.

[SkafteNicki]

@rchen19 please feel free to open a issue in the new torchmetrics repo https://github.com/PyTorchLightning/metrics (where all future metrics development will take place) and we can implement the changes in the documentation

[mircare]

Hi - has this been documented somewhere?