How to use F1 score and Accuracy for binary classification?

[aedoardo]

Hi! I'm having some doubts about a task that I'm facing. I'm doing an image binary classification and I have a question about the metrics.

In my task I have two classes that I defined as 0 and 1. Initially, I used the torchmetrics modules with the num_classes parameter equals to 2 and I saw that my F1 score and my Accuracy are always the same while the Recall is slightly different and the average is always the macro. The datasets that I'm using are balanced between the two classes.

Now, I changed the metrics' code in this way:

self.train_accuracy = torchmetrics.Accuracy(num_classes=1, average='macro', multiclass=False)
self.test_accuracy = torchmetrics.Accuracy(num_classes=1, average='macro', multiclass=False)
self.val_accuracy = torchmetrics.Accuracy(num_classes=1, average="macro", multiclass=False)

self.val_MCC = torchmetrics.MatthewsCorrCoef(num_classes=2, multiclass=False)
self.train_MCC = torchmetrics.MatthewsCorrCoef(num_classes=2, multiclass=False)
self.test_MCC = torchmetrics.MatthewsCorrCoef(num_classes=2, multiclass=False)

self.train_recall = torchmetrics.Recall(num_classes=1, average='macro', multiclass=False)
self.test_recall = torchmetrics.Recall(num_classes=1, average='macro', multiclass=False)

self.train_f1 = torchmetrics.F1Score(num_classes=1, average='macro', multiclass=False)
self.test_f1 = torchmetrics.F1Score(num_classes=1, average='macro', multiclass=False)

and even in this case I have always the same Accuracy and the same F1 Score.

I'm confused about the num_classes parameter: which one should I use for a binary classification? The first, with num_classes=2 or the second one with num_classes=1? And why I'm having always the same value for the F1 and the Accuracy even if I'm using the average macro? Is it due to the balanced dataset?

Just another information: in the training (and validation) step what I do is the following:

x, y = batch
y_hat = self(x)
loss = F.cross_entropy(y_hat, y)
output = torch.argmax(y_hat, dim=1)

and then I simply pass the output to the metrics with y as targets.

Thank you, have a nice day!

[celsofranssa]

And where do you call the metrics?
Tip: Since you are logging a collection of metrics, you might want to use MetricCollection.