How to use Accuracy with ignore class?

[noamzilo]

Please see same question on Stack Overflow.

When using Accuracy with a class that should be ignored, meaning it has labels but can never be predicted, the scoring is wrong, because it is calculated with the never predicted labels that should be ignored.

How to use Accuracy while ignoring some class?

Thanks :)

[SkafteNicki]

It is currently not supported in the accuracy metric, but we have an open PR for implementing that exact feature Lightning-AI/torchmetrics#155

Currently what you can is instead calculate the confusion matrix and then ignore some classes based on that (remember that the true positive/correctly classified are found on the diagonal of the confusion matrix):

ignore_index = 3
metric = ConfusionMatrix(num_classes=3)
confmat = metric(preds, target)
confmat = confmat[:2,:2] # remove last column and row corresponding to class 3
acc = confmat.trace() / confmat.sum()

[noamzilo]

Thanks! This is a nice answer, but makes me wonder why ALL metrics are not ConfusionMatrix based? only updates should go to the matrix, then it should have some properties such as accuracy, f1, mcc. Why are they distinct?

Also What I ended up doing (before reading this) was ignored_inds = label_batch == self._ignore_class_num and acc.update(log_probs[~ignored_inds, :], label_batch[~ignored_inds]). Does it make sense?

We can continue here or on SO, to your choosing, no need to copy everything on both :)

[SkafteNicki]

the reason not all metrics using confusion matrix currently is:

• memory: confusion matrix always requires you to store a num_class*num_class matrix, but this is inefficient since accuracy really only requires you to store 2*num_classes (if it should be able to store score per class)
• performance: confusion matrix implementation is generally slower than just computing true positive and number of samples