Difference MeanIoU vs Jaccard Score

[rot8]

Hey there,

I was wondering why the MeanIoU and BinaryJaccardIndex differ. Shouldn't -by definition- both yield the exact same result?

from torchmetrics.segmentation import MeanIoU
from torchmetrics.classification import BinaryJaccardIndex
import torch
torch.manual_seed(37)
num_classes = 2

preds = torch.randint(0, num_classes, (4, num_classes, 4, 4))
target = torch.randint(0, num_classes, (4, num_classes, 4, 4))


miou = MeanIoU(num_classes=num_classes)
miou_per_class = MeanIoU(num_classes=num_classes, per_class=True, include_background=True)
miou_per_class_no_background = MeanIoU(num_classes=num_classes, per_class=True, include_background=False)
miou(preds, target)
miou_per_class(preds, target)
miou_per_class_no_background(preds, target)

jaccard = BinaryJaccardIndex()
jaccard(preds, target)

• miou = 0.3378
• miou_per_class = [0.4169, 0.2587]
• miou_per_class_no_background = 0.2587
• jaccard = 0.3333

Best, rot8

[Borda]

Thanks for your question! Although MeanIoU and BinaryJaccardIndex are mathematically related (both based on the Jaccard index), they may differ slightly in practice due to implementation details.
Key points to consider:

• MeanIoU in torchmetrics computes the Intersection-over-Union per class and averages over classes. It has options to include or exclude the background class.
• BinaryJaccardIndex is designed for binary classification and computes the Jaccard similarity (IoU) directly on the binary outputs.
• Differences can arise if your inputs are formatted differently or if the metrics apply different thresholds or reduction strategies.
• Also, MeanIoU expects integer class labels, while BinaryJaccardIndex may expect binary tensors; passing multi-class tensors might yield different handling.
• Small variations are common due to the averaging methods or background exclusion in MeanIoU.

If you want the exact same result, make sure the inputs and parameters (e.g., number of classes, inclusion of background) match the expectations of each metric and consider using MeanIoU with per_class=True and background handling aligned.
Hope this clarifies the differences!