DiceMetric does not return NaN for empty classes

[benolmbrt]

Describe the bug
It is expected that the DiceMetric returns NaN values for classes that are not present in the ground truth, if the option ignore_empty is set to True. However, the metric returns instead a value of 0.

To Reproduce
Here is a minimal example to reproduce the bug. I'm creating two random segmentations with 8 classes,
and then manually remove classes 2 and 5 from the ground truth.

from monai.metrics import DiceMetric
import torch
from monai.networks import one_hot

# init metric
metric = DiceMetric(ignore_empty=True, get_not_nans=True, reduction='mean_batch')

# create ground truth and pred for a 8-classes problem
y = torch.randint(0, 8, (1, 1, 64, 64, 64))
y_hat = torch.randint(0, 8, (1, 1, 64, 64, 64))
y1h = one_hot(y, 8)
pred1h = one_hot(y_hat, 8)

# remove classes 2 and 5 from ground truth
y1h[:, 2] = torch.zeros(1, 64, 64, 64)
y1h[:, 5] = torch.zeros(1, 64, 64, 64)

# compute dice
metric(pred1h, y1h)
class_dices, not_nans_count = metric.aggregate()

print(class_dices)
print(not_nans_count)

Output :

class_dices = [0.1273, 0.1231, 0.0000, 0.1277, 0.1249, 0.0000, 0.1233, 0.1278]
not_nans_count = [1., 1., 0., 1., 1., 0., 1., 1.]

--> The Dice scores for class 2 and 5 are set to 0, not NaN.

Expected behavior
It is expected that the Dice values for class 2 and 5 are NaN, but it is 0.

Environment

I am using Monai 1.3.0 with Python 3.11.5

[KumoLiu]

Hi @benolmbrt, thanks for your reporting.
The behavior you're seeing with the DiceMetric is likely due to a misunderstanding of the ignore_empty parameter's functionality.
When the ignore_empty option is set to True, it's supposed to set a NaN value for cases where the ground truth is empty. However, this does not mean that a NaN will be returned for such classes when computing the metric. Instead, these cases are ignored during the calculation process.Thus, even if some classes are ignored (due to being absent in the ground truth when ignore_empty=True), the metric computation does not return NaN for these classes. Instead, it computes the metric over the non-ignored classes only.
If the DiceMetric were to return a NaN for an ignored class, this would also affect the computed average Dice score (since any mathematical operation with NaN results in NaN), rendering the metric not well-defined for batches that include at least one empty class in the ground truth.

I hope this helps and let me know if you have any other questions!
Thanks.