Optional Metrics

[EricZimmermann]

Hello,

Does Monai have any standardized way to do group-wise metrics? For example, if I have data coming from multiple cohorts (trials, sources, etc) I want to know how my model is performing on each individual cohort (ex dice cohort 1, dice cohort 2..) as well as globally (dice over entire set). If I attach a metric to an ignite engine, there is no guarantee that a single batch has samples coming from all cohorts. In this case, at event iteration completed, certain metrics may have empty or NoneType values and would break the metric trying to access a null value in an output.

I currently have a fix / wrapper for this kind of problem but I was wondering if there would be any native support coming? Also looking at work arounds for multiple datasets / loaders attached to an engine.

[Nic-Ma]

Hi @EricZimmermann ,

Currently, we usually compute metrics on non-nan values and do reduction when epoch completed, for example:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/metrics/meandice.py#L92

Thanks.

[EricZimmermann]

In the case where the value doesn't exist, would you set it to NaN directly or its implied the metric computes Nan and then is ignored?

[Nic-Ma]

Hi @EricZimmermann ,

For Nan values, we set to 0 in the array and ignore it when computing the average (use the not_nans count).

Thanks.

[EricZimmermann]

@Nic-Ma
Thank you for the reply. I think I misunderstood Monai metrics - I should be taking this up with ignite :)

[Nic-Ma]

Hi @EricZimmermann ,

I submitted a draft PR #3424 to clarify the reduction logic for Nan values.

Thanks.