How do you add additional metrics (BraTS tumor classes) to Ensemble Evaluator?

[bip5]

Hi, I am trying to add additional metrics to the EnsembleEvaluator used in the tutorial for cross validation. I was working with the BraTS dataset and need to add the tumor classes separately in addition to average dice.

[Nic-Ma]

Hi @bip5 ,

If you want to compute the metrics for every class, there are 2 methods: 1. Set the reduction arg of metric to compute average for every channel separately. 2. Split the prediction by SplitChanneld transform to save every class in a separate key and compute.
@yiheng-wang-nv May help share more concrete examples if you still have some issues.

Thanks.

[bip5]

Hi @Nic-Ma
Thank you for the reply

I tried to set the reduction arg as below:

   key_val_metric={
            "test_mean_dice": MeanDice(
                include_background=True,
                output_transform=from_engine(["pred", "label"]),
                reduction="mean_channel"
                
            )

When I tried this I received this error:
key metric is not a scalar value, skip the metric comparison with the current best metric.please set other metrics as the key metric, or change the reduction mode to 'mean'.

Is this the right place to set the reduction arg? Am I doing something wrong?

Thanks

[Nic-Ma]

Hi @bip5 ,

I think that's expected behavior because you are using this metric as the key metric to compare the best model, but it's not scalar value to compare now.
Please use it as additional metrics or use the second method based on SplitChanneld.

Thanks.

[bip5]

I tried adding additional metric as below
additional_metrics={ "Channelwise": MeanDice( include_background=True, output_transform=from_engine(["pred", "label"]), reduction="mean_channel") }
Unfortunately, the 'channel' reported in the final score was the individual test samples instead of the score across the channels in all samples being aggregated into one.

@yiheng-wang-nv Do you have any examples for either of the methods?

[Nic-Ma]

Hi @bip5 , for your case, I think you should use mean_batch reduction instead of mean_channel?

Thanks.

[bip5]

Hi @Nic-Ma , Thank you for your help. I now have the metrics as I want it.

[Nic-Ma]

No problem, please feel free to raise a ticket if you have any more questions.

Thanks.