Ouput the mean dice accuracy for each class.

[cugwu]

Hi, I've implemented the tutorial https://github.com/Project-MONAI/tutorials/blob/main/modules/cross_validation_models_ensemble.ipynb
related to the ensemble evaluator. Everything works properly and I have the following out:

INFO:ignite.engine.engine.EnsembleEvaluator:Engine run resuming from iteration 0, epoch 0 until 1 epochs
INFO:ignite.engine.engine.EnsembleEvaluator:Got new best metric of test_mean_dice: 0.9344435930252075
INFO:ignite.engine.engine.EnsembleEvaluator:Epoch[1] Complete. Time taken: 00:00:15
INFO:ignite.engine.engine.EnsembleEvaluator:Engine run complete. Time taken: 00:00:16

How can I gate also the test accuracy for each segmentation class?

I know from https://docs.monai.io/en/stable/metrics.html#metric that the Mean Dice score metric should output "Dice scores per batch and per class". How can I get this extra information using the ensemble evaluator?

My setups are:
def ensemble_evaluate(post_transforms, models, device, loader, inferer):
evaluator = EnsembleEvaluator(
device=device,
val_data_loader=loader,
pred_keys=["pred0", "pred1", "pred2", "pred3", "pred4"],
networks=models,
inferer=inferer,
postprocessing=post_transforms,
key_val_metric={
"test_mean_dice": MeanDice(
include_background=True,
# reduction=MetricReduction.MEAN,
output_transform=from_engine(["pred", "label"])
),
},
)
evaluator.run()

mean_post_transforms = Compose(
    [
        EnsureTyped(keys=["pred0", "pred1", "pred2", "pred3", "pred4"]),
        MeanEnsembled(
            keys=["pred0", "pred1", "pred2", "pred3", "pred4"],
            output_key="pred",
            weights=[0.95, 0.94, 0.95, 0.94, 0.90],
        ),
        # Activationsd(keys="pred", softmax=True),
        # AsDiscreted(keys="pred", threshold=0.5),
        AsDiscreted(keys="pred", argmax=True, to_onehot=args.out_channels),
        AsDiscreted(keys="label", to_onehot=args.out_channels),  # manca anche decollate batch?
        SaveImaged(keys="pred", meta_keys="image_meta_dict", output_dir=output_directory, output_postfix="mean",
                   output_ext=".nii.gz", resample=False)

    ]
)
# torch.cuda.empty_cache()
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
ensemble_evaluate(mean_post_transforms, models, torch.device(device), test_loader,
                  test_inferer)

Thanks in advance!

[KumoLiu]

Hi @cugwu, you can use MetricsSaver to save metrics values and details into expected files.

metrics_saver = MetricsSaver(
                save_dir=tempdir,
                metrics=["metric1", "metric2"],
                metric_details=["metric3", "metric4"],
                batch_transform=lambda x: x[PostFix.meta("image")],
                summary_ops=["mean", "median", "max", "5percentile", "95percentile", "notnans"],
                delimiter="\t",
            )

evaluator = EnsembleEvaluator(
val_handlers = metrics_saver
)