Confusion matrix in Tensorboard

[o-fares]

Hello,

I am using tensorboard to monitor my classification results, and I would like to check the confusion matrix during the training.
I already use the confusion_matrix function from monai.metrics and I tried to use a ConfusionMatrix handler, but I am struggling since there is no specific handler for Tensorboard.

Is there a proper way to display the confusion matrix directly in tensorboard by linking it to a handler ?

Thank you

[Nic-Ma]

Hi @o-fares ,

When you use ConfusionMatrix handler as a metrics in the evaluator, it will record metrics into the state.metrics.
Then TensorBoardStatsHandler can show it.
Refer to the MeanDice handler here:
https://github.com/Project-MONAI/tutorials/blob/master/modules/engines/unet_training_dict.py#L139

Thanks.

[o-fares]

Hi @Nic-Ma,
Thanks for your answer. I am already using it like this. What I wanted to do was to plot the confusion matrix like this: https://christianbernecker.medium.com/how-to-create-a-confusion-matrix-with-tensorboard-and-pytorch-3344ad5e7209

But I don't know how to link it to the SupervisedTrainer since I need to have the inputs and labels tensors. Any ideas ?

[o-fares]

@Nic-Ma
I succeeded in displaying the confusion matrix for by using trainer._iteration(trainer, next(iter(loader))) and the function presented in the link before. However, if I want to monitor the training and watch the confusion matrix evolve throughout the epochs, I need a custom Tensorboard handler.
I wanted to write something like this :

writer = SummaryWriter(tb_logdir_val)
    hand = TensorBoardHandler(summary_writer=writer)
    hand.attach(evaluator_val)
    conf_mat = createConfusionMatrix(trainer, val_loader, device)
    writer.add_figure("Confusion matrix", conf_mat, trainer.state.epoch)

    evaluator_val.add_event_handler(event_name=Events.EPOCH_COMPLETED, handler=hand)

but the handler is not supported.
Do you have an example of custom Tensorboard handler or maybe an alternative that could work ?

Thank you

[Nic-Ma]

Hi @o-fares ,

I see your point, that's an interesting feature. I think you can try below steps:

1. Set the reduction to save the raw data:
   https://github.com/Project-MONAI/MONAI/blob/dev/monai/handlers/confusion_matrix.py#L29
2. Refer to the example you shared, write a simple handler to call plot_2d_or_3d_image to plot the confusion matrix as image:
   https://github.com/Project-MONAI/MONAI/blob/dev/monai/handlers/tensorboard_handlers.py#L377

Thanks.