How to use different ignite handlers with hovernet?

[OmarAshkar]

Hello,
I am really struggling to setup good handlers to check training efficiency. I am using hovernet model following the ignite implementation here https://github.com/Project-MONAI/tutorials/blob/main/pathology/hovernet/training.py.

I want to get the following:

• A csv file with 4 columns: epoch, NP dice, NC dice, class prediction
• adding tensorboard NP and NC prediction per epoch

I found no quick way but use a combination of ClassificationSaver, MetricsSaver and TensorBoardImageHandler.

I added MetricsSaver, but it doesn't work as expected. It saves the single last mean_dice for NP calculated and overwrite the previous one. I need to keep track of epochs and add other metrics like mean_dice for NC.

The TensorBoardImageHandler and ClassificationSaver doesn't work at all for me. I don't know excatly how to apply batch_transform and output_transform and when to use them.

Any help would be useful especially if I can capture ignite.engine.state.batch and ignite.engine.state.output with training and extract what is needed without he need of the handlers.

Thank you in advance

[KumoLiu]

Hi @OmarAshkar,
In our tutorials, now we only calculate the NP dice. And if you want to calculate other metrics, you could set additional_metrics in SupervisedTrainer.

trainer = SupervisedTrainer(
   .......
    key_train_metric={
        "train_dice": MeanDice(
            include_background=False,
            output_transform=from_engine_hovernet(keys=["pred", "label"], nested_key=HoVerNetBranch.NP.value),
        )
    },
  additional_metrics={
       "nc_dice": MeanDice(
            include_background=False,
            output_transform=from_engine_hovernet(keys=["pred", "label"], nested_key=HoVerNetBranch.NC.value),
        )
   },
  ......
)

You may also need to change the post-processing since it only processes the NP branch now.
https://github.com/Project-MONAI/tutorials/blob/6be932f821c9ce4e795748dde7574beea8f8ea2c/pathology/hovernet/training.py#L268-L274

For how to save all the metrics you could use TensorBoardStatsHandler, by default it writes loss and metrics to tensorboard.

MONAI/monai/handlers/tensorboard_handlers.py

Lines 79 to 83 in 58c256e

	Default behaviors:
	- When EPOCH_COMPLETED, write each dictionary item in
	``engine.state.metrics`` to TensorBoard.
	- When ITERATION_COMPLETED, write each dictionary item in
	``self.output_transform(engine.state.output)`` to TensorBoard.

In addition, you could refer to this tutorial to learn about how to set batch_transform and output_transform.

Hope it can help you, thanks!

[KumoLiu]

You could set reduction="mean_batch" in the DiceMetric. Thanks!

[OmarAshkar]

Hi @KumoLiu, everything discussed works so far, but I have related issue. I am using MetricsSaver handler, but I cannot find meta_data key in engine.state.batch.

I have tried.
batch_transform = from_engine(keys= ["image_meta_dict"])
batch_transform = from_engine(keys= ["meta_data"])

So, I cannot use filename to keep record. Also, the files are overridden during each epoch.

1. how can I get meta_data from `engine.state.batch'
2. how can I keep different set of files per epoch.

Thanks alot for your help!

[KumoLiu]

Hi @omarmetrx, to get metadata, you could try batch_transform=lambda x: x.meta, for more details about how to use batch_transform and output_transform, you could refer to this tutorial.
https://github.com/Project-MONAI/tutorials/blob/main/modules/batch_output_transform.ipynb

Hope it can help you, thanks!

[OmarAshkar]

@KumoLiu unfortunately this did not work. I get AttributeError: 'list' object has no attribute 'meta' error.

[KumoLiu]

Then maybe try batch_transform=lambda x: [i['image'].meta for i in x]. Thanks!