post processing prediction masks using ensemble evaluator without/while saving

[bip5]

Hi Guys,

I'm using ensemble evaluator to evaluate some of my models. I think I've finally got it working and have managed to save some predictions using the SaveImaged handler. While its handy to be able to look at the saved predictions later, I was wondering if there was a way to work on the prediction as they are being evaluated and saved by ensemble evaluator. I'm working on BraTS segmentation data and was trying to output a csv with the predicted tumour size alongside the ground truth.

Thank you

[KumoLiu]

Hi @bip5, you could use CSVSaver to save the output to a CSV, for more information you can refer to the tutorials below.
https://github.com/Project-MONAI/tutorials/blob/31e5c8288b22d99f607efb6575be7b9ac46739c3/3d_classification/torch/densenet_evaluation_dict.py#L71-L81

Hope it can help you, thanks!

[bip5]

Hi @KumoLiu, Thank you for the suggestion. That example loads the images and makes a prediction with a single model. What I think need is to access the prediction from the ensemble evaluated using EnsembleEvaluator then calculate the size from that before saving it as a csv. I don't know how to access the prediction alone. This line val_outputs = model(val_images).argmax(dim=1) but inside EnsembleEvaluator using the ensemble. Do you have any suggestions?

Thank you

[KumoLiu]

Hi @bip5, you could use MeanEnsembled to access the prediction.
For more information, you could refer to this tutorial.

Hope it can help you, thanks!

[bip5]

Hi @KumoLiu , I already use MeanEnsembled in my transform. What I don't know is: how do I point to the prediction mask to make the tumour size one of the outputs of my ensemble_evaluate function? This is my code:

def ensemble_evaluate(post_transforms, models):
           # print(post_transforms.transforms)
           if args.val==1:
               evaluator = EnsembleEvaluator(
                   device=device,
                   val_data_loader=test_loader, #test dataloader - this is loading all 5 sets of data
                   pred_keys=["pred"+str(i) for i in range(len(models))], 
                   networks=models, # models defined above
                   inferer=SlidingWindowInferer(
                       roi_size=(192,192, 144), sw_batch_size=4, overlap=0),
                   postprocessing=post_transforms, # this is going to call post_transforms based on type of ensemble
                   key_val_metric={
                           "test_mean_dice": MeanDice(
                               include_background=True,
                               output_transform=from_engine(["pred", "label"]) ,reduction='mean_channel' 
                               
                           )},
                  
                   additional_metrics={ 
                       "Channelwise": MeanDice(
                       include_background=True,
                       output_transform=from_engine(["pred", "label"]),
                       reduction="mean_batch"),"Hausdorff":HausdorffDistance(
                       include_background=False,                      
                       output_transform=from_engine(["pred", "label"]),
                       reduction='mean_channel',
                       percentile=95)
                   }
               )
           else:
               evaluator = EnsembleEvaluator(
                   device=device,
                   val_data_loader=test_loader, #test dataloader - this is loading all 5 sets of data
                   pred_keys=["pred"+str(i) for i in range(len(models))], 
                   networks=models, # models defined above
                   inferer=SlidingWindowInferer(
                       roi_size=(192,192, 144), sw_batch_size=4, overlap=0),
                   postprocessing=post_transforms, # this is going to call post_transforms based on type of ensemble
                   key_val_metric={
                           "test_mean_dice": MeanDice(
                               include_background=True,
                               output_transform=from_engine(["pred", "label"]),reduction='mean_channel'  # takes all the preds and labels and turns them into one list each
                               
                           )},
                  
                   additional_metrics={ 
                       "Channelwise": MeanDice(
                       include_background=True,
                       output_transform=from_engine(["pred", "label"]),
                       reduction="mean_batch"),"Hausdorff":HausdorffDistance(
                       include_background=False,                      
                       output_transform=from_engine(["pred", "label"]),
                       reduction='mean_channel',
                       percentile=95)
                   }
               )

           evaluator.run()
           
           # print("validation stats: ",evaluator.get_validation_stats())
           mean_dice=evaluator.state.metrics['test_mean_dice']
           hausdorff=evaluator.state.metrics['Hausdorff']
           tumor_core=evaluator.state.metrics["Channelwise"][0]
           whole_tumor=evaluator.state.metrics["Channelwise"][1]
           enhancing_tumor=evaluator.state.metrics["Channelwise"][2]
           print("Mean Dice:",evaluator.state.metrics['test_mean_dice'],"metric_tc:",float(evaluator.state.metrics["Channelwise"][0]),"whole tumor:",float(evaluator.state.metrics["Channelwise"][1]),"enhancing tumor:",float(evaluator.state.metrics["Channelwise"][2]))#jbc
           
           return mean_dice,tumor_core,whole_tumor,enhancing_tumor,hausdorff
       `

[KumoLiu]

Hi @bip5, If you want to access individual prediction from EnsembleEvaluator, you can write your own post transform like MeanEnsembled. Here gives a simple example:

class CalculateTumorSize(Ensembled):
    def __init__(self, keys: KeysCollection, output_key: Optional[str] = None):
  
        cal_size_func = lambda x: [ np.count_nonzero(img) for img in img ]  # write your own func here!
        super().__init__(keys, cal_size_func , output_key)

Hope it can help you, thanks!

[bip5]

Hi @KumoLiu , Thanks for the suggestion. I've tried to write the classes as they're used in the additional_metrics portion. I think I've covered the basics but I'm stuck with this error: "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/monai/engines/workflow.py", line 242, in _register_metrics metric.attach(self, name) AttributeError: 'CountSize' object has no attribute 'attach'

My function and classes are the following:

    def countsize(y_pred):
        size=np.count_nonzero(y_pred)
        return size
    class CountSize(CumulativeIterationMetric):
        def __init__(self,
        reduction=MetricReduction.MEAN_CHANNEL,
        output_transform=lambda x:x,
        get_not_nans = False,
        ignore_empty = True,
        ):        
            super().__init__()
            self.reduction=reduction
        def _compute_tensor(self,y_pred):
            return countsize(y_pred)
    class PredSize(IgniteMetric):
        def __init__(self,
            reduction=MetricReduction.MEAN_CHANNEL,
            output_transform=lambda x:x,
            save_details=True
            ):        
            metric_fn=CountSize(
            reduction=reduction)
            super().__init__(metric_fn=metric_fn,output_transform=output_transform)
            self.reduction=reduction

Do you know how I could get past this error?

[bip5]

Hi @KumoLiu, Thank you for the hint, turned out I was calling CountSize where it mattered. I've run into another problem:

Current run is terminating due to exception: output must have length 2, got 4.
Engine run is terminating due to exception: output must have length 2, got 4.
Traceback (most recent call last):
  File "/scratch/a.bip5/BraTS 2021/./monai_eval.py", line 903, in <module>
    md1,tc1,wt1,et1,haus1=ensemble_evaluate(mean_post_transforms1, models1)
  File "/scratch/a.bip5/BraTS 2021/./monai_eval.py", line 739, in ensemble_evaluate
    evaluator.run()
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/monai/engines/evaluator.py", line 137, in run
    super().run()
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/monai/engines/workflow.py", line 282, in run
    super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 704, in run
    return self._internal_run()
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 783, in _internal_run
    self._handle_exception(e)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 466, in _handle_exception
    raise e
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 753, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 854, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 466, in _handle_exception
    raise e
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 841, in _run_once_on_dataset
    self._fire_event(Events.ITERATION_COMPLETED)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 421, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/metrics/metric.py", line 321, in iteration_completed
    self.update(output)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/metrics/metric.py", line 605, in wrapper
    func(self, *args, **kwargs)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/monai/handlers/ignite_metric.py", line 78, in update
    raise ValueError(f"output must have length 2, got {len(output)}.")
ValueError: output must have length 2, got 4.

[KumoLiu]

Hi @bip5, add output_transform just like MeanDiceto get y_pred and y from engine.

MeanDice(include_background=True, output_transform=from_engine(["pred", "label"]))

Thanks!

[bip5]

Thank you for your help, it's solved the previous issue but here's another. Seems like I've missed more than a few steps..

File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/monai/metrics/metric.py", line 213, in extend
    raise TypeError(
TypeError: split expects at least a 1-dimensional tensor. `data` should be a batch-first tensor or a list of channel-first tensors, got <class 'torch.Tensor'>

Printing out the data passed metric.py shows that the function there is receiving an empty tensor when it comes to the new metric

...
<monai.metrics.hausdorff_distance.HausdorffDistanceMetric object at 0x7fdc6d317040>
<class 'torch.Tensor'> type(d)
torch.Size([1, 2]) d.shape
<class 'torch.Tensor'> type(d_t)
torch.Size([1, 2]) d_t.shape
<__main__.CountSize object at 0x7fdc6d3170d0>
<class 'torch.Tensor'> type(d)
torch.Size([]) d.shape
<class 'torch.Tensor'> type(d_t)
torch.Size([]) d_t.shape

Not sure what it is that's causing this to happen, any ideas?

[bip5]

Just realised my countsize function returned a single integer which was seen as an empty size for a tensor which was the main cause for the error.

def countsize(y_pred,y):
        
        predsize=torch.tensor(torch.count_nonzero(y_pred[...,1]),device=y_pred.device)
        
        return predsize[None]

This seems to have fixed it for now however I'm getting a new error

...
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/monai/handlers/ignite_metric.py", line 92, in compute
    result = self.metric_fn.aggregate()
  File "/scratch/a.bip5/BraTS 2021/./monai_eval.py", line 450, in aggregate
    f, not_nans = do_metric_reduction(data, self.reduction)
  File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/monai/metrics/utils.py", line 93, in do_metric_reduction
    not_nans = not_nans.sum(dim=1)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

[KumoLiu]

Hi @bip5, since your metric is only one value, you couldn't use the MetricReduction.MEAN in MONAI which needs batch and channel dim. Try to use MetricReduction.MEAN_BATCH

MONAI/monai/metrics/utils.py

Lines 75 to 78 in f49cc96

	if reduction == MetricReduction.MEAN:
	# 2 steps, first, mean by channel (accounting for nans), then by batch
	not_nans = not_nans.sum(dim=1)
	f = torch.where(not_nans > 0, f.sum(dim=1) / not_nans, t_zero) # channel average

MONAI/monai/metrics/utils.py

Line 86 in f49cc96

elif reduction == MetricReduction.MEAN_BATCH:

Thanks!