Confusion Matrix Metric in Supervised Evaluator

[skrishnanv]

Hi all,
I am trying to compute the confusion matrix for the evaluation. In the sample code from various example

evaluator = create_supervised_evaluator(model=net,
                                                metrics=val_metrics,
                                                device=device,
                                                non_blocking=True,
                                                prepare_batch=prepare_batch,
                                                )

where

val_metrics = {"VAL_ACC": Accuracy(),
                       "ROC_AUC": ROCAUC(output_transform=roc_auc_trans),
                       #"CM":monai.handlers.ConfusionMatrix(output_transform=roc_auc_trans )
                       }
post_label = Compose([EnsureType(), AsDiscrete(num_classes=2, to_onehot=True)])
post_pred = Compose([EnsureType(), Activations(softmax=True)])

def roc_auc_trans(x):
    return [post_pred(i) for i in decollate_batch(x[0])], [post_label(i) for i in decollate_batch(x[1])]

If the line containing CFM is commented out, all works fine. However, if i uncomment the CFM,
the error I get is
File "C:\Users\Krishnan\anaconda3\envs\pytorch\lib\site-packages\monai\metrics\confusion_matrix.py", line 102, in _compute_tensor
return get_confusion_matrix(
File "C:\Users\Krishnan\anaconda3\envs\pytorch\lib\site-packages\monai\metrics\confusion_matrix.py", line 173, in get_confusion_matrix
tp = ((y_pred + y) == 2).float()
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
I do not understand how this is possible since the same output_transform is used for ROCAUC and ConfusionMatrix. I do not want to compute the matrix every validation cycle but I am not able to understand whats wrong in the code. I tried using ignite metrics but that is leading to another set of errors and is not relevant here. I plan use sklearn on the ypred and y but I was wondering what is wrong here.
Any help is appreciated.
Thanks!

[Nic-Ma]

Hi @skrishnanv ,

Seems the issue is your pred and label are not on the same device.
I would suggest to refer to this tutorial example if you prefer to the SupervisedEvaluator engine:
https://github.com/Project-MONAI/tutorials/blob/master/modules/engines/unet_evaluation_dict.py#L107

Thanks.

[skrishnanv]

Hi,
Thanks for your kind reply. I understood from the message same as you did. However, i am unable to understand why it happened. The code which i was working on was based on the MONAI bootcamp example here. I am unable to link to the exact section but code snippet is in section 5 "Start using ignite..." In that

# Validation

def roc_auc_trans(x):
    return [act(i) for i in decollate_batch(x[0])], [to_onehot(i) for i in decollate_batch(x[1])]

val_metrics = {
    'accuracy': Accuracy(), 
    'rocauc': ROCAUC(output_transform=roc_auc_trans)
}
evaluator = create_supervised_evaluator(
    net,
    val_metrics,
    device,
    True
)

works but as i mentioned on adding confusion matrix computation gives the error i posted earlier. If the pred and label are on different devices, i would expect the problem to happen with other computations. Moreover, I have ensured that all loaders use the GPU/cuda device. I am just not sure whether a) i have used the ConfusionMatrix incorrectly b) bug somewhere else. Using ignite's confusion matrix computer gives some error on batch shape. This is puzzling to me again because both the engine and the metrics are from ignite and i was hoping(thats the point of these wrappers) that it would "take care" :-)
if you could let me know if I have used the transform for monai.handlers.ConfusionMatrix() in val_metrics above incorrectly; in case time permits.. that would be very helpful.
For now it seems easier to extract the output on evaluation run and do the needful.
Thanks!

[Nic-Ma]

Hi @skrishnanv ,

Could you please help share all your test program for further analysis?
I think you can print XXX.device to identify where pred is moved to GPU and why missed to move the label.

Thanks in advance.

[o-fares]

Hi @skrishnanv @Nic-Ma,
I have the same problem with the ConfusionMatrix handler from Monai, even when forcing the device to be on the GPU, I get the same RuntimeError about the tensors being on the cpu and on the gpu. It seems to work only when the CPU is used.

[Nic-Ma]

Hi @o-fares ,

Could you please help share your test program for a deeper analysis?

Thanks in advance.

[skrishnanv]

I couldn't make a complete example. However, what I found is that if I detach the data structures during decollate, it seems to work fine. I will try to get an example soon.

…

On Thu, 6 Jan, 2022, 9:07 pm Fares Ouadahi, ***@***.***> wrote: Hi @skrishnanv <https://github.com/skrishnanv> @Nic-Ma <https://github.com/Nic-Ma>, I have the same problem with the ConfusionMatrix handler from Monai, even when forcing the device to be on the GPU, I get the same RuntimeError about the tensors being on the cpu and on the gpu. It seems to work only when the CPU is used. — Reply to this email directly, view it on GitHub <#3549 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALAFBNNUIOOLDOC6IEQ72WDUUWZK3ANCNFSM5K4KN7MA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[o-fares]

Hi @skrishnanv,
Thank you for your answer.
I am already using decollate_batch in the evaluator like this:

evaluator_val = create_supervised_evaluator(net, val_metrics, device, non_blocking=True,
                                            output_transform=lambda x, y, y_pred: ([post_pred(i) for i in decollate_batch(y_pred)], [gt_post(i) for i in decollate_batch(y)]),
                                            prepare_batch=prepare_batch)

with post_pred and gt_post including the post processing transforms.
Do you mean you used the Decollated transform ?

[skrishnanv]

I mean whatever you have written below; no changes. One change you can check is to use detach=False in decollate_batch. I seem to recall it worked(i do not know why, if it does for you).

…

On Thu, Jan 6, 2022 at 9:30 PM Fares Ouadahi ***@***.***> wrote: Hi @skrishnanv <https://github.com/skrishnanv>, Thank you for your answer. I am already using decollate_batch in the evaluator like this: evaluator_val = create_supervised_evaluator(net, val_metrics, device, non_blocking=True, output_transform=lambda x, y, y_pred: ([post_pred(i) for i in decollate_batch(y_pred)], [gt_post(i) for i in decollate_batch(y)]), prepare_batch=prepare_batch) with post_pred and gt_post including the post processing transforms. Do you mean you used the Decollated transform ? — Reply to this email directly, view it on GitHub <#3549 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALAFBNOOLNPSQADGS6RTSDTUUW4CDANCNFSM5K4KN7MA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[o-fares]

Yes it worked. Thanks a lot !

[skrishnanv]

@Nic-Ma , i hope this helps understand what works and what does not? i can put together an example in 2 days(after work) but the crux is as above: in decollate_batch(detach=False) seems to work otherwise no.

[Nic-Ma]

When you set detach=True, it may convert the GPU Tensor to scalar data, if False, keep the original Tensor type:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/utils.py#L392
Welcome your examples, maybe I can try to make it more clear.

Thanks in advance.