using get_confusion_matrix and confusion_matrix

[al3ms]

Hello,

I have to include confusion matrices such as sensitivity, precision, etc., in my project, but I am struggling with using such basic functions. I searched for examples, but I didn't find any. As I understand from the document, there are two functions that I have to apply.
1- get_confusion_matrix
2- ConfusionMatrixMetric

-I get the error: 'list' object has no attribute 'shape' when executing the first function.
-For the second function, as I understand, it needs the output of the first function (confusion_matrix). Am I right? I tried to run it without this parameter, and I got this value:

Confusion Metric: metatensor([[[2.0468e+06, 2.0898e+04, 2.8406e+04, 1.0170e+03],
[2.8406e+04, 1.0170e+03, 2.0468e+06, 2.0898e+04]]], device='cuda:0')

What are those values? Are those the expected output of the get_confusion_matrix function?
Anyone could help me with a correct code to calculate sensitivity and precision.

Thanks a lot.

This is my code:

sw_batch_size = 4
roi_size = (128, 128, 256)

post_pred = Compose([AsDiscrete(argmax=True, to_onehot=2)])
post_label = Compose([EnsureType(),AsDiscrete(to_onehot=2)])

confusionMetric = ConfusionMatrixMetric(include_background=True, metric_name=["sensitivity","specificity", "precision", "negative predictive value",
            "miss rate", "fall out", "false discovery rate", "false omission rate",
            "prevalence threshold", "threat score", "accuracy", "balanced accuracy",
            "f1 score", "matthews correlation coefficient", "fowlkes mallows index",
            "informedness", "markedness"])

    for t in test_loader:
        test_volume = t["vol"]
        test_label = t["seg"]
        test_label = test_label != 0
        test_volume, test_label = (test_volume.to(device), test_label.to(device),)
        test_outputs = model(test_volume)

        test_loss = loss_function(test_outputs, test_label)
        test_outputs2 = torch.sigmoid(test_outputs)

        test_outputs2 = [post_pred(i) for i in decollate_batch(test_outputs2)]
        test_label2 = [post_label(i) for i in decollate_batch(test_label)]

        gcm = get_confusion_matrix(test_outputs2,test_label2)
        cm = confusionMetric(test_outputs2,test_label2)

        print("get_confusion_matrix:", gcm)
        print("Confusion Metric:", cm)

[KumoLiu]

Hi @al3ms, You don't need to call get_confusion_matrix yourself, it will be called internally by ConfusionMatrixMetric.
You can find an example in this tutorial.
A simple demo here.

metric = ConfusionMatrixMetric(metric_name="F1", reduction="mean")

for batch in dataloader:
        inputs, labels = batch["input"], batch["label"]
        outputs = model(inputs)
        outputs = (torch.sigmoid(outputs) > cfg.clf_threshold).float()
        metric(outputs, labels)
score = metric.aggregate()[0]
metric.reset()

[al3ms]

Thanks dear for your support. I tried to get multi-metric either in the same line or spread:
confusion = ConfusionMatrixMetric(include_background=True,metric_name=['sensitivity', 'F1','precision'], reduction="mean")

f1 = ConfusionMatrixMetric(metric_name="F1", reduction="mean")
precision = ConfusionMatrixMetric(metric_name="precision", reduction="mean")
sensitivity = ConfusionMatrixMetric(metric_name="sensitivity", reduction="mean")

However, the resulting values are the same!

Test loss: 0.1625310778617859
Confusion Metric F1: tensor([0.9896], device='cuda:0')
Confusion Metric sensitivity: tensor([0.9896], device='cuda:0')
Confusion Metric sensitivity: tensor([0.9896], device='cuda:0')

[KumoLiu]

I noticed that you missed aggregate and reset in your code.

[al3ms]

It is there but I didn't copy all code:

        fscore = f1.aggregate()[0]
        f1.reset()
        prec = precision.aggregate()[0]
        precision.reset()
        sen = sensitivity.aggregate()[0]
        sensitivity.reset()

[KumoLiu]

refer to this one.