How to use hausdorff_distance,average_surface_distance, and MeanIoU

[al3ms]

Hi everyone,

I am facing problem while evaluating my model using the three metrics:
1-compute_hausdorff_distance
2-compute_average_surface_distance
3-MeanIoU

This is my result, it looks wrong =(inf,inf,0):

Test_loss: 0.0995
Test_Metric: 0.7005
hausdorff: inf
surface_distance: inf
IoU: 0.0000
current epoch: 42 current mean dice: 0.7005
best mean dice: 0.7005 at epoch: 42

My code:
1- Functions:

def hausdorff_metric(predicted, target):
    include_background=False, 
    hausdorff = compute_hausdorff_distance(predicted, target).item() 
    return hausdorff

def surface_distance(predicted, target):             
    surface_distance = compute_average_surface_distance(predicted, target).item() 
    return surface_distance

2- defining variables in before evaluation step:

                hd=0
                sd=0
                IoU=0
                epoch_hd = 0
                epoch_sd=0
                epoch_IoU = 0
                sid=0
                epoch_sdi = 0

3- for each image in evaluation:

                    hd = hausdorff_metric(test_outputs, test_label)
                    sd = surface_distance(test_outputs, test_label)

                    Iou = MeanIoU(test_outputs, test_label)  
                    # sdi = surface_dice(test_outputs, test_label)

                    epoch_IoU += IoU
                    epoch_hd += hd
                    epoch_sd += sd

4-print the average:

                epoch_hd /= test_step
                print('hausdorff: ' , epoch_hd)
                epoch_sd /= test_step
                print('surface_distance: ' , epoch_sd)
                epoch_IoU /= test_step
                print(f'IoU: {epoch_IoU:.4f}')

[KumoLiu]

Hi @al3ms, for these three metrics, you can use MeanIoU, SurfaceDistanceMetric, and HausdorffDistanceMetric. They all follow the same execution steps. You can refer to the code below:

MONAI/monai/metrics/metric.py

Lines 305 to 317 in e084885

	dice_metric = DiceMetric(include_background=True, reduction="mean")
	for val_data in val_loader:
	val_outputs = model(val_data["img"])
	val_outputs = [postprocessing_transform(i) for i in decollate_batch(val_outputs)]
	# compute metric for current iteration
	dice_metric(y_pred=val_outputs, y=val_data["seg"]) # callable to add metric to the buffer
	# aggregate the final mean dice result
	metric = dice_metric.aggregate().item()
	# reset the status for next computation round
	dice_metric.reset()

Hope it can help you, thanks!

[al3ms]

Thanks a lot dear for responding. I still get the same output: 'inf' for both surface distance and hausdorff. For the IoU I got an error about the size of the inputs. What should I do now? I appreciate your help, thanks.

[KumoLiu]

Hi @al3ms, could you please share more information, if possible, a prediction file and a small piece of code that I can reproduce it? Thanks.

[al3ms]

Yes sure, and thinks agin for your time.

This is the training code:

def train(model, data_in, loss, optim, max_epochs, model_dir, test_interval=1 , device=torch.device("cuda:0")):
    best_metric = -1
    best_metric_epoch = -1
    save_loss_train = []
    save_loss_test = []
    save_metric_train = []
    save_metric_test = []
    train_loader, test_loader = data_in
    time_start = time.perf_counter()

    meanIoU = MeanIoU(reduction="mean") 
    surfaceDistanceMetric = SurfaceDistanceMetric(reduction="mean") 
    hausdorffDistanceMetric = HausdorffDistanceMetric(reduction="mean") 

    for epoch in range(max_epochs):
        print("-" * 10)
        print(f"epoch {epoch + 1}/{max_epochs}")
        model.train()
        train_epoch_loss = 0
        train_step = 0
        epoch_metric_train = 0
        for batch_data in train_loader:
            
            train_step += 1

            volume = batch_data["vol"]
            label = batch_data["seg"]
            label = label != 0
            volume, label = (volume.to(device), label.to(device))

            optim.zero_grad()
            outputs = model(volume)
            
            train_loss = loss(outputs, label)
            
            train_loss.backward()
            optim.step()

            train_epoch_loss += train_loss.item()
            print(
                f"{train_step}/{len(train_loader) // train_loader.batch_size}, "
                f"Train_loss: {train_loss.item():.4f}")

            train_metric = dice_metric(outputs, label)
            epoch_metric_train += train_metric
            print(f'Train_dice: {train_metric:.4f}')

        print('-'*20)
        
        train_epoch_loss /= train_step
        print(f'Epoch_loss: {train_epoch_loss:.4f}')
        save_loss_train.append(train_epoch_loss)
        np.save(os.path.join(model_dir, 'loss_train.npy'), save_loss_train)
        
        epoch_metric_train /= train_step
        print(f'Epoch_metric: {epoch_metric_train:.4f}')

        save_metric_train.append(epoch_metric_train)
        np.save(os.path.join(model_dir, 'metric_train.npy'), save_metric_train)

        if (epoch + 1) % test_interval == 0:

            model.eval()
            with torch.no_grad():
                test_epoch_loss = 0
                test_metric = 0
                epoch_metric_test = 0
                test_step = 0
                hd=0
                sd=0
                IoU=0
                epoch_hd = 0
                epoch_sd=0
                epoch_IoU = 0
                sid=0
                epoch_sdi = 0

                for test_data in test_loader:

                    test_step += 1

                    test_volume = test_data["vol"]
                    test_label = test_data["seg"]
                    test_label = test_label != 0
                    test_volume, test_label = (test_volume.to(device), test_label.to(device),)
                    
                    test_outputs = model(test_volume)
                    
                  
                    test_loss = loss(outputs, test_label)
                    test_epoch_loss += test_loss.item()
                    test_metric = dice_metric(test_outputs, test_label)


                    # meanIoU(outputs,test_label)
                    surfaceDistanceMetric(outputs, test_label)
                    hausdorffDistanceMetric(outputs, test_label)


                    epoch_metric_test += test_metric
                    

                # iou = meanIoU.aggregate().item() 
                hd = hausdorffDistanceMetric.aggregate().item() 
                sd = surfaceDistanceMetric.aggregate().item() 

        # reset the status for next computation round 
                # meanIoU.reset() 
                hausdorffDistanceMetric.reset() 
                surfaceDistanceMetric.reset() 

                test_epoch_loss /= test_step
                print(f'test_loss_epoch: {test_epoch_loss:.4f}')
                save_loss_test.append(test_epoch_loss)
                np.save(os.path.join(model_dir, 'loss_test.npy'), save_loss_test)

                epoch_metric_test /= test_step
                print(f'test_dice_epoch: {epoch_metric_test:.4f}')
                save_metric_test.append(epoch_metric_test)
                np.save(os.path.join(model_dir, 'metric_test.npy'), save_metric_test)

                print('hausdorff: ' , hd)
                print('surface_distance: ' , sd)
                # print(f'IoU: {epoch_IoU:.4f}')
                # epoch_sdi /= test_step
                # print('surface_distance: ' , epoch_sdi)
                

                np.save(os.path.join(model_dir, 'hausdorff.npy'), epoch_hd)
                np.save(os.path.join(model_dir, 'surface_distance.npy'), epoch_sd)


                if epoch_metric_test > best_metric:
                    best_metric = epoch_metric_test
                    best_metric_epoch = epoch + 1
                    torch.save(model.state_dict(), os.path.join(
                        model_dir, "best_metric_model.pth"))
                
                print(
                    f"current epoch: {epoch + 1} current mean dice: {test_metric:.4f}"
                    f"\nbest mean dice: {best_metric:.4f} "
                    f"at epoch: {best_metric_epoch}"
                )


    print(
        f"train completed, best_metric: {best_metric:.4f} "
        f"at epoch: {best_metric_epoch}")
    time_elapsed = (time.perf_counter() - time_start)
    memMb=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024.0/1024.0
    print ("%5.1f secs %5.1f MByte" % (time_elapsed,memMb))

This is the simple output for two epochs:

----------
epoch 1/2
1/6, Train_loss: 0.5263
Train_dice: 0.4737
2/6, Train_loss: 0.5234
Train_dice: 0.4766
3/6, Train_loss: 0.5217
Train_dice: 0.4783
4/6, Train_loss: 0.5227
Train_dice: 0.4773
5/6, Train_loss: 0.5240
Train_dice: 0.4760
6/6, Train_loss: 0.5267
Train_dice: 0.4733
--------------------
Epoch_loss: 0.5241
Epoch_metric: 0.4759
test_loss_epoch: 0.4907
test_dice_epoch: 0.5150
hausdorff:  inf
surface_distance:  inf
current epoch: 1 current mean dice: 0.5052
best mean dice: 0.5150 at epoch: 1
----------
epoch 2/2
1/6, Train_loss: 0.5256
Train_dice: 0.4744
2/6, Train_loss: 0.5225
Train_dice: 0.4775
3/6, Train_loss: 0.5214
Train_dice: 0.4786
4/6, Train_loss: 0.5219
Train_dice: 0.4781
5/6, Train_loss: 0.5233
Train_dice: 0.4767
6/6, Train_loss: 0.5270
Train_dice: 0.4730
--------------------
Epoch_loss: 0.5236
Epoch_metric: 0.4764
test_loss_epoch: 0.4898
test_dice_epoch: 0.5173
hausdorff:  inf
surface_distance:  inf
current epoch: 2 current mean dice: 0.5098
best mean dice: 0.5173 at epoch: 2
train completed, best_metric: 0.5173 at epoch: 2
 20.5 secs   4.5 MByte

[KumoLiu]

I didn't see your post-processing, I think all these three metrics need binarized predictions, did you do post-processing on the predictions?

[al3ms]

Dear KumoLiu, I am truly grateful for your guidance as it has greatly benefited me.
You are right I mess post-processing and I didn't thought it is important. Now I will make sure if the output is correct.
Thanks again.