Can I load multiple labels into a monai.data.Dataset (for a multiple output model) for a single Nifti input image?

[Eschamp01]

Does MONAI support loading multiple labels in the monai.data.Dataset -> monai.data.Dataloader pipeline? I'd like to decrease the resolution of the label data a few times, and place each of these different resolutions in the Dataset. What would be the best way to do this?

[Nic-Ma]

Hi @Eschamp01 ,

Thanks for your interest here.
I don't quite understand what the "multiple labels" you mean, do you mean dictionary with keys?
Could you please help provide some example data for further discussion?

Thanks in advance.

[Eschamp01]

Hi Nic, Thank you for getting back to me so soon! I'm currently experimenting with lung tumour detection using MONAI and an architecture called multi-resolution convolutional neural networks. These networks are multi-output networks, giving outputs at each decoder dimension of a UNet model, so that the output resolutions (dimensions) may look like: (4,4), (8,8), (16,16), (32,32), (64,64), (128,128), (256,256), (512,512) for an input image with spacial dimensions (512,512). Here is an MCNN diagram from the original paper: [image: Screenshot 2022-06-14 at 17.51.56.jpg] The data I am working with is the Lung Tumours data set from the medical imaging decathlon: http://medicaldecathlon.com/ I have attached an example image and label to this email. They are in Nifti format (.nii.gz). For the different network output resolutions to be able to calculate a loss, they must be compared to a label (an array of zeros of the same dimension as the input image, with ones where a tumour exists), and so I would need to load in training labels at different resolutions to calculate each loss. I am using MONAI, and was wondering if this would be possible in the MONAI data loading process. Since posting, I have manipulated the label data within the PyTorch training process code in the following way. This does what I desired and doesn't noticeably slow down training. However, I would still really like to know if I can load in multiple data labels, as this is the correct way to load in data. inputs, labels = ( batch_data["image"].to(device), batch_data["label"].to(device), ) labels1 = make_block_reduce(labels.cpu().detach().numpy(), dim = (2,2,2,2), mode=np.max) labels2 = make_block_reduce(labels1, dim = (2,2,2,2), mode=np.max) labels3 = make_block_reduce(labels2, dim = (2,2,2,2), mode=np.max) labels4 = make_block_reduce(labels3, dim = (2,2,2,2), mode=np.max) optimizer.zero_grad() outputs, mcnn4s, mcnn3s, mcnn2s, mcnn1s = model(inputs) loss_orig = loss_function(outputs, labels) loss1 = loss_function(mcnn1s, torch.from_numpy(labels1).to(device)) loss2 = loss_function(mcnn2s, torch.from_numpy(labels2).to(device)) loss3 = loss_function(mcnn3s, torch.from_numpy(labels3).to(device)) loss4 = loss_function(mcnn4s, torch.from_numpy(labels4).to(device)) loss = loss_orig + loss1 + loss2 + loss3 + loss4 Here, make_block_reduce is a function that successively max pools the image with filter size (2,2,2), halving the x, y, and z dimensions. For info: from skimage.measure import block_reduce def make_block_reduce(input_layers, dim=(2,2,2,2), mode=np.max): stacked_layers = [block_reduce( image, dim, mode) for image in input_layers] return np.asarray(stacked_layers, dtype='float32') Please let me know if you think there is a good way to reduce the resolution of my label data and load it into the DataLoader object to be used in training. Thank you so much again for replying, Edward

…

On Tue, 14 Jun 2022 at 15:21, Nic Ma ***@***.***> wrote: Hi @Eschamp01 <https://github.com/Eschamp01> , Thanks for your interest here. I don't quite understand what the "multiple labels" you mean, do you mean dictionary with keys? Could you please help provide some example data for further discussion? Thanks in advance. — Reply to this email directly, view it on GitHub <#4500 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHW5HNHZPRWJSDOMTEMSKDTVPCIONANCNFSM5YXRPFZQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Nic-Ma]

Hi @Eschamp01 ,

Thanks for the explanation.
I think maybe you can refer to the tutorial of Brats multi-label segmentation:
https://github.com/Project-MONAI/tutorials/blob/main/3d_segmentation/brats_segmentation_3d.ipynb
@dongyang0122 @ericspod Please feel free to correct me if I misunderstood the question.

Thanks.

[dongyang0122]

Hi @Eschamp01, it is necessary to down-sample the ground truth labels on-the-fly in your use case. A better option would be using MONAI transforms monai.transforms.Resize or monai.transforms.Resized, which can resize the image/label (in either PyTorch tensor or numpy array format) efficiently. You can add dictionary keys like "label1", "label2" in the transform chain, and resize the ground truth labels when loading the data.