Affined Transformation Channel Dimensions

[ekurtulus]

I felt the need of asking this question to prevent both my and other user's future mistakes. I am working with multiple CT and BraTS datasets. In BraTS dataset, each patient has 4 MRI modalities (T1, T1w, T2, Flair) each of which is of size 155 x 240 x 240. My model input is 4 x 128 x 128 x 128 where 128 x 128 x 128 is a patch sampled from the same locations of each modality, and it is expected to return a tensor of same shape for segmentation. What should the input size of the Affined transform be (4 x 128 x 128 x 128) where we feed the whole input or (1 x 128 x 128 x 128) processing each modality separately and then concatenating or feeding them in the shape (128 x 128 x 128) and then concatenating ?

Furthermore, what should input size for CT datasets where, let's say, the size of each modality is of shape (50 x 512 x 512), (1 x 50 x 512 x 512) or (50 x 512 x 512) ?

[yiheng-wang-nv]

Hi @ekurtulus ,

What should the input size of the Affined transform be (4 x 128 x 128 x 128) where we feed the whole input or (1 x 128 x 128 x 128) processing each modality separately and then concatenating or feeding them in the shape (128 x 128 x 128) and then concatenating

Could you please re-describe your question? I cannot understand this sentence, thanks!

[ekurtulus]

Hi @yiheng-wang-nv

I am doing research on 3D volumetric segmentation with CT and MRI images. In my training pipeline, an iteration is defined as sampling a patch of size 128 x 128 x 128 from a selected CT / MRI image and processing it. Let's consider the BraTS dataset for example, my pipeline loads a patient (each patient has T1, T1w, T2, and Flair modalities) and samples a patch of size 128^3 from the same coordinates from each of the modalities. Thus, input size of my model is the concatenation of these patches meaning 4 x 128 x 128 x 128.

I am confused about when and how I should use monai.transforms in this context. There are 3 possible approaches and I am not sure which is correct.

First one:

transform = monai.transforms.Affined(args)
t1, t1ce, t2, flair = load_mri(path) # t1, t1ce, t2, and flair are different MRI modalities of the same patient where each has the shape 155 x 240 x 240
# Note that each patch is of size 128 x 128 x 128
input = torch.stack( [ transform(i) for i in [t1, t1ce, t2, flair] ] )

Second one:

transform = monai.transforms.Affined(args)
t1, t1ce, t2, flair = load_mri(path) # t1, t1ce, t2, and flair are different MRI modalities of the same patient where each has the shape 155 x 240 x 240
# Note that each patch is of size 128 x 128 x 128
input = torch.cat( [ transform(i.unsqueeze(0)) for i in [t1, t1ce, t2, flair] ] , dim=0)

Third one:

transform = monai.transforms.Affined(args)
t1, t1ce, t2, flair = load_mri(path) # t1, t1ce, t2, and flair are different MRI modalities of the same patient where each has the shape 155 x 240 x 240
# Note that each patch is of size 128 x 128 x 128
volume = torch.stack([t1, t1ce, t2, flair]) # shape = 4 x 128 x 128 x 128
input = transform(volume)

[yiheng-wang-nv]

Hi @ekurtulus , I think normally the third way is used. The input shape of transforms (not only for affined, but also for other transforms) should be (channels, H, W, D) for 3D images.

[ekurtulus]

Hi @yiheng-wang-nv , thanks a lot for your response !