Multimodality training with random crop augmentation

[ahxmeds]

Hi all,

Is there a way to set up multimodal training in a dictionary-based format where I am ALSO able to perform transformations such as random cropping of images (for example RandCropByPosNegLabeld)? I have PET (PT) and CT images with binary ground truth masks (GT). I want to concatenate the cropped portions (of spatial_size say (96, 96, 96)) of PT and CT together along the axis=0 (channel dimension), so the final input to the network must have the shape (B, 2, 96, 96, 96), where B is the number of batches. From the tutorials, I can understand how to concatenate two images along the channel dimension, and I also understand how to perform random cropping but I can't seem to find any way to do both together.

I am setting up my dictionary of filenames like this:

train_data = []
for i in range(len(gtpaths_train)):
    ctpath = ctpaths_train[i]
    ptpath = ptpaths_train[i]
    gtpath = gtpaths_train[i]
    train_data.append({'CT': ctpath, 'PT':ptpath, 'GT':gtpath})

And after that, I am defining the train_transform in this way:

train_transforms = Compose(
    [
        LoadImaged(keys=['CT', 'PT', 'GT']),
        EnsureChannelFirstd(keys=['CT', 'PT', 'GT']),
        ScaleIntensityRanged(keys=['CT'], a_min=-1024, a_max=1024, b_min=0, b_max=1, clip=True),
        ScaleIntensityd(keys=['PT'], minv=0, maxv=1),
        CropForegroundd(keys=['CT', 'PT', 'GT'], source_key='PT'),
        Orientationd(keys=['CT', 'PT', 'GT'], axcodes="RAS"),
        Spacingd(keys=['CT', 'PT', 'GT'], pixdim=(
            2.0, 2.0, 2.0), mode=('bilinear', 'bilinear', 'nearest')),
        RandCropByPosNegLabeld(
            keys=['CT', 'PT', 'GT'],
            label_key='GT',
            spatial_size=(96, 96, 96),
            pos=3,
            neg=1,
            num_samples=4,
            image_key='PT',
            image_threshold=0,
        ),
    ]
)

This technically gives num_sample=4 cropped regions for each (CT, PT, GT) triplet. Is there a way to concatenate every individual cropped CT and PT portion from this list of 4 regions within the definition of train_transforms itself? I would really appreciate it if someone could give me some pointers in the right direction. Thank you so much!

[KumoLiu]

Hi @ahxmeds, you can use ConcatItemsd to concatenate specified items from data dictionary together on the first dim to construct a big array.

ConcatItemsd(keys=['CT', 'PT'], name='CTPT')

Hope it can help you, thanks!