RandAffined does not work as expected

[Spenhouet]

Describe the bug

Given the following image and label:

When applying RandAffined, the image and labels no longer match:

To Reproduce

Steps to reproduce the behavior:

1. Load these files: example_files.zip
2. Execute this pipeline:

from pathlib import Path

import numpy as np
from monai.transforms.compose import Compose
from monai.transforms.io.dictionary import LoadImaged, SaveImaged
from monai.transforms.spatial.dictionary import RandAffined
from monai.transforms.utility.dictionary import (EnsureChannelFirstd, ToTensord)
from monai.utils.enums import GridSampleMode

path = Path('scratch/rand_affine')

T1_KEY = 't1'
LABEL_KEY = 'labels'
FILE_KEYS = [T1_KEY, LABEL_KEY]
SAMPLE_MODES = [GridSampleMode.BILINEAR, GridSampleMode.NEAREST]

data = {
    T1_KEY: path / 't1.nii.gz',
    LABEL_KEY: path / 'labels.nii.gz',
}

process = Compose([
    LoadImaged(FILE_KEYS),
    ToTensord(FILE_KEYS),
    EnsureChannelFirstd(FILE_KEYS),
    RandAffined(
        FILE_KEYS,
        rotate_range=5.0 * np.pi / 180.0,  # type: ignore
        prob=1.0,
        mode=SAMPLE_MODES,
        allow_missing_keys=True,
    ),
    SaveImaged(
        FILE_KEYS,
        output_dir=path,
        output_postfix=f"out",
        resample=False,
        output_dtype=np.float32,
        separate_folder=False,
        print_log=False,
    )
])

process([data])

Expected behavior
The same random affine transformation is applied to all images.
After the transform the labels still match the image.

Additional context
I did not yet identify the issue.

[wyli]

I couldn't replicate the issue locally with a macbook and visualise with itk-snap.

could you help provide more info?

[Spenhouet]

@wyli You did not use the data provided by me. Could you try it with that? It is part of the reproduction.

[wyli]

@wyli You did not use the data provided by me. Could you try it with that? It is part of the reproduction.

I downloaded and unzipped the file, there are two files, t1.nii.gz and labels.nii.gz. itk-snap refuses to display

the labels.nii.gz as segmentations for t1.nii.gz

[Spenhouet]

@wyli Why did you change this to a discussion?
You just used the wrong tool to visualize images. That is not the fault of this bug report and also does not change anything with respect to the bug.

Please reopen the bug report.

[Spenhouet]

I'm not a itk-snap user but if there is a problem in itk-snap probably good to report it to their project. Not really the issue I'm reporting here though.

Try using MRIcroGL or Slicer. Both work fine.

[Spenhouet]

@Nic-Ma @ericspod @wyli @rijobro This was closed due to an unrelated issue with itk-snap. I don't know why this was the reason for closing this bug report? Could you clarify and reopen the issue?

[rijobro]

Is it possible that what we're seeing here is due to differences in image and label sizes? I haven't checked the RandAffined code, but wouldn't be surprised if that's what's causing it. This is similar to your CropForegroundd problem, I don't know if there's a plan to support differing image/label sizes in MONAI as I suspect it would be a large undertaking.

[Spenhouet]

@rijobro Yes, it is. I reopened the issue and clarified that part: #3339

With respect to the "if", I really think this should be supported. Honestly, to me the "same image size" case is a rather toy dataset case. So to me that is honestly not a real question. I strongly believe that MONAI should support this.

[rijobro]

I respectfully disagree that DL use cases with matching image/label sizes are limited to toy examples, as in my experience they represent the overwhelming majority of cases that we use. I also think that your problem could be solved by simply cropping your larger images to match your smaller ones, or padding your smaller ones to match your larger ones. Seems like a fairly easy initial transform that would solve all these problems you're seeing.

[Spenhouet]

As I replied here #3182 (comment), it is not the case that there is an easy work around.
If it would help and you are interested, we could get on a call and I could show you our use case. This would clarify on why this is necessary.

I respectfully disagree that DL use cases with matching image/label sizes are limited to toy examples, as in my experience they represent the overwhelming majority of cases that we use.

My statement might have been badly worded. I do agree that the majority of use cases will have matching images. I was absolutely not stating that these are all toy examples. I meant to say that there are much more use cases beyond that. Like, if we deviate from the norm, we can no longer use MONAI?