Input to TestTimeAugmentation in combination with Lightning #6263

jamtheim · 2023-03-31T14:16:06Z

jamtheim
Mar 31, 2023

Dear Community,
I am trying to use the MONAI TestTimeAugmentation but cant figure out what input it requires.
I have tried to use the batch data in test_step from lightning as input but my transformation fails as it reports 4 dimensions rather than 3 (a random rotation). It seems to be a mismatch in batch dimension or similar somewhere.
It seems that TestTimeAugmentation wants a dictionary as input, but not sure of the contents of that.
Please help me with making this combination work.

Addon, maybe the relevant issue is: How do I combine TestTimeAugmentation and sliding window inference in lightning?

Answered by jamtheim

Apr 13, 2023

Hi again,
So I got it to work as intended. But know I have run into a problem and I could need some help.
As standard before validation and inference I am doing the transform:

inf_transforms = [
LoadImaged(keys=["image", "label"]),
EnsureChannelFirstd(keys=["image"]),
ScaleIntensityRanged(
keys=["image"],
a_min=conf.model.ScaleIntensityRanged_a_min,
a_max=conf.model.ScaleIntensityRanged_a_max,
b_min=conf.model.ScaleIntensityRanged_b_min,
b_max=conf.model.ScaleIntensityRanged_b_max,
clip=conf.model.ScaleIntensityRanged_clip,
)]

I cant figure out how to apply this as a standard transform before the data enters the test time augmentation module. I dont think I can put in the test time transf…

View full answer

KumoLiu · 2023-04-04T02:44:56Z

KumoLiu
Apr 4, 2023
Maintainer

Hi @jamtheim, thanks for your interest here, the input for TestTimeAugmentation is a dict, and each data should be shaped with (B, C, H, W, [D]). The dict is something like:

{
        "image": "imagesTr/img0001.nii.gz",
        "label": "labelsTr/label0001.nii.gz"
}

Hope it can help you! Thanks!

2 replies

jamtheim Apr 4, 2023
Author

Dear KumoLiu, thank you so much for reaching out and providing guideance. I will hereby provide some additional details and hoping for further help.
I have created the dictionary as suggested but I get the following error:

File "/XXXFilePath/test_time_augmentation.py", line 196, in call
b[self._pred_key] = self.inferrer_fn(b[self.image_key].to(self.device))
TypeError: to() received an invalid combination of arguments - got (list), but expected one of:

(torch.device device, torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
(torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
(Tensor tensor, bool non_blocking, bool copy, *, torch.memory_format memory_format)

I dont understand how the dataloader is connected when loading the model and I suspect this is the problem above.

The code i use is this:
from train import Net # This is the Lightning Class

Load the model file

model = Net.load_from_checkpoint(modelPath)

Disable randomness, dropout, etc for evaluation

model.eval()

inf_files = [
{"image": imagePath, "label": labelPath} for imagePath, labelPath in zip(infImagesPath, infLabelsPath)
]

# Make all items list items
for i in range(len(inf_files)):
    inf_files[i]['image'] = [inf_files[i]['image']]
    inf_files[i]['label'] = [inf_files[i]['label']]


# Select number of patients to use for inference
if conf.model.nrPatientsInf != 'all': # if not all
    inf_files = inf_files[:conf.model.nrPatientsInf]

  # Flip
transform_TTA = Compose(
    [
    LoadImaged(keys=["image", "label"]),
    EnsureChannelFirstd(keys=["image"]),
    RandFlipd(
    keys=["image", "label"],
    spatial_axis=0,
    prob=1.0,
    )]
    )

files = inf_files[0] #Example

_ = model.eval()
infFunction_here = lambda x: torch.softmax(model(x))

tt_aug = TestTimeAugmentation(
transform_TTA, batch_size=1, num_workers=0, inferrer_fn=infFunction_here, device=conf.model.inferenceGPUs)

# Get images
mode_tta, mean_tta, std_tta, vvc_tta = tt_aug(files, num_examples=10)

-- > error comes here

KumoLiu Apr 4, 2023
Maintainer

Hi @jamtheim, from the error message, I think the error is from the device you are setting, maybe check what is conf.model.inferenceGPUs, or simply set device="cuda:0" may help solve the problem.

Hope it can help you, thanks!

jamtheim · 2023-04-04T07:27:43Z

jamtheim
Apr 4, 2023
Author

Dear KumoLiu,
This got me in the right correction but another error came up and I am very much looking forward to your input on this.

Set GPU to index 1

device=torch.device("cuda:1" if torch.cuda.is_available() else "cpu")

Move model to GPU 1

model = model.to(torch.device(device))

Set inference function

infFunction_here = lambda x: torch.softmax(model(x))

tt_aug = TestTimeAugmentation(
transform_TTA, batch_size=1, num_workers=0, inferrer_fn=infFunction_here, device=device) # Set correct device

Get images

mode_tta, mean_tta, std_tta, vvc_tta = tt_aug(file, num_examples=10)

Error below:

0%| | 0/10 [00:00<?, ?it/s]
0%| | 0/10 [00:01<?, ?it/s]
Traceback (most recent call last):
File "", line 43, in
File "/mnt/mdstore2/Christian/YngreAlf/venv/lib/python3.8/site-packages/monai/data/test_time_augmentation.py", line 196, in call
b[self._pred_key] = self.inferrer_fn(b[self.image_key].to(self.device))
File "", line 36, in
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/mdstore2/Christian/YngreAlf/SegUnc/src/train.py", line 110, in forward
return self._model(x)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/mdstore2/Christian/YngreAlf/venv/lib/python3.8/site-packages/monai/networks/nets/unet.py", line 311, in forward
x = self.model(x)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/lib/python3/dist-packages/torch/nn/modules/container.py", line 204, in forward
input = module(input)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/mdstore2/Christian/YngreAlf/venv/lib/python3.8/site-packages/monai/networks/layers/simplelayers.py", line 130, in forward
y = self.submodule(x)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/lib/python3/dist-packages/torch/nn/modules/container.py", line 204, in forward
input = module(input)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/mdstore2/Christian/YngreAlf/venv/lib/python3.8/site-packages/monai/networks/layers/simplelayers.py", line 130, in forward
y = self.submodule(x)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/lib/python3/dist-packages/torch/nn/modules/container.py", line 204, in forward
input = module(input)
File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/mdstore2/Christian/YngreAlf/venv/lib/python3.8/site-packages/monai/networks/layers/simplelayers.py", line 133, in forward
return torch.cat([x, y], dim=self.dim)
File "/mnt/mdstore2/Christian/YngreAlf/venv/lib/python3.8/site-packages/monai/data/meta_tensor.py", line 268, in torch_function
ret = super().torch_function(func, types, args, kwargs)
File "/usr/lib/python3/dist-packages/torch/_tensor.py", line 1279, in torch_function
ret = func(*args, **kwargs)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 19 but got size 20 for tensor number 1 in the list.

1 reply

KumoLiu Apr 4, 2023
Maintainer

From the error message, seems the outputs have different spatial sizes. I noticed you are using RandFlip, just a guess, are the dimensions of the two axes you are changing the same? Before using TTA, maybe try to check the shape of your data. Hope it can help you, Thanks!

jamtheim · 2023-04-04T07:47:13Z

jamtheim
Apr 4, 2023
Author

I dont really get how the sliding windows inference is called at all, is this the root of the problem? If so, how should i fix it?

0 replies

jamtheim · 2023-04-04T11:07:15Z

jamtheim
Apr 4, 2023
Author

Dear KumoLiu, thanks for your guideance. It was a combination of wring device allocation and missing sliding window function. Below is working code:

# One sample file        
file = inf_files[0]

num_samples = 10

# Set GPU to  index 1
device=torch.device("cuda:1" if torch.cuda.is_available() else "cpu")
# Move model to GPU 1
model = model.to(torch.device(device))


# New inference function
roi_size = conf.model.slidingWindowInfSize
sw_batch_size = conf.model.inference_sw_batch_size
inferFunction = lambda x: torch.softmax( sliding_window_inference(x, roi_size, sw_batch_size, model.forward), dim=1 )

tt_aug = TestTimeAugmentation(transform_TTA, batch_size=1, num_workers=12, inferrer_fn=inferFunction, device=device, progress=True, output_device="cpu", return_full_data=False)  
transform_TTA.set_random_state(seed=123)

# Get images
mode_tta, mean_tta, std_tta, vvc_tta = tt_aug(file, num_examples=num_samples)

# Convert no numpy
mode_tta = mode_tta.numpy()
mean_tta = mean_tta.numpy()
std_tta = std_tta.numpy()

0 replies

jamtheim · 2023-04-13T11:49:55Z

jamtheim
Apr 13, 2023
Author

Hi again,
So I got it to work as intended. But know I have run into a problem and I could need some help.
As standard before validation and inference I am doing the transform:

inf_transforms = [
LoadImaged(keys=["image", "label"]),
EnsureChannelFirstd(keys=["image"]),
ScaleIntensityRanged(
keys=["image"],
a_min=conf.model.ScaleIntensityRanged_a_min,
a_max=conf.model.ScaleIntensityRanged_a_max,
b_min=conf.model.ScaleIntensityRanged_b_min,
b_max=conf.model.ScaleIntensityRanged_b_max,
clip=conf.model.ScaleIntensityRanged_clip,
)]

I cant figure out how to apply this as a standard transform before the data enters the test time augmentation module. I dont think I can put in the test time transform as I do not want it to be reversed. As stated in our previous discussion the test time augmentation takes a dictionary pair as input, thereby I cant figure out how to apply a standard transform beforehand.

Thankful for your help!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Input to TestTimeAugmentation in combination with Lightning #6263

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 5 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Input to TestTimeAugmentation in combination with Lightning #6263

Uh oh!

Uh oh!

jamtheim Mar 31, 2023

Replies: 5 comments · 3 replies

Uh oh!

KumoLiu Apr 4, 2023 Maintainer

Uh oh!

jamtheim Apr 4, 2023 Author

Load the model file

Disable randomness, dropout, etc for evaluation

Uh oh!

KumoLiu Apr 4, 2023 Maintainer

Uh oh!

jamtheim Apr 4, 2023 Author

Set GPU to index 1

Move model to GPU 1

Set inference function

Get images

Uh oh!

KumoLiu Apr 4, 2023 Maintainer

Uh oh!

jamtheim Apr 4, 2023 Author

Uh oh!

jamtheim Apr 4, 2023 Author

Uh oh!

jamtheim Apr 13, 2023 Author

jamtheim
Mar 31, 2023

Replies: 5 comments 3 replies

KumoLiu
Apr 4, 2023
Maintainer

jamtheim Apr 4, 2023
Author

KumoLiu Apr 4, 2023
Maintainer

jamtheim
Apr 4, 2023
Author

KumoLiu Apr 4, 2023
Maintainer

jamtheim
Apr 4, 2023
Author

jamtheim
Apr 4, 2023
Author

jamtheim
Apr 13, 2023
Author