OOM when predict in transforms.Invertd #4191

XIANYUNYEHE-DEL · 2022-04-28T02:26:49Z

XIANYUNYEHE-DEL
Apr 28, 2022

There are no error in my training process and most of prediction. But when i predict some file , there will be an error about cuda oom. I dont konw how to fix it. My transforms and prediction code at this.

val_dicts = [
    {"image": img_exam}
    for img_exam in img_exam[9:10]
]
labels = [1,2,3,4,5,6,7,8,9,10,11,12,13]
val_transform = transforms.Compose(
        [
            transforms.LoadImaged(keys="image"),
            transforms.AddChanneld(keys="image"),
            transforms.Orientationd(keys=["image"],
                                    axcodes="RAS"),
            transforms.Spacingd(keys=["image"],
                                pixdim=(1.5, 1.5, 2),
                                mode=("bilinear")),
            transforms.CropForegroundd(keys=["image"], source_key="image"),
            transforms.Resized(keys=["image"], spatial_size=[160,160,80], mode=("area")),
            transforms.ScaleIntensityRanged(keys=["image"],
                                            a_min=-135,
                                            a_max=215,
                                            b_min=0,
                                            b_max=1,
                                            clip=True),
        ]
    )
loader2 = Dataset(data=val_dicts,transform=val_transform)
test_loader = DataLoader(loader2,
                        batch_size=1,
                        shuffle=False,
                        num_workers=4,
                        pin_memory=True,
                        persistent_workers=True)
post_transforms = Compose([
    transforms.EnsureTyped(keys="pred"),
    transforms.Invertd(
        keys="pred",
        transform=val_transform,
        orig_keys="image",
        meta_keys="pred_meta_dict",
        orig_meta_keys="image_meta_dict",
        meta_key_postfix="meta_dict",
        nearest_interp=False,
        to_tensor=True,
    ),
    transforms.AsDiscreted(keys="pred", argmax=True),
    transforms.KeepLargestConnectedComponentd(keys="pred", applied_labels=labels),
    transforms.SaveImaged(keys="pred", meta_keys="pred_meta_dict", output_dir="./out", output_postfix="seg", resample=False, separate_folder=False),
])
check_data = first(test_loader)
print(check_data['image'].shape)

model = unet(in_channels=1, out_channels=14).cuda()
weights_dict = torch.load('./weight/best_metric_model_64.pth')
model.load_state_dict(weights_dict, strict=False)
model.eval()
with torch.no_grad():
    for idx, batch_data in enumerate(test_loader):
        data = batch_data['image']
        data = data.cuda()
        batch_data['pred'] = model(data)
        batch_data = [post_transforms(i) for i in decollate_batch(batch_data)]

RuntimeError                              Traceback (most recent call last)
File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py:89, in apply_transform(transform, data, map_items, unpack_items, log_stats)
     [88](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=87)         return [_apply_transform(transform, item, unpack_items) for item in data]
---> [89](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=88)     return _apply_transform(transform, data, unpack_items)
     [90](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=89) except Exception as e:

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py:53, in _apply_transform(transform, parameters, unpack_parameters)
     [51](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=50)     return transform(*parameters)
---> [53](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=52) return transform(parameters)

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py:286, in Spacingd.inverse(self, data)
    [285](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=284) # Apply inverse
--> [286](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=285) d[key], _, new_affine = inverse_transform(
    [287](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=286)     data_array=d[key],
    [288](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=287)     affine=meta_data["affine"],  # type: ignore
    [289](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=288)     mode=mode,
    [290](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=289)     padding_mode=padding_mode,
    [291](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=290)     align_corners=False if align_corners == TraceKeys.NONE else align_corners,
    [292](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=291)     dtype=dtype,
    [293](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=292)     output_spatial_shape=orig_size,
    [294](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=293) )
    [295](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py?line=294) meta_data["affine"] = new_affine  # type: ignore

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py:227, in Spacing.__call__(self, data_array, affine, mode, padding_mode, align_corners, dtype, output_spatial_shape)
    [220](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=219)     output_data = affine_xform(
    [221](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=220)         # AffineTransform requires a batch dim
    [222](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=221)         data_array_t.unsqueeze(0),
    [223](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=222)         convert_data_type(transform, torch.Tensor, data_array_t.device, dtype=_dtype)[0],
    [224](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=223)         spatial_size=output_shape if output_spatial_shape is None else output_spatial_shape,
    [225](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=224)     ).squeeze(0)
--> [227](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=226) output_data, *_ = convert_to_dst_type(output_data, data_array, dtype=torch.float32)
    [228](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/spatial/array.py?line=227) new_affine = to_affine_nd(affine_np, new_affine)  # type: ignore

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py:301, in convert_to_dst_type(src, dst, dtype, wrap_sequence)
    [300](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=299)     output_type = type(dst)
--> [301](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=300) return convert_data_type(data=src, output_type=output_type, device=device, dtype=dtype, wrap_sequence=wrap_sequence)

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py:261, in convert_data_type(data, output_type, device, dtype, wrap_sequence)
    [260](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=259) if output_type is torch.Tensor:
--> [261](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=260)     data = convert_to_tensor(data, dtype=dtype_, device=device, wrap_sequence=wrap_sequence)
    [262](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=261) elif output_type is np.ndarray:

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py:120, in convert_to_tensor(data, dtype, device, wrap_sequence)
    [119](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=118) if isinstance(data, torch.Tensor):
--> [120](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=119)     return data.to(dtype=dtype, device=device, memory_format=torch.contiguous_format)  # type: ignore
    [121](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=120) if isinstance(data, np.ndarray):
    [122](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=121)     # skip array of string classes and object, refer to:
    [123](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/utils/type_conversion.py?line=122)     # https://github.com/pytorch/pytorch/blob/v1.9.0/torch/utils/data/_utils/collate.py#L13

RuntimeError: CUDA out of memory. Tried to allocate 10.69 GiB (GPU 0; 39.59 GiB total capacity; 28.97 GiB already allocated; 8.20 GiB free; 29.13 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py:89, in apply_transform(transform, data, map_items, unpack_items, log_stats)
     [88](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=87)         return [_apply_transform(transform, item, unpack_items) for item in data]
---> [89](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=88)     return _apply_transform(transform, data, unpack_items)
     [90](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=89) except Exception as e:

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py:53, in _apply_transform(transform, parameters, unpack_parameters)
     [51](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=50)     return transform(*parameters)
---> [53](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=52) return transform(parameters)

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/post/dictionary.py:643, in Invertd.__call__(self, data)
    [642](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/post/dictionary.py?line=641) with allow_missing_keys_mode(self.transform):  # type: ignore
--> [643](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/post/dictionary.py?line=642)     inverted = self.transform.inverse(input_dict)
    [645](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/post/dictionary.py?line=644) # save the inverted data

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py:183, in Compose.inverse(self, data)
    [182](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py?line=181) for t in reversed(invertible_transforms):
--> [183](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py?line=182)     data = apply_transform(t.inverse, data, self.map_items, self.unpack_items, self.log_stats)
    [184](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py?line=183) return data

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py:113, in apply_transform(transform, data, map_items, unpack_items, log_stats)
    [112](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=111)         _log_stats(data=data)
--> [113](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=112) raise RuntimeError(f"applying transform {transform}") from e

RuntimeError: applying transform <bound method Spacingd.inverse of <monai.transforms.spatial.dictionary.Spacingd object at 0x7f26dcc14d00>>

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
/home/zhw/code/monai/predict.ipynb Cell 5' in <module>
     [10](vscode-notebook-cell://ssh-remote%2B10.11.65.5/home/zhw/code/monai/predict.ipynb#ch0000004vscode-remote?line=9) batch_data['pred'] = model(data)
     [11](vscode-notebook-cell://ssh-remote%2B10.11.65.5/home/zhw/code/monai/predict.ipynb#ch0000004vscode-remote?line=10) torch.cuda.empty_cache()
---> [12](vscode-notebook-cell://ssh-remote%2B10.11.65.5/home/zhw/code/monai/predict.ipynb#ch0000004vscode-remote?line=11) batch_data = [post_transforms(i) for i in decollate_batch(batch_data)]

/home/zhw/code/monai/predict.ipynb Cell 5' in <listcomp>(.0)
     [10](vscode-notebook-cell://ssh-remote%2B10.11.65.5/home/zhw/code/monai/predict.ipynb#ch0000004vscode-remote?line=9) batch_data['pred'] = model(data)
     [11](vscode-notebook-cell://ssh-remote%2B10.11.65.5/home/zhw/code/monai/predict.ipynb#ch0000004vscode-remote?line=10) torch.cuda.empty_cache()
---> [12](vscode-notebook-cell://ssh-remote%2B10.11.65.5/home/zhw/code/monai/predict.ipynb#ch0000004vscode-remote?line=11) batch_data = [post_transforms(i) for i in decollate_batch(batch_data)]

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py:173, in Compose.__call__(self, input_)
    [171](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py?line=170) def __call__(self, input_):
    [172](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py?line=171)     for _transform in self.transforms:
--> [173](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py?line=172)         input_ = apply_transform(_transform, input_, self.map_items, self.unpack_items, self.log_stats)
    [174](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/compose.py?line=173)     return input_

File ~/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py:113, in apply_transform(transform, data, map_items, unpack_items, log_stats)
    [111](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=110)     else:
    [112](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=111)         _log_stats(data=data)
--> [113](file:///home/zhw/miniconda3/envs/jupyter01/lib/python3.8/site-packages/monai/transforms/transform.py?line=112) raise RuntimeError(f"applying transform {transform}") from e

RuntimeError: applying transform <monai.transforms.post.dictionary.Invertd object at 0x7f26dcc6c460>

Answered by Nic-Ma

Apr 28, 2022

Hi @XIANYUNYEHE-DEL ,

Thanks for your interest here and experiments.
I think that may be because your inverse transform computation is on GPU Tensor, could you please help add EnsureTyped(keys="image") to your preprocessing transform chain and try again? Refer to:
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/torch/unet_inference_dict.py#L61

Thanks in advance.

View full answer

Nic-Ma · 2022-04-28T03:52:43Z

Nic-Ma
Apr 28, 2022
Maintainer

Hi @XIANYUNYEHE-DEL ,

Thanks for your interest here and experiments.
I think that may be because your inverse transform computation is on GPU Tensor, could you please help add EnsureTyped(keys="image") to your preprocessing transform chain and try again? Refer to:
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/torch/unet_inference_dict.py#L61

Thanks in advance.

2 replies

XIANYUNYEHE-DEL Apr 28, 2022
Author

Thanks for your reply. Your suggestion solved my problem effectively. Also thank your nice work about monai.

Nic-Ma Apr 28, 2022
Maintainer

No problem, please feel free to raise tickets if you face any other issues.

Thanks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

OOM when predict in transforms.Invertd #4191

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

OOM when predict in transforms.Invertd #4191

Uh oh!

XIANYUNYEHE-DEL Apr 28, 2022

Replies: 1 comment · 2 replies

Uh oh!

Nic-Ma Apr 28, 2022 Maintainer

Uh oh!

XIANYUNYEHE-DEL Apr 28, 2022 Author

Uh oh!

Nic-Ma Apr 28, 2022 Maintainer

XIANYUNYEHE-DEL
Apr 28, 2022

Replies: 1 comment 2 replies

Nic-Ma
Apr 28, 2022
Maintainer

XIANYUNYEHE-DEL Apr 28, 2022
Author

Nic-Ma Apr 28, 2022
Maintainer