RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByLabelClassesd object at 0x2b9c38c25340>

[lyangfan]

Hi, when I want to train a 3d Unet segmentation model , I meet the error at the 133th epoch:

Traceback (most recent call last):
  File "train.py", line 87, in <module>
    main()
  File "train.py", line 84, in main
    train(img_dir, label_dir)
  File "train.py", line 36, in train
    mean_loss = train_one_epoch(model, loss_function, optimizer, train_dataloader, device, epoch + 1)
  File "/public/home/three_whyz123/yhfu/yfliu/run/monai_seg/carcass_9_class_3dUnet/utils.py", line 63, in train_one_epoch
    for step, data in enumerate(data_loader):
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/transforms/transform.py", line 89, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/transforms/transform.py", line 53, in _apply_transform
    return transform(parameters)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/transforms/croppad/dictionary.py", line 1357, in __call__
    self.randomize(label, indices, image)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/transforms/croppad/dictionary.py", line 1344, in randomize
    indices_ = map_classes_to_indices(label, self.num_classes, image, self.image_threshold)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/transforms/utils.py", line 362, in map_classes_to_indices
    label_flat = ravel(any_np_pt(label[c : c + 1] if channels > 1 else label == c, 0))
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/transforms/utils_pytorch_numpy_unification.py", line 237, in any_np_pt
    x = torch.any(x, ax)
TypeError: any() received an invalid combination of arguments - got (bool, int), but expected one of:
 * (Tensor input, *, Tensor out)
 * (Tensor input, int dim, bool keepdim, *, Tensor out)
 * (Tensor input, name dim, bool keepdim, *, Tensor out)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/data/dataset.py", line 97, in __getitem__
    return self._transform(index)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/data/dataset.py", line 807, in _transform
    data = apply_transform(_transform, data)
  File "/public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/transforms/transform.py", line 113, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByLabelClassesd object at 0x2b9c38c25340>

It seems that some tranforms of the training stage are failded. I'm confused why it didn't fail in the beginning?And How can I solve this problem?

This is my transforms scripts:

from torchio.transforms import (
    RandomNoise,
    RandomBlur,
    RandomAnisotropy
)

from monai.transforms import (
    OneOf,
    RandCropByLabelClassesd,
    RandAffined,
    RandScaleIntensityd,
    EnsureChannelFirstd,
    Compose,
    CropForegroundd,
    LoadImaged,
    Orientationd,
    RandFlipd,
    ScaleIntensityRanged,
    Spacingd,
    NormalizeIntensityd,
    RandAdjustContrastd,
    ToTensord
)

def get_tio_dataset_augmentation():
    torchio_transforms = [
        [RandomNoise(
            p=0.15,
            std=(0, 0.1),
            include=["image"]
        ),
        RandomBlur(
            p=0.1,
            std=(0.5, 1.5),
            include=["image"]
        )],
        [RandomAnisotropy(
            p=0.125,
            downsampling=(1, 2),
            include=["image"]
        )]
    ]

    return torchio_transforms


def get_train_transforms():
    monai_transforms = [
        [LoadImaged(keys=["image", "label"]),
        EnsureChannelFirstd(keys=["image", "label"]),
        Orientationd(keys=["image", "label"], axcodes="RAS"),
        Spacingd(
            keys=["image", "label"],
            pixdim=(1.5, 1.5, 2.5),
            mode=("bilinear", "nearest"),
        ),
        CropForegroundd(keys=["image", "label"], source_key="image"),
        ScaleIntensityRanged(
            keys=["image"],
            a_min=-991,
            a_max=617,
            b_min=-991,
            b_max=617,
            clip=True,
        ),
        NormalizeIntensityd(
            keys=["image"],
            subtrahend=5.98,
            divisor=192.71,
        ),
        RandCropByLabelClassesd(
            keys=["image", "label"],
            label_key="label",
            spatial_size=(128, 128, 128),
            num_classes=9,
            num_samples=9,
        ),
        OneOf([
            RandAffined(
                keys=["image", "label"],
                prob=0.48,
                scale_range=(0.7, 1.4)),            
            RandAffined(
                keys=["image", "label"],
                prob=0.48,
                rotate_range =(-30, 30)),
            RandAffined(
                keys=["image", "label"],
                prob=0.24,
                scale_range=(0.7, 1.4),
                rotate_range =(-30, 30)),
            ]),
        ],
        [RandScaleIntensityd(
            keys=["image"],
            factors=0.3,
            prob=0.15
        )],
        [RandAdjustContrastd(
            keys=["image"],
            prob=0.15,
            gamma=(0.7, 1.5)
        ),
        RandFlipd(
            keys=["image", "label"],
            prob=0.5,
            spatial_axis=1
        ),
        ToTensord(keys=["image", "label"])
        ]
    ]

    tio_transforms = get_tio_dataset_augmentation()
    train_transforms = Compose(monai_transforms[0] + tio_transforms[0] + monai_transforms[1] + 
                                    tio_transforms[1] + monai_transforms[2])

    return train_transforms

def get_validation_transforms():
    val_transforms = Compose(
    [
        LoadImaged(keys=["image", "label"]),
        EnsureChannelFirstd(keys=["image", "label"]),
        Orientationd(keys=["image", "label"], axcodes="RAS"),
        Spacingd(
            keys=["image", "label"],
            pixdim=(1.5, 1.5, 2.5),
            mode=("bilinear", "nearest"),
        ),
        CropForegroundd(keys=["image", "label"], source_key="image"),
        ScaleIntensityRanged(
            keys=["image"],
            a_min=-991,
            a_max=617,
            b_min=-991,
            b_max=617,
            clip=True,
        ),
        NormalizeIntensityd(
            keys=["image"],
            subtrahend=5.98,
            divisor=192.71,
        ),
        ToTensord(keys=["image", "label"])
    ]
    )
    return val_transforms 

These are configs

MONAI version: 0.9.0
Numpy version: 1.22.3
Pytorch version: 1.13.1+cu117
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: af0e0e9f757558d144b655c63afcea3a4e0a06f5
MONAI __file__: /public/home/three_whyz123/miniconda3/envs/deep_learning/lib/python3.8/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 4.0.1
scikit-image version: 0.19.3
Pillow version: 9.1.0
Tensorboard version: 2.10.1
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.14.1+cu117
tqdm version: 4.64.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: NOT INSTALLED or UNKNOWN VERSION.
pandas version: 1.4.2
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: 3.0.0
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

Thanks for reply!
Yangfan Liu

[wyli]

it seems the transforms are used with a CacheDataset, could you please share how the dataset is initialised? I can try to reproduce if you provide more implementation details.

[lyangfan]

Thanks for your reply! The dataset script:

import os
import glob
import random
from monai.data import CacheDataset
from transforms import get_train_transforms, get_validation_transforms

def get_path_dict(image_dir, label_dir, train_dataset_rate=0.8):
    train_images_path = glob.glob(os.path.join(image_dir, '*.nii.gz'))
    train_labels_path = glob.glob(os.path.join(label_dir, '*.nii.gz'))
    data_dict = [{'image': image_path, 'label': label_path} for image_path, label_path in zip(train_images_path, train_labels_path)]
    random.shuffle(data_dict)
    train_dataset_size = int(len(data_dict) * train_dataset_rate)
    train_data_dict, val_data_dict = data_dict[:train_dataset_size], data_dict[train_dataset_size:]
    return train_data_dict, val_data_dict

def get_monai_dataset(image_dir, label_dir):
    train_data_dict, val_data_dict = get_path_dict(image_dir, label_dir)
    train_transforms = get_train_transforms()
    val_transforms = get_validation_transforms()
    train_dataset = CacheDataset(
        data=train_data_dict, transform=train_transforms,
        cache_rate=1.0, num_workers=4)
    val_dataset = CacheDataset(
        data=val_data_dict, transform=val_transforms,
        cache_rate=1.0, num_workers=4)
    return train_dataset, val_dataset

[wyli]

thanks, I still couldn't reproduce this issue, I saw another related discussion using PersistentDataset #2674. maybe it's hard disk issue if your system RAM is not large enough and the cache content are in the swap space.

[lyangfan]

I found the reason. As you say, the system RAM is not large enough.Thanks for your reply!