Problem with CacheDataset

[ctestagrose]

I have been using the tutorials to try and implement CacheDataset. However, I am receiving an error when trying to apply a transform to an image while using CacheDataset. The specific error I am receiving is:

Traceback (most recent call last):
  File "/workspace/MONAI/monai/transforms/transform.py", line 81, in apply_transform
    return [_apply_transform(transform, item, unpack_items) for item in data]
  File "/workspace/MONAI/monai/transforms/transform.py", line 81, in <listcomp>
    return [_apply_transform(transform, item, unpack_items) for item in data]
  File "/workspace/MONAI/monai/transforms/transform.py", line 53, in _apply_transform
    return transform(parameters)
  File "/workspace/MONAI/monai/transforms/io/array.py", line 186, in __call__
    filename = tuple(f"{Path(s).expanduser()}" for s in ensure_tuple(filename))  # allow Path objects
  File "/workspace/MONAI/monai/transforms/io/array.py", line 186, in <genexpr>
    filename = tuple(f"{Path(s).expanduser()}" for s in ensure_tuple(filename))  # allow Path objects
  File "/opt/conda/lib/python3.8/pathlib.py", line 1042, in __new__
    self = cls._from_parts(args, init=False)
  File "/opt/conda/lib/python3.8/pathlib.py", line 683, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/opt/conda/lib/python3.8/pathlib.py", line 667, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not numpy.ndarray

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train_cache.py", line 157, in <module>
    main()
  File "train_cache.py", line 73, in main
    train_ds = CacheDataset(
  File "/workspace/MONAI/monai/data/dataset.py", line 710, in __init__
    self._cache: List = self._fill_cache()
  File "/workspace/MONAI/monai/data/dataset.py", line 731, in _fill_cache
    return list(
  File "/opt/conda/lib/python3.8/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
  File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/workspace/MONAI/monai/data/dataset.py", line 751, in _load_cache_item
    item = apply_transform(_xform, item)
  File "/workspace/MONAI/monai/transforms/transform.py", line 106, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.io.array.LoadImage object at 0x7f1a835d7550>
terminate called without an active exception

This is currently how I am trying to implement CacheDataset:

    import sys
    import torch
    from monai.data import DataLoader
    from monai.transforms import (
        LoadImage,
        Resize,
        ResizeWithPadOrCrop, NormalizeIntensity,
        RepeatChannel, RandShiftIntensity,
        AddChannel, AsChannelFirst,
        AdjustContrast,
        ScaleIntensity,
        RandRotate,
        EnsureType,
        RandFlip, Compose)
    from torch.utils.tensorboard import SummaryWriter
    from monai.apps import DecathlonDataset
    from monai.data import CacheDataset, DataLoader, Dataset, decollate_batch

    from datalist_reader import DatalistReader
    from getarguments import get_arguments
    from model_def import ModelDefinition, MyModel

      class TrainDataset(torch.utils.data.Dataset):
      
          def __init__ (self, image_files, image_labels, transforms):
              self.image_files = image_files
              self.image_labels = image_labels
              self.transforms = transforms
  
          def __len__(self):
              return len(self.image_files)
  
          def __getitem__(self, index):
              return self.transforms(self.image_files[index]), torch.as_tensor(self.image_labels[index])
  
      def main():
  
          device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  
          temp_transform = Compose([LoadImage(image_only=True, reader='pilreader'), AsChannelFirst(channel_dim=2), Resize((299, 
                                        299)) ])
          train_ds = CacheDataset(
              data=TrainDataset(training_images, training_labels, temp_transform),
              transform=temp_transforms,
              cache_rate=1.0,
              num_workers=4
           )
          train_loader = DataLoader(train_ds, batch_size=int(args.training_batch_size), num_workers=4)
  
          validation_ds = CacheDataset(
              data=TrainDataset(validation_images, validation_labels, temp_transform),
              transform=temp_transforms,
              cache_rate=1.0,
              num_workers=4,
          )
          validation_loader = DataLoader(validation_ds, batch_size=int(args.validation_batch_size), num_workers=4)

I am using .jpg images

[Nic-Ma]

Hi @vikashg ,

May I know the format of your training_images? Seems it's not a list of image path strings.

Thanks.

[vikashg]

Hey @Nic-Ma The images are .jpg images. Also just another point I ran the same code without the cache dataset function and it worked. I just confirmed that the training images are indeed path strings. I printed them. Here is a snippet

/workspace/data/Training/B/1.2.840.113713.17.1538.104565410873071835590831837317229103099.jpg
/workspace/data/Training/D/1.2.840.113713.17.1538.20937298827674572594014918194588667912.jpg
/workspace/data/Training/C/1.2.840.113713.17.1538.12385883943647686737928376750204658165.jpg
/workspace/data/Training/A/1.2.840.113713.17.1538.33044068266680620359989931377135974992.jpg
/workspace/data/Training/C/1.2.840.113713.17.1538.128823555299441887560467283064047375311.jpg
Loading dataset:   0%|                                                                                                                                                      | 0/14692 [00:00<?, ?it/s]
=== Transform input info -- LoadImage ===
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 186.36923217773438)

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===

=== Transform input info -- LoadImage ===
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 167.54615783691406)

=== Transform input info -- LoadImage ===
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 167.54615783691406)

=== Transform input info -- LoadImage ===
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 167.54615783691406)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 168.84414672851562)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 167.54615783691406)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 168.84414672851562)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 168.84414672851562)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 184.510498046875)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 168.84414672851562)
Loading dataset:   0%|                                                                                                                                                      | 0/14692 [00:00<?, ?it/s]
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 184.510498046875)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 184.510498046875)
Data statistics:
Type: <class 'numpy.ndarray'> float32
Shape: (3, 200, 200)
Value range: (0.0, 184.510498046875)
Traceback (most recent call last):
  File "/workspace/MONAI/monai/transforms/transform.py", line 81, in apply_transform
    return [_apply_transform(transform, item, unpack_items) for item in data]

Well another weird thing is that I have written a small function to check data integrity where I load the files using the data loader. If they can be loaded then i use those images for training. So, I can read all the images using monai. The problem only arises when I use CacheDataset

Thanks

[Nic-Ma]

Hi @vikashg ,

Could you please help provide the full test program and some sample images or random images for test?
Then @yiheng-wang-nv can help reproduce it locally first.

Thanks in advance.

[wyli]

This is probably a usage question

train_ds = CacheDataset(
    data=TrainDataset(training_images, training_labels, temp_transform),
    transform=temp_transforms,
...

The temp_transforms (or temp_transform typo?)shouldn't be used in both TrainDataset and Cachedataset. (Converting to discussion)

[vikashg]

oh wow.. I wonder how I missed that. I called the TrainDataset inside CacheDataset. I am going through this example on [spleen segmentation] (https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/spleen_segmentation_3d.ipynb). However in that example train_files is a dictionary of images and labels and so train_transforms is composed of dictionary transfoms.

train_ds = CacheDataset(
    data=train_files, transform=train_transforms,
    cache_rate=1.0, num_workers=4)
# train_ds = monai.data.Dataset(data=train_files, transform=train_transforms)

# use batch_size=2 to load images and use RandCropByPosNegLabeld
# to generate 2 x 4 images for network training
train_loader = DataLoader(train_ds, batch_size=2, shuffle=True, num_workers=4)

I am trying to find out how to feed in cache dataset with images and labels. Maybe something like this would do

class TrainDataset(torch.utils.data.CacheDataset):
      
          def __init__ (self, image_files, image_labels, transforms):
              self.image_files = image_files
              self.image_labels = image_labels
              self.transforms = transforms
  
          def __len__(self):
              return len(self.image_files)
  
          def __getitem__(self, index):
              return self.transforms(self.image_files[index]), torch.as_tensor(self.image_labels[index])

Thoughts? and Thanks so much for pointing it out
Vikash

[vikashg]

Hi @wyli and @Nic-Ma, So I tried running CacheDataset after making the changes discussed. I still get this error

root@154a8cfa9518:/workspace/app# bash run_cache.sh 
Loading dataset:   0%|                                                                                                                                                          | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "train_cache.py", line 158, in <module>
    main()
  File "train_cache.py", line 74, in main
    train_ds = CacheDataset(
  File "/workspace/MONAI/monai/data/dataset.py", line 710, in __init__
    self._cache: List = self._fill_cache()
  File "/workspace/MONAI/monai/data/dataset.py", line 731, in _fill_cache
    return list(
  File "/opt/conda/lib/python3.8/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
  File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/workspace/MONAI/monai/data/dataset.py", line 745, in _load_cache_item
    item = self.data[idx]
KeyError: 0

My code is here just in case you want to have a look. Any suggestion on what I am missing here

[vikashg]

So another thing, I found that the above error is due to the fact that I was feeding in a dictionary to CacheDataset. and dictionary cannot be index, so when I feed a list it works just fine. However the reason that I was feeding in the dictionary was because I was following the spleen segmentation example Cell 5 in this tutorial

train_images = sorted(
    glob.glob(os.path.join(data_dir, "imagesTr", "*.nii.gz")))
train_labels = sorted(
    glob.glob(os.path.join(data_dir, "labelsTr", "*.nii.gz")))
data_dicts = [
    {"image": image_name, "label": label_name}
    for image_name, label_name in zip(train_images, train_labels)
]
train_files, val_files = data_dicts[:-9], data_dicts[-9:]

train_ds = CacheDataset(
    data=train_files, transform=train_transforms,
    cache_rate=1.0, num_workers=4)

Wondering why feeding the dictionary does not work in my case ? In addition if I feed the list and it caches, how can we feed the labels?
Thanks Looking forward to hearing from you

[Nic-Ma]

Hi @vikashg ,

I checked your code, seems you save all the data in a dict of lists:

train_data={'image': training_images, 'label': training_labels}
validation_data = {'image': validation_images, 'label': validation_labels}
train_ds = CacheDataset(
    data=train_data,
    transform=train_transforms,
    cache_rate=1.0, cache_num=10,
    num_workers=4
)

If you are using dict based transforms, the input datalist for dataset should be a list of dictionaries.
Then dataset will randomly index 1 item of the list then execute dict based transforms.
You can refer to:
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/torch/unet_evaluation_dict.py#L48

Thanks.