Batch read and transforms image errors of different bits

[Minxiangliu]

I want to use LoadImage to read some images. The attributes of the images are different. In the batch operation, after I read the 24bits image first, after reading the 8bits image, it will cause an error. If I use 1 batch to read, it will not be wrong.
How to solve it please?
Thanks in advance!

Code:

from monai.data import list_data_collate
from monai.data import CacheDataset, ThreadDataLoader
from monai.transforms.transform import MapTransform
from monai.transforms import (Compose, LoadImaged, Resized, AddChanneld)

class ConvertGray(MapTransform):
    def __init__(self, keys):
        super().__init__(keys)
    def __call__(self, data):
        d = dict(data)
        for key in self.key_iterator(d):
            if len(d[key].shape) == 3:
                R, G, B = d[key][:,:,0], d[key][:,:,1], d[key][:,:,2]
                imgGray = 0.2989 * R + 0.5870 * G + 0.1140 * B
                d[key] = imgGray
        return d

# CHNCXR_0548_1.png is 8bits
# CHNCXR_0658_1.png is 24bits
path1 = ['CHNCXR_0548_1.png','CHNCXR_0658_1.png']    #everything is normal
path2 = ['CHNCXR_0658_1.png','CHNCXR_0548_1.png']    #Using path2 and non1 batch will give an error

_files = [{"image": image_name} for image_name in path2]
_transforms = [LoadImaged(keys=['image']),
               ConvertGray(keys=['image']),
               AddChanneld(keys=['image']),
               Resized(keys=['image'], spatial_size=(256,256))]
dataset = CacheDataset(data=_files,
                       transform=Compose(_transforms),
                       cache_rate=1.0,
                       copy_cache=False)

dataloader = ThreadDataLoader(dataset, 
                              num_workers=0, 
                              batch_size=2,    #Error occurs when using path2 under non 1 batch.
                              shuffle=False,
                              drop_last=False,
                              collate_fn=list_data_collate)

for n,i in enumerate(dataloader):
    print(n,i['image'].shape)

Error:

Loading dataset: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  5.88it/s]
Exception in thread Thread-13:
Traceback (most recent call last):
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\_utils\collate.py", line 157, in default_collate
    return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\_utils\collate.py", line 157, in <dictcomp>     
    return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\_utils\collate.py", line 152, in default_collate
    return torch.tensor(batch)
TypeError: new(): invalid data type 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Anaconda3\envs\monai\lib\site-packages\monai\data\utils.py", line 272, in list_data_collate
    ret[k] = default_collate([d[k] for d in data])
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\_utils\collate.py", line 160, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\_utils\collate.py", line 160, in <dictcomp>     
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\_utils\collate.py", line 152, in default_collate
    return torch.tensor(batch)
TypeError: new(): invalid data type 'str'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Anaconda3\envs\monai\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "D:\Anaconda3\envs\monai\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py", line 45, in enqueue_values
    for src_val in self.src:
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py", line 530, in __next__
    data = self._next_data()
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py", line 570, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "D:\Anaconda3\envs\monai\lib\site-packages\torch\utils\data\_utils\fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "D:\Anaconda3\envs\monai\lib\site-packages\monai\data\utils.py", line 296, in list_data_collate
    raise TypeError(re_str) from re
TypeError: new(): invalid data type 'str'

Eenvironment:

MONAI version: 0.8.0
Numpy version: 1.21.5
Pytorch version: 1.11.0+cu113
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: 714d00dffe6653e21260160666c4c201ab66511b

Optional dependencies:
Pytorch Ignite version: 0.4.8
Nibabel version: 3.2.1
scikit-image version: 0.19.1
Pillow version: 9.0.0
Tensorboard version: 2.8.0
gdown version: 4.2.0
TorchVision version: 0.12.0+cu113
tqdm version: 4.62.3
lmdb version: 1.3.0
psutil version: 5.9.0
pandas version: 1.3.5
einops version: 0.4.0
transformers version: 4.16.0
mlflow version: 1.23.1

CHNCXR_0548_1.png
CHNCXR_0658_1.png

[Nic-Ma]

Hi @Minxiangliu ,

Thanks for your interest here.
I think maybe you can try to add a CastToType transform to make all the images have same data type when batching?
https://github.com/Project-MONAI/MONAI/blob/dev/monai/transforms/utility/array.py#L313

Thanks.

[Minxiangliu]

Hi, @Nic-Ma ,
Thanks for your reply, I have tried your method, but it doesn't solve the problem no matter where I add it. What else can I do please?

[Minxiangliu]

Hi, @Nic-Ma ,
Ok, the current solution, I have to use opencv to read in the image, convert it to grayscale and store it, and then use the same (except ConvertGray) transforms, everything works fine, I still don't know where the problem is.

import cv2 as cv
img = cv.imread('img.png')
img_gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
cv.imwrite('img.png', img_gray)

[Nic-Ma]

Seems if the images are not in the same format, it can't batch them together for the network.

Thanks.