Handling bad dicom images

[vikashg]

Hello all,
I was wondering if we could add a feature for checking the file integrity during the batch processing. Let me elaborate.
If we have say a file list of say 1000 dicom images. It will be so that there are few dicom images which are corrupted or are presentation states, that is they wont be readable by SimpleITK or Pydicom. So, if the dataloader encounters such a file (a corrupted file will typically be under 10.0 K in size compared to 100s of MB for a normal dicom file), that dataloader will ignore that file instead of breaking the training process

 File "/home/gupta/anaconda3/envs/monai/lib/python3.9/site-packages/SimpleITK/SimpleITK.py", line 8015, in Execute
    return _SimpleITK.ImageFileReader_Execute(self)
RuntimeError: Exception thrown in SimpleITK ImageFileReader_Execute: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:105:
sitk::ERROR: Unable to determine ImageIO reader for filename.dcm

[rijobro]

Alternatively you could just remove bad filenames as part of the pre-processing step:

def get_good_filenames(filenames):
    out = []
    loader = LoadImage()
    for file in filenames:
        try:
            loader(file)
        except:
            continue
        out.append(file)
    return out

filenames = get_good_filenames(all_filenames)

[vikashg]

I was thinking something more like

class TrainDataset(torch.utils.data.Dataset):
	def __init__(self, image_files, image_labels, transforms):
		self.image_files = image_files
		self.image_labels = image_labels
		self.transforms = transforms

	def __len__(self):
		return len(self.image_files)

	def __getitem__(self, index):
		try:
			LoadImage(self.image_files[index])
		except:
			continue
        	return self.transforms(self.image_files[index]), self.image_labels[index]

I know this is a wrong code as I am using try outside for loop but that was kind of the idea to check file integrity in real time. We can definitely triage the bad files before hand, but that was not what I was thinking.

[rijobro]

I can see why this code might be preferable as it looks tidier, but it would (a) be slower as on each epoch you would try to read the problematic images and (b) I doubt this would work. I doubt it would work because the dataset might return None for the problematic images. The DataLoader would then collate a list that would contain non-None entries for good images and None for problematic images, which using the default collate function, would raise an exception.

You could create your own collate function, but you would end up with batches of varying sizes depending on how many problematic images are in that given batch. And you would need to write more boiler plate code, at which point I feel like it would have been simpler to do it as a pre-processing step.

For reference, the collate function might look like this:

def remove_bad_collate(batch):
    batch = [b for b in batch if b is not None]
    return list_data_collate(batch)
train_loader = DataLoader(train_ds, collate_fn=remove_bad_collate)

[vikashg]

Thanks, yes I understand it will be slower. Infact training directly on dicom images is 100x slower than training on numpy arrays. But sometimes its preferable to train directly on dcms to have a cleaner end to end systems. I used to have such check in the generator function when I was working with keras. Unfortunately very new to the Pytorch way of doing things.
Thanks for the pointers.

[rijobro]

For speed concerns around the loading of DICOMs, you may want to look into PersistentDataset and CacheDataset if you haven't already: https://github.com/Project-MONAI/tutorials/blob/master/acceleration/dataset_type_performance.ipynb.

[vikashg]

Yes I have that in mind only after making the most basic stuff work.