"NotImpletementError: sharing CUDA metatensor across processes not implemented" when num_workers is not zero.

[Celtyee]

The situation is that I defined my personal Pytorch Dataset for DICOM files, the MONAI transformer is applied in the "getitem" function. The code snippet is as follows,
`class RCTDataset3D(torch.utils.data.Dataset, Randomizable):
def init(self, csv, mode=None, transform=None):
self.csv = csv.reset_index()
self.mode = mode
self.transform = transform

def __len__(self):
    return self.csv.shape[0]

def randomize(self) -> None:
    MAX_SEED = np.iinfo(np.uint32).max + 1
    self._seed = self.R.randint(MAX_SEED, dtype="uint32")

def __getitem__(self, index):
    self.randomize()
    index = index  
    row = self.csv.iloc[index]
    folder_name = search_dir_by_keyword(data_dir, row.name)
    folder_path = search_dir_by_keyword(folder_name, 'pre-operation')
    if os.path.exists(folder_path) is False:
        print(folder_path)
    dcm_lst = load_dicom_array(folder_path) if os.path.exists(folder_path) else []
    if len(dcm_lst) == 0:
        print(folder_path)
        exit()
    dcm_stack = torch.stack(dcm_lst)  # (d, h, w)
    data = dcm_stack
    if self.transform is not None:
        if isinstance(self.transform, Randomizable):
            self.transform.set_random_state(seed=self._seed)
        data = self.transform(dcm_stack)
    data = data.unsqueeze(0)  # (c, d, h, w)

    label = torch.zeros(num_classes)
    # one-hot label generation
    label[row[-1] - 1] = 1  

    return data, label`

The MONAI transformer is,
train_transforms = Compose([ScaleIntensity(), Resize((resize_h, resize_w)), RandAffine( prob=0.6, translate_range=(10, 10, 10), rotate_range=(np.pi * 4, np.pi * 4, np.pi * 4), scale_range=(0.15, 0.15, 0.15), padding_mode='border') ])
When I define the dataloader for the dataset and fetch the first data to check its shape,
`check_ds = RCTDataset3D(csv=pd.read_csv(train_csv), transform=train_transforms)
check_loader = DataLoader(check_ds, batch_size=batch_size, shuffle=True, num_workers=4)

im, labels = monai.utils.misc.first(check_loader)
logging.info(f"{type(im)}, {im.shape}, {labels}, {labels.shape}")`

an error raises,
site-packages/monai/data/__init__.py", line 142, in reduce_meta_tensor raise NotImplementedError("sharing CUDA metatensor across processes not implemented") NotImplementedError: sharing CUDA metatensor across processes not implemented.
I read the source code find there is an check for **.is_cuda() and raise this error message. Does anyone face this question and have any advices to fix it?

Major environment config:
python: 3.8
MONAI: 1.2.0

[wyli]

yes, the root cause is that cuda runtime doesn't support fork start method for multiprocessing https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing, I think you can try either make sure the transforms are operating on cpu data, or use monai's thread data loader (https://github.com/Project-MONAI/tutorials/blob/1.3.0/acceleration/fast_training_tutorial.ipynb), or use torch.multiprocessing.set_start_method('spawn').

[Celtyee]

Thank you for your reply. I find that when I turn the image value into torch.tensor, the tensor is on GPU beause of device argument. I remove the arg. to make sure that transform is applied on cpu data.