CUDA error: device-side assert triggered

[junxiant]

Hello everyone,

I was following the 3D segmentation with swin unetr on BTCV, which was working fine. Then I swapped over to my own datasets, and made changes to some of the transforms used. The training starts just fine, but then it triggers a cuda error. Setting

os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

did not help as it still returned the same error. I'm using a jupyter notebook. I don't think it was a memory error as nvidia-smi was showing 5gb/16gb usage when the model started training.

Could it be due to some of the transforms that I am using? These are the train transforms which I am using:

train_transforms = Compose(
    [
        LoadImaged(keys=["image", "label"], image_only=False),
        # AsDiscreted(keys=["label"], threshold=0.1),
        EnsureChannelFirstd(keys=["image", "label"]),
        # DataStatsd(keys=["image", "label"]),
        ScaleIntensityRanged(
            keys=["image"],
            a_min=-12.5,
            a_max=112.5,
            b_min=0.0,
            b_max=1.0,
            clip=True,
        ),
        CropForegroundd(keys=["image", "label"], source_key="image"),
        
        Orientationd(keys=["image", "label"], axcodes="RAS"),

        Spacingd(
            keys=["image", "label"],
            pixdim=(0.42, 0.42, 5.0), #0.45,0.42,0.5
            mode=("bilinear", "nearest"),
        ),
        
        # EnsureTyped(keys=["image", "label"], device=device, track_meta=False),
        
        RandCropByPosNegLabeld(
            keys=["image", "label"],
            label_key="label",
            spatial_size=(64, 64, 32),
            pos=1,
            neg=1,
            num_samples=num_samples,
            image_key="image",
            image_threshold=0,
            allow_smaller=True,
        ),
        # RandSpatialCropSamplesd(
        #     keys=["image", "label"],
        #     roi_size=(64, 64, 32),
        #     num_samples=num_samples,
        #     random_size=False,
        # ),        
        DivisiblePadd(keys=["image", "label"], k=32),
        RandFlipd(
            keys=["image", "label"],
            spatial_axis=[0],
            prob=0.10,
        ),
        RandFlipd(
            keys=["image", "label"],
            spatial_axis=[1],
            prob=0.10,
        ),
        RandFlipd(
            keys=["image", "label"],
            spatial_axis=[2],
            prob=0.10,
        ),
        RandRotate90d(
            keys=["image", "label"],
            prob=0.10,
            max_k=3,
        ),
        RandShiftIntensityd(
            keys=["image"],
            offsets=0.10,
            prob=0.50,
        ),
    ]
)

I was also using DiceFocalLoss with

loss_function = DiceFocalLoss(to_onehot_y=True, softmax=True)

Swapping over to DiceCELoss also returned the same error. I am using SwinUNetr with in_channels=1 and out_channels=2, these are the settings that I have

model = SwinUNETR(
    img_size=(64, 64, 32),
    in_channels=1,
    out_channels=2, # 1 for background, 1 for label
    feature_size=48,
    use_checkpoint=True,
).to(device)

with device set to torch.device("cuda")

Error:

Training (3 / 20000 Steps) (loss=0.75181):   3%|▎         | 4/155 [00:13<08:24,  3.34s/it]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-855e7bf4a41e> in <module>
     13 metric_values = []
     14 while global_step < max_iterations:
---> 15     global_step, dice_val_best, global_step_best = train(global_step, train_loader, dice_val_best, global_step_best)
     16 # model.load_state_dict(torch.load(os.path.join(root_dir, "best_metric_model.pt")))

<ipython-input-12-88c0fd3c91bd> in train(global_step, train_loader, dice_val_best, global_step_best)
     30             logit_map = model(x)
     31             loss = loss_function(logit_map, y)
---> 32         scaler.scale(loss).backward()
     33         epoch_loss += loss.item()
     34         scaler.unscale_(optimizer)

/opt/conda/lib/python3.8/site-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    394                 create_graph=create_graph,
    395                 inputs=inputs)
--> 396         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    397 
    398     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    171     # some Python versions print out the first line of a multi-line function
    172     # calls in the traceback and some print out the last line
--> 173     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    175         allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass

RuntimeError: CUDA error: device-side assert triggered

If the EnsureTyped transform is uncommented, it returns an error related to EnsureTyped:

Training (0 / 20000 Steps) (loss=0.94304):   1%|          | 1/155 [00:08<20:49,  8.11s/it]Exception in thread Thread-21:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 103, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 67, in _apply_transform
    return transform(parameters)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utility/dictionary.py", line 610, in __call__
    d[key] = self.converter(d[key])
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utility/array.py", line 531, in __call__
    out, *_ = convert_data_type(
  File "/opt/conda/lib/python3.8/site-packages/monai/utils/type_conversion.py", line 315, in convert_data_type
    data_ = convert_to_tensor(
  File "/opt/conda/lib/python3.8/site-packages/monai/utils/type_conversion.py", line 152, in convert_to_tensor
    return _convert_tensor(data).to(dtype=dtype, device=device, memory_format=torch.contiguous_format)
RuntimeError: CUDA error: device-side assert triggered

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 103, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 67, in _apply_transform
    return transform(parameters)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/compose.py", line 177, in __call__
    input_ = apply_transform(_transform, input_, self.map_items, self.unpack_items, self.log_stats)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 130, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.utility.dictionary.EnsureTyped object at 0x7f9cf89527f0>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.8/site-packages/monai/data/thread_buffer.py", line 49, in enqueue_values
    for src_val in self.src:
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 688, in __next__
    (data, worker_id) = self._next_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 770, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/monai/data/dataset.py", line 110, in __getitem__
    return self._transform(index)
  File "/opt/conda/lib/python3.8/site-packages/monai/data/dataset.py", line 904, in _transform
    return super()._transform(index)
  File "/opt/conda/lib/python3.8/site-packages/monai/data/dataset.py", line 96, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 130, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f9cf8952dc0>
Training (0 / 20000 Steps) (loss=0.94304):   1%|          | 1/155 [00:09<25:31,  9.94s/it]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-855e7bf4a41e> in <module>
     13 metric_values = []
     14 while global_step < max_iterations:
---> 15     global_step, dice_val_best, global_step_best = train(global_step, train_loader, dice_val_best, global_step_best)
     16 # model.load_state_dict(torch.load(os.path.join(root_dir, "best_metric_model.pt")))

<ipython-input-12-88c0fd3c91bd> in train(global_step, train_loader, dice_val_best, global_step_best)
     30             logit_map = model(x)
     31             loss = loss_function(logit_map, y)
---> 32         scaler.scale(loss).backward()
     33         epoch_loss += loss.item()
     34         scaler.unscale_(optimizer)

/opt/conda/lib/python3.8/site-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    394                 create_graph=create_graph,
    395                 inputs=inputs)
--> 396         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    397 
    398     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    171     # some Python versions print out the first line of a multi-line function
    172     # calls in the traceback and some print out the last line
--> 173     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    175         allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass

RuntimeError: CUDA error: device-side assert triggered

Package versions:

MONAI version: 1.2.dev2310
Numpy version: 1.23.5
Pytorch version: 1.12.1+cu113
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 7521d7701bb73057952650d982e38467d219ed5b
MONAI __file__: /opt/conda/lib/python3.8/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 5.0.1
scikit-image version: NOT INSTALLED or UNKNOWN VERSION.
Pillow version: 9.4.0
Tensorboard version: NOT INSTALLED or UNKNOWN VERSION.
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.13.1+cu113
tqdm version: 4.64.1
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.4
pandas version: 1.5.3
einops version: 0.6.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.

I'm not sure what else to try, or do I just have bad data?

Link to the notebook tutorial:
https://github.com/Project-MONAI/tutorials/blob/main/3d_segmentation/swin_unetr_btcv_segmentation_3d.ipynb

[KumoLiu]

Hi @junxiant, if you crop to (64, 64, 32), and random flip in three axes, the images may have different shapes. Maybe try to set ((64, 64, 64) and run again.

Hope it can help you, thanks!

[junxiant]

Hi @KumoLiu ,

So changing it to 64,64,64 didn't work.

I wasn't able to find out which part of the code was giving the error, so I made a fresh copy of the tutorial notebook and made minimal changes and it was working again