RuntimeError: DataLoader worker exited unexpectedly

[matteo-bastico]

I am using the CT-ORG dataset for segmentation of organs (https://wiki.cancerimagingarchive.net/display/Public/CT-ORG%3A+CT+volumes+with+multiple+organ+segmentations). After some training iterations, I have the following error:

Traceback (most recent call last):
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1163, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/multiprocessing/connection.py", line 262, in poll
    return self._poll(timeout)
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/multiprocessing/connection.py", line 429, in _poll
    r = wait([self], timeout)
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/multiprocessing/connection.py", line 936, in wait
    ready = selector.select(timeout)
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 44497) is killed by signal: Killed. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mbastico/Desktop/GitHub/UNETR/BTCV/main.py", line 251, in <module>
    main()
  File "/home/mbastico/Desktop/GitHub/UNETR/BTCV/main.py", line 108, in main
    main_worker(gpu=0, args=args)
  File "/home/mbastico/Desktop/GitHub/UNETR/BTCV/main.py", line 233, in main_worker
    accuracy = run_training(
  File "/home/mbastico/Desktop/GitHub/UNETR/BTCV/trainer.py", line 178, in run_training
    train_loss = train_epoch(
  File "/home/mbastico/Desktop/GitHub/UNETR/BTCV/trainer.py", line 57, in train_epoch
    for idx, batch_data in enumerate(loader):
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1359, in _next_data
    idx, data = self._get_data()
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1325, in _get_data
    success, data = self._try_get_data()
  File "/home/mbastico/anaconda3/envs/segmentation/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1176, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 44497) exited unexpectedly

I tried to use different number of workers and batch sizes, dataloader with and without cache, persistent_worker and pin_memory. In any case I get the same error or the execution is just killed when the number of workers is 0.

I thought the problem was related to the following warning on the data loading

pixdim[0] (qfac) should be 1 (default) or -1; setting qfac to 1

but after fixing the NIfTI headers, the training still doesn't work.

Furthermore, running the code on Google Colab Pro I have several tcmalloc: large alloc warnings. I also tried the training with other datasets and it always worked correctly. Is therefore the problem related only to the large size of the input volumes or there is another way to solve it?

Thank you for the help!

[KumoLiu]

Hi @matteo-bastico, can you try to set num_workers to 0 or reduce your batch size? Hope it can help you, thanks!

[matteo-bastico]

I already tried with num_workers=0 and the problem persists. I think that the sources of the issue are some CT volumes in which the metadata is somehow corrupted (eg, parameters such as xyzt_units, qoffset_x, srow_x are set to 0), as shown in the following. In particular, these samples give the pixdim[0] (qfac) should be 1 (default) or -1; setting qfac to 1 warning` when loading.

Removing these examples from the training set, the error doesn't show up (apart when num_workers is high and the RAM saturates). Can you confirm that this might be the cause of the issue? In that case, is there any way to fix the problem without getting rid of these samples?

Metadata
	sizeof_hdr: 348
	extents: 0
	session_error: 0
	dim_info: 0
	dim: [  3 512 512 299   0   0   0   0]
	intent_p1: 0.0
	intent_p2: 0.0
	intent_p3: 0.0
	intent_code: 0
	datatype: 16
	bitpix: 32
	slice_start: 0
	pixdim: [1.        1.367188  1.367188  3.2700195 0.        0.        0.
        0.       ]
	vox_offset: 0.0
	scl_slope: nan
	scl_inter: nan
	slice_end: 0
        slice_code: 0
	xyzt_units: 0
	cal_max: 0.0
	cal_min: 0.0
	slice_duration: 0.0
	toffset: 0.0
	glmax: 0
	glmin: 0
	qform_code: 0
	sform_code: 0
	quatern_b: 0.0
	quatern_c: 0.0
	quatern_d: 0.0
	qoffset_x: 0.0
        qoffset_y: 0.0
	qoffset_z: 0.0
	srow_x: [0. 0. 0. 0.]
	srow_y: [0. 0. 0. 0.]
	srow_z: [0. 0. 0. 0.]
	affine: tensor([[  -1.3672,    0.0000,    0.0000,  349.3165],
        [   0.0000,    1.3672,    0.0000, -349.3165],
        [   0.0000,    0.0000,    3.2700, -487.2329],
        [   0.0000,    0.0000,    0.0000,    1.0000]], dtype=torch.float64)
	original_affine: [[  -1.36718798    0.            0.          349.31652808]
        [   0.            1.36718798    0.         -349.31652808]
        [   0.            0.            3.27001953 -487.23291016]
        [   0.            0.            0.            1.        ]]
        as_closest_canonical: False
	spatial_shape: [512 512 299]
	space: RAS
	original_channel_dim: no_channel
	filename_or_obj: OrganSegmentations/volume-133.nii.gz

[KumoLiu]

Hi @matteo-bastico, if possible could you please share one error data so we could look more deeply into it? Thanks!

[matteo-bastico]

I am using the CT-ORG dataset available at https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=61080890. In particular, the error data are:

volume-131.nii.gz
volume-133.nii.gz
volume-135.nii.gz
volume-136.nii.gz
volume-137.nii.gz
volume-138.nii.gz
volume-139.nii.gz
volume-20.nii.gz
volume-70.nii.gz
volume-76.nii.gz