auto3dseg autorunner tensor mismatch error #7433

bilalcodehub · 2024-02-01T13:38:49Z

bilalcodehub
Feb 1, 2024

Hello everyone,

I intend to utilize autorunner for the 3D segmentation of lesions from CT scans.

I have formatted the data as requested. Please find the contents below, which outline the paths for testing and training keys inside the datalist_file.json file:
training

testing

I use the code below to run the model:

runner = AutoRunner(input=input)
runner.run()

The data analyzer successfully generates the stats file. You can access the file here: https://drive.google.com/file/d/1hv2QImkLAdGYFOKWU1SSHvkDG7zs_qNC/view?usp=sharing

However, I encounter the following tensor mismatch error when auto3dseg initiates the training. It appears to be related to issues with augmentation, as batches are not formed as anticipated. See the log below:

2024-01-19 22:44:57,109 - INFO - AutoRunner using work directory ./work_dir
2024-01-19 22:44:57,111 - INFO - Loading input config uls_work_dir/input.yaml
2024-01-19 22:44:57,120 - INFO - Datalist was copied to work_dir: /home/exsdatalab/mlworks/ULS23/work_dir/uls_datalist.json
2024-01-19 22:44:57,178 - INFO - Setting num_fold 5 based on the input datalist /home/exsdatalab/mlworks/ULS23/work_dir/uls_datalist.json.
2024-01-19 22:44:57,199 - INFO - Using user defined command running prefix , will override other settings
2024-01-19 22:44:57,200 - INFO - Running data analysis...
2024-01-19 22:44:57,200 - INFO - Found 2 GPUs for data analyzing!
100%|██████████| 15799/15799 [40:00<00:00,  6.58it/s]
2024-01-19 23:26:15,470 - INFO - Data spacing is not completely uniform. MONAI transforms may provide unexpected result
2024-01-19 23:26:15,471 - INFO - Writing data stats to /home/exsdatalab/mlworks/ULS23/work_dir/datastats.yaml.
2024-01-19 23:26:15,480 - INFO - Writing by-case data stats to /home/exsdatalab/mlworks/ULS23/work_dir/datastats_by_case.yaml, this may take a while.
2024-01-19 23:28:21,327 - INFO - BundleGen from https://github.com/Project-MONAI/research-contributions/releases/download/algo_templates/249bf4b.tar.gz
algo_templates.tar.gz: 104kB [00:00, 157kB/s]                                                                                                               
2024-01-19 23:28:22,008 - INFO - Downloaded: /tmp/tmppugsjrc0/algo_templates.tar.gz
2024-01-19 23:28:22,008 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmppugsjrc0/algo_templates.tar.gz.
2024-01-19 23:28:22,009 - INFO - Writing into directory: /home/exsdatalab/mlworks/ULS23/work_dir.
2024-01-19 23:28:22,145 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/dints_0

2024-01-19 23:28:22,209 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/dints_1
2024-01-19 23:28:22,272 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/dints_2
2024-01-19 23:28:22,335 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/dints_3
2024-01-19 23:28:22,399 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/dints_4
2024-01-19 23:28:22,481 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/segresnet_0
2024-01-19 23:28:22,517 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/segresnet_1
2024-01-19 23:28:22,552 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/segresnet_2
2024-01-19 23:28:22,586 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/segresnet_3
2024-01-19 23:28:22,621 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/segresnet_4
2024-01-19 23:28:22,644 - INFO - segresnet2d_0 is skipped! SegresNet2D is skipped due to median spacing of [0.796875, 0.796875, 1.5],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-01-19 23:28:22,660 - INFO - segresnet2d_1 is skipped! SegresNet2D is skipped due to median spacing of [0.796875, 0.796875, 1.5],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-01-19 23:28:22,676 - INFO - segresnet2d_2 is skipped! SegresNet2D is skipped due to median spacing of [0.796875, 0.796875, 1.5],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-01-19 23:28:22,692 - INFO - segresnet2d_3 is skipped! SegresNet2D is skipped due to median spacing of [0.796875, 0.796875, 1.5],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-01-19 23:28:22,708 - INFO - segresnet2d_4 is skipped! SegresNet2D is skipped due to median spacing of [0.796875, 0.796875, 1.5],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-01-19 23:28:22,902 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/swinunetr_0
2024-01-19 23:28:22,950 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/swinunetr_1
2024-01-19 23:28:22,996 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/swinunetr_2
2024-01-19 23:28:23,043 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/swinunetr_3
2024-01-19 23:28:23,090 - INFO - Generated:/home/exsdatalab/mlworks/ULS23/work_dir/swinunetr_4
2024-01-19 23:28:23,214 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '2', '/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/scripts/train.py', 'run', "--config_file='/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/hyper_parameters.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/hyper_parameters_search.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/network.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/network_search.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/transforms_infer.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/transforms_train.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/transforms_validate.yaml'"]
[2024-01-19 23:28:24,244] torch.distributed.run: [WARNING] 
[2024-01-19 23:28:24,244] torch.distributed.run: [WARNING] *****************************************
[2024-01-19 23:28:24,244] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-01-19 23:28:24,244] torch.distributed.run: [WARNING] *****************************************
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
dints_0 - training ...:   0%|          | 0/28 [00:00<?, ?round/s]
CRITICAL:dev_collate:> collate dict key "image" out of 5 keys
CRITICAL:dev_collate:>> collate/stack a list of tensors
CRITICAL:dev_collate:>> E: stack expects each tensor to be equal size, but got [1, 96, 96, 96] at entry 0 and [1, 86, 85, 96] at entry 21, shape [torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 86, 85, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96])] in collate([metatensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         ...,

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]]]]), metatensor([[[[0.0634, 0.0576, 0.0517,  ..., 0.0524, 0.0765, 0.0797],
          [0.0640, 0.0565, 0.0616,  ..., 0.0509, 0.0605, 0.0675],
          [0.0638, 0.0583, 0.0569,  ..., 0.0660, 0.0630, 0.0609],
          ...,
          [0.0590, 0.0616, 0.0673,  ..., 0.0588, 0.0588, 0.0638],
          [0.0525, 0.0562, 0.0516,  ..., 0.0695, 0.0635, 0.0614],
          [0.0709, 0.0590, 0.0557,  ..., 0.0566, 0.0675, 0.0682]],

         [[0.0714, 0.0794, 0.0719,  ..., 0.0709, 0.0580, 0.0593],
          [0.0673, 0.0498, 0.0631,  ..., 0.0577, 0.0542, 0.0570],
          [0.0596, 0.0662, 0.0560,  ..., 0.0614, 0.0679, 0.0689],
          ...,
          [0.0621, 0.0638, 0.0580,  ..., 0.0622, 0.0623, 0.0657],
          [0.0646, 0.0524, 0.0649,  ..., 0.0717, 0.0780, 0.0491],
          [0.0483, 0.0599, 0.0583,  ..., 0.0578, 0.0488, 0.0641]],

         [[0.0658, 0.0659, 0.0609,  ..., 0.0575, 0.0617, 0.0564],
          [0.0592, 0.0673, 0.0694,  ..., 0.0636, 0.0715, 0.0720],
          [0.0575, 0.0667, 0.0627,  ..., 0.0597, 0.0622, 0.0651],
          ...,
          [0.0645, 0.0624, 0.0652,  ..., 0.0640, 0.0589, 0.0597],
          [0.0595, 0.0599, 0.0574,  ..., 0.0662, 0.0689, 0.0599],
          [0.0611, 0.0642, 0.0709,  ..., 0.0657, 0.0689, 0.0570]],

         ...,

         [[0.0581, 0.0631, 0.0689,  ..., 0.0648, 0.0654, 0.0555],
          [0.0622, 0.0620, 0.0654,  ..., 0.0574, 0.0567, 0.0579],
          [0.0575, 0.0576, 0.0556,  ..., 0.0572, 0.0674, 0.0708],
          ...,
          [0.0537, 0.0690, 0.0651,  ..., 0.0641, 0.0595, 0.0711],
          [0.0708, 0.0593, 0.0508,  ..., 0.0660, 0.0562, 0.0500],
          [0.0535, 0.0543, 0.0694,  ..., 0.0553, 0.0561, 0.0682]],

         [[0.0720, 0.0635, 0.0630,  ..., 0.0596, 0.0659, 0.0582],
          [0.0679, 0.0653, 0.0564,  ..., 0.0577, 0.0644, 0.0511],
          [0.0540, 0.0592, 0.0717,  ..., 0.0536, 0.0618, 0.0613],
          ...,
          [0.0558, 0.0712, 0.0679,  ..., 0.0782, 0.0672, 0.0574],
          [0.0592, 0.0660, 0.0573,  ..., 0.0693, 0.0605, 0.0659],
          [0.0623, 0.0686, 0.0668,  ..., 0.0545, 0.0705, 0.0752]],

         [[0.0577, 0.0696, 0.0721,  ..., 0.0640, 0.0572, 0.0603],
          [0.0612, 0.0641, 0.0634,  ..., 0.0525, 0.0636, 0.0642],
          [0.0708, 0.0587, 0.0625,  ..., 0.0708, 0.0493, 0.0608],
          ...,
          [0.0601, 0.0664, 0.0523,  ..., 0.0586, 0.0645, 0.0571],
          [0.0615, 0.0539, 0.0648,  ..., 0.0689, 0.0621, 0.0498],
          [0.0738, 0.0717, 0.0700,  ..., 0.0658, 0.0608, 0.0525]]]]), metatensor([[[[-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          ...,
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231]],

         [[-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          ...,
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231]],

         [[-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          ...,
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231]],

         ...,

         [[-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          ...,
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231]],

         [[-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          ...,
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231]],

         [[-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          ...,
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231],
          [-0.0231, -0.0231, -0.0231,  ..., -0.0231, -0.0231, -0.0231]]]]), metatensor([[[[-0.0412, -0.0464, -0.0449,  ..., -0.0426, -0.0482, -0.0416],
          [-0.0513, -0.0389, -0.0412,  ..., -0.0569, -0.0532, -0.0421],
          [-0.0485, -0.0427, -0.0382,  ..., -0.0446, -0.0531, -0.0447],
          ...,
          [-0.0437, -0.0471, -0.0539,  ..., -0.0435, -0.0518, -0.0463],
          [-0.0489, -0.0465, -0.0453,  ..., -0.0370, -0.0465, -0.0423],
          [-0.0469, -0.0447, -0.0449,  ..., -0.0458, -0.0573, -0.0514]],

         [[-0.0428, -0.0476, -0.0365,  ..., -0.0365, -0.0402, -0.0531],
          [-0.0455, -0.0490, -0.0502,  ..., -0.0465, -0.0427, -0.0424],
          [-0.0457, -0.0459, -0.0375,  ..., -0.0505, -0.0448, -0.0478],
          ...,
          [-0.0387, -0.0492, -0.0527,  ..., -0.0452, -0.0493, -0.0549],
          [-0.0526, -0.0508, -0.0479,  ..., -0.0485, -0.0459, -0.0385],
          [-0.0506, -0.0498, -0.0513,  ..., -0.0537, -0.0484, -0.0396]],

         [[-0.0456, -0.0472, -0.0452,  ..., -0.0424, -0.0531, -0.0500],
          [-0.0430, -0.0457, -0.0472,  ..., -0.0503, -0.0533, -0.0455],
          [-0.0455, -0.0417, -0.0429,  ..., -0.0354, -0.0435, -0.0421],
          ...,
          [-0.0420, -0.0483, -0.0535,  ..., -0.0430, -0.0544, -0.0533],
          [-0.0538, -0.0511, -0.0432,  ..., -0.0464, -0.0403, -0.0499],
          [-0.0464, -0.0358, -0.0470,  ..., -0.0394, -0.0537, -0.0425]],

         ...,

         [[-0.0403, -0.0485, -0.0388,  ..., -0.0439, -0.0463, -0.0423],
          [-0.0479, -0.0487, -0.0452,  ..., -0.0438, -0.0501, -0.0468],
          [-0.0482, -0.0498, -0.0463,  ..., -0.0521, -0.0410, -0.0526],
          ...,
          [-0.0430, -0.0452, -0.0455,  ..., -0.0500, -0.0478, -0.0476],
          [-0.0411, -0.0437, -0.0441,  ..., -0.0400, -0.0485, -0.0491],
          [-0.0453, -0.0450, -0.0374,  ..., -0.0431, -0.0473, -0.0538]],

         [[-0.0453, -0.0452, -0.0417,  ..., -0.0415, -0.0421, -0.0363],
          [-0.0508, -0.0417, -0.0489,  ..., -0.0434, -0.0454, -0.0449],
          [-0.0486, -0.0506, -0.0436,  ..., -0.0418, -0.0460, -0.0483],
          ...,
          [-0.0511, -0.0463, -0.0477,  ..., -0.0511, -0.0475, -0.0434],
          [-0.0413, -0.0445, -0.0390,  ..., -0.0489, -0.0432, -0.0525],
          [-0.0477, -0.0428, -0.0382,  ..., -0.0442, -0.0439, -0.0473]],

         [[-0.0462, -0.0464, -0.0431,  ..., -0.0447, -0.0490, -0.0363],
          [-0.0472, -0.0401, -0.0439,  ..., -0.0495, -0.0423, -0.0504],
          [-0.0438, -0.0451, -0.0444,  ..., -0.0500, -0.0442, -0.0503],
          ...,
          [-0.0506, -0.0514, -0.0452,  ..., -0.0472, -0.0414, -0.0463],
          [-0.0470, -0.0532, -0.0433,  ..., -0.0462, -0.0505, -0.0402],
          [-0.0416, -0.0483, -0.0449,  ..., -0.0463, -0.0453, -0.0414]]]]), metatensor([[[[0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          ...,
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479]],

         [[0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          ...,
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479]],

         [[0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          ...,
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479]],

         ...,

         [[0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          ...,
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479]],

         [[0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          ...,
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479]],

         [[0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          ...,
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479],
          [0.0479, 0.0479, 0.0479,  ..., 0.0479, 0.0479, 0.0479]]]]), metatensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         ...,

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]]]]), metatensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         ...,

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]]]]), metatensor([[[[-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          ...,
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389]],

         [[-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          ...,
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389]],

         [[-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          ...,
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389]],

         ...,

         [[-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          ...,
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389]],

         [[-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          ...,
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389]],

         [[-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          ...,
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389],
          [-0.0389, -0.0389, -0.0389,  ..., -0.0389, -0.0389, -0.0389]]]]), metatensor([[[[0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          ...,
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079]],

         [[0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          ...,
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079]],

         [[0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          ...,
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079]],

         ...,

         [[0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          ...,
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079]],

         [[0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          ...,
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079]],

         [[0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          ...,
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079],
          [0.0079, 0.0079, 0.0079,  ..., 0.0079, 0.0079, 0.0079]]]]), metatensor([[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

         ...,

         [[0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          [0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          [0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          [0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          [0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          [0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          [0.0000, 0.0000, 0.0000,  ..., 0.0018, 0.0018, 0.0018],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]]]])] ... )
CRITICAL:dev_collate:> collate dict key "label" out of 5 keys
CRITICAL:dev_collate:>> collate/stack a list of tensors
CRITICAL:dev_collate:> collate dict key "fold" out of 5 keys
CRITICAL:dev_collate:> collate dict key "image_meta_dict" out of 5 keys
CRITICAL:dev_collate:>> collate dict key "sizeof_hdr" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "extents" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "session_error" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim_info" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p1" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p2" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p3" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "datatype" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "bitpix" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_start" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "pixdim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "vox_offset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_slope" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_inter" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_end" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "xyzt_units" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_max" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_min" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_duration" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "toffset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmax" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmin" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "sform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_b" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_c" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_d" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "original_affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "as_closest_canonical" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "spatial_shape" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "space" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "original_channel_dim" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "filename_or_obj" out of 43 keys
CRITICAL:dev_collate:> collate dict key "label_meta_dict" out of 5 keys
CRITICAL:dev_collate:>> collate dict key "sizeof_hdr" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "extents" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "session_error" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim_info" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p1" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p2" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p3" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "datatype" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "bitpix" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_start" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "pixdim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "vox_offset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_slope" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_inter" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_end" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "xyzt_units" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_max" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_min" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_duration" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "toffset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmax" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmin" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "sform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_b" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_c" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_d" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "original_affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "as_closest_canonical" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "spatial_shape" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "space" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "original_channel_dim" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "filename_or_obj" out of 43 keys
dints_0 - training ...:   0%|          | 0/28 [46:03<?, ?round/s]
Traceback (most recent call last):
  File "/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/scripts/train.py", line 984, in <module>
    fire.Fire()
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/scripts/train.py", line 564, in run
    for batch_data in train_loader:
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 495, in list_data_collate
    ret = collate_meta_tensor(data)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 464, in collate_meta_tensor
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 464, in <dictcomp>
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 454, in collate_meta_tensor
    collated = default_collate(batch)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 123, in collate
    return collate_fn_map[collate_type](batch, collate_fn_map=collate_fn_map)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 161, in collate_tensor_fn
    out = elem.new(storage).resize_(len(batch), *list(elem.size()))
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 282, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/_tensor.py", line 1386, in __torch_function__
    ret = func(*args, **kwargs)
RuntimeError: Trying to resize storage that is not resizable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 508, in list_data_collate
    raise RuntimeError(re_str) from re
RuntimeError: Trying to resize storage that is not resizable

[2024-01-20 00:15:16,790] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 433420 closing signal SIGTERM
[2024-01-20 00:15:17,454] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 433419) of binary: /home/exsdatalab/mambaforge/bin/python3.10
Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/scripts/train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-20_00:15:16
  host      : exsdatalab
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 433419)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[5], line 2
      1 runner = AutoRunner(input=input)
----> 2 runner.run()

File ~/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/auto_runner.py:806, in AutoRunner.run(self)
    804 if len(history) > 0:
    805     if not self.hpo:
--> 806         self._train_algo_in_sequence(history)
    807     else:
    808         self._train_algo_in_nni(history)

File ~/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/auto_runner.py:658, in AutoRunner._train_algo_in_sequence(self, history)
    656 algo = algo_dict[AlgoKeys.ALGO]
    657 if has_option(algo.train, "device_setting"):
--> 658     algo.train(self.train_params, self.device_setting)
    659 else:
    660     algo.train(self.train_params)

File ~/mlworks/ULS23/work_dir/algorithm_templates/dints/scripts/algo.py:490, in DintsAlgo.train(self, train_params, device_setting, search)
    488 cmd, devices_info = self._create_cmd(dints_train_params)
    489 cmd = "OMP_NUM_THREADS=1 " + cmd
--> 490 return self._run_cmd(cmd, devices_info)

File ~/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:251, in BundleAlgo._run_cmd(self, cmd, devices_info)
    249     return _run_cmd_bcprun(cmd, n=self.device_setting["NUM_NODES"], p=self.device_setting["n_devices"])
    250 elif int(self.device_setting["n_devices"]) > 1:
--> 251     return _run_cmd_torchrun(
    252         cmd, nnodes=1, nproc_per_node=self.device_setting["n_devices"], env=ps_environ, check=True
    253     )
    254 else:
    255     return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)

File ~/mambaforge/lib/python3.10/site-packages/monai/auto3dseg/utils.py:502, in _run_cmd_torchrun(cmd, **kwargs)
    500     torchrun_list += [f"--{arg}", str(params.pop(arg))]
    501 torchrun_list += cmd_list
--> 502 return run_cmd(torchrun_list, run_cmd_verbose=True, **params)

File ~/mambaforge/lib/python3.10/site-packages/monai/utils/misc.py:874, in run_cmd(cmd_list, **kwargs)
    872     monai.apps.utils.get_logger("run_cmd").info(f"{cmd_list}")
    873 try:
--> 874     return subprocess.run(cmd_list, **kwargs)
    875 except subprocess.CalledProcessError as e:
    876     if not debug:

File ~/mambaforge/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    524     retcode = process.poll()
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['torchrun', '--nnodes', '1', '--nproc_per_node', '2', '/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/scripts/train.py', 'run', "--config_file='/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/hyper_parameters.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/hyper_parameters_search.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/network.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/network_search.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/transforms_infer.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/transforms_train.yaml,/home/exsdatalab/mlworks/ULS23/work_dir/dints_0/configs/transforms_validate.yaml'"]' returned non-zero exit status 1.

Could someone please provide guidance on what might be incorrect with my configurations?

Any assistance to resolve this issue would be greatly appreciated.

Thank you and
Kind regards,
Bilal

KumoLiu · 2024-02-02T09:38:50Z

KumoLiu
Feb 2, 2024
Maintainer

Hi @bilalcodehub, you may add a ResizeWithPadOrCrop at the end of the transform list to ensure all the images have the same shape.
https://github.com/Project-MONAI/research-contributions/blob/main/auto3dseg/algorithm_templates/dints/configs/transforms_train.yaml

Hi @dongyang0122, is there any reason we don't add this transform to ensure the same shape in the suto3dseg?
Thanks.

8 replies

KumoLiu Feb 14, 2024
Maintainer

Hi @bilalcodehub,

as suggested but struggling to find materials on adding transforms.

I didn't get the point here, could you explain on this further? Thanks.

bilalcodehub Feb 14, 2024
Author

Thanks @KumoLiu: I'm currently facing difficulty running the auto3dseg code by manually adding ResizeWithPadOrCrop code as shown below in the transform_train.yaml file to handle dimension mismatches in our dataset.

 - _target_: ResizeWithPadOrCrop
    keys: ["@image_key", "@label_key"]
    patch_size: [256, 256, 128]
    method: pad
    prob: 0.1

I couldn't figure out what is wrong with this code. I would greatly appreciate any guidance to fix this issue and any reading materials explaining how to add these transforms manually. Thank you for your assistance.

KumoLiu Feb 14, 2024
Maintainer

I think you should use the dictionary version, ResizeWithPadOrCropd instead of ResizeWithPadOrCrop?

KumoLiu Feb 14, 2024
Maintainer

And didn't see 'pad' mode here. Please check the doc-string for instructions, thanks.

MONAI/monai/transforms/croppad/dictionary.py

Line 1178 in ff43028

    
                   mode: available modes for numpy array:{``"constant"``, ``"edge"``, ``"linear_ramp"``, ``"maximum"``,

bilalcodehub Feb 16, 2024
Author

Thanks for the update, @KumoLiu. I've tried running the auto3dseg for the dints model with folds=1, and I've implemented the transform at the end of the transform_train.yaml file as you described (see below). While I no longer encounter the transform validation error, I'm still facing a dimension mismatch error:

  - _target_: ResizeWithPadOrCropd
    keys: ['@image_key', '@label_key']
    spatial_size: [256, 256, 128]
    prob: 0.5

The new error message is given below:

2024-02-16 09:08:27,213 - INFO - The keys num_warmup_epochs cannot be found in the uls_work_dir/dints_0/configs/hyper_parameters.yaml for training. Skipped overriding key num_warmup_epochs.
2024-02-16 09:08:27,214 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '2', 'uls_work_dir/dints_0/scripts/train.py', 'run', "--config_file='uls_work_dir/dints_0/configs/hyper_parameters.yaml,uls_work_dir/dints_0/configs/hyper_parameters_search.yaml,uls_work_dir/dints_0/configs/network.yaml,uls_work_dir/dints_0/configs/network_search.yaml,uls_work_dir/dints_0/configs/transforms_infer.yaml,uls_work_dir/dints_0/configs/transforms_train.yaml,uls_work_dir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=2']
[2024-02-16 09:08:28,254] torch.distributed.run: [WARNING] 
[2024-02-16 09:08:28,254] torch.distributed.run: [WARNING] *****************************************
[2024-02-16 09:08:28,254] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-02-16 09:08:28,254] torch.distributed.run: [WARNING] *****************************************
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
dints_0 - training ...:   0%|          | 0/100 [00:00<?, ?round/s]
CRITICAL:dev_collate:> collate dict key "image" out of 5 keys
CRITICAL:dev_collate:>> collate/stack a list of tensors
CRITICAL:dev_collate:>> E: stack expects each tensor to be equal size, but got [1, 96, 96, 96] at entry 0 and [1, 96, 86, 96] at entry 10, shape [torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 86, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 96, 96, 96]), torch.Size([1, 86, 96, 96]), torch.Size([1, 96, 86, 96]), torch.Size([1, 96, 96, 96])] in collate([metatensor([[[[-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          ...,
          [ 0.6000,  0.5891,  0.5853,  ...,  0.3169,  0.3112,  0.3025],
          [ 0.6205,  0.6205,  0.6205,  ...,  0.3280,  0.3071,  0.3006],
          [ 0.6205,  0.6205,  0.6205,  ...,  0.3289,  0.3283,  0.3365]],

         [[-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          ...,
          [ 0.6205,  0.6205,  0.6204,  ...,  0.3200,  0.3370,  0.3446],
          [ 0.6205,  0.6205,  0.6205,  ...,  0.3451,  0.3289,  0.3255],
          [ 0.6205,  0.6205,  0.6205,  ...,  0.3290,  0.3194,  0.3297]],

         [[-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          ...,
          [ 0.6205,  0.6205,  0.6205,  ...,  0.3327,  0.3302,  0.3233],
          [ 0.6205,  0.6205,  0.6205,  ...,  0.3166,  0.3192,  0.3298],
          [ 0.6205,  0.6205,  0.6205,  ...,  0.3182,  0.3074,  0.3125]],

         ...,

         [[-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0257, -0.0449, -0.0757],
          [-0.0879, -0.0879, -0.0879,  ...,  0.1599,  0.1541,  0.1479],
          ...,
          [ 0.3126,  0.3110,  0.3038,  ...,  0.6202,  0.6205,  0.6147],
          [ 0.2881,  0.2927,  0.3003,  ...,  0.6119,  0.6104,  0.6033],
          [ 0.2918,  0.2998,  0.3047,  ...,  0.5977,  0.6001,  0.5996]],

         [[-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0142, -0.0295, -0.0554],
          [-0.0879, -0.0879, -0.0879,  ...,  0.1653,  0.1654,  0.1564],
          ...,
          [ 0.2633,  0.2975,  0.3053,  ...,  0.6205,  0.6205,  0.6205],
          [ 0.2744,  0.3017,  0.3195,  ...,  0.6205,  0.6205,  0.6205],
          [ 0.3091,  0.3046,  0.3069,  ...,  0.6205,  0.6205,  0.6205]],

         [[-0.0879, -0.0879, -0.0879,  ..., -0.0879, -0.0879, -0.0879],
          [-0.0879, -0.0879, -0.0879,  ..., -0.0077, -0.0174, -0.0418],
          [-0.0879, -0.0879, -0.0879,  ...,  0.1715,  0.1804,  0.1734],
          ...,
          [ 0.2637,  0.2516,  0.2606,  ...,  0.6205,  0.6205,  0.6205],
          [ 0.3001,  0.2765,  0.2916,  ...,  0.6205,  0.6205,  0.6205],
          [ 0.3550,  0.3307,  0.3173,  ...,  0.6205,  0.6205,  0.6205]]]]), metatensor([[[[0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          ...,
          [0.6107, 0.6366, 0.6860,  ..., 0.7911, 0.6900, 0.5903],
          [0.7148, 0.7164, 0.7257,  ..., 0.8253, 0.7136, 0.5901],
          [0.7219, 0.7360, 0.7280,  ..., 0.8693, 0.7295, 0.6092]],

         [[0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          ...,
          [0.6132, 0.6315, 0.6656,  ..., 0.7444, 0.6509, 0.5838],
          [0.6761, 0.6545, 0.6525,  ..., 0.7666, 0.6463, 0.5550],
          [0.6759, 0.6694, 0.6675,  ..., 0.8443, 0.6781, 0.5629]],

         [[0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          ...,
          [0.7350, 0.6916, 0.6458,  ..., 0.6762, 0.5690, 0.5259],
          [0.6610, 0.5683, 0.5172,  ..., 0.7630, 0.5922, 0.4974],
          [0.6205, 0.6082, 0.6179,  ..., 0.7773, 0.6368, 0.5094]],

         ...,

         [[0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          ...,
          [0.7059, 0.6116, 0.6441,  ..., 0.6881, 0.7396, 0.7780],
          [0.7437, 0.7355, 0.7297,  ..., 0.6660, 0.7248, 0.7438],
          [0.6561, 0.7126, 0.7377,  ..., 0.7392, 0.7988, 0.8211]],

         [[0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          ...,
          [0.7758, 0.7815, 0.7845,  ..., 0.7425, 0.8115, 0.8174],
          [0.7145, 0.7515, 0.7661,  ..., 0.7017, 0.7646, 0.7768],
          [0.6002, 0.6649, 0.7141,  ..., 0.7478, 0.7800, 0.7660]],

         [[0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          [0.0699, 0.0699, 0.0699,  ..., 0.0699, 0.0699, 0.0699],
          ...,
          [0.7378, 0.7815, 0.8141,  ..., 0.7254, 0.7495, 0.7569],
          [0.6883, 0.7184, 0.7552,  ..., 0.7612, 0.7543, 0.7427],
          [0.5243, 0.5886, 0.6729,  ..., 0.7729, 0.7714, 0.7700]]]]), metatensor([[[[0.3087, 0.2770, 0.2597,  ..., 0.3378, 0.3316, 0.3236],
          [0.3747, 0.3394, 0.2913,  ..., 0.3982, 0.3999, 0.3943],
          [0.3787, 0.3650, 0.3348,  ..., 0.3982, 0.4179, 0.4062],
          ...,
          [0.9218, 0.9227, 0.9055,  ..., 0.9671, 0.9671, 0.9671],
          [0.8790, 0.9264, 0.8945,  ..., 0.9671, 0.9671, 0.9671],
          [0.8471, 0.8520, 0.8205,  ..., 0.9671, 0.9671, 0.9671]],

         [[0.3248, 0.3135, 0.3107,  ..., 0.3241, 0.3204, 0.3020],
          [0.3665, 0.3759, 0.3529,  ..., 0.3139, 0.3188, 0.3127],
          [0.3397, 0.3550, 0.3442,  ..., 0.3269, 0.3246, 0.3149],
          ...,
          [0.8226, 0.8222, 0.8224,  ..., 0.9671, 0.9671, 0.9671],
          [0.8386, 0.9007, 0.8819,  ..., 0.9671, 0.9671, 0.9671],
          [0.8181, 0.9024, 0.9171,  ..., 0.9671, 0.9671, 0.9671]],

         [[0.2867, 0.2875, 0.2739,  ..., 0.3047, 0.3100, 0.3100],
          [0.3135, 0.3436, 0.3314,  ..., 0.2869, 0.2895, 0.2918],
          [0.3031, 0.3153, 0.3215,  ..., 0.3103, 0.3183, 0.3096],
          ...,
          [0.8445, 0.8074, 0.7645,  ..., 0.9671, 0.9671, 0.9671],
          [0.8060, 0.8080, 0.7994,  ..., 0.9671, 0.9671, 0.9671],
          [0.8923, 0.9477, 0.9279,  ..., 0.9671, 0.9671, 0.9671]],

         ...,

         [[0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          [0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          [0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          ...,
          [0.4020, 0.4242, 0.4640,  ..., 0.3230, 0.3119, 0.2823],
          [0.4541, 0.4724, 0.4698,  ..., 0.3228, 0.3138, 0.2930],
          [0.4187, 0.4407, 0.4338,  ..., 0.3018, 0.3247, 0.3367]],

         [[0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          [0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          [0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          ...,
          [0.3443, 0.3442, 0.3588,  ..., 0.3016, 0.2735, 0.2427],
          [0.3477, 0.3691, 0.3849,  ..., 0.2914, 0.2739, 0.2615],
          [0.2938, 0.3290, 0.3652,  ..., 0.2693, 0.2778, 0.3093]],

         [[0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          [0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          [0.0304, 0.0304, 0.0304,  ..., 0.0304, 0.0304, 0.0304],
          ...,
          [0.2971, 0.2820, 0.3004,  ..., 0.2917, 0.2657, 0.2503],
          [0.2920, 0.3217, 0.3531,  ..., 0.2779, 0.2552, 0.2567],
          [0.2525, 0.2882, 0.3278,  ..., 0.2703, 0.2763, 0.3159]]]]), metatensor([[[[0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          ...,
          [1.0581, 1.0581, 1.0581,  ..., 0.6206, 0.6224, 0.6410],
          [1.0581, 1.0581, 1.0581,  ..., 0.6192, 0.6218, 0.6407],
          [1.0373, 1.0341, 1.0310,  ..., 0.6194, 0.6212, 0.6404]],

         [[0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          ...,
          [1.0581, 1.0581, 1.0581,  ..., 0.6196, 0.6283, 0.6439],
          [1.0581, 1.0581, 1.0581,  ..., 0.6176, 0.6275, 0.6436],
          [1.0420, 1.0370, 1.0320,  ..., 0.6177, 0.6269, 0.6433]],

         [[0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          ...,
          [1.0581, 1.0581, 1.0581,  ..., 0.6185, 0.6340, 0.6469],
          [1.0581, 1.0581, 1.0581,  ..., 0.6159, 0.6332, 0.6466],
          [1.0488, 1.0458, 1.0421,  ..., 0.6161, 0.6327, 0.6463]],

         ...,

         [[0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          ...,
          [0.6466, 0.6414, 0.6339,  ..., 0.6289, 0.6367, 0.6435],
          [0.7013, 0.7049, 0.6853,  ..., 0.6354, 0.6467, 0.6598],
          [0.6685, 0.6606, 0.6673,  ..., 0.6389, 0.6511, 0.6658]],

         [[0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          ...,
          [0.6436, 0.6397, 0.6324,  ..., 0.6386, 0.6365, 0.6339],
          [0.7036, 0.7021, 0.6783,  ..., 0.6415, 0.6398, 0.6385],
          [0.6658, 0.6607, 0.6704,  ..., 0.6396, 0.6396, 0.6394]],

         [[0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          [0.0581, 0.0581, 0.0581,  ..., 0.0581, 0.0581, 0.0581],
          ...,
          [0.6422, 0.6376, 0.6343,  ..., 0.6070, 0.6153, 0.6243],
          [0.7022, 0.6946, 0.6742,  ..., 0.6093, 0.6180, 0.6264],
          [0.6653, 0.6638, 0.6681,  ..., 0.6121, 0.6193, 0.6264]]]]), metatensor([[[[0.7067, 0.7250, 0.7453,  ..., 0.7859, 0.7802, 0.7835],
          [0.6906, 0.7126, 0.7403,  ..., 0.7976, 0.8149, 0.8309],
          [0.6712, 0.6985, 0.7319,  ..., 0.8907, 0.8996, 0.9079],
          ...,
          [0.8760, 0.8718, 0.8676,  ..., 0.2509, 0.2645, 0.2692],
          [0.8549, 0.8562, 0.8575,  ..., 0.2360, 0.2585, 0.2606],
          [0.8508, 0.8371, 0.8234,  ..., 0.2539, 0.2497, 0.2426]],

         [[0.7412, 0.7685, 0.7663,  ..., 0.8231, 0.8337, 0.8271],
          [0.7261, 0.7556, 0.7610,  ..., 0.8265, 0.8258, 0.8296],
          [0.7138, 0.7436, 0.7546,  ..., 0.8299, 0.8438, 0.8617],
          ...,
          [0.8706, 0.8664, 0.8622,  ..., 0.2492, 0.2661, 0.2728],
          [0.8566, 0.8579, 0.8592,  ..., 0.2398, 0.2563, 0.2550],
          [0.8334, 0.8197, 0.8060,  ..., 0.2582, 0.2484, 0.2398]],

         [[0.8291, 0.8983, 0.9196,  ..., 0.6881, 0.6959, 0.7133],
          [0.7991, 0.8626, 0.8816,  ..., 0.6934, 0.7075, 0.7215],
          [0.7728, 0.8289, 0.8465,  ..., 0.6981, 0.7102, 0.7216],
          ...,
          [0.8646, 0.8617, 0.8583,  ..., 0.2482, 0.2684, 0.2756],
          [0.8444, 0.8462, 0.8506,  ..., 0.2428, 0.2534, 0.2498],
          [0.7976, 0.7922, 0.7874,  ..., 0.2623, 0.2460, 0.2368]],

         ...,

         [[0.5911, 0.5803, 0.5447,  ..., 0.2999, 0.2913, 0.2849],
          [0.5349, 0.5319, 0.5295,  ..., 0.2593, 0.2636, 0.2675],
          [0.5477, 0.5423, 0.5385,  ..., 0.2611, 0.2602, 0.2577],
          ...,
          [0.5727, 0.5704, 0.5683,  ..., 0.5721, 0.5740, 0.5710],
          [0.5730, 0.5781, 0.5850,  ..., 0.5726, 0.5757, 0.5698],
          [0.6068, 0.6112, 0.6130,  ..., 0.5723, 0.5783, 0.5700]],

         [[0.5800, 0.5660, 0.5377,  ..., 0.2885, 0.2795, 0.2705],
          [0.5380, 0.5330, 0.5312,  ..., 0.2518, 0.2519, 0.2521],
          [0.5497, 0.5450, 0.5386,  ..., 0.2511, 0.2505, 0.2500],
          ...,
          [0.5877, 0.5856, 0.5833,  ..., 0.5715, 0.5781, 0.5855],
          [0.5818, 0.5793, 0.5775,  ..., 0.5704, 0.5735, 0.5818],
          [0.5769, 0.5765, 0.5795,  ..., 0.5702, 0.5693, 0.5777]],

         [[0.5687, 0.5525, 0.5318,  ..., 0.2770, 0.2680, 0.2590],
          [0.5398, 0.5342, 0.5330,  ..., 0.2520, 0.2521, 0.2522],
          [0.5516, 0.5477, 0.5381,  ..., 0.2504, 0.2498, 0.2492],
          ...,
          [0.5880, 0.5922, 0.5954,  ..., 0.5576, 0.5762, 0.5890],
          [0.5900, 0.5872, 0.5837,  ..., 0.5613, 0.5786, 0.5925],
          [0.5820, 0.5802, 0.5797,  ..., 0.5649, 0.5815, 0.5941]]]]), metatensor([[[[2.6957e-01, 2.5551e-01, 2.4190e-01,  ..., 4.4140e-01,
           4.4127e-01, 4.3923e-01],
          [2.7959e-01, 2.7036e-01, 2.5887e-01,  ..., 4.4115e-01,
           4.4107e-01, 4.3847e-01],
          [2.6611e-01, 2.6608e-01, 2.5829e-01,  ..., 4.4062e-01,
           4.4079e-01, 4.3451e-01],
          ...,
          [1.7500e-01, 1.7338e-01, 1.7570e-01,  ..., 1.6004e-04,
           2.6135e-05, 0.0000e+00],
          [2.0493e-01, 2.0361e-01, 2.0284e-01,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [2.4304e-01, 2.4546e-01, 2.4291e-01,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         [[2.5178e-01, 2.4025e-01, 2.2677e-01,  ..., 4.3320e-01,
           4.2713e-01, 4.2216e-01],
          [2.5148e-01, 2.4467e-01, 2.3480e-01,  ..., 4.3635e-01,
           4.3109e-01, 4.2214e-01],
          [2.4789e-01, 2.4868e-01, 2.4178e-01,  ..., 4.4214e-01,
           4.3635e-01, 4.1878e-01],
          ...,
          [1.7635e-01, 1.7235e-01, 1.7148e-01,  ..., 3.9213e-05,
           5.7063e-06, 0.0000e+00],
          [2.0997e-01, 2.0447e-01, 2.0052e-01,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [2.4149e-01, 2.4397e-01, 2.4242e-01,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         [[2.1427e-01, 2.0855e-01, 2.0003e-01,  ..., 4.3194e-01,
           4.2109e-01, 4.1179e-01],
          [2.1856e-01, 2.1286e-01, 2.0622e-01,  ..., 4.3685e-01,
           4.2356e-01, 4.0702e-01],
          [2.3777e-01, 2.3588e-01, 2.2959e-01,  ..., 4.4509e-01,
           4.2726e-01, 3.9895e-01],
          ...,
          [1.8037e-01, 1.7892e-01, 1.7714e-01,  ..., 1.2981e-05,
           7.5930e-20, 0.0000e+00],
          [2.1082e-01, 2.0623e-01, 2.0418e-01,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [2.2867e-01, 2.3471e-01, 2.3928e-01,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         ...,

         [[4.3966e-01, 4.4077e-01, 4.4248e-01,  ..., 6.2423e-01,
           6.6016e-01, 6.9527e-01],
          [4.4991e-01, 4.4453e-01, 4.3807e-01,  ..., 5.4013e-01,
           5.7908e-01, 6.2202e-01],
          [4.4666e-01, 4.4259e-01, 4.3651e-01,  ..., 4.8746e-01,
           5.1618e-01, 5.4908e-01],
          ...,
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         [[4.2864e-01, 4.2718e-01, 4.2641e-01,  ..., 5.8616e-01,
           6.2470e-01, 6.6886e-01],
          [4.4344e-01, 4.3780e-01, 4.3196e-01,  ..., 5.1671e-01,
           5.4842e-01, 5.8813e-01],
          [4.4636e-01, 4.4299e-01, 4.3807e-01,  ..., 4.8425e-01,
           5.0378e-01, 5.2673e-01],
          ...,
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         [[4.1474e-01, 4.1286e-01, 4.1190e-01,  ..., 5.5147e-01,
           5.8684e-01, 6.3506e-01],
          [4.2925e-01, 4.2427e-01, 4.2031e-01,  ..., 4.9684e-01,
           5.2020e-01, 5.5438e-01],
          [4.3713e-01, 4.3504e-01, 4.3213e-01,  ..., 4.8026e-01,
           4.9274e-01, 5.0606e-01],
          ...,
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]]]]), metatensor([[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0608, 0.0173, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.3417, 0.3329, 0.3142],
          ...,
          [0.5631, 0.5530, 0.5744,  ..., 1.0000, 0.9919, 0.9978],
          [0.5373, 0.5480, 0.5293,  ..., 0.9858, 0.9757, 0.9818],
          [0.5474, 0.5543, 0.5439,  ..., 0.9712, 0.9705, 0.9737]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0345, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.3532, 0.3057, 0.2638],
          ...,
          [0.5461, 0.5382, 0.5666,  ..., 0.8675, 0.8429, 0.8545],
          [0.5685, 0.5789, 0.5824,  ..., 0.7608, 0.7278, 0.7649],
          [0.5866, 0.5684, 0.5434,  ..., 0.6963, 0.7106, 0.7308]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0027, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.3089, 0.2764, 0.2237],
          ...,
          [0.5479, 0.5390, 0.5384,  ..., 0.5897, 0.6215, 0.6041],
          [0.5360, 0.5641, 0.5918,  ..., 0.4493, 0.4995, 0.5420],
          [0.5392, 0.5495, 0.5521,  ..., 0.4865, 0.5085, 0.5198]],

         ...,

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.9557, 0.9504, 0.9129,  ..., 0.5634, 0.5512, 0.5702],
          [1.0000, 1.0000, 1.0000,  ..., 0.5576, 0.5484, 0.5707],
          [1.0000, 1.0000, 1.0000,  ..., 0.5875, 0.5992, 0.6116]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.8715, 0.8879, 0.8525,  ..., 0.5669, 0.5774, 0.5858],
          [1.0000, 0.9995, 0.9991,  ..., 0.5957, 0.6192, 0.6164],
          [1.0000, 1.0000, 1.0000,  ..., 0.6323, 0.6355, 0.6316]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.7899, 0.7503, 0.6994,  ..., 0.5679, 0.5760, 0.5929],
          [0.9895, 0.9838, 0.9785,  ..., 0.6146, 0.6474, 0.6549],
          [1.0000, 1.0000, 1.0000,  ..., 0.6167, 0.6226, 0.6324]]]]), metatensor([[[[0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          ...,
          [0.7013, 0.7855, 1.0097,  ..., 0.6400, 0.6313, 0.6433],
          [0.7416, 0.9822, 1.0205,  ..., 0.6531, 0.6572, 0.6530],
          [0.7859, 1.0137, 1.0205,  ..., 0.5612, 0.5617, 0.5738]],

         [[0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          ...,
          [0.7186, 0.9421, 1.0205,  ..., 0.6280, 0.6234, 0.6135],
          [0.8894, 1.0205, 1.0205,  ..., 0.6526, 0.6705, 0.6757],
          [0.9769, 1.0205, 1.0205,  ..., 0.6016, 0.5981, 0.5964]],

         [[0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          ...,
          [0.8628, 1.0175, 1.0205,  ..., 0.5996, 0.6109, 0.6234],
          [1.0101, 1.0205, 1.0205,  ..., 0.6332, 0.6539, 0.6628],
          [1.0205, 1.0205, 1.0205,  ..., 0.6313, 0.6272, 0.6161]],

         ...,

         [[0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          ...,
          [0.5368, 0.5428, 0.5347,  ..., 0.6339, 0.6381, 0.6275],
          [0.4150, 0.4090, 0.4503,  ..., 0.6278, 0.6290, 0.6324],
          [0.3546, 0.3622, 0.3839,  ..., 0.5990, 0.5990, 0.6201]],

         [[0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          ...,
          [0.3950, 0.3938, 0.3991,  ..., 0.6393, 0.6416, 0.6113],
          [0.3051, 0.2950, 0.3345,  ..., 0.5811, 0.5953, 0.6142],
          [0.3447, 0.3392, 0.3360,  ..., 0.5712, 0.5904, 0.6149]],

         [[0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          [0.0910, 0.0910, 0.0910,  ..., 0.0910, 0.0910, 0.0910],
          ...,
          [0.3520, 0.3285, 0.3377,  ..., 0.5786, 0.6226, 0.6147],
          [0.3169, 0.2862, 0.3129,  ..., 0.5287, 0.5549, 0.5804],
          [0.3525, 0.3525, 0.3389,  ..., 0.5001, 0.5346, 0.5749]]]]), metatensor([[[[0.2649, 0.2645, 0.2643,  ..., 0.2726, 0.2655, 0.2615],
          [0.2617, 0.2608, 0.2610,  ..., 0.2725, 0.2655, 0.2615],
          [0.2587, 0.2519, 0.2446,  ..., 0.2724, 0.2654, 0.2614],
          ...,
          [0.5780, 0.5780, 0.5798,  ..., 0.6450, 0.6406, 0.6348],
          [0.5780, 0.5780, 0.5801,  ..., 0.6696, 0.6718, 0.6725],
          [0.5780, 0.5780, 0.5804,  ..., 0.6582, 0.6717, 0.6836]],

         [[0.2651, 0.2644, 0.2646,  ..., 0.2540, 0.2511, 0.2511],
          [0.2620, 0.2608, 0.2611,  ..., 0.2540, 0.2510, 0.2511],
          [0.2601, 0.2538, 0.2459,  ..., 0.2539, 0.2510, 0.2511],
          ...,
          [0.5780, 0.5779, 0.5789,  ..., 0.6342, 0.6223, 0.6098],
          [0.5780, 0.5779, 0.5792,  ..., 0.6808, 0.6708, 0.6580],
          [0.5780, 0.5779, 0.5795,  ..., 0.7130, 0.7151, 0.7140]],

         [[0.2657, 0.2645, 0.2646,  ..., 0.2557, 0.2575, 0.2609],
          [0.2626, 0.2610, 0.2610,  ..., 0.2557, 0.2575, 0.2610],
          [0.2610, 0.2553, 0.2475,  ..., 0.2557, 0.2575, 0.2611],
          ...,
          [0.5781, 0.5779, 0.5781,  ..., 0.5916, 0.5916, 0.5976],
          [0.5781, 0.5779, 0.5782,  ..., 0.6451, 0.6335, 0.6231],
          [0.5781, 0.5779, 0.5785,  ..., 0.7309, 0.7213, 0.7088]],

         ...,

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.6340, 0.6072, 0.5695,  ..., 0.2343, 0.2390, 0.2450],
          [0.6338, 0.6067, 0.5690,  ..., 0.2553, 0.2545, 0.2610],
          [0.6336, 0.6062, 0.5685,  ..., 0.2767, 0.2760, 0.2686]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.4732, 0.4414, 0.4207,  ..., 0.2335, 0.2376, 0.2438],
          [0.4727, 0.4412, 0.4204,  ..., 0.2556, 0.2536, 0.2588],
          [0.4722, 0.4409, 0.4202,  ..., 0.2765, 0.2775, 0.2708]],

         [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          ...,
          [0.3678, 0.3497, 0.3400,  ..., 0.2327, 0.2367, 0.2425],
          [0.3675, 0.3496, 0.3399,  ..., 0.2559, 0.2539, 0.2573],
          [0.3672, 0.3495, 0.3398,  ..., 0.2764, 0.2776, 0.2725]]]]), metatensor([[[[1.1500, 1.1548, 1.1476,  ..., 0.3513, 0.3514, 0.3518],
          [1.1423, 1.1567, 1.1513,  ..., 0.3555, 0.3668, 0.3825],
          [1.1345, 1.1573, 1.1546,  ..., 0.3484, 0.3835, 0.4137],
          ...,
          [0.3391, 0.3389, 0.3232,  ..., 0.2957, 0.2997, 0.3065],
          [0.3589, 0.3561, 0.3352,  ..., 0.3114, 0.3220, 0.3260],
          [0.3537, 0.3494, 0.3407,  ..., 0.3050, 0.3091, 0.3092]],

         [[1.1352, 1.1456, 1.1251,  ..., 0.3316, 0.3363, 0.3472],
          [1.1280, 1.1508, 1.1331,  ..., 0.3843, 0.3934, 0.4028],
          [1.1204, 1.1533, 1.1406,  ..., 0.4186, 0.4207, 0.4164],
          ...,
          [0.3467, 0.3479, 0.3292,  ..., 0.3060, 0.3129, 0.3176],
          [0.3581, 0.3579, 0.3422,  ..., 0.3355, 0.3391, 0.3389],
          [0.3526, 0.3482, 0.3420,  ..., 0.3274, 0.3272, 0.3259]],

         [[1.1110, 1.1226, 1.0778,  ..., 0.3403, 0.3548, 0.3671],
          [1.1067, 1.1369, 1.0963,  ..., 0.4020, 0.4110, 0.4200],
          [1.1022, 1.1425, 1.1142,  ..., 0.4166, 0.4133, 0.4096],
          ...,
          [0.3533, 0.3556, 0.3375,  ..., 0.3122, 0.3146, 0.3127],
          [0.3563, 0.3584, 0.3470,  ..., 0.3292, 0.3210, 0.3110],
          [0.3524, 0.3475, 0.3459,  ..., 0.3181, 0.3063, 0.2971]],

         ...,

         [[0.4074, 0.4110, 0.4146,  ..., 0.3804, 0.3638, 0.2749],
          [0.4163, 0.4086, 0.4049,  ..., 0.3431, 0.3211, 0.2403],
          [0.3659, 0.3409, 0.3259,  ..., 0.3110, 0.2773, 0.1985],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

         [[0.4001, 0.3987, 0.3974,  ..., 0.3751, 0.3657, 0.2977],
          [0.4081, 0.4032, 0.3966,  ..., 0.3359, 0.3241, 0.2613],
          [0.3423, 0.3281, 0.3102,  ..., 0.3108, 0.2798, 0.2124],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

         [[0.3718, 0.3668, 0.3576,  ..., 0.3659, 0.3679, 0.3204],
          [0.3856, 0.3797, 0.3629,  ..., 0.3313, 0.3262, 0.2762],
          [0.3154, 0.2963, 0.2838,  ..., 0.3116, 0.2793, 0.2167],
          ...,
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]]]])] ... )
CRITICAL:dev_collate:> collate dict key "label" out of 5 keys
CRITICAL:dev_collate:>> collate/stack a list of tensors
CRITICAL:dev_collate:> collate dict key "fold" out of 5 keys
CRITICAL:dev_collate:> collate dict key "image_meta_dict" out of 5 keys
CRITICAL:dev_collate:>> collate dict key "sizeof_hdr" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "extents" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "session_error" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim_info" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p1" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p2" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p3" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "datatype" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "bitpix" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_start" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "pixdim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "vox_offset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_slope" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_inter" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_end" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "xyzt_units" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_max" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_min" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_duration" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "toffset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmax" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmin" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "sform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_b" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_c" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_d" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "original_affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "as_closest_canonical" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "spatial_shape" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "space" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "original_channel_dim" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "filename_or_obj" out of 43 keys
CRITICAL:dev_collate:> collate dict key "label_meta_dict" out of 5 keys
CRITICAL:dev_collate:>> collate dict key "sizeof_hdr" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "extents" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "session_error" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim_info" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "dim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p1" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p2" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_p3" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "intent_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "datatype" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "bitpix" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_start" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "pixdim" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "vox_offset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_slope" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "scl_inter" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_end" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "xyzt_units" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_max" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "cal_min" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "slice_duration" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "toffset" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmax" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "glmin" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "sform_code" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_b" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_c" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "quatern_d" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "qoffset_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_x" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_y" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "srow_z" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "original_affine" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "as_closest_canonical" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "spatial_shape" out of 43 keys
CRITICAL:dev_collate:>>> collate/stack a list of numpy arrays
CRITICAL:dev_collate:>>> collate/stack a list of tensors
CRITICAL:dev_collate:>> collate dict key "space" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "original_channel_dim" out of 43 keys
CRITICAL:dev_collate:>> collate dict key "filename_or_obj" out of 43 keys
dints_0 - training ...:   0%|          | 0/100 [13:31<?, ?round/s]
Traceback (most recent call last):
  File "/home/exsdatalab/mlworks/ULS23/uls_work_dir/dints_0/scripts/train.py", line 984, in <module>
    fire.Fire()
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/exsdatalab/mlworks/ULS23/uls_work_dir/dints_0/scripts/train.py", line 564, in run
    for batch_data in train_loader:
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 3.
Original Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 495, in list_data_collate
    ret = collate_meta_tensor(data)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 464, in collate_meta_tensor
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 464, in <dictcomp>
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 454, in collate_meta_tensor
    collated = default_collate(batch)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 123, in collate
    return collate_fn_map[collate_type](batch, collate_fn_map=collate_fn_map)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 161, in collate_tensor_fn
    out = elem.new(storage).resize_(len(batch), *list(elem.size()))
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 282, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/_tensor.py", line 1386, in __torch_function__
    ret = func(*args, **kwargs)
RuntimeError: Trying to resize storage that is not resizable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/data/utils.py", line 508, in list_data_collate
    raise RuntimeError(re_str) from re
RuntimeError: Trying to resize storage that is not resizable

[2024-02-16 09:22:54,018] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1264425 closing signal SIGTERM
[2024-02-16 09:22:54,432] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1264424) of binary: /home/exsdatalab/mambaforge/bin/python3.10
Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
uls_work_dir/dints_0/scripts/train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-02-16_09:22:54
  host      : exsdatalab
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1264424)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[23], line 4
      2 for algo_dict in history:
      3     algo = algo_dict[AlgoKeys.ALGO]
----> 4     algo.train(train_param)  # can use default params by `algo.train()`
      5     acc = algo.get_score()
      6     algo_to_pickle(algo, template_path=algo.template_path, best_metric=acc)

File ~/mlworks/ULS23/uls_work_dir/algorithm_templates/dints/scripts/algo.py:490, in DintsAlgo.train(self, train_params, device_setting, search)
    488 cmd, devices_info = self._create_cmd(dints_train_params)
    489 cmd = "OMP_NUM_THREADS=1 " + cmd
--> 490 return self._run_cmd(cmd, devices_info)

File ~/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:251, in BundleAlgo._run_cmd(self, cmd, devices_info)
    249     return _run_cmd_bcprun(cmd, n=self.device_setting["NUM_NODES"], p=self.device_setting["n_devices"])
    250 elif int(self.device_setting["n_devices"]) > 1:
--> 251     return _run_cmd_torchrun(
    252         cmd, nnodes=1, nproc_per_node=self.device_setting["n_devices"], env=ps_environ, check=True
    253     )
    254 else:
    255     return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)

File ~/mambaforge/lib/python3.10/site-packages/monai/auto3dseg/utils.py:502, in _run_cmd_torchrun(cmd, **kwargs)
    500     torchrun_list += [f"--{arg}", str(params.pop(arg))]
    501 torchrun_list += cmd_list
--> 502 return run_cmd(torchrun_list, run_cmd_verbose=True, **params)

File ~/mambaforge/lib/python3.10/site-packages/monai/utils/misc.py:874, in run_cmd(cmd_list, **kwargs)
    872     monai.apps.utils.get_logger("run_cmd").info(f"{cmd_list}")
    873 try:
--> 874     return subprocess.run(cmd_list, **kwargs)
    875 except subprocess.CalledProcessError as e:
    876     if not debug:

File ~/mambaforge/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    524     retcode = process.poll()
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['torchrun', '--nnodes', '1', '--nproc_per_node', '2', 'uls_work_dir/dints_0/scripts/train.py', 'run', "--config_file='uls_work_dir/dints_0/configs/hyper_parameters.yaml,uls_work_dir/dints_0/configs/hyper_parameters_search.yaml,uls_work_dir/dints_0/configs/network.yaml,uls_work_dir/dints_0/configs/network_search.yaml,uls_work_dir/dints_0/configs/transforms_infer.yaml,uls_work_dir/dints_0/configs/transforms_train.yaml,uls_work_dir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=2']' returned non-zero exit status 1.

Do you have any insights into why the proposed transform is not getting applied and how this can be fixed?

Many thanks and kind regards,

auto3dseg autorunner tensor mismatch error #7433

Uh oh!

Uh oh!

bilalcodehub Feb 1, 2024

Replies: 1 comment · 8 replies

Uh oh!

KumoLiu Feb 2, 2024 Maintainer

Uh oh!

KumoLiu Feb 14, 2024 Maintainer

Uh oh!

Uh oh!

bilalcodehub Feb 14, 2024 Author

Uh oh!

KumoLiu Feb 14, 2024 Maintainer

Uh oh!

KumoLiu Feb 14, 2024 Maintainer

Uh oh!

Uh oh!

bilalcodehub Feb 16, 2024 Author

bilalcodehub
Feb 1, 2024

Replies: 1 comment 8 replies

KumoLiu
Feb 2, 2024
Maintainer

KumoLiu Feb 14, 2024
Maintainer

bilalcodehub Feb 14, 2024
Author

KumoLiu Feb 14, 2024
Maintainer

KumoLiu Feb 14, 2024
Maintainer

bilalcodehub Feb 16, 2024
Author