Transform issue during training

[pritamrungta]

Training a deepgrow_3d model

image -> dataset/train1.nii.gz
label -> dataset/labels/final/train1.nii.gz
request -> {"num_epochs": 50, "datalist": [{"image": "train1.nii.gz", "label": "train1.nii.gz"}]}

get this error

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 89, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 53, in _apply_transform
    return transform(parameters)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py", line 377, in __call__
    d[key], old_affine, new_affine = self.ornt_transform(d[key], affine=meta_data["affine"])
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monai/transforms/spatial/array.py", line 326, in __call__
    spatial_ornt = nib.orientations.ornt_transform(src, dst)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/nibabel/orientations.py", line 128, in ornt_transform
    raise ValueError("Unable to find out axis %d in start_ornt" %
ValueError: Unable to find out axis 0 in start_ornt
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monailabel/interfaces/utils/app.py", line 132, in <module>
    run_main()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monailabel/interfaces/utils/app.py", line 117, in run_main
    result = a.train(request)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 366, in train
    result = task(request, self.datastore())
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monailabel/tasks/train/basic_train.py", line 350, in __call__
    datalist = self.pre_process(req, datastore)
  File "/home/ubuntu/deepgrow/lib/train.py", line 88, in pre_process
    datalist = create_dataset(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monai/apps/deepgrow/dataset.py", line 101, in create_dataset
    data = transforms({image_key: image, label_key: label})
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monai/transforms/compose.py", line 173, in __call__
    input_ = apply_transform(_transform, input_, self.map_items, self.unpack_items, self.log_stats)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 113, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.spatial.dictionary.Orientationd object at 0x7f1c9bdcc520>
[2022-03-16 08:56:01,683] [2426] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:76) - Return code: 1

[wyli]

this is probably a usage question, could you please share some data samples and the transforms you are using? (converting to a discussion)

[pritamrungta]

The following are image and label samples:
image -> train1.nii.gz
label -> train1.nii.gz

started server using:
monailabel start_server --app deepgrow --studies dataset

and ran training using the curl command
POST /train/deepgrow_3d
data = {'max_epochs': 50, 'datalist': [{'image': 'train1', 'label': 'train1'}]}

and then this error showed up on the server logs.
let me know if I could provide more information.
Thanks

[wyli]

thanks, I don't have details of your preprcoessing transforms, but perhaps you missed EnsureChannelFirstD.
I downloaded the volumes and these work fine:

from monai.transforms import LoadImageD, OrientationD, EnsureChannelFirstD, Compose

xform = Compose([LoadImageD(keys=("img", "seg")), 
                 EnsureChannelFirstD(keys=("img", "seg")), 
                 OrientationD(keys=("img", "seg"), axcodes="RAS")])
output = xform({"img":"/Users/Downloads/train1.nii.gz", 
                "seg": "/Users/Downloads/train1_seg.nii.gz"})

[SachidanandAlle]

Deepgrow flattens the image/label for each label/organ type..
https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/deepgrow/lib/train.py#L81-L97

Now coming to the problem.. The first error talks about
raise ValueError("Unable to find out axis %d in start_ornt" %

Which means, how you got this label? Looks some issue with that.. lets do this.. download sample image/label from MSD Dataset and try one of those image/labels..

You can also download MSD dataset by using monailabel
monailabel datasets --help

[pritamrungta]

I got this dataset from https://www.kaggle.com/c/testzl/data

[SachidanandAlle]

https://msd-for-monai.s3-us-west-2.amazonaws.com/Task09_Spleen.tar

You can pick one or two image/label from here and try same action.. I see it's more related to the data you were using.. this trial is just to isolate the issue.. and later someone can dig deep into the test data which you were trying.. what meta info is missing there.. that Loading such labels fail.. (it might fail even for basic segmentation app also while training.. my guess) if you are running some Orientation on top of it..

[SachidanandAlle]

Also the second error ValueError: operands could not be broadcast together with shapes (4,) (3,) says the label has something extra.. in that case you need to handle the pre-transforms accordingly.. lets not believe one set of pre-transform works for all kind of data/images

the end goal of pre-transform shall be to prepare what we need to feed to the network for training/inference.. in case of deepgrow it is
Number of Batch, Channel (3 => Image, Foreground clicks, Background clicks), 3D array..

or in simple words: NCHWD

[pritamrungta]

Yes, the other dataset works. And thanks for the explanation. It was so helpful.