Auto3dSeg SwinUnetR with Multiple inputs

[a-parida12]

I am trying the auto3dseg to do a segmentation task. The aim is to be able to input 3 different MRI modalities, which needs to be stacked as channels to be input into swinuter and produce a single channel output segmention.

Based on the HECKTOR22 example this is how input yaml looks like

modality: MRI                        # primary modality
dataroot: '/data/'           # dataset location
datalist: dataset_folds.json      # a list of filenames
class_names: ['out']    # names for tensorboard
extra_modalities: {image1 : MRI, image2: MRI}    # a second modality
roi_size: [96, 96, 96]

but the code still launches and shows me single channel input. How can I modify the functionality.

[KumoLiu]

Hi @a-parida12, It seems that auto3dseg is not concat the data for you; perhaps you should do a concat yourself before putting the data into it.
https://github.com/Project-MONAI/research-contributions/blob/main/auto3dseg/algorithm_templates/swinunetr/configs/transforms_train.yaml
Also, remember to update the input_channels.
https://github.com/Project-MONAI/research-contributions/blob/fecb45270c9ad0fedb75fce255fb3e63140935f9/auto3dseg/algorithm_templates/swinunetr/configs/network.yaml#L5

Hope it helps, thanks!

[a-parida12]

Thanks for helping me debug the issue. I ended up modifying the swinunetr template as per suggestion.

runner = AutoRunner(input="input.yaml", 
                    algos= {"algname":dict(_target_='swinunetr.scripts.algo.SwinunetrAlgo', template_path = "swinunetr")},
                    hpo=True,
                    work_dir="./work_dir")

Thanks for your help @KumoLiu

[che85]

I think AutoRunner conveys the wrong idea if one has to use workarounds to get Auto3DSeg to use multiple inputs, even though you can provide that information in the task configuration and add that add additional imageXYZ in the datalist file.

a) If extra_modalities was provided, but it's not used, there should be a notification.
b) notify the user if datalist.json provides additional keys (e.g.image2)

I just prepared all the data to then find out that it's not using all the inputs I am providing.

@diazandr3s, maybe you have some more insights

[diazandr3s]

Hi @che85,

I haven't trained myself the Auto3DSeg on a multi-sequence/multi-modality task. But it should work well as it won the BRATS challenge.
Did you follow this example to create the JSON and YAML files? https://github.com/Project-MONAI/tutorials/tree/main/auto3dseg/tasks/hecktor22

How do you know the models are not being trained on all modalities?

Please let us know,

[che85]

@diazandr3s Yes, followed the same schema as done in https://github.com/Project-MONAI/tutorials/tree/main/auto3dseg/tasks/hecktor22

The extra modality does not show up in the analyzer output datasets.yaml, so I am assuming that these files are never touched.

I can also just randomly rename the image2 to anything and it won't complain (considering I don't use the following part of the task.

# settings below are optional, but will accelerate the training speed
# compared to fully automated settings. Here, we add a custom transform to
# pre-crop an approximate head and neck region to reduce image size to 200x200x310mm
custom_data_transforms:
  - key: after_resample_transforms
    path: '.'
    transform: {_target_: hecktor_crop_neck_region.HecktorCropNeckRegion, box_size: [200, 200, 310]}

I will check with the actual BRATS to see if my assumption is right and get back to you on that.

[diazandr3s]

Yes, please keep us posted!

@dongyang0122 is there a way to confirm Auto3DSeg uses all sequences?

[che85]

After digging a little more it seems that only segresnet is using multiple inputs.

Is there any documentation of what configuration can be set solely from the input yaml?

For example, setting roi_size: [96, 96, 96] as used above, doesn't seem to have any effect.

[diazandr3s]

Hi @che85,

Agree. We need more use cases with different settings.
In most of my use cases, I use Auto3DSeg with the SegResNet architecture only. It works quite well.
Regarding the ROI size, these are the flags I used in the YAML file:

roi_size: [128, 128, 32]
auto_scale_allowed: False

Hope this helps,