tips and tricks for training on large volumes

[joho84]

Hi monai-team, hi all,
first of all thank you very much for the great work you continue doing here!
I am using monai to segment multiple small regions in a large 3D volume (with multiple channels). There might be up to 1000 segments (between 50 and 500 voxels in volume) in a volume of around 500x500x2000 voxels. Overall the training works but I would like to increase the training speed and segmentation performance.
It would be great if you could share some best practices or ideas for the following questions:

• What would be the fastest way to train the network? (At the moment I crop patches from the large volume in a pre-processing step and then use CachDataset to cache the patches for training.)
• What would be a reasonable cache rate in this scenario? Is there a better way to deal with the data loading? (I did have a look at the tutorials but they mostly deal with volumes which can be loaded to GPU right away.)
• Do you have any recommondations to improve the performance for the many small segmentations? (I guess the network struggles a bit with the high class imbalance.)

Thank you very much in advance!

[KumoLiu]

Hi @joho84,
First of all, we appreciate your interest in MONAI and hope it will be very beneficial to you.
For your first questions, I strongly recommend you to read fast_training_tutorial for more tips to optimize your training program.
Typically, the cache rate depends on the hardware, possibly you might try several experiments to discover the ideal cache rate.
Finally, you can try FocalLoss which down-weights loss from high confidence correct predictions or use pos_weight in nn. BCEWithLogitsLoss to give greater penalty weight for fewer categories.
Hope these can help you!

[joho84]

Hi @KumoLiu, hi all,
thank you very much for your quick help!
I did look at the fast_training_tutorial but this mostly focuses on profiling. I was hoping you or someone else could already share some tips what worked for him/her as a starting point for my optimization.

• In especially, I would be interested in how others deal with large volumes which cannot be handeled as easily as some of the example data in the tutorials?
• Is there a better way to deal with large volumes than cropping them in a pre-processing step, saving the smaller volumes and then loading those volumes for training? (I found this to be faster than cropping in the actual training process but might be mistaken.)
• Are there any experiences on how the size of the cropped cubes influences the training and performance of the model?

As for your suggestions for the adapted loss, I will try those. Thank you very much!