Fine-Tuning MAISI for Head/Neck CT Always Falls Back to Chest Anatomy

[Mudassar-MLE]

Hi,

First, thank you to the MAISI team for the incredible work. I am currently exploring MAISI for neck CT, but I’m running into several challenges. There don’t seem to be many anatomies supported for head/neck CT. Even the existing models are unable to generate meaningful images of the brain, head, or neck, although the documentation mentions that indexing “1” should support head-neck CT.

I tried fine-tuning ControlNet following the provided instructions, adding around 40 new labels with 5–10 cases. However, the model did not converge. After experimenting with different parameters and epochs, it showed slight convergence, but the outputs still mostly resembled chest anatomy with a rough neck contour.

In addition, I reviewed the configs/image_median_statistics.json file in the HuggingFace repo and couldn’t find anything related to neck anatomy. Dataset preparation is not an issue on my side—I am generating embeddings through the MAISI scripts, and masks are 100% verified.

This leaves me confused: do I need to train ControlNet from scratch (and potentially each model from scratch), or is MAISI simply not designed to handle neck scenarios? Also, what would be a good dataset size to train on? In community discussions, I noticed that results were inconsistent—some had issues even with 300 cases, while others reported good results with ~700 cases.

Any guidance or advice on this would be really appreciated!

@guopengf @Can-Zhao

[Can-Zhao]

It looks like you were using maisi-ddpm? It is a known issue that this version was not good at head and neck. Have you tried maisi-rflow?

[Mudassar-MLE]

I mainly followed the Hugging Face code scripts, but I also tried the maisi-rflow models available on GitHub as well as the models published on Hugging Face. Since the Hugging Face script was last updated about five months ago, I’m assuming it corresponds to the MAISI-rflow implementation.
Secondly, I’d like to get your thoughts on the required dataset size for fine-tuning when working with more than 40 new labels. Also, in the image_median_statistics.json file, I couldn’t find any fields related to head/neck anatomy.

[Can-Zhao]

There is a parameter in command line that controlls the version you use "--version maisi3d-rflow".
If you have used "--version maisi3d-rflow" but still see bad results, it is possible that the spacing and spatial size is not within supported range. But the inference code should give you a warning for that if you used this code as the tutorial shows:

export MONAI_DATA_DIRECTORY=<dir_you_will_download_data>
python -m scripts.inference -c ./configs/config_maisi3d-rflow.json -i ./configs/config_infer.json -e ./configs/environment_maisi3d-rflow.json --random-seed 0 --version maisi3d-rflow

[sara-create]

@Can-Zhao I have a related question. For the rFlow checkpoints, is it possible to constrain the model to generate only a specific region, for example chest CT? I couldn’t find where this can be configured. (In my case, I am specifically using only the diffusion model, not ControlNet => not the function LDMSampler)

[Can-Zhao]

No. We removed this part. As we found it requires extra effort for users to generate labels when they prepare training data, especially some data has mixed regions. For users who are interested in inference only, Controlnet can restrict regions by giving the anatomy structure list.