LongiSeg on a fluoroscopy video

[KanielDatz]

Hello,
I came across your repository and I'm trying to understand if it fits my needs.

I'm currently using the excellent nnUNet ResEnc for a task involving 2D fluoroscopy videos, where I'm performing segmentation frame by frame. I'm looking for ways to incorporate temporal features into the segmentation process. So far, I've tried using adjacent frames as different modalities in the nnUNet input, and also by stacking frames into a 3D volume.

From what I understand, your project is designed for multiple scans of the same patient. I'm wondering if there's a way to repurpose it so that I can treat the frames of a video as if they were separate scans of the same patient—so, for example, a 30-frame video would be treated as 30 “scans” of a single case?

Before diving deeper into the project, I’d appreciate your thoughts on whether this kind of adaptation would be feasible.

Thanks!

[ykirchhoff]

Hi @KanielDatz,

we have never tried using LongiSeg with 2d data, but it is definitely possible to do and might be worth giving it a try. If you have the frames as individual files you only need to create a patientsTr.json indicating the frames per video in the correct order and then you should be good to go. LongiSeg will however only use one prior scan due to memory limitations in 3d, for 2d it might make sense to extend that further. But I guess as a baseline it shouldn't be too much work to get it started just as it is.

Let me know, if you have any further questions or run into any issues!

Best,
Yannick

[KanielDatz]

Thanks for the quick and helpful response!

I’ve successfully trained LongiSeg using the nnUNetTrainerDiceTopK10Loss, which has shown the best performance for our 2D fluoroscopy segmentation task when used with regular nnUNet. Interestingly, the results were quite comparable to the standard nnUNet setup, no improvement.

Just to make sure I’ve configured the patientsTr.json correctly, here's a snippet from the file:

{
  "0000.00.10": [
    "0000.00.10.0000",
    "0000.00.10.0015",
    "0000.00.10.0030",
    "0000.00.10.0045"
  ],
  "0001.00.00": [
    "0001.00.00.0000",
    "0001.00.00.0015",
    "0001.00.00.0030",
    "0001.00.00.0045"
  ],
  "0001.00.01": [
    "0001.00.01.0000",
    "0001.00.01.0015",
    "0001.00.01.0030",
    "0001.00.01.0045",
    "0001.00.01.0060"
  ]
}

Each file name follows the format:
<patient ID>.<procedure ID>.<fluoro sequence ID>.<frame index>

Let me know if this structure aligns with how LongiSeg expects the data.

Additionally, I was wondering if there are any plans to integrate the ResEnc architecture into LongiSeg? We've observed notable improvements in our use case when using ResEnc, and it would be great to leverage that in the longitudinal setting as well.

Lastly, could you point me to the relevant parts of the codebase where I could modify the pipeline to use more than one prior scan? longi_dataset, longi_data_loader, nnUNetTrainerLongi files, any more hints?

Thanks again!

[KanielDatz]

I just started looking into the code and noticed that the TopK10 trainer I used inherits from nnUNetTrainerNoLongi, which I assume is the standard nnU-Net trainer without longitudinal fusion. Does that mean I actually didn't use the LongiSeg architecture at all during training? 😄

[ykirchhoff]

Hi @KanielDatz,

yeah unfortunately that trainer doesn't use longitudinal context at all.
Your patientsTr.json looks fine, that should work as is.
You can use any architecture available for nnUNet with LongiSeg. This part of the code, located in the difference weighting repo, actually takes care of that and allows you to have ResEncL as a backbone network if you like.
Some files you need to modify are:

• longi_dataset: needs to be adapted to load multiple frames
• longi_data_loader: same as for the dataset
• architecture: change input_channels in the backbone
• LongiSegTrainer: especially train_step, validation_step and perform_actual_validation
• merge and split transform: used to be compatible with normal nnUNet augmentations, also has to be changed in the LongiSegTrainer

These are at least all files that I could think of right now.

Best,
Yannick