MONAI Label and 3D Slicer: CUDA out of memory

[andmog]

I am new at MONAI Label.

I am trying the Radiology demo of MONAI Label.
In 3D Slicer I segmented with scribbles 8 spleens of MSD dataset and trained the spleen_segmentation model.

Then I used that pretrained model to see how it works on unseen data. I clicked Run in the Auto Segmentation menu of 3D Slicer using the segmentation_spleen model and was able to view the spleen. Then I tried to refine the model.
When using deepgrow_3d I got an error in 3D Slicer and one in Terminal (CUDA out of memory).
Please, see attached two screenshots.
I am using a Nvidia Geforce RTX 2080 (8 GB of memory).
Could you help me?

Thank you in advance.

Andrea

[SachidanandAlle]

Deepgrow 3D and DeepEdit models need more GPU.. 16 to 24 GB.. may be you try to restart to release existing gpu memory.. and start only one task at a time..

[andmog]

Thank you.
I tried to restart the MONAI Label server and also 3D Slicer. However the problem persists.
The only solution is to reboot the computer. Is it there another way to release the GPU memory to avoid restarting the computer?

[diazandr3s]

Hi @andmog,

Thanks for opening this discussion.
Did you start the MONAI Label server with both segmentation and deepgrow model? If yes, how do you plan to use the deepgrow model: for inference or for training? Using both models might need more GPU memory.
My suggestion is to keep using segmentation model with scribbles or grow from seeds functionality. Here is a video series showing these two: https://www.youtube.com/watch?v=3HTh2dqZqew&list=PLtoSVSQ2XzyD4lc-lAacFBzOdv5Ou-9IA&index=3

I hope this helps

[andmog]

Hi @diazandr3s,

Thank you for your help.
I started MONAI Label server adding the --conf models all for simplicity.
I used scribbles to create 7-8 segmented spleens.
Then in Active Learning I selected 'segmentation_spleen' in the dropdown menu. I trained that model.
Then I created a new app (basically a copy of Radiology) to try a new scenario.
I restarted the MONAI Label server again with --conf models all for simplicity.
I clicked on Next Sample. In Auto Segmentation I clicked on Run to infer selecting 'segmentation_spleen' in the dropdown menu.
What I needed was to refine the inferred model before clicking on Submit label.
In Smart/Edit when I click on deepgrow_3d I can work. See attached a screenshot.
If I repeat the process and want to refine a second image using deepgrow_3d then I get the crash.
I need to restart the computer. Restarting 3DSlicer and MONAI Label server is not enough.

Thanks

[diazandr3s]

Thanks for the detailed post, @andmog.
As @SachidanandAlle suggested, in order to use DeepGrow along with the segmentation model you'd need a bigger GPU.
My suggestion would be to use the segmentation model along with scribbles instead: --conf models segmentation

[andmog]

Thanks @diazandr3s
I watched your turorial (https://www.youtube.com/watch?v=3HTh2dqZqew&list=PLtoSVSQ2XzyD4lc-lAacFBzOdv5Ou-9IA&index=3).
It is very clear.
I hope you will add new ones in the future.
Sometimes when I click on Next Sample 3D Slicer does not retrieve a new model. Is it normal?
See attached a screenshot

[diazandr3s]

It is not normal. Next time please post the logs you get from the MONAI Label server to see what's going on there :)

[andmog]

@diazandr3s
Here is a screenshot of the MONAI Label server. Thank you

[diazandr3s]

Thanks, @andmog.
It's strange, I don't see any issue from the logs.

[SachidanandAlle]

why some files are starting with ._ (hidden files)

these 2 errors are different.. the recent one when u clicked next sample.. looks like more of actual image loading issue in the 3D slicer.. possibly due to incorrect images (._).. u can check 3D slicer logs.. also some logs when u open python interpreter in 3D slicer

the top one is GPU memory not enough.. u can check nvidia-smi and see current usage for the GPU

[pieper]

why some files are starting with ._ (hidden files)

I saw this same issue on some of the sample files I downloaded - I needed to delete these to so they weren't loaded with Next Sample. I think it's a mac file.

[SachidanandAlle]

https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/main.py#L165-L169

[andmog]

@SachidanandAlle @pieper

I found that there were hidden files with the same name of the images but starting with ._
I deleted them and fixed. Thanks.
I am using Ubuntu 20.04. I simply downloaded the spleen dataset from https://drive.google.com/drive/folders/1HqEgzS8BV2c7xYNrZdEAnrHk7osJJ--2

I will check next time with nvidia-smi the usage of my GPU.

Thanks

[pieper]

https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/main.py#L165-L169

Yes, I hadn't run into this problem download with MONAI. But the built-in download seemed slow so I found a copy on aws s3 and downloaded them with the cli. I put them in the dataset manually and needed to delete the extra OS files like @andmog did.