Guidance on Using SAM with Endoscopy App in MONAILabel

[bilalcodehub]

Dear MONAILabel Team,

Could you kindly direct me to any available tutorials or documentation on integrating SAM with the Endoscopy App in MONAILabel? I would greatly appreciate any guidance.

Looking forward to your advice.

Best regards,
Bilal

[diazandr3s]

Hi @bilalcodehub,

Thanks for asking this.
If you use the latest MONAI Label version, it'll come with SAM2/SAM2.1. Please refer to this part: https://github.com/Project-MONAI/MONAILabel/tree/main?tab=readme-ov-file#sam-2

Hope this helps,

[bilalcodehub]

Hi @diazandr3s,

Thank you for your reply.

I’ve started the MONAI Label server (running from latest Docker), but I’m not sure if the SAM model gets deployed automatically to Nuclio. Nothing seems to happen after the server starts.

Do I need to manually deploy the SAM model to Nuclio myself?
If so, could you point me to the steps to do that?

Any guidance on what might be going wrong would be much appreciated.

Thanks again!

Best regards,
Bilal

[bilalcodehub]

Do I need to run the following command to deploy SAM to Nuclio for CVAT integration?

./deploy.sh <function> <model> <functions_dir>

[bilalcodehub]

Never mind. ./deploy.sh command worked. Excellent.

[bilalcodehub]

Still getting an error with SAM interactive use. When I add a positive point on the image, I get the following error in CVAT UI:

500 Server Error: Internal Server Error for url: http://host.docker.internal:32802/

Checked both dockers logs monailabel and sam nuclio docker, so it seems like same_2d is not registered with monailabel server.

So I attempted the following command but it seems like sam_2d is not a valid model in the list of models supported by endoscopy app.

root@blssai-server01:/opt/monailabel# monailabel start_server --app surgvu/endoscopy --studies surgvu/dataset/images --conf models sam_2d,tooltracking --conf epistemic_enabled true --conf epistemic_top_k 84 --conf cvat_segment_size 84 --conf cvat_labels_file surgvu/endoscopy/labels.csv
Using PYTHONPATH=/usr:

[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: version = False
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: app = /opt/monailabel/surgvu/endoscopy
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: studies = /opt/monailabel/surgvu/dataset/images
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: verbose = INFO
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: conf = [['models', 'sam_2d,tooltracking'], ['epistemic_enabled', 'true'], ['epistemic_top_k', '84'], ['cvat_segment_size', '84'], ['cvat_labels_file', 'surgvu/endoscopy/labels.csv']]
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: host = 0.0.0.0
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: port = 8000
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: uvicorn_app = monailabel.app:app
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile = None
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: ssl_certfile = None
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile_password = None
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: ssl_ca_certs = None
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: workers = None
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: limit_concurrency = None
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: access_log = False
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: root_path =
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: log_level = info
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: log_config = None
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: dryrun = False
[2025-03-29 01:57:46,622] [844] [MainThread] [INFO] (main:285) - USING:: action = start_server
[2025-03-29 01:57:46,623] [844] [MainThread] [INFO] (main:296) -
Allow Origins: ['*']
[2025-03-29 01:57:47,653] [844] [MainThread] [INFO] (uvicorn.error:82) - Started server process [844]
[2025-03-29 01:57:47,654] [844] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup.
App Init...
[2025-03-29 01:57:47,654] [844] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: /opt/monailabel/surgvu/endoscopy; studies: /opt/monailabel/surgvu/dataset/images; conf: {'models': 'sam_2d,tooltracking', 'epistemic_enabled': 'true', 'epistemic_top_k': '84', 'cvat_segment_size': '84', 'cvat_labels_file': 'surgvu/endoscopy/labels.csv'}
[2025-03-29 01:57:47,664] [844] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2025-03-29 01:57:47,690] [844] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.tooltracking.ToolTracking'>
[2025-03-29 01:57:47,690] [844] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.inbody.InBody'>
[2025-03-29 01:57:47,691] [844] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>

---

Invalid Model(s) are provided: ['sam_2d']
Following are the available models. You can pass comma (,) separated names to pass multiple
all, deepedit, inbody, tooltracking

[2025-03-29 01:57:47,692] [844] [MainThread] [ERROR] (uvicorn.error:121) - Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 732, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/usr/local/lib/python3.10/dist-packages/monailabel/app.py", line 49, in lifespan
instance = app_instance()
File "/usr/local/lib/python3.10/dist-packages/monailabel/interfaces/utils/app.py", line 50, in app_instance
app = c(app_dir=app_dir, studies=studies, conf=conf)
File "/opt/monailabel/surgvu/endoscopy/main.py", line 73, in init
exit(-1)
File "/usr/lib/python3.10/_sitebuiltins.py", line 26, in call
raise SystemExit(code)
SystemExit: -1

I might be doing it all wrong. I will appreciate guidance to fix it.

Thanks in advance!

[bilalcodehub]

@SachidanandAlle: Have you any idea how to resolve this error and make the SAM work with the endoscopy app?