pathology DSA plugin connection error

[hw-ju]

start server:

/monailabel/scripts/monailabel start_server --app sample-apps/pathology --studies http://0.0.0.0:8080/api/v1

server seems to be running ok according to its output:

[2022-07-24 11:24:55,000] [260077] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

try to use MONAI Label Annotation, and it always fails, but Detects Nuclei can run:

click on the failed job, its log output shows Connection timed out

 [2022-07-24 18:25:58,716] INFO: Running container: image: projectmonai/monailabel-dsa:latest args: ['MONAILabelAnnotation', '--analysis_level', '0', '--analysis_roi', '9816, 12977, 6732, 6295', '--analysis_tile_size', '1024', '--api-url', 'http://girder:8080/api/v1/', '--extra_params', '{}', '--girder-token', '3xVadzm23oN63KjOAH3p8bhk0YDBD1CNk9feoQQItIA6X49mNtGghNDZ87EGtW5o', '--min_fgnd_frac', '-1', '--min_poly_area', '80', '--model_name', 'deepedit_nuclei', '--server', 'http://192.168.1.37:8000', '/mnt/girder_worker/e263e6dd2c0d45568c1b749732cdc23d/CMU-1.svs', '/mnt/girder_worker/e263e6dd2c0d45568c1b749732cdc23d/MONAILabel Annotations-outputAnnotationFile.anot'] runtime: None kwargs: {'tty': False, 'volumes': {'/tmp/tmpfxe5u43p': {'bind': '/mnt/girder_worker/e263e6dd2c0d45568c1b749732cdc23d', 'mode': 'rw'}}, 'detach': True, 'network': 'container:b7902399502e4062a1e80e5a2e5f8f9fd2e158fbf95a640d8cefa22567caa3a5'}
INFO:root:CLI Parameters ...

INFO:root:USING:: inputImageFile = /mnt/girder_worker/e263e6dd2c0d45568c1b749732cdc23d/CMU-1.svs
INFO:root:USING:: outputAnnotationFile = /mnt/girder_worker/e263e6dd2c0d45568c1b749732cdc23d/MONAILabel Annotations-outputAnnotationFile.anot
INFO:root:USING:: analysis_level = 0
INFO:root:USING:: analysis_roi = [9816.0, 12977.0, 6732.0, 6295.0]
INFO:root:USING:: analysis_tile_size = 1024.0
INFO:root:USING:: girderApiUrl = http://girder:8080/api/v1/
INFO:root:USING:: extra_params = {}
INFO:root:USING:: girderToken = 3xVadzm23oN63KjOAH3p8bhk0YDBD1CNk9feoQQItIA6X49mNtGghNDZ87EGtW5o
INFO:root:USING:: min_fgnd_frac = -1.0
INFO:root:USING:: min_poly_area = 80.0
INFO:root:USING:: model_name = deepedit_nuclei
INFO:root:USING:: server = http://192.168.1.37:8000
INFO:root:>> Reading input image ... 

INFO:root:Run MONAILabel Task... and collect the annotations: [9816.0, 12977.0] => [6732.0, 6295.0]
INFO:root:For Server Logs click/open:  http://192.168.1.37:8000/logs/?refresh=3
Traceback (most recent call last):
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 174, in <module>
    main(CLIArgumentParser().parse_args())
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 168, in main
    fetch_annotations(args, tiles)
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 55, in fetch_annotations
    _, res = client.wsi_infer(model=args.model_name, image_in=image, body=body, output=output)
  File "/opt/monailabel/dsa/cli/client.py", line 186, in wsi_infer
    status, form, _ = MONAILabelUtils.http_method("POST", self._server_url, selector, body)
  File "/opt/monailabel/dsa/cli/client.py", line 282, in http_method
    conn.request(method, selector, body=body, headers=headers)
  File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/usr/local/lib/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
DockerException: Non-zero exit code from docker container (1).
  File "/opt/venv/lib/python3.9/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/slicer_cli_web/slicer_cli_web/girder_worker_plugin/direct_docker_run.py", line 87, in __call__
    super().__call__(*args, **kwargs)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 308, in __call__
    super(DockerTask, self).__call__(*args, **kwargs)
  File "/opt/girder_worker/girder_worker/task.py", line 154, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/opt/venv/lib/python3.9/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/slicer_cli_web/slicer_cli_web/girder_worker_plugin/direct_docker_run.py", line 123, in run
    return _docker_run(task, **kwargs)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 371, in _docker_run
    _run_select_loop(task, container, read_streams, write_streams)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 222, in _run_select_loop
    raise DockerException('Non-zero exit code from docker container (%d).' % exit_code)

I tried building dsa plugin docker image on my own, and tried shutting down and restarting server and DSA, but no luck. Any suggestions?

I also tried http://0.0.0.0:8000/ for MONAI Label Address, but get ConnectionRefusedError:

[2022-07-24 18:49:47,524] INFO: Running container: image: projectmonai/monailabel-dsa:latest args: ['MONAILabelAnnotation', '--analysis_level', '0', '--analysis_roi', '-1, -1, -1, -1', '--analysis_tile_size', '1024', '--api-url', 'http://girder:8080/api/v1/', '--extra_params', '{}', '--girder-token', '3VlD3w7uR8oxTaz9egKFY4hKr1DP7ZNAqHeioAQdFoOaX229GEez6f3zT4fyc9JG', '--min_fgnd_frac', '-1', '--min_poly_area', '80', '--model_name', 'deepedit_nuclei', '--server', 'http://0.0.0.0:8000/', '/mnt/girder_worker/8e686e264f4c478bb483a9213ee7626d/CMU-2.svs', '/mnt/girder_worker/8e686e264f4c478bb483a9213ee7626d/MONAILabel Annotations-outputAnnotationFile.anot'] runtime: None kwargs: {'tty': False, 'volumes': {'/tmp/tmpp_8hkx4m': {'bind': '/mnt/girder_worker/8e686e264f4c478bb483a9213ee7626d', 'mode': 'rw'}}, 'detach': True, 'network': 'container:7a9678326254160db42b7204298a82e8039924bf37614cdbd355deacf3eb9a8f'}
INFO:root:CLI Parameters ...

INFO:root:USING:: inputImageFile = /mnt/girder_worker/8e686e264f4c478bb483a9213ee7626d/CMU-2.svs
INFO:root:USING:: outputAnnotationFile = /mnt/girder_worker/8e686e264f4c478bb483a9213ee7626d/MONAILabel Annotations-outputAnnotationFile.anot
INFO:root:USING:: analysis_level = 0
INFO:root:USING:: analysis_roi = [-1.0, -1.0, -1.0, -1.0]
INFO:root:USING:: analysis_tile_size = 1024.0
INFO:root:USING:: girderApiUrl = http://girder:8080/api/v1/
INFO:root:USING:: extra_params = {}
INFO:root:USING:: girderToken = 3VlD3w7uR8oxTaz9egKFY4hKr1DP7ZNAqHeioAQdFoOaX229GEez6f3zT4fyc9JG
INFO:root:USING:: min_fgnd_frac = -1.0
INFO:root:USING:: min_poly_area = 80.0
INFO:root:USING:: model_name = deepedit_nuclei
INFO:root:USING:: server = http://0.0.0.0:8000/
INFO:root:>> Reading input image ... 

INFO:root:Run MONAILabel Task... and collect the annotations: [-1.0, -1.0] => [-1.0, -1.0]
INFO:root:For Server Logs click/open:  http://0.0.0.0:8000/logs/?refresh=3
Traceback (most recent call last):
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 174, in <module>
    main(CLIArgumentParser().parse_args())
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 168, in main
    fetch_annotations(args, tiles)
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 55, in fetch_annotations
    _, res = client.wsi_infer(model=args.model_name, image_in=image, body=body, output=output)
  File "/opt/monailabel/dsa/cli/client.py", line 186, in wsi_infer
    status, form, _ = MONAILabelUtils.http_method("POST", self._server_url, selector, body)
  File "/opt/monailabel/dsa/cli/client.py", line 282, in http_method
    conn.request(method, selector, body=body, headers=headers)
  File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/usr/local/lib/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
DockerException: Non-zero exit code from docker container (1).
  File "/opt/venv/lib/python3.9/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/slicer_cli_web/slicer_cli_web/girder_worker_plugin/direct_docker_run.py", line 87, in __call__
    super().__call__(*args, **kwargs)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 308, in __call__
    super(DockerTask, self).__call__(*args, **kwargs)
  File "/opt/girder_worker/girder_worker/task.py", line 154, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/opt/venv/lib/python3.9/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/slicer_cli_web/slicer_cli_web/girder_worker_plugin/direct_docker_run.py", line 123, in run
    return _docker_run(task, **kwargs)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 371, in _docker_run
    _run_select_loop(task, container, read_streams, write_streams)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 222, in _run_select_loop
    raise DockerException('Non-zero exit code from docker container (%d).' % exit_code)

[SachidanandAlle]

Where you able to access the same IP:port from your browser.. monailabel will show the API page on the browser..

[diazandr3s]

Hi @hw-ju,

Thanks for opening this discussion.

Are you running the MONAI Label server on the same PC? As @SachidanandAlle suggested, please paste the internal IP address.
In this video, you can see how you can do that https://drive.google.com/file/d/19j7CMx7c8nn96ZL75L1a26usw8pHW-Sj/view?usp=sharing ... Specifically from minute 1:50

Here is shown how you could cancel previous jobs in DSA: #873 (comment)

I hope this helps,

[hw-ju]

@diazandr3s @SachidanandAlle
Thanks a lot for your help! The video is super helpful!
After following the video, during a failed run using MONAI Label annotation within DSA, I found the following error in the MONAI Label server side output:

NVIDIA RTX A6000 with CUDA capability sm_86 is not compatible with the current PyTorch installation. 
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70. 
If you want to use the NVIDIA RTX A6000 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

I fixed that by creating a new conda env with appropriate pytorch version installed using command suggested by https://pytorch.org/get-started/locally/. Below are the commands I used if anyone run into same problem:

conda create -n monailabel-env python=3.9 

conda activate monailabel-env 

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116 

pip3 install monailabel

So happy it can work now :)