An unexpected error happened during the inference of the pretrained segmentation model #5058

Ci-Yelson · 2022-08-04T03:08:03Z

Ci-Yelson
Aug 4, 2022

Describe the bug
Hi, when I use the monai label example app with the pretrained segmentation model for multi-label, I face this problem:

2022-08-03 11:03:33.426 | INFO     | monai.transforms.utils:get_largest_connected_component_mask:961 - testtest [] testtest []
[2022-08-03 11:03:33,428] [858] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/conda/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/fastapi/applications.py", line 212, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
    File "/code/monailabel/endpoints/infer.py", line 177, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/code/monailabel/endpoints/infer.py", line 160, in run_inference
    result = instance.infer(request)
  File "/code/monailabel/interfaces/app.py", line 260, in infer
    result_file_name, result_json = task(request)
  File "/code/monailabel/interfaces/tasks/infer.py", line 299, in __call__
    data = self.run_post_transforms(data, self.post_transforms(data))
  File "/code/monailabel/interfaces/tasks/infer.py", line 360, in run_post_transforms
    return run_transforms(data, transforms, log_prefix="POST")
  File "/code/monailabel/interfaces/utils/transform.py", line 93, in run_transforms
    data = t(data)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 258, in __call__
    d[key] = self.converter(d[key])
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/array.py", line 351, in __call__
    mask = get_largest_connected_component_mask(foreground, self.connectivity)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utils.py", line 962, in get_largest_connected_component_mask
    comp = x_label == vals[cp.ndarray.argmax(counts)]
  File "cupy/_core/core.pyx", line 863, in cupy._core.core.ndarray.argmax
  File "cupy/_core/core.pyx", line 880, in cupy._core.core.ndarray.argmax
  File "cupy/_core/_routines_statistics.pyx", line 105, in cupy._core._routines_statistics._ndarray_argmax
  File "cupy/_core/_reduction.pyx", line 564, in cupy._core._reduction._SimpleReductionKernel.__call__
  File "cupy/_core/_reduction.pyx", line 331, in cupy._core._reduction._AbstractReductionKernel._call
ValueError: zero-size array to reduction operation cupy_argmax which has no identity

It seens the counts is a null list [ ] and for only one dicom image, the error happened. But i dont know why this could happen and the image looks like fine,. Can anyone help?

Nic-Ma · 2022-08-04T03:18:27Z

Nic-Ma
Aug 4, 2022
Maintainer

Hi @SachidanandAlle ,

Have you seen a similar issue in the MONAI Label app?

Thanks in advance.

0 replies

SachidanandAlle · 2022-08-04T05:56:57Z

SachidanandAlle
Aug 4, 2022
Collaborator

Can you try other sample images.. if possible some .nii.gz ones..

Before that please share more logs/lines above.. we print image size, type for every transforms etc..

lets try to understand if its coming due what..
Looks like inference didn't result a single positive value.. that can be checked through some kind of unit test

0 replies

Ci-Yelson · 2022-08-04T06:58:35Z

Ci-Yelson
Aug 4, 2022
Author

Can you try other sample images.. if possible some .nii.gz ones..

Before that please share more logs/lines above.. we print image size, type for every transforms etc..

lets try to understand if its coming due what.. Looks like inference didn't result a single positive value.. that can be checked through some kind of unit test

Thanks for your reply, and this is the logs with more details. And in fact, this error just occured for one dcm image, and others are fine, and for now I cant get more sample images, so I am sorry for that I cant give you more infomation.

[2022-08-04 06:42:55,129] [95] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:274) - Infer Request (final): {'largest_cc': True, 'model': 'segmentation', 'image': '/tmp/tmpz71cfzjs.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint16', 'result_compress': False, 'description': 'A pre-trained model for volumetric (3D) segmentation over 3D Images', 'device': 'cuda'}
[2022-08-04 06:42:55,137] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - PRE - Run Transform(s)
[2022-08-04 06:42:55,137] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Input Keys: ['largest_cc', 'model', 'image', 'result_extension', 'result_dtype', 'result_compress', 'description', 'device', 'image_path']
[2022-08-04 06:42:55,789] [95] [Thread-2] [INFO] (monailabel.datastore.local:538) - Invalidate count: 0
[2022-08-04 06:42:55,820] [95] [Thread-2] [INFO] (monailabel.datastore.local:538) - Invalidate count: 0
[2022-08-04 06:42:57,637] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (LoadImaged): Time: 2.4988; image: (512, 512, 119)(float32)
[2022-08-04 06:42:57,638] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (AddChanneld): Time: 0.0000; image: (1, 512, 512, 119)(float32)
[2022-08-04 06:43:01,064] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (Spacingd): Time: 3.4255; image: (1, 360, 360, 296)(float32)
[2022-08-04 06:43:01,345] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (ScaleIntensityRanged): Time: 0.2805; image: (1, 360, 360, 296)(float32)
[2022-08-04 06:43:01,345] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (EnsureTyped): Time: 0.0002; image: torch.Size([1, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:01,345] [95] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:454) - Inferer:: SlidingWindowInferer => {'roi_size': (160, 160, 160), 'sw_batch_size': 1, 'overlap': 0.25, 'mode': <BlendMode.CONSTANT: 'constant'>, 'sigma_scale': 0.125, 'padding_mode': <PytorchPadMode.CONSTANT: 'constant'>, 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'roi_weight_map': None}
[2022-08-04 06:43:01,348] [95] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:364) - Infer model path: /code/apps/radiology/model/pretrained_segmentation.pt
[2022-08-04 06:43:05,049] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - POST - Run Transform(s)
[2022-08-04 06:43:05,049] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - POST - Input Keys: ['largest_cc', 'model', 'image', 'result_extension', 'result_dtype', 'result_compress', 'description', 'device', 'image_path', 'image_meta_dict', 'image_transforms', 'pred']
[2022-08-04 06:43:05,049] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - POST - Transform (EnsureTyped): Time: 0.0001; image: torch.Size([1, 360, 360, 296])(torch.float32); pred: torch.Size([14, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:05,053] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - POST - Transform (Activationsd): Time: 0.0030; image: torch.Size([1, 360, 360, 296])(torch.float32); pred: torch.Size([14, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:05,120] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - POST - Transform (AsDiscreted): Time: 0.0674; image: torch.Size([1, 360, 360, 296])(torch.float32); pred: torch.Size([1, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:11,116] [95] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/conda/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/fastapi/applications.py", line 212, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/code/monailabel/endpoints/infer.py", line 177, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/code/monailabel/endpoints/infer.py", line 160, in run_inference
    result = instance.infer(request)
  File "/code/monailabel/interfaces/app.py", line 260, in infer
    result_file_name, result_json = task(request)
  File "/code/monailabel/interfaces/tasks/infer.py", line 299, in __call__
    data = self.run_post_transforms(data, self.post_transforms(data))
  File "/code/monailabel/interfaces/tasks/infer.py", line 360, in run_post_transforms
    return run_transforms(data, transforms, log_prefix="POST")
  File "/code/monailabel/interfaces/utils/transform.py", line 92, in run_transforms
    data = t(data)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 258, in __call__
    d[key] = self.converter(d[key])
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/array.py", line 351, in __call__
    mask = get_largest_connected_component_mask(foreground, self.connectivity)
  File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utils.py", line 959, in get_largest_connected_component_mask
    comp = x_label == vals[cp.ndarray.argmax(counts)]
  File "cupy/_core/core.pyx", line 941, in cupy._core.core.ndarray.argmax
  File "cupy/_core/core.pyx", line 958, in cupy._core.core.ndarray.argmax
  File "cupy/_core/_routines_statistics.pyx", line 105, in cupy._core._routines_statistics._ndarray_argmax
  File "cupy/_core/_reduction.pyx", line 567, in cupy._core._reduction._SimpleReductionKernel.__call__
  File "cupy/_core/_reduction.pyx", line 332, in cupy._core._reduction._AbstractReductionKernel._call
ValueError: zero-size array to reduction operation cupy_argmax which has no identity

In addition, if I mannually set comp = false, it seens it worked fine.

0 replies

SachidanandAlle · 2022-08-08T13:59:27Z

SachidanandAlle
Aug 8, 2022
Collaborator

And for thus image you can run none of deepedit or segmentation or segmentation spleen. model..
True?

0 replies

Ci-Yelson · 2022-08-08T14:33:35Z

Ci-Yelson
Aug 8, 2022
Author

And for thus image you can run none of deepedit or segmentation or segmentation spleen. model.. True?

This error just happened when use the pretrained segmentation model for multi-label. Other single label models worked fine.

0 replies

SachidanandAlle · 2022-08-09T07:41:51Z

SachidanandAlle
Aug 9, 2022
Collaborator

Will it be OK if you can share the image temporarily? This will help to debug and fix the issue quickly

0 replies

Ci-Yelson · 2022-08-09T07:56:14Z

Ci-Yelson
Aug 9, 2022
Author

Will it be OK if you can share the image temporarily? This will help to debug and fix the issue quickly

I‘m sorry I cant share the image. But I noticed that this image was upside down compared to other images, that the pelvis was on top. Hope this will help you.

0 replies

SachidanandAlle · 2022-08-09T16:17:45Z

SachidanandAlle
Aug 9, 2022
Collaborator

Them my only guess is it has not generated any non zero results after inference..
There is config to disable largest cc.. you can enable it in 3d slicer under edit/settings/monailabel (check developer mode) and it show those config options for train/infer etc..

0 replies

Ci-Yelson · 2022-08-10T06:05:40Z

Ci-Yelson
Aug 10, 2022
Author

Them my only guess is it has not generated any non zero results after inference.. There is config to disable largest cc.. you can enable it in 3d slicer under edit/settings/monailabel (check developer mode) and it show those config options for train/infer etc..

I used the OHIF instead of 3d slicer. And my MONAIlabel version is MONAI Label 0.3.2，monai version is as below:

monai-weekly[nibabel, skimage, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, lmdb, psutil, openslide]==0.10.dev2227

0 replies

SachidanandAlle · 2022-08-10T06:11:24Z

SachidanandAlle
Aug 10, 2022
Collaborator

The version is bit old.. and few things are improved in 4.2 version..

I suggest to uninstall existing monai and monailabel and try latest version of monailabel from pip

0 replies

Ci-Yelson · 2022-08-10T06:16:35Z

Ci-Yelson
Aug 10, 2022
Author

The version is bit old.. and few things are improved in 4.2 version..

I suggest to uninstall existing monai and monailabel and try latest version of monailabel from pip

OK. I will try the latest version, thank you very much.

0 replies

jlvahldiek · 2022-08-30T19:21:37Z

jlvahldiek
Aug 30, 2022

Them my only guess is it has not generated any non zero results after inference.. There is config to disable largest cc.. you can enable it in 3d slicer under edit/settings/monailabel (check developer mode) and it show those config options for train/infer etc..

Discovered the same issue (ValueError: zero-size array to reduction operation cupy_argmax which has no identity) with version 0..4.2 and 3D Slicer. Disabling largest cc as discribed by @SachidanandAlle resolved the problem and segmentation is displayed.

0 replies

An unexpected error happened during the inference of the pretrained segmentation model #5058

Uh oh!

Ci-Yelson Aug 4, 2022

Replies: 12 comments

Uh oh!

Nic-Ma Aug 4, 2022 Maintainer

Uh oh!

SachidanandAlle Aug 4, 2022 Collaborator

Uh oh!

Uh oh!

Ci-Yelson Aug 4, 2022 Author

Uh oh!

Uh oh!

SachidanandAlle Aug 8, 2022 Collaborator

Uh oh!

Ci-Yelson Aug 8, 2022 Author

Uh oh!

SachidanandAlle Aug 9, 2022 Collaborator

Uh oh!

Ci-Yelson Aug 9, 2022 Author

Uh oh!

SachidanandAlle Aug 9, 2022 Collaborator

Uh oh!

Ci-Yelson Aug 10, 2022 Author

Uh oh!

SachidanandAlle Aug 10, 2022 Collaborator

Uh oh!

Ci-Yelson Aug 10, 2022 Author

Uh oh!

jlvahldiek Aug 30, 2022

Ci-Yelson
Aug 4, 2022

Nic-Ma
Aug 4, 2022
Maintainer

SachidanandAlle
Aug 4, 2022
Collaborator

Ci-Yelson
Aug 4, 2022
Author

SachidanandAlle
Aug 8, 2022
Collaborator

Ci-Yelson
Aug 8, 2022
Author

SachidanandAlle
Aug 9, 2022
Collaborator

Ci-Yelson
Aug 9, 2022
Author

SachidanandAlle
Aug 9, 2022
Collaborator

Ci-Yelson
Aug 10, 2022
Author

SachidanandAlle
Aug 10, 2022
Collaborator

Ci-Yelson
Aug 10, 2022
Author

jlvahldiek
Aug 30, 2022