RuntimeError: quantile() input tensor is too large

[AAttarpour]

Hello MONAI team,

I hope you are doing great!

I want to report two problems I have confronted recently with MONAI; I would appreciate it if someone can help me.

I have been using monai-weekly==0.9.dev2214 and DynUnet for 3D microscopy data; data patches have the size of 512x512x512. The training was fine and I wanted to do MC dropout. Implementing this in PyTorch/MONAI is easy; I need to set the model's dropout layers to train at test time. However, I got a problem; when I printed the model's architecture, there were no dropout layers in the model even though I turned it on during training. This is how I defined the model:

kernel_sizes = [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]
strides = [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]
filters = [16, 32, 64, 128, 256]
model = DynUNet(
    spatial_dims=3,
    in_channels=1,
    out_channels=2,
    kernel_size=kernel_sizes,
    strides=strides,
    filters=filters,
    upsample_kernel_size=strides[1:],
    res_block=True,
    deep_supervision=True,
    deep_supr_num=2,
    dropout = 0.25,
    norm_name = Norm.BATCH
)

I uninstalled the monai from my virtualenv and installed the latest version; I defined the same DynUnet model, loaded the trained model (I had with monai-weekly==0.9.dev2214), and printed the model architecture again; I realized that there are actually dropout layers; so, I assume there is sth related to the MONAI version. However, I am confronted with another problem with this new MONAI (version 0.10).
I use ScaleIntensityRangePercentilesd transform. In the new MONAI, it gives me this error from torch when it wants to use this transform:
RuntimeError: quantile() input tensor is too large I firstly thought sth went wrong with my virtualenv when I uninstalled and installed MONAI again, but I tested in google colab and it gave me the same error. Here is how I tested it:

!pip install -q monai-weekly
from monai.transforms import ScaleIntensityRangePercentilesd
from monai.networks.nets import DynUnet
import numpy as np
from monai.config import print_config

print_config()
MONAI version: 0.10.dev2232
Numpy version: 1.21.6
Pytorch version: 1.12.0+cu113
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: bedd7dde553c107e5b8f633b830e3e7e0a70b5e3
MONAI __file__: /usr/local/lib/python3.7/dist-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 3.0.2
scikit-image version: 0.18.3
Pillow version: 7.1.2
Tensorboard version: 2.8.0
gdown version: 4.4.0
TorchVision version: 0.13.0+cu113
tqdm version: 4.64.0
lmdb version: 0.99
psutil version: 5.4.8
pandas version: 1.3.5
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.

img = np.ones((512,512,512), dtype='float32')
print(isinstance(img, np.ndarray))
img[100:200, 100:200, 100:200] = 10
seg = np.ones((512,512,512), dtype='float32')
transform = ScaleIntensityRangePercentilesd(keys=["image"], lower=0.05,
                                            upper=99.95, b_min=0, b_max=1,
                                            clip=True, relative=False)
img_dict = {'image': img, 'label': seg}
img_dict_transformed = transform(img_dict)

RuntimeError                              Traceback (most recent call last)
[<ipython-input-4-e0f9f6c40a55>](https://localhost:8080/#) in <module>()
      7                                             clip=True, relative=False)
      8 img_dict = {'image': img, 'label': seg}
----> 9 img_dict_transformed = transform(img_dict)

3 frames
[/usr/local/lib/python3.7/dist-packages/monai/transforms/utils_pytorch_numpy_unification.py](https://localhost:8080/#) in percentile(x, q, dim, keepdim, **kwargs)
    109     else:
    110         q = convert_to_dst_type(q / 100.0, x)[0]
--> 111         result = torch.quantile(x, q, dim=dim, keepdim=keepdim)
    112     return result
    113 

RuntimeError: quantile() input tensor is too large

This isn't a problem with the previous MONAI; in the google colab, I installed monai-weekly==0.9.dev2214 again and there was no error.

Thank you.

[Nic-Ma]

Hi @rijobro ,

About the quantile() input tensor is too large issue here, have you seen it when you develop the numpy & torch APIs unification?

Thanks in advance.

[yiheng-wang-nv]

Hi @AAttarpour , for the dropout issue, it was wrong in old MONAI version and had been fixed. Therefore, in the latest version it exists.

[AAttarpour]

Hi @yiheng-wang-nv,
Thanks a lot for your answer! Would you mind telling me that it was a printing problem only in the previous monai or does that mean there were no dropout layers in the DynUnet? In other words, should I repeat training the model with new MONAI? thanks again!

[yiheng-wang-nv]

There was no dropout layer for the original DynUnet, and later an arg to add dropout layers were wrongly added (only added the arg, but layers were not added due to some bug), and now this issue has been fixed. You can also refer to:
#4590

[AAttarpour]

Great! thanks a lot for solving the bug :)

[wyli]

There's a known issue for torch.quantile pytorch/pytorch#64947

[rijobro]

Could we use a try/catch statement and fall back to numpy?

[Nic-Ma]

@rijobro Sounds good to me, and maybe same solution for: #4765 (comment)

Thanks.

[AAttarpour]

Thanks a lot @rijobro @Nic-Ma! I know there is a bug related to this in pytorch; however, I was looking to utils_pytorch_numpy_unification.py; in this script in the percentile function we can find these:

if isinstance(x, np.ndarray):
        result = np.percentile(x, q, axis=dim, keepdims=keepdim, **kwargs)
    else:
        q = convert_to_dst_type(q / 100.0, x)[0]
        result = torch.quantile(x, q, dim=dim, keepdim=keepdim)

so basically the function shouldn't use torch since my input images are np.ndarrays and thus isinstance(x, np.ndarray) gives us True

[wyli]

thanks for the discussions, now it's addressed by #4877