DynUnet inconsistency

[AAttarpour]

Hello MONAI team,

Based on previous discussion #4851, the problem of DynUnet is solved. In the monai-weekly version 0.10.dev2235, there are dropout layers in the DynUnet model. However, I have another problem with this new model. I use 3D input with the size of 128x128x128. Using the previous DynUnet in the monai-weekly version 0.9.dev2214, I could use a batch size of 24 on our GPU server. Here is how I defined the model:

kernel_sizes = [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]
strides = [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]
filters = [16, 32, 64, 128, 256]
model = DynUNet(
    spatial_dims=3,
    in_channels=1,
    out_channels=2,
    kernel_size=kernel_sizes,
    strides=strides,
    filters=filters,
    upsample_kernel_size=strides[1:],
    res_block=True,
    deep_supervision=True,
    deep_supr_num=2,
    dropout = 0.25,
    norm_name = Norm.BATCH
)

However, with the same inputs, model, and GPU server I cannot use a batch size of more than 3 in this new version. If I increase it to 24 (what I had before), it gives me random errors; errors such as:
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution
or
KeyError: Caught KeyError in DataLoader worker process 1.
This difference in batch size is not logical. Is there something wrong with my virtual env? Or is it sth wrong with the model?
It should be noted that for both of them I use torch version 1.10; updating torch to version 1.12 didn't solve the problem. I would really appreciate it if someone can help me.

[Nic-Ma]

Hi @yiheng-wang-nv ,

Could you please help share some best practices for this question?

Thanks in advance.

[markvvw]

我建议最好是把dynunet重写下或者写个关于deep supervision=True时代码的例子（只是一部分即可）