CUDNN_STATUS_MAPPING_ERROR after GPU upgrade

[KSchmidtACR]

Your question
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Hello,

I had been successfully running the BraTS-toolkit segmentor on a Tesla V-100 GPU via a p3.2xlarge AWS EC2 instance. AWS will not be supporting the p3.2xlarge after the end of this year, so I've been trying to get the segmentor to work using an L4 GPU via a g6.4xlarge EC2. It has been failing with a CUDNN_STATUS_MAPPING_ERROR.

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=8 : invalid device function /usr/local/lib/python3.6/dist-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images. warn("Anti-aliasing will be enabled by default in skimage 0.15 to " Traceback (most recent call last): File "predict_patient.py", line 998, in <module> None, None, trainer.patch_size, True) File "predict_patient.py", line 714, in predict_preprocessed_data_return_softmax return self.network.predict_3D(data, do_mirroring, num_repeats, use_train_mode, batch_size, mirror_axes, tiled, tile_in_z, step, min_size, use_gaussian=use_gaussian)[2] File "predict_patient.py", line 185, in predict_3D mirror_axes, regions_class_order, pad_border_mode, pad_kwargs=pad_kwargs) File "predict_patient.py", line 283, in _internal_predict_3D_3Dconv stacked = self._internal_maybe_mirror_and_pred_3D(data, num_repeats, mirror_axes, do_mirroring) File "predict_patient.py", line 244, in _internal_maybe_mirror_and_pred_3D p = self.inference_apply_nonlin(self(a)) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "predict_patient.py", line 589, in forward x = self.conv_blocks_context[d](x) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "predict_patient.py", line 364, in forward return self.blocks(x) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py", line 91, in forward input = module(input) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "predict_patient.py", line 76, in forward x = self.conv(x) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 421, in forward self.padding, self.dilation, self.groups) RuntimeError: CuDNN error: CUDNN_STATUS_MAPPING_ERROR

I noticed there was another CUDNN_STATUS_MAPPING_ERROR issue from a few months ago, and the conclusion was that the GPU was too new. Is that the case here as well? IThanks!

[neuronflow]

@KSchmidtACR, I apologize for overlooking your issue. Each algorithm has different GPU requirements. It is a good idea to experiment with different algorithms.

Also, I advise you to have a look at BraTS orchestrato,r offering newer algorithms: https://github.com/BrainLesion/BraTS