Embrace or Drop Cudnn

[MahmoudAshraf97]

Hi,
Cudnn is only used for Conv operations, it doesn't make sense to include it as a dependency with all the conflicts it causes with pytorch just for the conv operation, we either use it in more operations especially attention since it provides good kernels or write a conv kernel and drop it entirely

[jordimas]

My proposal is to remove the cuDNN dependency completely:

• Conv1D can be implemented directly in CUDA with some performance loss, but it is barely used in transformers and appears mainly in Whisper, where it is not a dominant operation.
• Although other kernels (softmax, layer norm, activations) are implemented with cuDNN, we already have efficient CUDA implementations for them with negligible performance difference.
• This change would significantly reduce dependency issues, which are common and affect many users across different CUDA/cuDNN version combinations.

I am already working on a PR that includes a new CUDA-native Conv1D implementation.

If you think differently please let me know

[MahmoudAshraf97]

Tbh, I'm not that expert when it comes to CT2 internals, but before removing cudnn, I'd suggest using it for attention kernels first since this is the bulk of the computation and see if it's better than the current implementation, if not then we remove it and use cublas/cutlass directly

[jordimas]

That's a good point. Digging into the attention kernels is going to take a significant amount of time for me. I'm going then to keep the cuDNN support. Disable it when compiling by default with -DWITH_CUDNN=OFF and providing a Conv1D implementation (which I already have) which does not uses cuDNN. If, in the future, we want to explore using cuDNN attention kernels or add more functionality, we can revisit it.

[jordimas]

Hello. I did #1949 I will appreciate if you can test it. It will be an important change for faster-whisper consumers. Thanks

[MahmoudAshraf97]

Thanks, I'm OOO until next sunday and I don't have my PC will test it when I'm back

[winstxnhdw]

Hey @MahmoudAshraf97

it doesn't make sense to include it as a dependency with all the conflicts it causes with pytorch just for the conv operation

I fail to see where the conflicts with PyTorch is occurring. PyTorch is only installed for the converter. OTOH, I've never had any of these issues you mentioned with PyTorch and CTranslate2.

[MahmoudAshraf97]

Pytorch usually uses cudnn, not necessarily with CT2, but most users need both in the same environment, if you don't do this then you won't face any issues

[winstxnhdw]

I’ve used both in the same environment with no issues. At most, I had to manually add the specific cuDNN wheel and populate LD_LIBRARY_PATH.