Recently, I found when I used the new cuda toolkit in Dockerfile like
FROM nvidia/cuda:12.3.1-base-ubuntu{20|22}.04
This will unaligned the host's cuda toolkit. Regarded to the nvidia-compute engine, in host's cuda-12.3.1, it will use official
libnvidia-compute-545
However, in docker container, the ppa:graphics-drivers will bring in
libnvidia-compute-550
And this will break the opencl detection inside the container and it will complain the mismatch of NVLM driver. Hence, I already tried to remove the ppa repository to prevent the usage of too-new compute engine whiel container is building.
Thanks for serious consideration about such incompatibility since it is alinged to recent nVidia's official cuda release.