The NVIDIA driver's CUDA version is 12.2 which is older than the ptxas CUDA version (12.3.107). Because the driver is older than the ptxas version, XLA is disabling parallel compilation, which may slow down compilation. You should update your NVIDIA driver or use the NVIDIA-provided CUDA forward compatibility packages.

[VachanVY]

Hi,
getting this message when importing jax

2024-03-10 21:46:26.964590: W external/xla/xla/service/gpu/nvptx_compiler.cc:698] The NVIDIA driver's CUDA version is 12.2 which is older than the ptxas CUDA version (12.3.107). Because the driver is older than the ptxas version, XLA is disabling parallel compilation, which may slow down compilation. You should update your NVIDIA driver or use the NVIDIA-provided CUDA forward compatibility packages.

what should I do to get rid of this message?

[jakevdp]

Hi - thanks for the question! The error lays out the issue pretty clearly I think: you have CUDA 12.2 installed, and JAX requires CUDA 12.3. To fix this, you should upgrade your CUDA libraries. Can you say more about how you installed JAX?

[kvablack]

Oh, I should also add that this is solvable by installing an NVIDIA forward compatibility package (as the error message suggests). However, I tried this and got some random segfaults many hours into training, which scared me into uninstalling it pretty quick.

[hawkinsp]

We actually have plans to support parallel compilation without the driver. It's just blocked on us migrating across the board to CUDA 12.4 to work around an NVIDIA bug. So hopefully you'll stop seeing this warning in the next few weeks.

[hawkinsp]

Incidentally, usually on cloud you can choose a NVIDIA driver inside your VM freely. At least, that's the way it works on GCP, I can't speak to other vendors.

[kvablack]

Oh that's great news, thanks! Looking forward to it :)

[quanvuong]

friendly ping to see if we have parallel compilation without the driver yet. Thank you!

[xujiboy]

I am getting similar warning on a Amazon Linux 2 machine:

2024-08-07 01:22:52.431656: W external/xla/xla/service/gpu/nvptx_compiler.cc:760] The NVIDIA driver's CUDA version is 12.4 which is older than the ptxas CUDA version (12.6.20). Because the driver is older than the ptxas version, XLA is disabling parallel compilation, which may slow down compilation. You should update your NVIDIA driver or use the NVIDIA-provided CUDA forward compatibility packages.

But when I check ptxas --version I got the following:

ptxas: NVIDIA (R) Ptx optimizing assembler
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:14:54_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

which indicates that ptxas is also on CUDA 12.4. This conflicts with the warning "The NVIDIA driver's CUDA version is 12.4 which is older than the ptxas CUDA version (12.6.20)". Am I missing something?

[heleidsn]

same issue
The NVIDIA driver's CUDA version is 12.4 which is older than the ptxas CUDA version (12.5.40).

[MalAstronomy]

same issue.
The NVIDIA driver's CUDA version is 12.4 which is older than the ptxas CUDA version (12.6.68)
But ptxas --version is,

ptxas: NVIDIA (R) Ptx optimizing assembler
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:15:50_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0

[claireyyang]

I ran into pretty much the same warning (NVIDIA driver's CUDA version is 12.4 which is older than the ptxas CUDA version (12.6.85) and was able to resolve it by running pip install --upgrade "jax[cuda12]", since I installed jax via pip and am not tied to a particular jax version.

It seems like all it did was upgrade some of the pip dependencies, since it printed out Successfully installed jax-0.4.35 jax-cuda12-pjrt-0.4.35 jax-cuda12-plugin-0.4.35 jaxlib-0.4.34 nvidia-cudnn-cu12-9.5.1.17. The warning isn't showing up anymore at least anymore, so maybe this may help someone else who also runs across this issue.