tensorflow-gpu 2.18.0: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice.

[otavio-silva]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

When running TensorFlow installed from tensorflow-gpu, any sort of operation that would use GPU acceleration gives the error:

W0000 00:00:1739677334.593337    3089 gpu_backend_lib.cc:579] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
  ./cuda_sdk_lib
  python.runfiles/cuda_nvcc
  python/cuda_nvcc

  /usr/local/cuda
  /home/azakun/miniforge3/envs/tf/lib/python3.11/site-packages/tensorflow/python/platform/../../../nvidia/cuda_nvcc
  /home/azakun/miniforge3/envs/tf/lib/python3.11/site-packages/tensorflow/python/platform/../../../../nvidia/cuda_nvcc
  /home/azakun/miniforge3/envs/tf/lib/python3.11/site-packages/tensorflow/python/platform/../../cuda
  .
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
W0000 00:00:1739677334.594935    3089 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739677334.594948    3081 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739677334.594984    3080 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739677334.597365    3089 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
W0000 00:00:1739677334.598724    3081 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
W0000 00:00:1739677334.599706    3080 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
W0000 00:00:1739677334.601815    3081 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
W0000 00:00:1739677334.602845    3080 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
2025-02-16 00:42:14.603342: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-02-16 00:42:14.604857: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-02-16 00:42:14.604912: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/azakun/miniforge3/envs/tf/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/azakun/miniforge3/envs/tf/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 6002, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnknownError: {{function_node __wrapped__Pow_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:Pow] name:

Installed packages

# packages in environment at /home/azakun/miniforge3/envs/tf:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.1.0              pyhd8ed1ab_1    conda-forge
astunparse                1.6.3              pyhd8ed1ab_3    conda-forge
brotli-python             1.1.0           py311hfdbb021_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.4               hb9d3cd8_0    conda-forge
ca-certificates           2025.1.31            hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
certifi                   2025.1.31          pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py311hf29c0ef_0    conda-forge
charset-normalizer        3.4.1              pyhd8ed1ab_0    conda-forge
cuda-crt-tools            12.8.61              ha770c72_1    conda-forge
cuda-cudart               12.8.57              h5888daf_1    conda-forge
cuda-cudart_linux-64      12.8.57              h3f2d84a_1    conda-forge
cuda-cupti                12.8.57              hbd13f7d_0    conda-forge
cuda-nvcc-tools           12.8.61              he02047a_1    conda-forge
cuda-nvrtc                12.8.61              hbd13f7d_0    conda-forge
cuda-nvtx                 12.8.55              hbd13f7d_0    conda-forge
cuda-nvvm-tools           12.8.61              he02047a_1    conda-forge
cuda-version              12.8                 h5d125a7_3    conda-forge
cudnn                     9.7.1.26             h969bcc4_0    conda-forge
flatbuffers               24.12.23             h8f4948b_0    conda-forge
gast                      0.6.0              pyhd8ed1ab_0    conda-forge
giflib                    5.2.2                hd590300_0    conda-forge
google-pasta              0.2.0              pyhd8ed1ab_2    conda-forge
grpcio                    1.67.1          py311h9789449_1    conda-forge
h2                        4.2.0              pyhd8ed1ab_0    conda-forge
h5py                      3.12.1          nompi_py311h5ed33ec_103    conda-forge
hdf5                      1.14.4          nompi_h2d575fe_105    conda-forge
hpack                     4.1.0              pyhd8ed1ab_0    conda-forge
hyperframe                6.1.0              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
importlib-metadata        8.6.1              pyha770c72_0    conda-forge
keras                     3.8.0              pyh753f3f9_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
libabseil                 20240722.0      cxx17_hbbce691_4    conda-forge
libaec                    1.1.3                h59595ed_0    conda-forge
libblas                   3.9.0           28_h59b9bed_openblas    conda-forge
libcblas                  3.9.0           28_he106b2a_openblas    conda-forge
libcublas                 12.8.3.14            h9ab20c4_0    conda-forge
libcufft                  11.3.3.41            hbd13f7d_0    conda-forge
libcurand                 10.3.9.55            hbd13f7d_0    conda-forge
libcurl                   8.12.1               h332b0f4_0    conda-forge
libcusolver               11.7.2.55            h9ab20c4_0    conda-forge
libcusparse               12.5.7.53            hbd13f7d_0    conda-forge
libedit                   3.1.20250104    pl5321h7949ede_0    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.6                h2dba641_0    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran               14.2.0               h69a702a_1    conda-forge
libgfortran5              14.2.0               hd5240d6_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libgrpc                   1.67.1               h25350d4_1    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           28_h7ac8fdf_openblas    conda-forge
liblzma                   5.6.4                hb9d3cd8_0    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjitlink              12.8.61              hbd13f7d_0    conda-forge
libopenblas               0.3.28          pthreads_h94d23a6_1    conda-forge
libpng                    1.6.46               h943b412_0    conda-forge
libprotobuf               5.28.3               h6128344_1    conda-forge
libre2-11                 2024.07.02           hbbce691_2    conda-forge
libsqlite                 3.48.0               hee588c1_1    conda-forge
libssh2                   1.11.1               hf672d98_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
markdown                  3.6                pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_1    conda-forge
markupsafe                3.0.2           py311h2dc5d0c_1    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_1    conda-forge
ml_dtypes                 0.4.0           py311h7db5c69_2    conda-forge
namex                     0.0.8              pyhd8ed1ab_1    conda-forge
nccl                      2.25.1.1             ha44e49d_0    conda-forge
ncurses                   6.5                  h2d0b736_3    conda-forge
numpy                     2.2.3           py311h5d046bc_0    conda-forge
openssl                   3.4.1                h7b32b05_0    conda-forge
opt_einsum                3.4.0              pyhd8ed1ab_1    conda-forge
optree                    0.14.0          py311hd18a35c_1    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
pip                       25.0.1             pyh8b19718_0    conda-forge
protobuf                  5.28.3          py311hfdbb021_0    conda-forge
pycparser                 2.22               pyh29332c3_1    conda-forge
pygments                  2.19.1             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha55dd90_7    conda-forge
python                    3.11.11         h9e4cc4f_1_cpython    conda-forge
python-flatbuffers        25.2.10            pyhbc23db3_0    conda-forge
python_abi                3.11                    5_cp311    conda-forge
re2                       2024.07.02           h9925aae_2    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.32.3             pyhd8ed1ab_1    conda-forge
rich                      13.9.4             pyhd8ed1ab_1    conda-forge
setuptools                75.8.0             pyhff2d567_0    conda-forge
six                       1.17.0             pyhd8ed1ab_0    conda-forge
snappy                    1.2.1                h8bd8927_1    conda-forge
tensorboard               2.18.0             pyhd8ed1ab_1    conda-forge
tensorboard-data-server   0.7.0           py311hafd3f86_2    conda-forge
tensorflow                2.18.0          cuda126py311h673d4f6_200    conda-forge
tensorflow-base           2.18.0          cuda126py311h593f62b_200    conda-forge
tensorflow-estimator      2.18.0          cuda126py311h3eb005d_200    conda-forge
tensorflow-gpu            2.18.0          cuda126py311h418687c_200    conda-forge
termcolor                 2.5.0              pyhd8ed1ab_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
typing-extensions         4.12.2               hd8ed1ab_1    conda-forge
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tzdata                    2025a                h78e105d_0    conda-forge
urllib3                   2.3.0              pyhd8ed1ab_0    conda-forge
werkzeug                  3.1.3              pyhd8ed1ab_1    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
wrapt                     1.17.2          py311h9ecbd09_0    conda-forge
zipp                      3.21.0             pyhd8ed1ab_1    conda-forge
zstandard                 0.23.0          py311hbc35293_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Environment info

mamba version : 1.5.12
     active environment : tf
    active env location : /home/azakun/miniforge3/envs/tf
            shell level : 1
       user config file : /home/azakun/.condarc
 populated config files : /home/azakun/miniforge3/.condarc
                          /home/azakun/.condarc
          conda version : 24.11.2
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.11.2=0
                          __cuda=12.8=0
                          __glibc=2.39=0
                          __linux=5.15.167.4=0
                          __unix=0=0
       base environment : /home/azakun/miniforge3  (writable)
      conda av data dir : /home/azakun/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/azakun/miniforge3/pkgs
                          /home/azakun/.conda/pkgs
       envs directories : /home/azakun/miniforge3/envs
                          /home/azakun/.conda/envs
               platform : linux-64
             user-agent : conda/24.11.2 requests/2.32.3 CPython/3.12.8 Linux/5.15.167.4-microsoft-standard-WSL2 ubuntu/24.04.2 glibc/2.39 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.12
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[h-vetinari]

Do you have CUDA_DIR set to anything? Because I see you have cuda-nvvm-tools in your environment, which contains

[$PREFIX/]nvvm/libdevice/libdevice.10.bc

and your error says

Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.

Can you try what happens when you do export CUDA_DIR=$CONDA_PREFIX? If that doesn't work, please also try downgrading your cuda-version to 12.6, which is what this package has been built with.

[hmaarrfk]

Second suggestion, can you try the suggestions in #296 (comment) for the workaround...

[otavio-silva]

@hmaarrfk @h-vetinari tried your suggestions and none of them worked, unfortunately.
The last TensorFlow version on conda-forge that seems to work for me is 2.16.2. I can install and use it with no issues, but 2.17 and above have this issue.

[hmaarrfk]

I'm sorry, i can't help, i just noticed you are using WSL. I have no experience with it.

[otavio-silva]

I don't think WSL is related to this, I installed TensorFlow 2.17 from Anaconda channel and it works as intended. I could be wrong of course, but I think it maybe something related to how conda-forge is setting environment variables?
I forgot to mention but the CUDA_DIR environment variable was not set to anything.

[hmaarrfk]

I don't think WSL is related

That is OK. I just don't really want to think about anything that touches WSL personally.

I am happy to read and consider any fixes you and others find and integrate them into conda-forge. I just can't help.
If you can recreate on a pure linux install, I'm happy to reconsider.

Other maintainers may have different opinions.

If the Anaconda channel works, you can certainly use their channel.

[otavio-silva]

@hmaarrfk I have made a fresh install of Pop!_OS and was able to reproduce the error. I tried the fixes suggested above, made no difference.
Output of mamba list:

# packages in environment at /home/azakun/miniforge3/envs/tf:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.1.0              pyhd8ed1ab_1    conda-forge
astunparse                1.6.3              pyhd8ed1ab_3    conda-forge
brotli-python             1.1.0           py311hfdbb021_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.4               hb9d3cd8_0    conda-forge
ca-certificates           2025.1.31            hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
certifi                   2025.1.31          pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py311hf29c0ef_0    conda-forge
charset-normalizer        3.4.1              pyhd8ed1ab_0    conda-forge
cuda-crt-tools            12.6.85              ha770c72_0    conda-forge
cuda-cudart               12.6.77              h5888daf_0    conda-forge
cuda-cudart_linux-64      12.6.77              h3f2d84a_0    conda-forge
cuda-cupti                12.6.80              hbd13f7d_0    conda-forge
cuda-nvcc-tools           12.6.85              he02047a_0    conda-forge
cuda-nvrtc                12.6.85              hbd13f7d_0    conda-forge
cuda-nvtx                 12.6.77              hbd13f7d_0    conda-forge
cuda-nvvm-tools           12.6.85              he02047a_0    conda-forge
cuda-version              12.6                 h7480c83_3    conda-forge
cudnn                     9.7.1.26             h969bcc4_0    conda-forge
flatbuffers               24.12.23             h8f4948b_0    conda-forge
gast                      0.6.0              pyhd8ed1ab_0    conda-forge
giflib                    5.2.2                hd590300_0    conda-forge
google-pasta              0.2.0              pyhd8ed1ab_2    conda-forge
grpcio                    1.67.1          py311h9789449_1    conda-forge
h2                        4.2.0              pyhd8ed1ab_0    conda-forge
h5py                      3.12.1          nompi_py311h5ed33ec_103    conda-forge
hdf5                      1.14.4          nompi_h2d575fe_105    conda-forge
hpack                     4.1.0              pyhd8ed1ab_0    conda-forge
hyperframe                6.1.0              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
importlib-metadata        8.6.1              pyha770c72_0    conda-forge
keras                     3.8.0              pyh753f3f9_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
libabseil                 20240722.0      cxx17_hbbce691_4    conda-forge
libaec                    1.1.3                h59595ed_0    conda-forge
libblas                   3.9.0           28_h59b9bed_openblas    conda-forge
libcblas                  3.9.0           28_he106b2a_openblas    conda-forge
libcublas                 12.6.4.1             h5888daf_1    conda-forge
libcufft                  11.3.0.4             hbd13f7d_0    conda-forge
libcurand                 10.3.7.77            hbd13f7d_0    conda-forge
libcurl                   8.12.1               h332b0f4_0    conda-forge
libcusolver               11.7.1.2             h5888daf_1    conda-forge
libcusparse               12.5.4.2             hbd13f7d_0    conda-forge
libedit                   3.1.20250104    pl5321h7949ede_0    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.6                h2dba641_0    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran               14.2.0               h69a702a_1    conda-forge
libgfortran5              14.2.0               hd5240d6_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libgrpc                   1.67.1               h25350d4_1    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           28_h7ac8fdf_openblas    conda-forge
liblzma                   5.6.4                hb9d3cd8_0    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjitlink              12.6.85              hbd13f7d_0    conda-forge
libopenblas               0.3.28          pthreads_h94d23a6_1    conda-forge
libpng                    1.6.46               h943b412_0    conda-forge
libprotobuf               5.28.3               h6128344_1    conda-forge
libre2-11                 2024.07.02           hbbce691_2    conda-forge
libsqlite                 3.48.0               hee588c1_1    conda-forge
libssh2                   1.11.1               hf672d98_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
markdown                  3.6                pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_1    conda-forge
markupsafe                3.0.2           py311h2dc5d0c_1    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_1    conda-forge
ml_dtypes                 0.4.0           py311h7db5c69_2    conda-forge
namex                     0.0.8              pyhd8ed1ab_1    conda-forge
nccl                      2.25.1.1             ha44e49d_0    conda-forge
ncurses                   6.5                  h2d0b736_3    conda-forge
numpy                     2.2.3           py311h5d046bc_0    conda-forge
openssl                   3.4.1                h7b32b05_0    conda-forge
opt_einsum                3.4.0              pyhd8ed1ab_1    conda-forge
optree                    0.14.0          py311hd18a35c_1    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
pip                       25.0.1             pyh8b19718_0    conda-forge
protobuf                  5.28.3          py311hfdbb021_0    conda-forge
pycparser                 2.22               pyh29332c3_1    conda-forge
pygments                  2.19.1             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha55dd90_7    conda-forge
python                    3.11.11         h9e4cc4f_1_cpython    conda-forge
python-flatbuffers        25.2.10            pyhbc23db3_0    conda-forge
python_abi                3.11                    5_cp311    conda-forge
re2                       2024.07.02           h9925aae_2    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.32.3             pyhd8ed1ab_1    conda-forge
rich                      13.9.4             pyhd8ed1ab_1    conda-forge
setuptools                75.8.0             pyhff2d567_0    conda-forge
six                       1.17.0             pyhd8ed1ab_0    conda-forge
snappy                    1.2.1                h8bd8927_1    conda-forge
tensorboard               2.18.0             pyhd8ed1ab_1    conda-forge
tensorboard-data-server   0.7.0           py311hafd3f86_2    conda-forge
tensorflow                2.18.0          cuda126py311h673d4f6_200    conda-forge
tensorflow-base           2.18.0          cuda126py311h593f62b_200    conda-forge
tensorflow-estimator      2.18.0          cuda126py311h3eb005d_200    conda-forge
tensorflow-gpu            2.18.0          cuda126py311h418687c_200    conda-forge
termcolor                 2.5.0              pyhd8ed1ab_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
typing-extensions         4.12.2               hd8ed1ab_1    conda-forge
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tzdata                    2025a                h78e105d_0    conda-forge
urllib3                   2.3.0              pyhd8ed1ab_0    conda-forge
werkzeug                  3.1.3              pyhd8ed1ab_1    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
wrapt                     1.17.2          py311h9ecbd09_0    conda-forge
zipp                      3.21.0             pyhd8ed1ab_1    conda-forge
zstandard                 0.23.0          py311hbc35293_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Output of mamba info:

          mamba version : 1.5.12
     active environment : tf
    active env location : /home/azakun/miniforge3/envs/tf
            shell level : 1
       user config file : /home/azakun/.condarc
 populated config files : /home/azakun/miniforge3/.condarc
                          /home/azakun/.condarc
          conda version : 24.11.3
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.11.3=0
                          __cuda=12.7=0
                          __glibc=2.35=0
                          __linux=6.9.3=0
                          __unix=0=0
       base environment : /home/azakun/miniforge3  (writable)
      conda av data dir : /home/azakun/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/azakun/miniforge3/pkgs
                          /home/azakun/.conda/pkgs
       envs directories : /home/azakun/miniforge3/envs
                          /home/azakun/.conda/envs
               platform : linux-64
             user-agent : conda/24.11.3 requests/2.32.3 CPython/3.12.8 Linux/6.9.3-76060903-generic pop/22.04 glibc/2.35 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.12
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[hmaarrfk]

Thanks!

Just for your reference, "this" changed

I don't even think 12.8 is easy to install ATM so it really helps to have something that can be recreated on a linux server.

Can you provide the "minimum code" since I have CIs that use tensorflow 2.18 and they seem to be fine....

[hmaarrfk]

conda info
[...]

       virtual packages : __archspec=1=zen
                          __conda=24.11.3=0
                          __cuda=12.4=0
                          __glibc=2.35=0
                          __linux=6.8.0=0
                          __unix=0=0

mamba create --name tf "tensorflow=2.18*=cuda*" python=3.11 ipython --channel conda-forge --override-channels
mamba activate tf
ipython

In [1]: import tensorflow as tf
   ...:
   ...: # Ensure TensorFlow is using GPU
   ...: if not tf.config.list_physical_devices('GPU'):
   ...:     raise RuntimeError("No GPU found. Ensure TensorFlow is installed with GPU support.")
   ...:
   ...: # Define two tensors
   ...: a = tf.constant([1.0, 2.0, 3.0, 4.0], dtype=tf.float32)
   ...: b = tf.constant([5.0, 6.0, 7.0, 8.0], dtype=tf.float32)
   ...:
   ...: # Perform addition on the GPU
   ...: with tf.device('/GPU:0'):
   ...:     result = tf.add(a, b)
   ...:
   ...: print("Result:", result.numpy())
2025-02-16 15:06:54.833211: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1739736414.863864  380799 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1739736414.873554  380799 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-16 15:06:54.894738: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
I0000 00:00:1739736417.293538  380799 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 46864 MB memory:  -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:43:00.0, compute capability: 8.6
Result: [ 6.  8. 10. 12.]

what GPU do you have?

[otavio-silva]

Thanks!

Just for your reference, "this" changed

I don't even think 12.8 is easy to install ATM so it really helps to have something that can be recreated on a linux server.

Can you provide the "minimum code" since I have CIs that use tensorflow 2.18 and they seem to be fine....

I think this change is because I was trying the other suggestion to downgrade cuda-version to 12.6, but I made a fresh new environment by just doing:

mamba create -n tf2 python=3.11
mamba activate tf2
mamba install tensorflow-gpu  

And the code I was running in the terminal (with the outputs):

Python 3.11.11 | packaged by conda-forge | (main, Dec  5 2024, 14:17:24) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2025-02-16 17:11:50.504300: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-02-16 17:11:50.511363: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1739736710.520736   11499 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1739736710.523985   11499 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-16 17:11:50.533588: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> m = tf.random.normal((4, 5))
I0000 00:00:1739736720.098686   11499 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14264 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3080 Ti Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
>>> m**2
W0000 00:00:1739736723.863369   11575 gpu_backend_lib.cc:579] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
  ./cuda_sdk_lib
  python.runfiles/cuda_nvcc
  python/cuda_nvcc
  
  /usr/local/cuda
  /home/azakun/miniforge3/envs/tf2/lib/python3.11/site-packages/tensorflow/python/platform/../../../nvidia/cuda_nvcc
  /home/azakun/miniforge3/envs/tf2/lib/python3.11/site-packages/tensorflow/python/platform/../../../../nvidia/cuda_nvcc
  /home/azakun/miniforge3/envs/tf2/lib/python3.11/site-packages/tensorflow/python/platform/../../cuda
  .
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
W0000 00:00:1739736723.864422   11575 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739736723.864557   11582 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739736723.864611   11576 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739736723.864643   11577 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739736723.864679   11579 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739736723.864691   11583 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739736723.864726   11578 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
W0000 00:00:1739736723.864776   11581 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
error: libdevice not found at ./libdevice.10.bc
2025-02-16 17:12:03.865146: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-02-16 17:12:03.866133: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-02-16 17:12:03.866153: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/azakun/miniforge3/envs/tf2/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/azakun/miniforge3/envs/tf2/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 6002, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnknownError: {{function_node __wrapped__Pow_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:Pow] name:

[hmaarrfk]

Ok indeed

python -c "import tensorflow as tf; tf.random.normal((4, 5)) ** 2"

does recreate things on my machine.

mamba install cuda-nvcc

XLA_FLAGS=--xla_gpu_cuda_data_dir=${CONDA_PREFIX}/targets/x86_64-linux \
python -c "import tensorflow as tf; tf.random.normal((4, 5)) ** 2"

fixes things. sorry it is same issue as #296 (comment) but my "fix" was very scattered, I've documented it below.

[hmaarrfk]

If this doesn't fix things on WSL, please let me know so I can close this issue. Otherwise I'm going to close 1 week in favor of #296

[otavio-silva]

Ok indeed

python -c "import tensorflow as tf; tf.random.normal((4, 5)) ** 2"

does recreate things on my machine.

mamba install cuda-nvcc

XLA_FLAGS=--xla_gpu_cuda_data_dir=${CONDA_PREFIX}/targets/x86_64-linux \
python -c "import tensorflow as tf; tf.random.normal((4, 5)) ** 2"

fixes things. sorry it is same issue as #296 (comment) but my "fix" was very scattered, I've documented it below.

Unfortunately, the error persists still.