cuda wsl undefined symbol: ncclMemFree

[jschoch]

Trying to get some acceleration working, rocm failed and now i'm having issues with CUDA on a 3070, the precompiled didn't work and build_xla also had the same issue. What is a know working combination of nvidia deps for WSL?

(.venv) schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang++ XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.compile xla
==> xla
Compiling 5 files (.ex)
Generated xla app
rm -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        ln -s "/home/schoch/dev/docgen/deps/xla/extension" /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        cd /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20 && \
        bazel build --define "framework_shared_object=false" -c opt   --config=cuda --repo_env=HERMETIC_CUDA_VERSION="12.8.0" --repo_env=HERMETIC_CUDNN_VERSION="9.8.0" --action_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES="sm_50,sm_60,sm_70,sm_80,sm_90,sm_100,compute_120" --action_env=TF_NVCC_CLANG="1" --@local_config_cuda//:cuda_compiler=nvcc --repo_env=CC=clang --repo_env=CXX=clang++ --copt=-Wno-error=unused-command-line-argument --copt=-Wno-gnu-offsetof-extensions --copt=-Qunused-arguments --copt=-Wno-error=c23-extensions //xla/extension:xla_extension && \
        mkdir -p /home/schoch/.cache/xla/0.9.1/build/ && \
        cp -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/bazel-bin/xla/extension/xla_extension.tar.gz /home/schoch/.cache/xla/0.9.1/build/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz
INFO: Reading 'startup' options from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/.bazelrc:
  Inherited 'common' options: --noenable_bzlmod --noincompatible_enable_cc_toolchain_resolution
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  Inherited 'common' options: --announce_rc --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --config=short_logs --@rules_python//python/config_settings:precompile=force_disabled
INFO: Found applicable config definition build:short_logs in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:cuda in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --config=cuda_version --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda_version in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env=HERMETIC_CUDA_VERSION=12.6.3 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --repo_env=HERMETIC_NVSHMEM_VERSION=3.2.5
INFO: Found applicable config definition build:linux in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --experimental_guard_against_concurrent_changes
Computing main repo mapping:
DEBUG: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:73:14: Cannot find clang, either correct your path, or set the CLANG_CUDA_COMPILER_PATH or CC environment variables
INFO: Repository local_config_cuda instantiated at:
  /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/WORKSPACE:87:15: in <toplevel>
Repository rule cuda_configure defined at:
  /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:705:33: in <toplevel>
ERROR: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:127:20: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 684, column 28, in _cuda_configure_impl
                _create_cuda_repository(repository_ctx)
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 668, column 38, in _create_cuda_repository
                _create_local_cuda_repository(repository_ctx)
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 611, column 44, in _create_local_cuda_repository
                _create_cuda_ptx_copts_list(repository_ctx, cuda_config.cuda_version) +
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 297, column 21, in _create_cuda_ptx_copts_list
                if not _is_clang(cc):
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 127, column 20, in _is_clang
                return "clang" in cc
Error: unsupported binary operation: string in NoneType
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: unsupported binary operation: string in NoneType
make: *** [Makefile:24: /home/schoch/.cache/xla/0.9.1/build/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz] Error 2
could not compile dependency :xla, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile xla --force", update it with "mix deps.update xla" or clean it with "mix deps.clean xla"
==> oracle_knowledge_graph
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. If you are using
Ubuntu or any other Debian-based system, install the packages
"build-essential". Also install "erlang-dev" package if not
included in your Erlang/OTP version. If you're on Fedora, run
"dnf group install 'Development Tools'".

(.venv) schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang++ XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.clean xla
* Cleaning xla
(.venv) schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang++ XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.compile xla
** (Mix) Cannot compile dependency :xla because it isn't available, run "mix deps.get" first
(.venv) schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang++ XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.get xla
Resolving Hex dependencies...
Resolution completed in 0.126s
Unchanged:
  axon 0.7.0
  bumblebee 0.6.3
  castore 1.0.14
  cc_precompiler 0.1.10
  certifi 2.15.0
  complex 0.6.0
  db_connection 2.8.0
  decimal 2.3.0
  earmark_parser 1.4.44
  ecto 3.13.2
  ecto_sql 3.13.2
  ecto_sqlite3 0.21.0
  elixir_make 0.9.0
  ex_doc 0.38.2
  excoveralls 0.18.5
  exla 0.10.0
  exqlite 0.33.0
  file_system 1.1.0
  fine 0.1.1
  floki 0.38.0
  hackney 1.25.0
  html2markdown 0.3.0
  httpoison 2.2.3
  idna 6.1.1
  jason 1.4.4
  libgraph 0.16.0
  makeup 1.2.1
  makeup_elixir 1.0.1
  makeup_erlang 1.0.2
  meck 0.9.2
  metrics 1.0.1
  mimerl 1.4.0
  nimble_parsec 1.4.2
  nimble_pool 1.1.0
  nx 0.10.0
  nx_image 0.1.2
  nx_signal 0.2.0
  ortex 0.1.10
  parse_trans 3.4.1
  polaris 0.1.0
  poolboy 1.5.2
  progress_bar 3.0.0
  rustler 0.36.2
  rustler_precompiled 0.8.3
  safetensors 0.1.3
  ssl_verify_fun 1.1.7
  stb_image 0.6.10
  stream_data 1.2.0
  telemetry 1.3.0
  text_chunker 0.2.0
  tokenizers 0.5.1
  toml 0.7.0
  unicode_util_compat 0.7.1
  unpickler 0.1.0
  unzip 0.12.0
  xla 0.9.1
  yamerl 0.10.0
  yaml_elixir 2.11.0
* Getting xla (Hex package)
(.venv) schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang++ XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.compile xla
==> xla
Compiling 5 files (.ex)
Generated xla app
rm -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        ln -s "/home/schoch/dev/docgen/deps/xla/extension" /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        cd /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20 && \
        bazel build --define "framework_shared_object=false" -c opt   --config=cuda --repo_env=HERMETIC_CUDA_VERSION="12.8.0" --repo_env=HERMETIC_CUDNN_VERSION="9.8.0" --action_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES="sm_50,sm_60,sm_70,sm_80,sm_90,sm_100,compute_120" --action_env=TF_NVCC_CLANG="1" --@local_config_cuda//:cuda_compiler=nvcc --repo_env=CC=clang --repo_env=CXX=clang++ --copt=-Wno-error=unused-command-line-argument --copt=-Wno-gnu-offsetof-extensions --copt=-Qunused-arguments --copt=-Wno-error=c23-extensions //xla/extension:xla_extension && \
        mkdir -p /home/schoch/.cache/xla/0.9.1/build/ && \
        cp -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/bazel-bin/xla/extension/xla_extension.tar.gz /home/schoch/.cache/xla/0.9.1/build/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz
INFO: Reading 'startup' options from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/.bazelrc:
  Inherited 'common' options: --noenable_bzlmod --noincompatible_enable_cc_toolchain_resolution
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  Inherited 'common' options: --announce_rc --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --config=short_logs --@rules_python//python/config_settings:precompile=force_disabled
INFO: Found applicable config definition build:short_logs in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:cuda in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --config=cuda_version --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda_version in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env=HERMETIC_CUDA_VERSION=12.6.3 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --repo_env=HERMETIC_NVSHMEM_VERSION=3.2.5
INFO: Found applicable config definition build:linux in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --experimental_guard_against_concurrent_changes
Computing main repo mapping:
DEBUG: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:73:14: Cannot find clang, either correct your path, or set the CLANG_CUDA_COMPILER_PATH or CC environment variables
INFO: Repository local_config_cuda instantiated at:
  /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/WORKSPACE:87:15: in <toplevel>
Repository rule cuda_configure defined at:
  /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:705:33: in <toplevel>
ERROR: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:127:20: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 684, column 28, in _cuda_configure_impl
                _create_cuda_repository(repository_ctx)
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 668, column 38, in _create_cuda_repository
                _create_local_cuda_repository(repository_ctx)
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 611, column 44, in _create_local_cuda_repository
                _create_cuda_ptx_copts_list(repository_ctx, cuda_config.cuda_version) +
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 297, column 21, in _create_cuda_ptx_copts_list
                if not _is_clang(cc):
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 127, column 20, in _is_clang
                return "clang" in cc
Error: unsupported binary operation: string in NoneType
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: unsupported binary operation: string in NoneType
make: *** [Makefile:24: /home/schoch/.cache/xla/0.9.1/build/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz] Error 2
could not compile dependency :xla, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile xla --force", update it with "mix deps.update xla" or clean it with "mix deps.clean xla"
==> oracle_knowledge_graph
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. If you are using
Ubuntu or any other Debian-based system, install the packages
"build-essential". Also install "erlang-dev" package if not
included in your Erlang/OTP version. If you're on Fedora, run
"dnf group install 'Development Tools'".

(.venv) schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.compile xla
==> xla
Compiling 5 files (.ex)
Generated xla app
rm -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        ln -s "/home/schoch/dev/docgen/deps/xla/extension" /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        cd /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20 && \
        bazel build --define "framework_shared_object=false" -c opt   --config=cuda --repo_env=HERMETIC_CUDA_VERSION="12.8.0" --repo_env=HERMETIC_CUDNN_VERSION="9.8.0" --action_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES="sm_50,sm_60,sm_70,sm_80,sm_90,sm_100,compute_120" --action_env=TF_NVCC_CLANG="1" --@local_config_cuda//:cuda_compiler=nvcc --repo_env=CC=clang --repo_env=CXX=clang++ --copt=-Wno-error=unused-command-line-argument --copt=-Wno-gnu-offsetof-extensions --copt=-Qunused-arguments --copt=-Wno-error=c23-extensions //xla/extension:xla_extension && \
        mkdir -p /home/schoch/.cache/xla/0.9.1/build/ && \
        cp -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/bazel-bin/xla/extension/xla_extension.tar.gz /home/schoch/.cache/xla/0.9.1/build/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz
INFO: Reading 'startup' options from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/.bazelrc:
  Inherited 'common' options: --noenable_bzlmod --noincompatible_enable_cc_toolchain_resolution
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  Inherited 'common' options: --announce_rc --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --config=short_logs --@rules_python//python/config_settings:precompile=force_disabled
INFO: Found applicable config definition build:short_logs in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:cuda in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --config=cuda_version --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda_version in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env=HERMETIC_CUDA_VERSION=12.6.3 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --repo_env=HERMETIC_NVSHMEM_VERSION=3.2.5
INFO: Found applicable config definition build:linux in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --experimental_guard_against_concurrent_changes
Computing main repo mapping:
DEBUG: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:73:14: Cannot find clang, either correct your path, or set the CLANG_CUDA_COMPILER_PATH or CC environment variables
INFO: Repository local_config_cuda instantiated at:
  /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/WORKSPACE:87:15: in <toplevel>
Repository rule cuda_configure defined at:
  /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:705:33: in <toplevel>
ERROR: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl:127:20: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 684, column 28, in _cuda_configure_impl
                _create_cuda_repository(repository_ctx)
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 668, column 38, in _create_cuda_repository
                _create_local_cuda_repository(repository_ctx)
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 611, column 44, in _create_local_cuda_repository
                _create_cuda_ptx_copts_list(repository_ctx, cuda_config.cuda_version) +
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 297, column 21, in _create_cuda_ptx_copts_list
                if not _is_clang(cc):
        File "/home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/cuda/hermetic/cuda_configure.bzl", line 127, column 20, in _is_clang
                return "clang" in cc
Error: unsupported binary operation: string in NoneType
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: unsupported binary operation: string in NoneType
make: *** [Makefile:24: /home/schoch/.cache/xla/0.9.1/build/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz] Error 2
could not compile dependency :xla, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile xla --force", update it with "mix deps.update xla" or clean it with "mix deps.clean xla"
==> oracle_knowledge_graph
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. If you are using
Ubuntu or any other Debian-based system, install the packages
"build-essential". Also install "erlang-dev" package if not
included in your Erlang/OTP version. If you're on Fedora, run
"dnf group install 'Development Tools'".

(.venv) schoch@xeon:~/dev/docgen$ which clang
(.venv) schoch@xeon:~/dev/docgen$ which clang++
(.venv) schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.compile xla^C
(.venv) schoch@xeon:~/dev/docgen$ vi ~/.bashrc
(.venv) schoch@xeon:~/dev/docgen$ source ~/.bashrc
schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang XLA_BUILD=true XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.compile xla
==> xla
Compiling 5 files (.ex)
Generated xla app
rm -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        ln -s "/home/schoch/dev/docgen/deps/xla/extension" /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/xla/extension && \
        cd /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20 && \
        bazel build --define "framework_shared_object=false" -c opt   --config=cuda --repo_env=HERMETIC_CUDA_VERSION="12.8.0" --repo_env=HERMETIC_CUDNN_VERSION="9.8.0" --action_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES="sm_50,sm_60,sm_70,sm_80,sm_90,sm_100,compute_120" --action_env=TF_NVCC_CLANG="1" --@local_config_cuda//:cuda_compiler=nvcc --repo_env=CC=clang --repo_env=CXX=clang++ --copt=-Wno-error=unused-command-line-argument --copt=-Wno-gnu-offsetof-extensions --copt=-Qunused-arguments --copt=-Wno-error=c23-extensions //xla/extension:xla_extension && \
        mkdir -p /home/schoch/.cache/xla/0.9.1/build/ && \
        cp -f /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/bazel-bin/xla/extension/xla_extension.tar.gz /home/schoch/.cache/xla/0.9.1/build/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz
INFO: Reading 'startup' options from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/.bazelrc:
  Inherited 'common' options: --noenable_bzlmod --noincompatible_enable_cc_toolchain_resolution
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  Inherited 'common' options: --announce_rc --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --config=short_logs --@rules_python//python/config_settings:precompile=force_disabled
INFO: Found applicable config definition build:short_logs in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:cuda in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --config=cuda_version --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda_version in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --repo_env=HERMETIC_CUDA_VERSION=12.6.3 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --repo_env=HERMETIC_NVSHMEM_VERSION=3.2.5
INFO: Found applicable config definition build:linux in file /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/tensorflow.bazelrc: --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --experimental_guard_against_concurrent_changes
Computing main repo mapping:
Loading:
Loading: 2 packages loaded
Analyzing: target //xla/extension:xla_extension (3 packages loaded, 0 targets configured)
Analyzing: target //xla/extension:xla_extension (3 packages loaded, 0 targets configured)

Analyzing: target //xla/extension:xla_extension (50 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (50 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (51 packages loaded, 26 targets configured)

Analyzing: target //xla/extension:xla_extension (148 packages loaded, 4393 targets configured)

DEBUG: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/nvidia_common_rules.bzl:275:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvrtc/linux-x86_64/cuda_nvrtc-linux-x86_64-12.8.61-archive.tar.xz
DEBUG: /home/schoch/.cache/xla_build/xla-870d90fd098c480fb8a426126bd02047adb2bc20/third_party/gpus/nvidia_common_rules.bzl:275:10: Downloading and extracting https://developer.download.nvidia.com/compute/cuda/redist/nvidia_driver/linux-x86_64/nvidia_driver-linux-x86_64-570.86.10-archive.tar.xz
Analyzing: target //xla/extension:xla_extension (219 packages loaded, 6519 targets configured)

Analyzing: target //xla/extension:xla_extension (261 packages loaded, 12820 targets configured)

Analyzing: target //xla/extension:xla_extension (275 packages loaded, 14978 targets configured)

Analyzing: target //xla/extension:xla_extension (278 packages loaded, 19223 targets configured)

Analyzing: target //xla/extension:xla_extension (278 packages loaded, 19414 targets configured)

Analyzing: target //xla/extension:xla_extension (278 packages loaded, 19414 targets configured)

Analyzing: target //xla/extension:xla_extension (278 packages loaded, 19414 targets configured)

Analyzing: target //xla/extension:xla_extension (279 packages loaded, 20054 targets configured)

Analyzing: target //xla/extension:xla_extension (283 packages loaded, 23510 targets configured)

Analyzing: target //xla/extension:xla_extension (283 packages loaded, 23510 targets configured)

Analyzing: target //xla/extension:xla_extension (287 packages loaded, 25074 targets configured)

Analyzing: target //xla/extension:xla_extension (289 packages loaded, 26050 targets configured)

Analyzing: target //xla/extension:xla_extension (289 packages loaded, 26050 targets configured)

Analyzing: target //xla/extension:xla_extension (289 packages loaded, 26050 targets configured)

Analyzing: target //xla/extension:xla_extension (289 packages loaded, 26050 targets configured)

Analyzing: target //xla/extension:xla_extension (289 packages loaded, 26050 targets configured)

Analyzing: target //xla/extension:xla_extension (289 packages loaded, 26050 targets configured)

Analyzing: target //xla/extension:xla_extension (290 packages loaded, 26060 targets configured)

Analyzing: target //xla/extension:xla_extension (290 packages loaded, 26060 targets configured)

Analyzing: target //xla/extension:xla_extension (290 packages loaded, 26060 targets configured)

INFO: Analyzed target //xla/extension:xla_extension (291 packages loaded, 26093 targets configured).
[1,691 / 10,745] Creating runfiles tree bazel-out/k8-opt-exec-ST-0465588ec812/bin/third_party/implib_so/make_stub.runfiles [for tool]; 0s local ... (14 actions, 13 running)
[1,736 / 10,745] Creating runfiles tree bazel-out/k8-opt-exec-ST-0465588ec812/bin/third_party/implib_so/make_stub.runfiles [for tool]; 1s local ... (15 actions, 13 running)
[1,764 / 10,745] Creating runfiles tree bazel-out/k8-opt-exec-ST-0465588ec812/bin/third_party/implib_so/make_stub.runfiles [for tool]; 2s local ... (16 actions, 15 running)
[1,783 / 10,837] Creating runfiles tree bazel-out/k8-opt-exec-ST-0465588ec812/bin/third_party/implib_so/make_stub.runfiles [for tool]; 3s local ... (16 actions, 15 running)
[1,795 / 12,091] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 2s local ... (16 actions, 15 running)
[1,834 / 12,091] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 3s local ... (16 actions, 15 running)
[1,864 / 12,091] Compiling xla/stream_executor/gpu/scoped_activate_context.cc; 2s local ... (17 actions, 15 running)
[1,882 / 12,091] Compiling google/fft.cc; 2s local ... (16 actions, 15 running)


<SNIP>

[12,090 / 12,091] [Prepa] Executing genrule //xla/extension:xla_extension
[12,090 / 12,091] Executing genrule //xla/extension:xla_extension; 1s local
[12,090 / 12,091] Executing genrule //xla/extension:xla_extension; 11s local
INFO: Found 1 target...
Target //xla/extension:xla_extension up-to-date:
  bazel-bin/xla/extension/xla_extension.tar.gz
INFO: Elapsed time: 3562.426s, Critical Path: 535.16s
INFO: 12091 processes: 2247 internal, 9844 local.
INFO: Build completed successfully, 12091 total actions
schoch@xeon:~/dev/docgen$ CC=/usr/lib/llvm-18/bin/clang  XLA_TARGET=cuda12 MIX_ENV=dev  mix deps.compile exla
==> exla
Unpacking /home/schoch/.cache/xla/0.9.1/download/xla_extension-0.9.1-x86_64-linux-gnu-cuda12.tar.gz into /home/schoch/dev/docgen/deps/exla/cache
Using libexla.so from /home/schoch/.cache/xla/exla/elixir-1.18.4-erts-16.0.2-xla-0.9.1-exla-0.10.0-6ot23boihvqnj5b2fz4iye6qmm/libexla.so
EXLA_CPU_ONLY is not set, checking for nvcc availability
CUDA is available.
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/exla.cc -o cache/0.10.0/objs/exla.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/exla_client.cc -o cache/0.10.0/objs/exla_client.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/exla_mlir.cc -o cache/0.10.0/objs/exla_mlir.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/ipc.cc -o cache/0.10.0/objs/ipc.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/eigh_f32.cc -o cache/0.10.0/objs/custom_calls/eigh_f32.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/eigh_f64.cc -o cache/0.10.0/objs/custom_calls/eigh_f64.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/lu_bf16.cc -o cache/0.10.0/objs/custom_calls/lu_bf16.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/lu_f16.cc -o cache/0.10.0/objs/custom_calls/lu_f16.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/lu_f32.cc -o cache/0.10.0/objs/custom_calls/lu_f32.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/lu_f64.cc -o cache/0.10.0/objs/custom_calls/lu_f64.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/qr_bf16.cc -o cache/0.10.0/objs/custom_calls/qr_bf16.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/qr_f16.cc -o cache/0.10.0/objs/custom_calls/qr_f16.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/qr_f32.cc -o cache/0.10.0/objs/custom_calls/qr_f32.o
g++ -fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3 -c c_src/exla/custom_calls/qr_f64.cc -o cache/0.10.0/objs/custom_calls/qr_f64.o
nvcc -Xcompiler "-fPIC -I/home/schoch/.asdf/installs/erlang/28.0.2/erts-16.0.2/include -I/home/schoch/dev/docgen/deps/fine/include -Icache/xla_extension/include -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -std=c++17 -w -O3" -DCUDA_ENABLED -c c_src/exla/exla_cuda.cc -o cache/0.10.0/objs/exla_cuda.o
nvcc warning : Support for offline compilation for architectures prior to '<compute/sm/lto>_75' will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
g++ cache/0.10.0/objs/exla.o cache/0.10.0/objs/exla_client.o cache/0.10.0/objs/exla_mlir.o cache/0.10.0/objs/ipc.o cache/0.10.0/objs/custom_calls/eigh_f32.o cache/0.10.0/objs/custom_calls/eigh_f64.o cache/0.10.0/objs/custom_calls/lu_bf16.o cache/0.10.0/objs/custom_calls/lu_f16.o cache/0.10.0/objs/custom_calls/lu_f32.o cache/0.10.0/objs/custom_calls/lu_f64.o cache/0.10.0/objs/custom_calls/qr_bf16.o cache/0.10.0/objs/custom_calls/qr_f16.o cache/0.10.0/objs/custom_calls/qr_f32.o cache/0.10.0/objs/custom_calls/qr_f64.o cache/0.10.0/objs/exla_cuda.o -o cache/libexla.so -Lcache/xla_extension/lib -lxla_extension -shared -fvisibility=hidden -Wl,-rpath,'$ORIGIN/xla_extension/lib'
Compiling 23 files (.ex)

11:48:19.777 [error] Process #PID<0.274.0> raised an exception
** (RuntimeError) Failed to load NIF library.
Follow the steps in the :exla README Troubleshooting section for more information.

:load_failed
Failed to load NIF library /home/schoch/dev/docgen/_build/dev/lib/exla/priv/libexla: '/home/schoch/dev/docgen/_build/dev/lib/exla/priv/xla_extension/lib/libxla_extension.so: undefined symbol: ncclMemFree'

    lib/exla/nif.ex:13: EXLA.NIF.__on_load__/0
    (kernel 10.3.2) code_server.erl:1388: anonymous fn/1 in :code_server.schedule_on_load/4

11:48:19.780 [warning] The on_load function for module Elixir.EXLA.NIF returned:
{%RuntimeError{
   message: "Failed to load NIF library.\nFollow the steps in the :exla README Troubleshooting section for more information.\n\n:load_failed\nFailed to load NIF library /home/schoch/dev/docgen/_build/dev/lib/exla/priv/libexla: '/home/schoch/dev/docgen/_build/dev/lib/exla/priv/xla_extension/lib/libxla_extension.so: undefined symbol: ncclMemFree'\n"
 },
 [
   {EXLA.NIF, :__on_load__, 0,
    [file: ~c"lib/exla/nif.ex", line: 13, error_info: %{...}]},
   {:code_server, :"-schedule_on_load/4-fun-0-", 1,
    [file: ~c"code_server.erl", line: 1388]}
 ]}

Generated exla app

[jonatanklosko]

I believe you are missing apt-get install libnccl2. We actually don't need NCCL, but unfortunately the current .so depends on it.

the precompiled didn't work

Didn't work in which way, the same nccl error? If that's the case, then it should work fine once you install it.

[jschoch]

maybe a path problem ?

history |grep libnccl2
  131  sudo apt install libnccl2 libnccl-dev
  152  sudo apt install libnccl2 libnccl-dev
  198  sudo apt install libnccl2 libnccl-dev
  271  sudo apt install libnccl2 libnccl-dev
  272  sudo apt upgrade libnccl2 libnccl-dev

[jonatanklosko]

Hmm, could be. What is your CUDA, cuDNN and NCCL versions?

For the record, here is what we install in Livebook Docker image (currently with CUDA 12.8):

https://github.com/livebook-dev/livebook/blob/a9c360ee9d8af76adcdd9475af1b5c74b989fddb/Dockerfile#L24