Compilation with cuda 10.2

[llan-ml]

I'm just wondering if the jaxlib still supports building upon cuda 10.2. I tried to build jaxlib==v0.3.20, and it raises the error nvcc fatal : Value 'c++17' is not defined for option 'std'.

The full log is as follows:

± |tag:jax-v0.3.20 U:1 ✗| → python build/build.py --enable_cuda --bazel_startup_options="--host_jvm_args=-Djavax.net.ssl.trustStore=/home/ma-user/cacerts" --bazel_startup_options="--host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit" --noenable_tpu --noenable_rocm --noenable_remote_tpu --cuda_version=10.2 --cudnn_version=7

     _   _  __  __
    | | / \ \ \/ /
 _  | |/ _ \ \  /
| |_| / ___ \/  \
 \___/_/   \/_/\_\


Bazel binary path: ./bazel-5.1.1-linux-x86_64
Bazel version: 5.1.1
Python binary path: /home/ma-user/.pyenv/versions/3.9.12/envs/jax-cuda102/bin/python
Python version: 3.9
NumPy version: 1.21.5
MKL-DNN enabled: yes
Target CPU: x86_64
Target CPU features: release
CUDA enabled: yes
CUDA version: 10.2
CUDNN version: 7
NCCL enabled: yes
TPU enabled: no
Remote TPU enabled: no
ROCm enabled: no
Plugin device enabled: no

Building XLA and installing it in the jaxlib source tree...
./bazel-5.1.1-linux-x86_64 --host_jvm_args=-Djavax.net.ssl.trustStore=/home/ma-user/cacerts --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit run --verbose_failures=true :build_wheel -- --output_path=/home/ma-user/jax/dist --cpu=x86_64
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /home/ma-user/jax/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /home/ma-user/jax/.bazelrc:
  Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_plugin_device=false
INFO: Reading rc options for 'run' from /home/ma-user/jax/.jax_configure.bazelrc:
  Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/home/ma-user/.pyenv/versions/3.9.12/envs/jax-cuda102/bin/python --action_env=PYENV_ROOT --python_path=/home/ma-user/.pyenv/versions/3.9.12/envs/jax-cuda102/bin/python --action_env TF_CUDA_VERSION=10.2 --action_env TF_CUDNN_VERSION=7 --distinct_host_configuration=false --config=avx_posix --config=mkl_open_source_only --config=cuda
INFO: Found applicable config definition build:short_logs in file /home/ma-user/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:avx_posix in file /home/ma-user/jax/.bazelrc: --copt=-mavx --host_copt=-mavx
INFO: Found applicable config definition build:mkl_open_source_only in file /home/ma-user/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:cuda in file /home/ma-user/jax/.bazelrc: --repo_env TF_NEED_CUDA=1 --action_env TF_CUDA_COMPUTE_CAPABILITIES=sm_35,sm_52,sm_60,sm_70,compute_80 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=true --define=xla_python_enable_gpu=true
INFO: Found applicable config definition build:linux in file /home/ma-user/jax/.bazelrc: --config=posix --copt=-Wno-unknown-warning-option --copt=-Wno-stringop-truncation --copt=-Wno-array-parameter
INFO: Found applicable config definition build:posix in file /home/ma-user/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++17 --host_cxxopt=-std=c++17WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/daf51682b439096eb62b6bb378b3a8f55a593e55.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/039cef36705e93539aaa34e544268b835ce2ab14.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/protocolbuffers/upb/archive/9effcbcb27f0a665f9f345030188c0b291e32482.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/rules_go/releases/download/v0.34.0/rules_go-v0.34.0.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
DEBUG: /home/ma-user/.cache/bazel/_bazel_ma-user/234f287aee29af8cc1e4cd117398b982/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:118:10:
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
Loading:
Loading: 0 packages loaded
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class java.io.FileNotFoundException GET
returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/openxla/stablehlo/archive/55814280317cc6db499bf2137750bcfa46ca452e.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/pybind/pybind11_abseil/archive/2c4932ed6f6204f1656e245838f4f5eae69d2e29.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/NVIDIA/cudnn-frontend/archive/refs/tags/v0.7.1.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //build:build_wheel (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 12] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[583 / 2,470] Compiling tensorflow/compiler/xla/service/custom_call_status.cc; 1s local ... (72 actions running)
[595 / 2,482] Compiling tensorflow/compiler/xla/service/custom_call_status.cc; 2s local ... (72 actions, 71 running)
[598 / 2,482] Compiling absl/container/internal/raw_hash_set.cc; 3s local ... (72 actions, 71 running)
[603 / 2,487] Compiling absl/container/internal/raw_hash_set.cc; 4s local ... (72 actions, 71 running)
[611 / 2,496] Compiling absl/container/internal/raw_hash_set.cc; 5s local ... (72 actions, 71 running)
[613 / 2,496] Compiling absl/container/internal/raw_hash_set.cc; 6s local ... (72 actions running)
[620 / 2,516] Compiling absl/container/internal/raw_hash_set.cc; 8s local ... (72 actions running)
[621 / 2,516] Compiling absl/container/internal/raw_hash_set.cc; 9s local ... (72 actions, 71 running)
[634 / 2,516] Compiling tensorflow/tsl/platform/scanner.cc; 10s local ... (72 actions running)
[639 / 2,516] Compiling tensorflow/tsl/platform/scanner.cc; 12s local ... (72 actions running)
[641 / 2,516] Compiling tensorflow/tsl/platform/scanner.cc; 15s local ... (71 actions running)
[653 / 2,668] Compiling tensorflow/tsl/platform/default/mutex.cc; 17s local ... (72 actions running)
[667 / 2,668] Compiling tensorflow/tsl/platform/str_util.cc; 20s local ... (72 actions running)
[678 / 2,668] Compiling tensorflow/tsl/platform/default/logging.cc; 23s local ... (72 actions running)
[691 / 2,668] Compiling tensorflow/tsl/platform/numbers.cc; 27s local ... (72 actions running)
[712 / 2,668] Compiling src/google/protobuf/generated_message_table_driven_lite.cc; 32s local ... (71 actions, 70 running)
[729 / 2,668] Compiling src/google/protobuf/generated_message_table_driven_lite.cc; 37s local ... (72 actions running)
[756 / 2,668] Compiling mlir/tools/mlir-tblgen/AttrOrTypeFormatGen.cpp; 42s local ... (72 actions running)
[783 / 2,668] Compiling mlir/tools/mlir-tblgen/AttrOrTypeFormatGen.cpp; 49s local ... (72 actions running)
[814 / 2,668] Compiling mlir/tools/mlir-tblgen/AttrOrTypeDefGen.cpp; 57s local ... (71 actions running)
[877 / 2,845] Compiling mlir/tools/mlir-tblgen/AttrOrTypeDefGen.cpp; 66s local ... (72 actions running)
[956 / 2,898] Compiling mlir/tools/mlir-tblgen/OpFormatGen.cpp; 76s local ... (72 actions running)
[1,037 / 3,083] Compiling mlir/tools/mlir-tblgen/OpFormatGen.cpp; 88s local ... (72 actions, 71 running)
[1,075 / 3,083] Compiling mlir/tools/mlir-tblgen/OpFormatGen.cpp; 102s local ... (72 actions running)
[1,112 / 3,083] Compiling mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp; 117s local ... (72 actions running)
[1,168 / 3,136] Compiling mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp; 135s local ... (72 actions running)
[1,614 / 3,466] Compiling src/idl_gen_java.cpp; 79s local ... (72 actions running)
[1,716 / 3,618] Compiling src/idl_gen_cpp.cpp; 102s local ... (72 actions running)
[1,837 / 3,618] Compiling src/idl_parser.cpp; 126s local ... (72 actions, 71 running)
[1,958 / 3,728] Compiling src/google/protobuf/descriptor.cc; 103s local ... (72 actions running)
[2,140 / 3,818] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp; 62s local ... (72 actions running)
[2,315 / 3,870] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp; 103s local ... (72 actions running)
[2,527 / 4,097] Compiling llvm/utils/TableGen/GlobalISelEmitter.cpp; 147s local ... (72 actions running)
ERROR: /home/ma-user/jax/jaxlib/cuda/BUILD:243:13: Compiling jaxlib/cuda/cuda_prng_kernels.cu.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
  (cd /home/ma-user/.cache/bazel/_bazel_ma-user/234f287aee29af8cc1e4cd117398b982/execroot/__main__ && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/x86_64-linux-gnu/ \
    PATH=/home/ma-user/.pyenv/versions/3.9.12/envs/jax-cuda102/bin:/home/ma-user/.pyenv/libexec:/home/ma-user/.pyenv/plugins/python-build/bin:/home/ma-user/.pyenv/plugins/pyenv-virtualenv/bin:/home/ma-user/.pyenv/plugins/pyenv-update/bin:/home/ma-user/.pyenv/plugins/pyenv-installer/bin:/home/ma-user/.pyenv/plugins/pyenv-doctor/bin:/home/ma-user/.pyenv/shims:/home/ma-user/.pyenv/bin:/modelarts/authoring/notebook-conda/bin:/home/ma-user/anaconda3/envs/PyTorch-1.8/bin:/usr/local/openmpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ma-user/modelarts-dev/ma-cli/bin:/modelarts/authoring/script/entrypoint/deps/ssh/bin:/home/ma-user/anaconda3/envs/PyTorch-1.8/bin:/home/ma-user/.fzf/bin \
    PWD=/proc/self/cwd \
    PYENV_ROOT=/home/ma-user/.pyenv \
    TF_CUDA_COMPUTE_CAPABILITIES=sm_35,sm_52,sm_60,sm_70,compute_80 \
    TF_CUDA_VERSION=10.2 \
    TF_CUDNN_VERSION=7 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/jaxlib/cuda/_objs/cuda_prng_kernels_impl/cuda_prng_kernels.cu.pic.d '-frandom-seed=bazel-out/k8-opt/bin/jaxlib/cuda/_objs/cuda_prng_kernels_impl/cuda_prng_kernels.cu.pic.o' -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -DHAVE_SYS_UIO_H -DTF_USE_SNAPPY -iquote . -iquote bazel-out/k8-opt/bin -iquote external/org_tensorflow -iquote bazel-out/k8-opt/bin/external/org_tensorflow -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/snappy -iquote bazel-out/k8-opt/bin/external/snappy -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/local_config_rocm -iquote bazel-out/k8-opt/bin/external/local_config_rocm -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers_virtual -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cublas_headers_virtual -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include -isystem external/org_tensorflow/third_party/eigen3/mkl_include -isystem bazel-out/k8-opt/bin/external/org_tensorflow/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/local_config_rocm/rocm -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm -isystem external/local_config_rocm/rocm/rocm/include -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include -isystem external/local_config_rocm/rocm/rocm/include/rocrand -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/rocrand -isystem external/local_config_rocm/rocm/rocm/include/roctracer -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/roctracer -isystem external/local_config_cuda/cuda/cublas/include -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cublas/include -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -fno-canonical-system-headers -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '-fvisibility=hidden' -Wno-sign-compare -Wno-unknown-warning-option -Wno-stringop-truncation -Wno-array-parameter
'-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.' -mavx '-std=c++17' -x cuda '-DGOOGLE_CUDA=1' '-Xcuda-fatbinary=--compress-all' '--cuda-gpu-arch=sm_35' '--cuda-gpu-arch=sm_52' '--cuda-gpu-arch=sm_60' '--cuda-gpu-arch=sm_70' '--cuda-include-ptx=sm_80' '--cuda-gpu-arch=sm_80' -c jaxlib/cuda/cuda_prng_kernels.cu.cc -o bazel-out/k8-opt/bin/jaxlib/cuda/_objs/cuda_prng_kernels_impl/cuda_prng_kernels.cu.pic.o)
# Configuration: a8c06e910a0003330080b94e49917b21db70969682d4362f9464efa2a69b5659
# Execution platform: @local_execution_config_platform//:platform
nvcc fatal   : Value 'c++17' is not defined for option 'std'
Target //build:build_wheel failed to build
INFO: Elapsed time: 389.693s, Critical Path: 300.48s
INFO: 3030 processes: 1100 internal, 1930 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
  File "/home/ma-user/jax/build/build.py", line 569, in <module>
    main()
  File "/home/ma-user/jax/build/build.py", line 564, in main
    shell(command)
  File "/home/ma-user/jax/build/build.py", line 55, in shell
    output = subprocess.check_output(cmd)
  File "/home/ma-user/.pyenv/versions/3.9.12/envs/jax-cuda102/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/ma-user/.pyenv/versions/3.9.12/envs/jax-cuda102/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-5.1.1-linux-x86_64', '--host_jvm_args=-Djavax.net.ssl.trustStore=/home/ma-user/cacerts', '--host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit', 'run', '--verbose_failures=true', ':build_wheel', '--', '--output_path=/home/ma-user/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.

Can anyone help? Thanks!

[hawkinsp]

We support CUDA 11.1 or greater at the moment. Any reason you can't update?

[llan-ml]

Thanks for your reply. Our internal cloud service still uses cuda 10.2, and I still need to use the old jax for now. 😞

[hawkinsp]

Note that it's actually quite feasible to just ship your own CUDA installation around, provided:
a) you don't mind that it's pretty big in size
b) you have a sufficiently new NVIDIA driver.

We're actually planning to switch JAX to use pip-installed CUDA wheels that are currently CUDA 11.8. The only piece you would need to bring yourself is the driver.

[llan-ml]

@hawkinsp Thanks! I'm already in a docker container and the version of nvidia driver is 440.xx.xx. Do you know if the driver could be updated inside the container?

[hawkinsp]

No, it cannot, with one caveat below. You would need to have whoever administers the system to update the NVIDIA driver.

It's possible that NVIDIA's forward-compatibility packages would work if you're using a datacenter GPU (https://docs.nvidia.com/deploy/cuda-compatibility/index.html#forward-compatibility-title). I'm not sure what the exact requirements are there though.

[llan-ml]

@hawkinsp Thank you very much for your help! The NVIDIA's forward-compatibility packages work well. I can use the latest jax now in a container with an old nvidia driver.