net_zstd build failure

[gcapodagAMD]

Issue type

Build/Install

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

merge-250714

Custom code

Yes

OS platform and distribution

Linux Ubuntu 22.04

Mobile device

No response

Python version

3.10

Bazel version

8.4.1

GCC/compiler version

11.4.0

CUDA/cuDNN version

No response

GPU model and memory

gfx90a

Current behavior?

The build fails with this error:

ERROR: /home/sysadmin/.cache/bazel/_bazel_sysadmin/f84c89eaa02a68c63a7b150a8bf830ca/external/net_zstd/BUILD.bazel:15:11: Compiling decompress/huf_decompress_amd64.S failed: absolute path inclusion(s) found in rule '@@net_zstd//:zstdlib':
the source file 'decompress/huf_decompress_amd64.S' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/opt/rocm-6.4.1/lib/llvm/lib/clang/19/include/cet.h'
Target //tensorflow/tools/pip_package:wheel failed to build
ERROR: /tmp/tensorflow-upstream/tensorflow/tools/pip_package/BUILD:292:9 Action tensorflow/tools/pip_package/wheel_house/tensorflow_rocm-2.20.0.dev20250716-cp310-cp310-linux_x86_64.whl failed: absolute path inclusion(s) found in rule '@@net_zstd//:zstdlib':
the source file 'decompress/huf_decompress_amd64.S' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/opt/rocm-6.4.1/lib/llvm/lib/clang/19/include/cet.h'

Standalone code to reproduce the issue

Run this script: https://github.com/amd/HPCTrainingDock/blob/main/extras/scripts/tensorflow_setup.sh
like this to reproduce the issue:


git clone https://github.com/amd/HPCTrainingDock.git
cd HPCTrainingDock
vi extras/scripts/tensorflow_setup.sh (change (this)[https://github.com/amd/HPCTrainingDock/blob/8caed60a50b747f2383b7cc98517f9a261b31776/extras/scripts/tensorflow_setup.sh#L10] line to version merge-250714)
./bare_system/test_install.sh --rocm-version 6.4.1 --use-makefile 1 --distro ubuntu --distro-version 22.04 --amdgpu-gfxmodel gfx90a
make tensorflow


The `test_install.sh` will use Docker (or Podman) to get you on a bare Ubuntu 22.04 container and doing `make tensorflow` will install some base OS packages, ROCm 6.4.1 and then tensorflow with the above script.

You can also run the script directly with these options:

./extras/script/tensorflow_setup.sh --build-tensorflow 1 --rocm-version 6.4.1 --amdgpu-gfxmodel gfx90a --install-path (INSTALL PATH where you have write access) --module-path (MODULE FILE PATH where you have write access) --git-branch merge-250714

Relevant log output

[gcapodagAMD]

@hsharsha any updates on this?

[0xra0]

Has anyone solved this problem?

[hsharsha]

Looking into it. Have you tired it on newer Rocm 7.0.2 tensorflow images? something like rocm/tensorflow-build@sha256:abd4ae15bab1292ba5cb6f7feab43e167b34963f991fc911478e9de65d54b1a3

[0xra0]

Looking into it. Have you tired it on newer Rocm 7.0.2 tensorflow images? something like rocm/tensorflow-build@sha256:abd4ae15bab1292ba5cb6f7feab43e167b34963f991fc911478e9de65d54b1a3

But I need to use with Rocm 6.4.4, tensorflow-upstream branch r2.20-rocm-enhanced

[0xra0]

Looking into it. Have you tired it on newer Rocm 7.0.2 tensorflow images? something like rocm/tensorflow-build@sha256:abd4ae15bab1292ba5cb6f7feab43e167b34963f991fc911478e9de65d54b1a3

Any fix ?

[0xra0]

@hsharsha

[hsharsha]

I am trying with https://github.com/ROCm/tensorflow-upstream/commits/merge-20251021 and 6.4 With port 2222 I got port already in use and I am trying with 3333

[0xra0]

merge-20251021

The same error

[0xra0]

merge-20251021

The same error

ERROR: /home/home/.cache/bazel/_bazel_home/c0b57fd22c2c29419a7970d135015b3e/external/net_zstd/BUILD.bazel:15:11: Compiling decompress/huf_decompress_amd64.S failed: absolute path inclusion(s) found in rule '@@net_zstd//:zstdlib':
the source file 'decompress/huf_decompress_amd64.S' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
'/usr/lib/llvm18/lib/clang/18/include/cet.h'
Target @@net_zstd//:zstdlib failed to build
INFO: Elapsed time: 317.929s, Critical Path: 1.18s
INFO: 24 processes: 14 internal, 10 local.
ERROR: Build did NOT complete successfully

[hsharsha]

INFO: Analyzed target //tensorflow/tools/pip_package:wheel (993 packages loaded, 88772 targets configured).
ERROR: /home/sysadmin/.cache/bazel/_bazel_sysadmin/f84c89eaa02a68c63a7b150a8bf830ca/external/net_zstd/BUILD.bazel:15:11: Compiling decompress/huf_decompress_amd64.S failed: absolute path inclusion(s) found in rule '@@net_zstd//:zstdlib':
the source file 'decompress/huf_decompress_amd64.S' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/opt/rocm-6.4.1/lib/llvm/lib/clang/19/include/cet.h'
Target //tensorflow/tools/pip_package:wheel failed to build
ERROR: /tmp/tensorflow-upstream/tensorflow/tools/pip_package/BUILD:313:9 Action tensorflow/tools/pip_package/wheel_house/tensorflow_rocm-2.21.0.dev0+selfbuilt-cp310-cp310-linux_x86_64.whl failed: absolute path inclusion(s) found in rule '@@net_zstd//:zstdlib':****

where does it clone tensorflow?

[0xra0]

merge-20251021

The same error

ERROR: /home/home/.cache/bazel/_bazel_home/c0b57fd22c2c29419a7970d135015b3e/external/net_zstd/BUILD.bazel:15:11: Compiling decompress/huf_decompress_amd64.S failed: absolute path inclusion(s) found in rule '@@net_zstd//:zstdlib': the source file 'decompress/huf_decompress_amd64.S' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): '/usr/lib/llvm18/lib/clang/18/include/cet.h' Target @@net_zstd//:zstdlib failed to build INFO: Elapsed time: 317.929s, Critical Path: 1.18s INFO: 24 processes: 14 internal, 10 local. ERROR: Build did NOT complete successfully

git clone --branch merge-20251021 --depth 1 https://github.com/ROCm/tensorflow-upstream.git

[hsharsha]

Looks like it clones in /tmp but it is getting cleared out

/opt/rocmplus-6.4.1/tensorflow/bazel /tmp
/tmp
Cloning into 'tensorflow-upstream'...
remote: Enumerating objects: 2180912, done.
remote: Counting objects: 100% (166/166), done.
remote: Compressing objects: 100% (106/106), done.
remote: Total 2180912 (delta 83), reused 60 (delta 60), pack-reused 2180746 (from 3)
Receiving objects: 100% (2180912/2180912), 1.25 GiB | 2.12 MiB/s, done.
Resolving deltas: 100% (1785703/1785703), done.
Note: switching to '4e5e68826b7906358eff167f82607d847bae7ae6'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Updating files: 100% (35227/35227), done.
6.4.1
You have bazel 7.4.1 installed.
Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]
Do you wish to build TensorFlow with CUDA support? [y/N]: No CUDA support will be enabled for TensorFlow.

Please specify the path to clang executable. [Default is /opt/rocm-6.4.1/llvm/bin/clang]:

WARNING: current clang installation is not a release version.

You have Clang 19.0.0 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:

Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: Not configuring the WORKSPACE for Android builds.

Also it is using llvm19 from /opt/rocm. Is it possible to install system llvm18 and use it for compilation as we use this in general
https://github.com/ROCm/tensorflow-upstream/blob/develop-upstream/.bazelrc#L392

[gcapodagAMD]

@hsharsha why are you not using clang from ROCm?