clang-offload-wrapper creating broken .o file

[ZuseZ4]

Currently on llvm 21.1.0 rc3: rust-lang/rust@3fb4be2

I am working on reproducing the behavior of clang's offloading pipeline in rustc. As part of that, I run clang on a minimal openmp (kernel mode) example, and look at the output from -save-temps, --dry-run and -###. I figured the first steps out, so I am now especially looking at the clang-linker-wrapper which seems to be doing plenty of things.

I started compiling the c++ example file via myclang++ -fuse-ld=lld -fopenmp -fopenmp-offload-mandatory --offload-arch=gfx90a omp_bare.cpp -o bare -save-temps. myclang is based on the same llvm commit as rustc.

So far I generated a lib.ll (host, x86-64) and a device.ll (target amdgcn-amd-amdhsa) file from the same rust source file "lib.rs", using rustc.

I then run the following 3 commands, which I got from the clang invocation above (slightly cleaned up)

clang-offload-packager -o host.out --image=file=device.ll,triple=amdgcn-amd-amdhsa,arch=gfx90a,kind=openmp

clang-21 -cc1 -triple x86_64-unknown-linux-gnu -S -dumpdir asdf -save-temps=cwd -disable-free -clear-ast-before-backend -main-file-name lib.rs -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -ferror-limit 19 -fopenmp -fopenmp-offload-mandatory -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fembed-offload-object=host.out -fopenmp-targets=amdgcn-amd-amdhsa -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o host.s -x ir lib.ll

clang-21 -cc1as -triple x86_64-unknown-linux-gnu -filetype obj -main-file-name lib.rs -target-cpu x86-64 -dwarf-version=5 -mrelocation-model pic -o host.o host.s

So far, so good. I now have device.ll host.o host.out host.s lib.ll in my build folder. The last step of the clang offload invocation is (again slightly simplified)

clang-linker-wrapper --should-extract=gfx90a --device-linker=amdgcn-amd-amdhsa=-lompdevice --host-triple=x86_64-unknown-linux-gnu --linker-path=/path/rust/build/x86_64-unknown-linux-gnu/lld/bin/ld.lld --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -pie -dynamic-linker -o bare /lib/../lib64/Scrt1.o /lib/../lib64/crti.o /path/gcc/x86_64/12/crtbeginS.o -L/path/gcc/x86_64/12  -L/path/gcc/x86_64/12/../../../../lib64 -L/lib/../lib64 -L/usr/lib64 -L/lib -L/usr/lib host.o -lstdc++ -lm -lomp -lomptarget -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /path/gcc/x86_64/12/crtendS.o /lib/../lib64/crtn.o

It gives

ld.lld: error: host-amdgcn-amd-amdhsa-gfx90a.o:1: malformed number: '
>>> ; ModuleID = 'lib-openmp-amdgcn-amd-amdhsa-gfx90a.bc'
>>>              ^
clang: error: ld.lld command failed with exit code 1 (use -v to see invocation)

Inspecting the host-amdgcn-amd-amdhsa-gfx90a.o file (where does it even get generated?) shows that it's just our device.ll file, which we started with. The clang-linker-wrapper somehow creates a copy and incorrectly names it a .o file.

If I append --save-temps and --dry-run to the clang-linker-wrapper invocation above, I get the following output:

"/path/prog/llvm/bin/clang" --no-default-config -o bare.amdgcn.gfx90a.img --target=amdgcn-amd-amdhsa -mcpu=gfx90a -flto -Wl,--no-undefined host-amdgcn-amd-amdhsa-gfx90a.o -Wl,--save-temps -Xlinker -lompdevice

 "/path/rust/build/x86_64-unknown-linux-gnu/lld/bin/ld.lld" {some-args}

It already fails at the clang invocation, since host-amdgcn-amd-amdhsa-gfx90a.o is an incorrectly named device.ll file. clang-linker-wrapper also must create that file in some way before calling clang, but does so in a way that's not printed by --dry-run.

I'll add more details later.

[ZuseZ4]

My device.ll file

; ModuleID = 'r.77ca1dff4c037088-cgu.0'
source_filename = "r.77ca1dff4c037088-cgu.0"
target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
target triple = "amdgcn-amd-amdhsa"

; Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn memory(argmem: write)
define amdgpu_kernel void @kernel_1(ptr noundef writeonly captures(none) %x) unnamed_addr #0 {
start:
  %0 = addrspacecast ptr %x to ptr addrspace(1)
  store float 2.100000e+01, ptr addrspace(1) %0, align 4
  ret void
}

attributes #0 = { mustprogress nofree norecurse nosync nounwind willreturn memory(argmem: write) "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,8" "target-cpu"="gfx90a" "uniform-work-group-size"="false" }

!llvm.module.flags = !{!0}
!llvm.ident = !{!1, !1, !1}

!0 = !{i32 8, !"PIC Level", i32 2}
!1 = !{!"rustc version 1.91.0-nightly (b6401657d 2025-08-12)"}

and my host.ll:

device.ll.txt

I have manually updated the ModuleID and source_filename to match the naming conventions of clang, without success.

[jhuber6]

Is that textual LLVM-IR? I don't think that the linker handles that, I'd expect LLVM-bitcode.

[kevinsala]

Is this correct --image=file=device.ll,[...] in the clang-offload-packager command? It probably needs the .bc.

[ZuseZ4]

Thanks you two, I somehow assumed that .bc and .ll would be interchangeable. I'll give it a try.

[jhuber6]

Thanks you two, I somehow assumed that .bc and .ll would be interchangeable. I'll give it a try.

LLVM bitcode and ELF have magic bytes so the implementation can tell what it is by looking at it. The textual format is text so we can't determine what it is by just reading a few bytes into the file.

[ZuseZ4]

That's sad, but also makes sense. I had switched back and forth between .ll and .bc since rustc doesn't fully generate the correct IR yet so I had to verify or correct things by hand. I guess at some point I started to get them mixed up.