how to enable --instrument bolt runtime library bolt_rt_instr.a in x86_64 when compiling llvm-bolt in x86

[zhouyu0502]

My goal is to use llvm-bolt --instrument a binary executable（from arm64）on an x86_64 server
I ran into this issue. Here are the steps I took:
我的目的是为arm64 架构的设备执行 二进制文件的 插装，遇到了这个问题，以下是我的操作步骤

1. 在x86_64的服务器上编译好的 llvm_bolt 和 arm64架构上的可执行文件binfromarm64
2. 在服务器上执行./llvm-bolt binfromarm64 -instrument -o binfromarm64_instr
   BOLT-INFO: shared object or position-independent executable detected
   BOLT-INFO: Target architecture: aarch64
   BOLT-INFO: BOLT version: 6abb626
   BOLT-INFO: first alloc address is 0x0
   BOLT-INFO: creating new program header table at address 0x200000, offset 0x200000
   BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
   BOLT-INFO: enabling relocation mode
   BOLT-INFO: forcing -jump-tables=move for instrumentation
   BOLT-INFO: number of removed linker-inserted veneers: 0
   BOLT-INFO: 0 out of 12 functions in the binary (0.0%) have non-empty execution profile
   BOLT-INSTRUMENTER: Number of indirect call site descriptors: 0
   BOLT-INSTRUMENTER: Number of indirect call target descriptors: 9
   BOLT-INSTRUMENTER: Number of function descriptors: 9
   BOLT-INSTRUMENTER: Number of branch counters: 9
   BOLT-INSTRUMENTER: Number of ST leaf node counters: 17
   BOLT-INSTRUMENTER: Number of direct call counters: 0
   BOLT-INSTRUMENTER: Total number of counters: 26
   BOLT-INSTRUMENTER: Total size of counters: 208 bytes (static alloc memory)
   BOLT-INSTRUMENTER: Total size of string table emitted: 237 bytes in file
   BOLT-INSTRUMENTER: Total size of descriptors: 2056 bytes in file
   BOLT-INSTRUMENTER: Profile will be saved to file /tmp/prof.fdata
   BOLT-INFO: Starting stub-insertion pass
   BOLT-INFO: Inserted 0 stubs in the hot area and 0 stubs in the cold area. Shared 0 times, iterated 1 times.
   BOLT-INFO: padding code to 0x600000 to accommodate hot text
   BOLT-ERROR: linking object with arch x86_64 into context with arch aarch64
   错误的原因是编出来的libbolt_rt_instr.a 是x86_64架构的，请问我该如何在x86服务器上编出支持的arm64架构的libbolt_rt_instr.a？我试过交叉编译，但是生成的插装后的可执行文件无法执行。
   The error is caused by the fact that the built libbolt_rt_instr.a is for the x86_64 architecture. How can I produce an libbolt_rt_instr.a that supports the arm64 architecture on an x86 server? I tried cross-compiling, but the resulting instrumented executable cannot run.

[llvmbot]

@llvm/issue-subscribers-bolt

Author: None (zhouyu0502)

My goal is to use llvm-bolt --instrument a binary executable（from arm64）on an x86_64 server I ran into this issue. Here are the steps I took: 我的目的是为arm64 架构的设备执行 二进制文件的 插装，遇到了这个问题，以下是我的操作步骤 1. 在x86_64的服务器上编译好的 llvm_bolt 和 arm64架构上的可执行文件binfromarm64 2. 在服务器上执行./llvm-bolt binfromarm64 -instrument -o binfromarm64_instr BOLT-INFO: shared object or position-independent executable detected BOLT-INFO: Target architecture: aarch64 BOLT-INFO: BOLT version: 6abb626 BOLT-INFO: first alloc address is 0x0 BOLT-INFO: creating new program header table at address 0x200000, offset 0x200000 BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it. BOLT-INFO: enabling relocation mode BOLT-INFO: forcing -jump-tables=move for instrumentation BOLT-INFO: number of removed linker-inserted veneers: 0 BOLT-INFO: 0 out of 12 functions in the binary (0.0%) have non-empty execution profile BOLT-INSTRUMENTER: Number of indirect call site descriptors: 0 BOLT-INSTRUMENTER: Number of indirect call target descriptors: 9 BOLT-INSTRUMENTER: Number of function descriptors: 9 BOLT-INSTRUMENTER: Number of branch counters: 9 BOLT-INSTRUMENTER: Number of ST leaf node counters: 17 BOLT-INSTRUMENTER: Number of direct call counters: 0 BOLT-INSTRUMENTER: Total number of counters: 26 BOLT-INSTRUMENTER: Total size of counters: 208 bytes (static alloc memory) BOLT-INSTRUMENTER: Total size of string table emitted: 237 bytes in file BOLT-INSTRUMENTER: Total size of descriptors: 2056 bytes in file BOLT-INSTRUMENTER: Profile will be saved to file /tmp/prof.fdata BOLT-INFO: Starting stub-insertion pass BOLT-INFO: Inserted 0 stubs in the hot area and 0 stubs in the cold area. Shared 0 times, iterated 1 times. BOLT-INFO: padding code to 0x600000 to accommodate hot text BOLT-ERROR: linking object with arch x86_64 into context with arch aarch64 错误的原因是编出来的libbolt_rt_instr.a 是x86_64架构的，请问我该如何在x86服务器上编出支持的arm64架构的libbolt_rt_instr.a？我试过交叉编译，但是生成的插装后的可执行文件无法执行。 The error is caused by the fact that the built libbolt_rt_instr.a is for the x86_64 architecture. How can I produce an libbolt_rt_instr.a that supports the arm64 architecture on an x86 server? I tried cross-compiling, but the resulting instrumented executable cannot run.

[bgergely0]

Hi!

You need to cross compile the libbolt_rt_instr.a library. I had this PR to enable it automatically: #101180

The patch is out of date however (it does not apply w/o conflicts).

But you should be able build it for aarch64 by putting --target=aarch64-linux-gnu in here, but keep in mind that the whole runtime/CMakeLists.txt file is not written with cross compilation in mind.

[zhouyu0502]

Hi!

You need to cross compile the libbolt_rt_instr.a library. I had this PR to enable it automatically: #101180

The patch is out of date however (it does not apply w/o conflicts).

But you should be able build it for aarch64 by putting --target=aarch64-linux-gnu in here, but keep in mind that the whole runtime/CMakeLists.txt file is not written with cross compilation in mind.

thanks for your answer i still have some questions
1.can i use PR #101180 ,is this still useful？
2.have tryed this change
step 1.add following changes in bolt/runtime/CMakeLists.txt

set(CMAKE_C_COMPILER /usr/bin/aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER /usr/bin/aarch64-linux-gnu-g++)

set(BOLT_RT_FLAGS
-target=aarch64-linux-gnu
step2.

[bgergely0]

1.can i use PR #101180 ,is this still useful？

You can try, but it does not work out of the box. With some work (merge conflicts), you might be able to use it.

The --target=... flag is only valid for Clang. As opposed to GCC compilers, Clang is a cross compiler by default. You can set CC=clang, and CXX=clang++ to use the --target flag, OR use the aarch64-linux-gnu-g++ cross compiler, but not both.

[zhouyu0502]

hello now have successfully compile bolt_rt in arm64
but i still come across some problems

1.llvm-bolt testbin -instrument -o test.instr.1

i cannot exe test.instr.1 in arm64, shows "Illegal instruction (core dumped)"
2.llvm-bolt testbin -instrumentation-file=/data/test.fdata -o test.instr.2

i can exe test.instr.2 but can not find test.fdata anyway i feel very confused

[bgergely0]

1. llvm-bolt testbin -instrument -o test.instr.1

Can you run the instrumented binary in gdb/lldb and post a backtrace of the crash?

Are you sure the instrumentation lib is aarch64? What is the diff you applied to the BOLT sources to get it to build for aarch64?

2. llvm-bolt testbin -instrumentation-file=/data/test.fdata -o test.instr.2

Based on the BOLT output below, the --instrument flag is also needed. I don't think setting zhe instrumentation-file is enough. You should see lines starting with BOLT-INSTRUMENTER, just like in the output of the first attempt.