[BOLT] Instrumented aarch64 clang-21 crashes with illegal instruction or hangs #153123
Description
When building LLVM 21.1.0-rc2 for kernel.org, I have noticed an infinite loop or Illegal instruction crash when running a clang binary instrumented by BOLT on some AArch64 machines (but not others).
$ llvm-bolt --instrument --instrumentation-file=/tmp/clang.fdata --instrumentation-file-append-pid -o clang.inst clang-21
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: d8e9216c27b82b4292e83437d58aebf594adb111
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x6c00000, offset 0x6c00000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-WARNING: 1 collisions detected while hashing binary objects. Use -v=1 to see the list.
BOLT-INFO: number of removed linker-inserted veneers: 0
BOLT-INFO: 0 out of 129351 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 47228
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 127221
BOLT-INSTRUMENTER: Number of function descriptors: 127221
BOLT-INSTRUMENTER: Number of branch counters: 1381499
BOLT-INSTRUMENTER: Number of ST leaf node counters: 678042
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 2059541
BOLT-INSTRUMENTER: Total size of counters: 16476328 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 14682724 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 144601856 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /tmp/clang.fdata
BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 3810 stubs in the hot area and 0 stubs in the cold area. Shared 64123 times, iterated 4 times.
BOLT-INFO: padding code to 0x10400000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x121639fc
BOLT-INFO: clear procedure is 0x121600f0
BOLT-INFO: setting __bolt_runtime_start to 0x121639fc
BOLT-INFO: setting __bolt_runtime_fini to 0x12163a8c
BOLT-INFO: setting __hot_start to 0x6e00000
BOLT-INFO: setting __hot_end to 0x10318b94
Using this same exact binary...
On my Honeycomb LX2, which has an SOC with Cortex-A72 cores:
$ ./clang-21 --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp
$ ./clang.inst --version
fish: Job 1, './clang.inst --version' terminated by signal SIGILL (Illegal instruction)
On an Ampere Altra system, which has Neoverse-N1 cores:
$ ./clang-21 --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp
$ ./clang.inst --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp
In a virtual machine on my M1 Max Mac Studio:
$ ./clang-21 --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/nathan
$ ./clang.inst --version
<hangs indefinitely>
I have uploaded the original and instrumented binaries above here, compressed with zstd -19 to stay under GitHub's single file size limit.