[BOLT][AArch64] Application crash with segment fault after bolt instrument sometimes

[mrzhouyh]

My application crash with segment fault after BOLT instrument on arm64 ( Kunpeng 920 5250 acctually ), this issue happens somtimes, not always. And it never appear on x86.

the intrument command is :
llvm-bolt $bin -o ${BOLTDIR}/${filename}.inst -instrument --instrumentation-file=${BOLTDIR}/${filename}.fdata

After check the core file, it seems that segment fault raised when TLS accessed like:
`__thread int hg_trigger_success = 0;

void func() {
/** do something /
hg_trigger_success = 1; /* segment fault here*/
/** do something */
}
`

0x0000ffffbefa96f4 <+1396>: msr nzcv, x1
0x0000ffffbefa96f8 <+1400>: ldp x0, x1, [sp], #16
0x0000ffffbefa96fc <+1404>: b 0xffffbefa9778 <crt_hg_progress+1528>
0x0000ffffbefa9700 <+1408>: cbnz w27, 0xffffbefaa5a4 <crt_hg_progress+5156>
0x0000ffffbefa9704 <+1412>: mrs x16, tpidr_el0
0x0000ffffbefa9708 <+1416>: adrp x0, 0xffffbea90000 crt_hlc_localtime_get2@got.plt
0x0000ffffbefa970c <+1420>: ldr x3, [x0, #3904]
0x0000ffffbefa9710 <+1424>: add x0, x0, #0xf40
0x0000ffffbefa9714 <+1428>: stp x0, x1, [sp, #-16]!
0x0000ffffbefa9718 <+1432>: mov x0, x3
0x0000ffffbefa971c <+1436>: movk x1, #0x0, lsl #48
0x0000ffffbefa9720 <+1440>: movk x1, #0x0, lsl #32
0x0000ffffbefa9724 <+1444>: movk x1, #0x0, lsl #16
0x0000ffffbefa9728 <+1448>: movk x1, #0x264
0x0000ffffbefa972c <+1452>: stp x0, x1, [sp, #-16]!
0x0000ffffbefa9730 <+1456>: adrp x0, 0xffffbf22c000 <swim_updates_parse+18100>
--Type for more, q to quit, c to continue without paging--
0x0000ffffbefa9734 <+1460>: add x0, x0, #0x594
0x0000ffffbefa9738 <+1464>: blr x0
0x0000ffffbefa973c <+1468>: mov w17, #0x1 // #1
0x0000ffffbefa9740 <+1472>: subs w26, w26, w7
=> 0x0000ffffbefa9744 <+1476>: str w17, [x16, x0]
0x0000ffffbefa9748 <+1480>: b.eq 0xffffbefa9778 <crt_hg_progress+1528> // b.none
0x0000ffffbefa974c <+1484>: stp x0, x1, [sp, #-16]!
0x0000ffffbefa9750 <+1488>: mrs x1, nzcv

Is here anybody know why this issus happen, and how to fix it?

[llvmbot]

@llvm/issue-subscribers-bolt

Author: None (mrzhouyh)

My application crash with segment fault after BOLT instrument on arm64 ( Kunpeng 920 5250 acctually ), this issue happens somtimes, not always. And it never appear on x86.

the intrument command is :
llvm-bolt $bin -o ${BOLTDIR}/${filename}.inst -instrument --instrumentation-file=${BOLTDIR}/${filename}.fdata

After check the core file, it seems that segment fault raised when TLS accessed like:
`__thread int hg_trigger_success = 0;

void func() {
/** do something /
hg_trigger_success = 1; /* segment fault here*/
/** do something */
}
`

0x0000ffffbefa96f4 &lt;+1396&gt;: msr nzcv, x1 0x0000ffffbefa96f8 &lt;+1400&gt;: ldp x0, x1, [sp], #<!-- -->16 0x0000ffffbefa96fc &lt;+1404&gt;: b 0xffffbefa9778 &lt;crt_hg_progress+1528&gt; 0x0000ffffbefa9700 &lt;+1408&gt;: cbnz w27, 0xffffbefaa5a4 &lt;crt_hg_progress+5156&gt; 0x0000ffffbefa9704 &lt;+1412&gt;: mrs x16, **tpidr_el0** 0x0000ffffbefa9708 &lt;+1416&gt;: adrp x0, 0xffffbea90000 &lt;crt_hlc_localtime_get2@<!-- -->got.plt&gt; 0x0000ffffbefa970c &lt;+1420&gt;: ldr x3, [x0, #<!-- -->3904] 0x0000ffffbefa9710 &lt;+1424&gt;: add x0, x0, #<!-- -->0xf40 0x0000ffffbefa9714 &lt;+1428&gt;: stp x0, x1, [sp, #-16]! 0x0000ffffbefa9718 &lt;+1432&gt;: mov x0, x3 0x0000ffffbefa971c &lt;+1436&gt;: movk x1, #<!-- -->0x0, lsl #<!-- -->48 0x0000ffffbefa9720 &lt;+1440&gt;: movk x1, #<!-- -->0x0, lsl #<!-- -->32 0x0000ffffbefa9724 &lt;+1444&gt;: movk x1, #<!-- -->0x0, lsl #<!-- -->16 0x0000ffffbefa9728 &lt;+1448&gt;: movk x1, #<!-- -->0x264 0x0000ffffbefa972c &lt;+1452&gt;: stp x0, x1, [sp, #-16]! 0x0000ffffbefa9730 &lt;+1456&gt;: adrp x0, 0xffffbf22c000 &lt;swim_updates_parse+18100&gt; --Type &lt;RET&gt; for more, q to quit, c to continue without paging-- 0x0000ffffbefa9734 &lt;+1460&gt;: add x0, x0, #<!-- -->0x594 0x0000ffffbefa9738 &lt;+1464&gt;: blr x0 0x0000ffffbefa973c &lt;+1468&gt;: mov w17, #<!-- -->0x1 // #<!-- -->1 0x0000ffffbefa9740 &lt;+1472&gt;: subs w26, w26, w7 =&gt; 0x0000ffffbefa9744 &lt;+1476&gt;: str w17, [**x16**, x0] 0x0000ffffbefa9748 &lt;+1480&gt;: b.eq 0xffffbefa9778 &lt;crt_hg_progress+1528&gt; // b.none 0x0000ffffbefa974c &lt;+1484&gt;: stp x0, x1, [sp, #-16]! 0x0000ffffbefa9750 &lt;+1488&gt;: mrs x1, nzcv

Is here anybody know why this issus happen, and how to fix it?

[aaupov]

Does the binary have relocations preserved with -Wl,--emit-relocs?
Can you please check if TLS access relocation is correctly updated, by dumping the containing function/code range with objdump -r before and after BOLT

[bgergely0]

Can use also post the full output BOLT produced while instrumenting your binary?

[yavtuk]

@mrzhouyh hello, can you check with PR

#141918

=> 0x0000ffffbefa9744 <+1476>: str w17, [x16, x0]

instrumentation snippet can overwrite x16

[mrzhouyh]

Does the binary have relocations preserved with -Wl,--emit-relocs? Can you please check if TLS access relocation is correctly updated, by dumping the containing function/code range with objdump -r before and after BOLT

@aaupov Thanks for your advice. Yes, I'm sure the binary have relocations. Here is dump info:

before:

[root@node22 /]# readelf -r /usr/lib64/libcart.so | grep TLS
000000420fd0  027200000407 R_AARCH64_TLSDESC 00000000000000b8 __gcov_indirect_call + 0
000000420fe0  02b400000407 R_AARCH64_TLSDESC 0000000000000038 hg_trigger_success + 0
000000420fc0  033700000407 R_AARCH64_TLSDESC 0000000000000030 crt_progress_cnt + 0
000000420f80  000000000407 R_AARCH64_TLSDESC                    40
000000420f90  000000000407 R_AARCH64_TLSDESC                    0
000000420fa0  000000000407 R_AARCH64_TLSDESC                    60
000000420fb0  000000000407 R_AARCH64_TLSDESC                    98

objdump -r -d $bin:
000000000016edb0 <crt_hg_progress>:
**
  16edc4:       a9056bf9        stp     x25, x26, [sp, #80]
  16edc8:       d53bd059        mrs     x25, tpidr_el0
**
  16edf4:       f8606b20        ldr     x0, [x25, x0]
**
  16ef64:       d0001580        adrp    x0, 420000 <crt_hlc_localtime_get2@@Base+0x84560>
  16ef68:       f947f002        ldr     x2, [x0, #4064]
  16ef6c:       913f8000        add     x0, x0, #0xfe0
  16ef70:       d63f0040        blr     x2
  16ef74:       910da2f7        add     x23, x23, #0x368
  16ef78:       52800022        mov     w2, #0x1                        // #1
  16ef7c:       b8206b22        str     w2, [x25, x0]

after:

[root@node21 /]# readelf -r /usr/lib64/libcart.so | grep TLS
000000200ef0  000000000407 R_AARCH64_TLSDESC                    40
000000200f00  000000000407 R_AARCH64_TLSDESC                    0
000000200f10  000000000407 R_AARCH64_TLSDESC                    60
000000200f20  000000000407 R_AARCH64_TLSDESC                    98
000000200f30  031c00000407 R_AARCH64_TLSDESC 0000000000000030 crt_progress_cnt + 0
000000200f40  029d00000407 R_AARCH64_TLSDESC 0000000000000038 hg_trigger_success + 0

objdump -r -d $bin:
0000000000758440 <crt_hg_progress>:
**
  7589c4:       d53bd050        mrs     x16, tpidr_el0
**
  7589ec:       a9bf07e0        stp     x0, x1, [sp, #-16]!
  7589f0:       90001260        adrp    x0, 9a4000 <swim_updates_parse+0x46b4>
  7589f4:       91165000        add     x0, x0, #0x594
  7589f8:       d63f0000        blr     x0
  7589fc:       52800031        mov     w17, #0x1                       // #1
  758a00:       6b07035a        subs    w26, w26, w7
  758a04:       b8206a11        str     w17, [x16, x0]
  758a08:       54000180        b.eq    758a38 <crt_hg_progress+0x5f8>  // b.none

Can you help me look up useful infos? I not good at this, thank you.

[mrzhouyh]

@mrzhouyh hello, can you check with PR

#141918

=> 0x0000ffffbefa9744 <+1476>: str w17, [x16, x0]

instrumentation snippet can overwrite x16

@yavtuk Thank you, it seems the same issue, i try to apply the path to tag 21.1.2, a compile error raise:

llvm-project-llvmorg-21.1.2/bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp:1966: error: 'VK_None' is not a member of 'llvm::MCSymbolRefExpr'

I add VK_None in MCSymbolRefExpr manually to fix it. I will try to retest later.

[mrzhouyh]

Can use also post the full output BOLT produced while instrumenting your binary?

@bgergely0 Since llvm-bolt is executing multiple .so concurrently, the output got mixed up. I'll modify the script and run it again later, then post the complete output log.

[mrzhouyh]

@mrzhouyh hello, can you check with PR
#141918
=> 0x0000ffffbefa9744 <+1476>: str w17, [x16, x0]
instrumentation snippet can overwrite x16

@yavtuk Thank you, it seems the same issue, i try to apply the path to tag 21.1.2, a compile error raise:

llvm-project-llvmorg-21.1.2/bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp:1966: error: 'VK_None' is not a member of 'llvm::MCSymbolRefExpr'

I add VK_None in MCSymbolRefExpr manually to fix it. I will try to retest later.

This patch seems to have fixed my issue . I will keep monitoring it for a few more days. Thanks for your help. @yavtuk @bgergely0 @aaupov

[paschalis-mpeis]

Let's keep this open until we land @yavtuk patch.