Differences in LLDB and GDB Behavior When Stepping Into Inline Functions Twice

[edumoot]

We found that:

• When debugging a binary compiled with -O3, -O2, or -O1, GDB only steps into the first instance of an inline function, while LLDB steps in twice.
• Additionally, GDB halts at line 25 and LLDB at line 26, even though neither line appears in the debug line table.

Steps to Reproduce:

1. Compile and debug the binary in LLVM 19.1.7 context.

$clang -g -O3 6.c -o 6_O3.out

$lldb
(lldb) file 6_O3.out
(lldb) b main
(lldb) r
* thread #1, name = '6_O3.out', stop reason = breakpoint 1.1
    frame #0: 0x0000555555555130 6_O3.out`main [inlined] compute_result at 6.c:19:16
   16  	    int32_t secondary_var = -9L;
   17  	
   18  	    global_counter = local_values[7];
-> 19  	    temp_var = global_array[1] % 0xFCL;  
   20  	    secondary_var &= (temp_var | (++global_value));
   21  	    return global_union;
   22  	}
(lldb) s

* thread #1, name = '6_O3.out', stop reason = step in
    frame #0: 0x0000555555555136 6_O3.out`main at 6.c:26:19
   23  	
   24  	int main(void) {
   25  	    int result0 = compute_result().f0;
-> 26  	    int result1 = compute_result().f1;
   27  	    global_union.f0++;
   28  	    return result0 == result1;
   29  	}
(lldb) s

* thread #1, name = '6_O3.out', stop reason = step in
    frame #0: 0x0000555555555136 6_O3.out`main [inlined] compute_result at 6.c:19:16
   16  	    int32_t secondary_var = -9L;
   17  	
   18  	    global_counter = local_values[7];
-> 19  	    temp_var = global_array[1] % 0xFCL;  
   20  	    secondary_var &= (temp_var | (++global_value));
   21  	    return global_union;
   22  	}
(lldb) s

* thread #1, name = '6_O3.out', stop reason = step in
    frame #0: 0x000055555555513c 6_O3.out`main [inlined] compute_result at 6.c:20:35
   17  	
   18  	    global_counter = local_values[7];
   19  	    temp_var = global_array[1] % 0xFCL;  
-> 20  	    secondary_var &= (temp_var | (++global_value));
   21  	    return global_union;
   22  	}
   23  
(lldb) s

* thread #1, name = '6_O3.out', stop reason = step in
    frame #0: 0x0000555555555136 6_O3.out`main at 6.c:26:19
   23  	
   24  	int main(void) {
   25  	    int result0 = compute_result().f0;
-> 26  	    int result1 = compute_result().f1;
   27  	    global_union.f0++;
   28  	    return result0 == result1;
   29  	}

2. Debug the binary using GDB 15.

$ gdb
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

(gdb) file 6_O3.out
(gdb) b main
(gdb) r
[...]
Breakpoint 1, main () at 6.c:25
25	    int result0 = compute_result().f0;
(gdb) s
compute_result () at 6.c:19
19	    temp_var = global_array[1] % 0xFCL;  
(gdb) s
compute_result () at 6.c:20
20	    secondary_var &= (temp_var | (++global_value));
(gdb) s
main () at 6.c:27
27	    global_union.f0++;
(gdb) s
28	    return result0 == result1;

3. line information

Address            Line   Column File   ISA Discriminator OpIndex Flags
------------------ ------ ------ ------ --- ------------- ------- -------------
0x0000000000001130     19     16      0   0             0       0  is_stmt prologue_end
0x0000000000001136     19     16      0   0             0       0 
0x000000000000113c     20     35      0   0             0       0  is_stmt
0x0000000000001143     27     20      0   0             0       0  is_stmt
0x0000000000001149     28      5      0   0             0       0  is_stmt
0x000000000000114f     28      5      0   0             0       0  is_stmt end_sequence

4. cat 6.c

File: 6.c
01: #include <stdint.h>
02: 
03: union U1 {
04:     int8_t f0;
05:     const int8_t f1;
06: };
07: 
08: static uint16_t global_counter = 1UL;
09: static volatile uint32_t global_array[10] = {0UL};
10: static uint32_t global_value = 4294967286UL;
11: static union U1 global_union = {-2L};
12: 
13: static union U1 compute_result(void) { 
14:     int8_t local_values[8] = {0xA0L};  
15:     int32_t temp_var = 0x5BF2D3F5L;
16:     int32_t secondary_var = -9L;
17: 
18:     global_counter = local_values[7];
19:     temp_var = global_array[1] % 0xFCL;  
20:     secondary_var &= (temp_var | (++global_value));
21:     return global_union;
22: }
23: 
24: int main(void) {
25:     int result0 = compute_result().f0;
26:     int result1 = compute_result().f1;
27:     global_union.f0++;
28:     return result0 == result1;
29: }
30: 

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Yachao Zhu (edumoot)

We found that: - When debugging a binary compiled with -O3, -O2, or -O1, GDB only steps into the first instance of an inline function, while LLDB steps in twice. - Additionally, GDB halts at line 25 and LLDB at line 26, even though neither line appears in the debug line table.

Steps to Reproduce:

1. Compile and debug the binary in LLVM 19.1.7 context.

$clang -g -O3 6.c -o 6_O3.out

$lldb
(lldb) file 6_O3.out
(lldb) b main
(lldb) r
* thread #<!-- -->1, name = '6_O3.out', stop reason = breakpoint 1.1
    frame #<!-- -->0: 0x0000555555555130 6_O3.out`main [inlined] compute_result at 6.c:19:16
   16  	    int32_t secondary_var = -9L;
   17  	
   18  	    global_counter = local_values[7];
-&gt; 19  	    temp_var = global_array[1] % 0xFCL;  
   20  	    secondary_var &amp;= (temp_var | (++global_value));
   21  	    return global_union;
   22  	}
(lldb) s

* thread #<!-- -->1, name = '6_O3.out', stop reason = step in
    frame #<!-- -->0: 0x0000555555555136 6_O3.out`main at 6.c:26:19
   23  	
   24  	int main(void) {
   25  	    int result0 = compute_result().f0;
-&gt; 26  	    int result1 = compute_result().f1;
   27  	    global_union.f0++;
   28  	    return result0 == result1;
   29  	}
(lldb) s

* thread #<!-- -->1, name = '6_O3.out', stop reason = step in
    frame #<!-- -->0: 0x0000555555555136 6_O3.out`main [inlined] compute_result at 6.c:19:16
   16  	    int32_t secondary_var = -9L;
   17  	
   18  	    global_counter = local_values[7];
-&gt; 19  	    temp_var = global_array[1] % 0xFCL;  
   20  	    secondary_var &amp;= (temp_var | (++global_value));
   21  	    return global_union;
   22  	}
(lldb) s

* thread #<!-- -->1, name = '6_O3.out', stop reason = step in
    frame #<!-- -->0: 0x000055555555513c 6_O3.out`main [inlined] compute_result at 6.c:20:35
   17  	
   18  	    global_counter = local_values[7];
   19  	    temp_var = global_array[1] % 0xFCL;  
-&gt; 20  	    secondary_var &amp;= (temp_var | (++global_value));
   21  	    return global_union;
   22  	}
   23  
(lldb) s

* thread #<!-- -->1, name = '6_O3.out', stop reason = step in
    frame #<!-- -->0: 0x0000555555555136 6_O3.out`main at 6.c:26:19
   23  	
   24  	int main(void) {
   25  	    int result0 = compute_result().f0;
-&gt; 26  	    int result1 = compute_result().f1;
   27  	    global_union.f0++;
   28  	    return result0 == result1;
   29  	}

2. Debug the binary using GDB 15.

$ gdb
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

(gdb) file 6_O3.out
(gdb) b main
(gdb) r
[...]
Breakpoint 1, main () at 6.c:25
25	    int result0 = compute_result().f0;
(gdb) s
compute_result () at 6.c:19
19	    temp_var = global_array[1] % 0xFCL;  
(gdb) s
compute_result () at 6.c:20
20	    secondary_var &= (temp_var | (++global_value));
(gdb) s
main () at 6.c:27
27	    global_union.f0++;
(gdb) s
28	    return result0 == result1;

3. line information

Address            Line   Column File   ISA Discriminator OpIndex Flags
------------------ ------ ------ ------ --- ------------- ------- -------------
0x0000000000001130     19     16      0   0             0       0  is_stmt prologue_end
0x0000000000001136     19     16      0   0             0       0 
0x000000000000113c     20     35      0   0             0       0  is_stmt
0x0000000000001143     27     20      0   0             0       0  is_stmt
0x0000000000001149     28      5      0   0             0       0  is_stmt
0x000000000000114f     28      5      0   0             0       0  is_stmt end_sequence

4. cat 6.c

File: 6.c
01: #include <stdint.h>
02: 
03: union U1 {
04:     int8_t f0;
05:     const int8_t f1;
06: };
07: 
08: static uint16_t global_counter = 1UL;
09: static volatile uint32_t global_array[10] = {0UL};
10: static uint32_t global_value = 4294967286UL;
11: static union U1 global_union = {-2L};
12: 
13: static union U1 compute_result(void) { 
14:     int8_t local_values[8] = {0xA0L};  
15:     int32_t temp_var = 0x5BF2D3F5L;
16:     int32_t secondary_var = -9L;
17: 
18:     global_counter = local_values[7];
19:     temp_var = global_array[1] % 0xFCL;  
20:     secondary_var &= (temp_var | (++global_value));
21:     return global_union;
22: }
23: 
24: int main(void) {
25:     int result0 = compute_result().f0;
26:     int result1 = compute_result().f1;
27:     global_union.f0++;
28:     return result0 == result1;
29: }
30: 

[edumoot]

The questions is: When the debugger encounters an inline function, should it stop at the caller function's line to indicate the function call, or step directly into the callee function?

Neither LLDB nor GDB can directly stop at the function's line to indicate the function call by setting a breakpoint at the specified line number, specifically at line 25 or line 26 in this case, because these lines are not included in the line table.

(lldb) b 25
Breakpoint 1: where = 6_O3.out`main + 19 at 6.c:27:20, address = 0x0000000000001143
(lldb) br delete

(lldb) b 26
Breakpoint 2: where = 6_O3.out`main + 19 at 6.c:27:20, address = 0x0000000000001143


(gdb) b 25
Breakpoint 6 at 0x555555555143: file 6.c, line 27.
(gdb) br delete
(gdb) b 26
Breakpoint 5 at 0x555555555143: file 6.c, line 27.

The core question becomes: whether, when applying inline optimization and generating debug information, these kinds of line numbers should be included in the line table? @dwblaikie

[dwblaikie]

The questions is: When the debugger encounters an inline function, should it stop at the caller function's line to indicate the function call, or step directly into the callee function?

As an abstract ideal - yes, we should stop at the caller, then the callee. Put simply, we should be able to navigate the program as if inlining hadn't happened, as much as possible. This is true with other optimizations, etc (source locations - we show that the variable is in a register, a constant, back in memory, etc, such that the user mostly doesn't have to know about those details - same with inlining).

Today, a sufficiently heroic DWARF consumer could implement this, but it's not easy given what DWARF provides - eg: the line table doesn't encode these extra locations, so a consumer would need to parse the DWARF DIE tree to discover the extra file/line locations and use those for extra breakpoints.

Variable locations also don't have a way to express different values at the same instruction location - GCC's location view extension (being standardized in DWARFv6 if I recall correctly) may provide a way to do this.

The core question becomes: whether, when applying inline optimization and generating debug information, these kinds of line numbers should be included in the line table? @dwblaikie

Fundamentally, imo, the line table can't encode these line numbers - the line table only encodes one line per instruction, so if there's no instruction associated with the line (or the instruction is ambiguous/has multiple lines associated with it) - the DWARF line table can't represent that.

LLDB does manage to provide something like location "views" - doing some of the heroics, but not going all the way (as you found with trying to set a breakpoint at the line directly)

I don't think this is a bug in GDB or LLDB - it could be a feature request for GDB, but they could reasonably decline to implement it due to priorities or because perhaps they don't like the inconsistency around breakpoints that it introduces/you observe in LLDB.