[LLDB] `si` (step instruction) stops multiple times at the same PC starting from LLDB 21

[patryk4815]

Starting with LLDB 21, the si (step instruction) command may stop multiple times at the same program counter (PC).
This behavior was not present in earlier releases (e.g. LLDB 16-20), where only a single stop event would occur.

Compile bug binary

cat << EOF > bug.S
.section .text
.global _start
_start:
mov x1, 1
foo:
mov x1, 2
bar:
mov x1, 3
EOF

zig cc -target aarch64-freestanding bug.S -o bug.elf

Repro

$ lldb ./bug.elf
(lldb) target create "./bug.elf"
Current executable set to '/Users/psondej/projekty/pwndbg/bug.elf' (aarch64).
(lldb) version
lldb version 21.1.1
(lldb) b foo
Breakpoint 1: where = bug.elf`foo, address = 0x0000000001010124
(lldb) b bar
Breakpoint 2: where = bug.elf`bar, address = 0x0000000001010128
(lldb) process launch -s
Process 353436 stopped
* thread #1, name = 'bug.elf', stop reason = signal SIGSTOP
    frame #0: 0x0000000001010120 bug.elf`_start at bug.S:4
   1   	.section .text
   2   	.global _start
   3   	_start:
-> 4   	mov x1, 1
   5   	foo:
   6   	mov x1, 2
   7   	bar:
Process 353436 launched: '/Users/psondej/projekty/pwndbg/bug.elf' (aarch64)
(lldb) si
Process 353436 stopped
* thread #1, name = 'bug.elf', stop reason = instruction step into
    frame #0: 0x0000000001010124 bug.elf`foo at bug.S:6
   3   	_start:
   4   	mov x1, 1
   5   	foo:
-> 6   	mov x1, 2
   7   	bar:
   8   	mov x1, 3
(lldb) si
Process 353436 stopped
* thread #1, name = 'bug.elf', stop reason = breakpoint 1.1
    frame #0: 0x0000000001010124 bug.elf`foo at bug.S:6
   3   	_start:
   4   	mov x1, 1
   5   	foo:
-> 6   	mov x1, 2
   7   	bar:
   8   	mov x1, 3
>>>

Actual behavior:

The debugger stops multiple times at the same instruction:

• once due to the step
• once again due to the breakpoint at the same address

Expected

Im not sure what is expected. Maybe si should trigger a stop only once per instruction, not multiple times at the same PC.

Maybe related with:

• [lldb] Inconsistent behaviour when stepping onto a breakpoint instruction #156650

[llvmbot]

@llvm/issue-subscribers-lldb

Author: None (patryk4815)

Starting with LLDB 21, the `si` (step instruction) command may stop multiple times at the same program counter (PC). This behavior was not present in earlier releases (e.g. LLDB 16-20), where only a single stop event would occur.

Compile bug binary

cat << EOF > bug.S
.section .text
.global _start
_start:
mov x1, 1
foo:
mov x1, 2
bar:
mov x1, 3
EOF

zig cc -target aarch64-freestanding bug.S -o bug.elf

Repro

$ lldb ./bug.elf
(lldb) target create "./bug.elf"
Current executable set to '/Users/psondej/projekty/pwndbg/bug.elf' (aarch64).
(lldb) version
lldb version 21.1.1
(lldb) b foo
Breakpoint 1: where = bug.elf`foo, address = 0x0000000001010124
(lldb) b bar
Breakpoint 2: where = bug.elf`bar, address = 0x0000000001010128
(lldb) process launch -s
Process 353436 stopped
* thread #<!-- -->1, name = 'bug.elf', stop reason = signal SIGSTOP
    frame #<!-- -->0: 0x0000000001010120 bug.elf`_start at bug.S:4
   1   	.section .text
   2   	.global _start
   3   	_start:
-&gt; 4   	mov x1, 1
   5   	foo:
   6   	mov x1, 2
   7   	bar:
Process 353436 launched: '/Users/psondej/projekty/pwndbg/bug.elf' (aarch64)
(lldb) si
Process 353436 stopped
* thread #<!-- -->1, name = 'bug.elf', stop reason = instruction step into
    frame #<!-- -->0: 0x0000000001010124 bug.elf`foo at bug.S:6
   3   	_start:
   4   	mov x1, 1
   5   	foo:
-&gt; 6   	mov x1, 2
   7   	bar:
   8   	mov x1, 3
(lldb) si
Process 353436 stopped
* thread #<!-- -->1, name = 'bug.elf', stop reason = breakpoint 1.1
    frame #<!-- -->0: 0x0000000001010124 bug.elf`foo at bug.S:6
   3   	_start:
   4   	mov x1, 1
   5   	foo:
-&gt; 6   	mov x1, 2
   7   	bar:
   8   	mov x1, 3
&gt;&gt;&gt;

Actual behavior:

The debugger stops multiple times at the same instruction:

• once due to the step
• once again due to the breakpoint at the same address

Expected

si should trigger a stop only once per instruction, not multiple times at the same PC.

[DavidSpickett]

Reproduces on Linux.

.section .text
.globl main
.type main, %function
main:
mov x1, 1
foo:
mov x1, 2
bar:
mov x1, 3

$ gcc /tmp/test.s -o /tmp/test.o -g

$ ./bin/lldb /tmp/test.o
(lldb) target create "/tmp/test.o"
Current executable set to '/tmp/test.o' (aarch64).
(lldb) b main
Breakpoint 1: where = test.o`main, address = 0x0000000000000714
(lldb) b foo
Breakpoint 2: where = test.o`foo, address = 0x0000000000000718
(lldb) b bar
Breakpoint 3: where = test.o`bar, address = 0x000000000000071c
(lldb) run
Process 2868479 launched: '/tmp/test.o' (aarch64)
Process 2868479 stopped
* thread #1, name = 'test.o', stop reason = breakpoint 1.1
    frame #0: 0x0000aaaaaaaa0714 test.o`main at test.s:5
   2   	.globl main
   3   	.type main, %function
   4   	main:
-> 5   	mov x1, 1
   6   	foo:
   7   	mov x1, 2
   8   	bar:
(lldb) si
Process 2868479 stopped
* thread #1, name = 'test.o', stop reason = instruction step into
    frame #0: 0x0000aaaaaaaa0718 test.o`foo at test.s:7
   4   	main:
   5   	mov x1, 1
   6   	foo:
-> 7   	mov x1, 2
   8   	bar:
   9   	mov x1, 3
(lldb) 
Process 2868479 stopped
* thread #1, name = 'test.o', stop reason = breakpoint 2.1
    frame #0: 0x0000aaaaaaaa0718 test.o`foo at test.s:7
   4   	main:
   5   	mov x1, 1
   6   	foo:
-> 7   	mov x1, 2
   8   	bar:
   9   	mov x1, 3
(lldb) 
Process 2868479 stopped
* thread #1, name = 'test.o', stop reason = instruction step into
    frame #0: 0x0000aaaaaaaa071c test.o`bar at test.s:9
   6   	foo:
   7   	mov x1, 2
   8   	bar:
-> 9   	mov x1, 3
(lldb) 

ni (next instruction) does not do this.

[DavidSpickett]

$ git bisect bad
b666ac3b63e01bfa602318c877ea2395fea53f89 is the first bad commit
commit b666ac3b63e01bfa602318c877ea2395fea53f89
Author: Jason Molenda <jmolenda@apple.com>
Date:   Thu Feb 13 11:30:10 2025 -0800

    [lldb] Change lldb's breakpoint handling behavior, reland (#126988)

#126988

Same energy as #156650 for sure but the reproducer there does not change behaviour when this change is removed.

From the report there:

As an aside, the rationale for the intended behaviour makes sense (LLDB does not want to "lose" stop reasons)

Same type of deal here. One stop is us finishing the step, the other is us hitting the breakpoint. We could think of them as one event that combines the two things or two events.

From the PR description:

In this patch, I change lldb to only set a thread's stop reason to breakpoint-hit when we've actually executed the instruction/triggered the breakpoint.

Sounds like exactly what is happening here.

@jasonmolenda sounds like your change is acting as expected, right?

[jimingham]

We've also been discussing this here

#156650

The main problem is that with a fuzzy rule for "when to consider a breakpoint hit" makes reasoning about when to count the breakpoint as hit difficult, and leads inaccurate hit counts when debugging multithreaded programs. So we introduced the rule that "we only count the breakpoint as hit when the trap is executed." Step-i steps to the next instruction but doesn't execute it, so that doesn't count as a breakpoint hit.

This made all the accounting solid and we don't get lots of flakey failures in our concurrent breakpoint tests anymore. This was the one wart that remained.

However, we felt that was worth it to get a setup we could make robust. And if you know the rule it makes sense what's going on - it is predictable.

It also has the benefit that lldb is more accurately doing what you told it to. For instance, when you step over an instruction, you've only executed that instruction, not the one after it with the breakpoint. So it would be kind of weird to trigger a breakpoint on an instruction you haven't yet executed...

[jimingham]

BTW, one way to get the behavior you want would be to change step-i so that if it has changed the PC, and to a PC that contains a breakpoint, it also executes the next instruction, triggering the trap. That way we wouldn't have to make the process of breakpoint accounting harder, but you'd still see the behavior you want, which is stepping to an instruction with a breakpoint also triggers the breakpoint.