[lldb] Inconsistent behaviour when stepping onto a breakpoint instruction

[SLTozer]

Reproducer steps:

int a(int) {return 2;}
int foo() {
  int x = a(1);
  return x;
}
int bar() {
  int x = 2;
  return x;
}
int main() {
  return foo() + bar();
}

Building this reproducer file with clang++ -O0 -g and setting breakpoints on all the lines in foo and bar (3, 4, 7, 8), we see the following behaviour: when we step on the first line in foo (line 3) and step over (n), we stop at line 4 with "stop reason = breakpoint 2.1 -2.1". When we continue (c) and arrive at the first line of bar (line 7), we step over again; this time, we stop at line 8 with "stop reason = step over", and when we continue again we stop at line 8 again with "stop reason = breakpoint 4.1". I've attached a bash log demonstrating this.

Firstly, this appears to be a regression; performing the same action with my system LLDB (version 18.1.8), in both cases I see just "stop reason = breakpoint 2.1" and "stop reason = breakpoint 4.1", with no negative breakpoints or step over visible (log also attached); though I haven't tested it, this likely started with the behaviour change in #105594, in which as far as I understand it the behaviour in bar is the intended outcome. Secondly, running with log enable lldb step, the divergence between the two results appears to start with ThreadPlanStepRange::SetNextBranchBreakpoint; in foo we see Setting breakpoint -2 (site 3) to run to address 0x555555555155, while in bar we see skipping invalid run_to_address; I've attached a full log-enabled session in case it's of any use.

As an aside, the rationale for the intended behaviour makes sense (LLDB does not want to "lose" stop reasons), but does create the need for workarounds in tooling, and feels slightly confusing in general use; is there any feasible way that when there are multiple reasons for stopping at an instruction, they can simply all be returned at once, rather than needing to use continue to advance across different stop reasons at the same instruction? The current behaviour does feel "odd", but it can certainly continue be worked around if there are fundamental limitations (and I'm grateful to @jasonmolenda for writing the current workaround for Dexter!).

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Stephen Tozer (SLTozer)

Reproducer steps: ```cpp int a(int) {return 2;} int foo() { int x = a(1); return x; } int bar() { int x = 2; return x; } int main() { return foo() + bar(); } ```

Building this reproducer file with clang++ -O0 -g and setting breakpoints on all the lines in foo and bar (3, 4, 7, 8), we see the following behaviour: when we step on the first line in foo (line 3) and step over (n), we stop at line 4 with "stop reason = breakpoint 2.1 -2.1". When we continue (c) and arrive at the first line of bar (line 7), we step over again; this time, we stop at line 8 with "stop reason = step over", and when we continue again we stop at line 8 again with "stop reason = breakpoint 4.1". I've attached a bash log demonstrating this.

Firstly, this appears to be a regression; performing the same action with my system LLDB (version 18.1.8), in both cases I see just "stop reason = breakpoint 2.1" and "stop reason = breakpoint 4.1", with no negative breakpoints or step over visible (log also attached); though I haven't tested it, this likely started with the behaviour change in #105594, in which as far as I understand it the behaviour in bar is the intended outcome. Secondly, running with log enable lldb step, the divergence between the two results appears to start with ThreadPlanStepRange::SetNextBranchBreakpoint; in foo we see Setting breakpoint -2 (site 3) to run to address 0x555555555155, while in bar we see skipping invalid run_to_address; I've attached a full log-enabled session in case it's of any use.

As an aside, the rationale for the intended behaviour makes sense (LLDB does not want to "lose" stop reasons), but does create the need for workarounds in tooling, and feels slightly confusing in general use; is there any feasible way that when there are multiple reasons for stopping at an instruction, they can simply all be returned at once, rather than needing to use continue to advance across different stop reasons at the same instruction? The current behaviour does feel "odd", but it can certainly continue be worked around if there are fundamental limitations (and I'm grateful to @jasonmolenda for writing the current workaround for Dexter!).

[jimingham]

The big change to lldb around this time was to track whether we had actually run the instruction we are stopped on or not, and only say the breakpoint was hit if the breakpoint instruction was run. For instance, if you step up to the first instruction of a new source line, but don't execute that instruction, even though the breakpoint is at the current pc, we don't yet want to say the breakpoint was hit. We only want to say that when you continue to actually run the first instruction on the new line.
Being accurate about that is necessary to get consistent reporting of breakpoint hits, particularly when you have multiple threads all stopping at breakpoints at the same time. So we don't want to lose that accuracy.

The overall point is that in these cases there aren't really multiple reasons for the stop to have happened. If you walk to an instruction but haven't executed it yet, the stop reasons is that your step process took you to that instruction. But since you haven't executed the trap instruction yet, you haven't yet stopped because of the breakpoint. You will only do that when you continue.

That also has some useful behaviors. If I step up to but haven't yet executed a line of code, I wouldn't want the breakpoint command on that instruction to trigger and continue the process out from under me.

I don't think we want to go back to the state where we are pretending we've executed breakpoint instructions we haven't yet.

[SLTozer]

If you walk to an instruction but haven't executed it yet, the stop reasons is that your step process took you to that instruction. But since you haven't executed the trap instruction yet, you haven't yet stopped because of the breakpoint.

This does help explain the rationale for getting "step over" before hitting the breakpoint. Following this logic also explains the inconsistent behaviour, I believe - from a single step command (n), LLDB may internally choose different strategies to perform the step, one of which involves setting an internal breakpoint (with a negative ID). Because we define hitting a breakpoint as executing the trap instruction, this means that we get different debugger outcomes based on the strategy chosen - if LLDB used an internal breakpoint, then that breakpoint is hit at the same time as the user breakpoint, and so we hit both breakpoints at once; if LLDB did not use an internal breakpoint, then we see "step over" before the breakpoint.

As an example, if I set a breakpoint command on the user breakpoint in the reproducer, then when I perform a step onto that breakpoint the command will indeed execute and continue the process without a stop:

(lldb) target create "repro"
Current executable set to '/home/gbtozers/dev/upstream-llvm/repro' (x86_64).
(lldb) b 3
Breakpoint 1: where = repro`foo() + 8 at repro.cpp:3:11, address = 0x0000000000001148
(lldb) breakpoint set --line 4 -C continue
Breakpoint 2: where = repro`foo() + 21 at repro.cpp:4:10, address = 0x0000000000001155
(lldb) r
Process 30764 launched: '/home/gbtozers/dev/upstream-llvm/repro' (x86_64)
Process 30764 stopped
* thread #1, name = 'repro', stop reason = breakpoint 1.1
    frame #0: 0x0000555555555148 repro`foo() at repro.cpp:3:11
   1    int a(int) {return 2;}
   2    int foo() {
-> 3      int x = a(1);
   4      return x;
   5    }
   6    int bar() {
   7      int x = 2;
(lldb) n
(lldb)  continue
Process 30764 resuming
Command #1 'continue' continued the target.
Process 30764 exited with status = 4 (0x00000004)

[SLTozer]

So to focus in on the issue with the clarification above, we have the following reproducer:

int a(int) {return 2;}
int main() {
  int x = a(1);
  return x;
}

When we build this with clang -O0 -g and debug with current lldb we observe:

(lldb) b 3
Breakpoint 1: where = repro`main + 15 at repro.cpp:3:11, address = 0x000000000000114f
(lldb) b 4
Breakpoint 2: where = repro`main + 28 at repro.cpp:4:10, address = 0x000000000000115c
(lldb) r
Process 10699 launched: '/home/gbtozers/dev/upstream-llvm/repro' (x86_64)
Process 10699 stopped
* thread #1, name = 'repro', stop reason = breakpoint 1.1
    frame #0: 0x000055555555514f repro`main at repro.cpp:3:11
   1    int a(int) {return 2;}
   2    int main() {
-> 3      int x = a(1);
   4      return x;
   5    }
(lldb) n
Process 10699 stopped
* thread #1, name = 'repro', stop reason = breakpoint 2.1 -2.1
    frame #0: 0x000055555555515c repro`main at repro.cpp:4:10
   1    int a(int) {return 2;}
   2    int main() {
   3      int x = a(1);
-> 4      return x;
   5    }
(lldb)

The expected outcome is that on the final step we should see stop reason = step over, but instead we see stop reason = breakpoint 2.1 -2.1.

[jimingham]

I haven't confirmed this in the code, but it looks like the bug is not about when we choose to say whether we've hit a breakpoint or not, but rather that to step over line 3, we seem to be setting a breakpoint on the first instruction of line 4 and running to that breakpoint. That's what breakpoint -2.1 is for.

But next over line 3 should mean "run all the instructions in line 3 leaving the PC at the first instruction of line 4 without executing any instructions of line 4." That's the principle we seem to be violating. When there's not another branch between the current PC and the end of the range of the current line, we should either step-i till we're out of the range or set a breakpoint on the last instruction of the range and then step-i over that.

[labath]

But next over line 3 should mean "run all the instructions in line 3 leaving the PC at the first instruction of line 4 without executing any instructions of line 4." That's the principle we seem to be violating. When there's not another branch between the current PC and the end of the range of the current line, we should either step-i till we're out of the range or set a breakpoint on the last instruction of the range and then step-i over that.

I don't think we should be forcing an additional step operation (and all the associated round-trips) to make this work.

The way I see it, by placing a trap on the first instruction of line 4, we really are:

• executing all instructions of line 3
• not executing any instructions of line 4 (because those instructions have been replaced by a trap)
• leaving the PC at the beginning of line 4

Making stepping slower to satisfy this invariant seems like a bad tradeoff, and I don't think it's necessary to implement the desired behavior (*). We already have sufficient information to figure out what happened. If we wanted to, we could say that the stepping breakpoint takes priority and we avoid reporting the "real" breakpoint hit (and also not bump its hit count and skip running its condition check). Conceptually, it could be thought of as placing the stepping trap opcode "over" the other breakpoint, so that the other bkpt is not really hit. This would be similar to what happens in the debug stub, if it implements single-step via setting a breakpoint on the next instruction (and the next insn already contains a breakpoint).

(*) I'm not sure that is the desired behavior, as I guess that would mean that next time we do a step, we report stop-reason=breakpoint without moving the PC. I can see how that could be useful sometimes, but I suspect it would be annoying in most cases. I think it might be best to have the ability to report multiple stop reasons ("we've finished stepping over, and by the way, we've also hit a breakpoint").

[jimingham]

Yes, you are right we aren't formally breaking our contract w.r.t. the instruction since we haven't actually executed it yet. But from the user's perspective, we shouldn't have executed their breakpoint either since that's seen to be the same as executing their instruction. So reporting it is wrong.

It may seem a little awkward, but the odd thing really is "I told you to step through the code in line 3, and you told me you hit a breakpoint at an instruction that's actually on line 4." The user really should be asking "Why are you running instructions beyond the range I told you to execute?"

More importantly, we don't guarantee that we will ALWAYS get to the instruction outside the current step range with breakpoint and run. For instance, if a branch instruction takes you outside the range, that will be done with an instruction single step - which wouldn't trigger the breakpoint. So the behavior becomes "sometimes when there's a breakpoint at the first instruction outside the step range we report it and sometimes we don't." We shouldn't be unpredictable like that. We should instead either always trigger breakpoints on the instruction we arrive at outside the step range or never do. I think never do makes much more sense, since after all that wasn't an instruction you asked us to run yet...

Note, you wouldn't have to slow stepping in general for this to work correctly. This would only come into play when lldb knows it can run to code outside the current range using a breakpoint and continue. At that point, it can check whether that target instruction has a user breakpoint. If it doesn't - which would be the majority of the time - it would do what we do now: use a breakpoint to do the run, but suppress the hit and report the stop reason as "step completed". However, if there was a user breakpoint there, it would instead put the step-implementation breakpoint one instruction prior and step over that. One extra step-i when you step to an instruction that has a user breakpoint won't appreciably slow stepping, and it would be quite straight-forward to implement

The other option, saying "in some cases, I don't consider ALL the breakpoint locations owning a particular site get hit when the site gets hit" seems much more complicated to me, and likely to be more fragile.

[SLTozer]

As another consumer of LLDB's API, @labath's description of the expected/desired debugger behaviour seems more natural to me. The definition/contract you've given for hitting a breakpoint is, "a breakpoint has only been hit when the corresponding trap instruction is executed". But as a user, when I set a breakpoint on an instruction my intent isn't "replace this instruction with a trap and notify me when that trap is executed", it's "pause the program's execution immediately before executing this instruction" - the trap instruction is an internal implementation detail that I may not know about, and should not need to know about, in which case the difference between "step to this instruction" and "breakpoint this instruction and continue" would be non-existent. Is there a motivating example for this breakpoint principle, or some functionality that can't be realized without it?

[jimingham]

The contract that I really want to have with the user w.r.t. breakpoints is that the accounting for breakpoint hits needs to be robust - once per time that the instruction you told me to put a breakpoint on gets executed. Having breakpoints whose hit count and actions are untrustworthy is not at all desirable. That's what started us (really Jason) on this whole journey. What we found was that if you have to guess whether to consider a breakpoint hit just from the fact that the pc is at the breakpoint instruction, this accounting gets really tricky and fragile. OTOH, the rule "a breakpoint is only hit when the trap is executed" made this problem tractable. So what I was stating as a principle was really the way it was possible to reason accurately enough to make this accounting robust. The other principle I think is important to maintain is that when stepping, lldb should only do what the user tells it to. If I tell lldb to step over line 5, I don't mean for it to do any of the things I told it to do when executing line 6. For instance, if I have an auto-continue breakpoint at line 6, I would be super annoyed if stepping over line 5 triggered the breakpoint and caused the process to continue, since I haven't told lldb to execute line 6 yet. lldb should also be accurate about telling you what it did. It would be really confusing if I "step over" a line of code that doesn't have a breakpoint on it, and have the stop reason be "hit the breakpoint in code I haven't told lldb to execute yet". So I'd really rather lldb not do that. Jim

…

On Sep 11, 2025, at 10:57 AM, Stephen Tozer ***@***.***> wrote: SLTozer left a comment (llvm/llvm-project#156650) <#156650 (comment)> As another consumer of LLDB's API, @labath <https://github.com/labath>'s description of the expected/desired debugger behaviour seems more natural to me. The definition/contract you've given for hitting a breakpoint is, "a breakpoint has only been hit when the corresponding trap instruction is executed". But as a user, when I set a breakpoint on an instruction my intent isn't "replace this instruction with a trap and notify me when that trap is executed", it's "pause the program's execution immediately before executing this instruction" - the trap instruction is an internal implementation detail that I may not know about, and should not need to know about, in which case the difference between "step to this instruction" and "breakpoint this instruction and continue" would be non-existent. Is there a motivating example for this breakpoint principle, or some functionality that can't be realized without it? — Reply to this email directly, view it on GitHub <#156650 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADUPVW76L5IIMKBP63W3MLT3SGZX7AVCNFSM6AAAAACFQKV46SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOBSGA4DEMBQGM>. You are receiving this because you are on a team that was mentioned.

[SLTozer]

As far as I can see, there are two ways we could define breakpoint hits - a trap-based definition, as you've explained here, and a PC-based definition, where the PC is at the breakpoint instruction's address (such that the breakpoint instruction will be executed next when the program resumes execution). Neither definition appears to violate any contract with the user - the promise given by a breakpoint is that we will execute everything up to but not including the breakpoint instruction.

I agree that the breakpoint accounting needs to be robust and reliable, but I don't think I understand yet what it is that makes this difficult if LLDB uses the PC-based definition. From a user perspective, i.e. without any reference to internal implementation details, is there any observable difference between "we have paused execution with the PC at the breakpoint's address" and "the breakpoint's trap has been executed"? For any simple program I can think of, it is impossible for a user to distinguish the two by design, as 1) execution will always resume from the breakpoint instruction, and 2) executing the trap should not in itself result in any change to the user-visible program state. So my first question is, under what circumstances does using trap-based breakpoint hits improve reliability over PC-based breakpoint hits?

The continue breakpoint example makes sense to me as there is an undesirable user outcome, where a user sets a breakpoint with a continue command intending for it to be transparently passed-over, and the result is that LLDB continues past a "step over" that lands on that breakpoint. But if LLDB used the PC-based breakpoint hit definition and explicitly documented it, then this outcome would be predictable (or at least explained) to a user. If the user does wants a transparent breakpoint that won't pause execution but also won't continue out of a step, then LLDB has functionality specifically for this: setting the breakpoint with --auto-continue true does exactly that with both current LLDB, and with lldb-18 where the old step-onto-breakpoint behaviour exists:

(lldb) b 3
Breakpoint 1: where = repro`main + 15 at repro.cpp:3:11, address = 0x000000000000114f
(lldb) breakpoint set --line 4 --auto-continue true -C "fr v"
Breakpoint 2: where = repro`main + 28 at repro.cpp:4:10, address = 0x000000000000115c
(lldb) r
Process 9896 launched: '/home/gbtozers/dev/upstream-llvm/repro' (x86_64)
Process 9896 stopped
* thread #1, name = 'repro', stop reason = breakpoint 1.1
    frame #0: 0x000055555555514f repro`main at repro.cpp:3:11
   1    int a(int) {return 2;}
   2    int main() {
-> 3      int x = a(1);
   4      return x;
   5    }
(lldb) n
(lldb)  fr v
(int) x = 2
Process 9896 stopped
* thread #1, name = 'repro', stop reason = step over
    frame #0: 0x000055555555515c repro`main at repro.cpp:4:10
   1    int a(int) {return 2;}
   2    int main() {
   3      int x = a(1);
-> 4      return x;
   5    }

Therefore it seems that a user has control over what LLDB does: a breakpoint set with -C continue will continue out of a step that lands on it, while a breakpoint set with --auto-continue true will not. This seems, at least to my current understanding, to be predictable and to enable users to achieve whichever result they want with their breakpoints. So my other questions are, does the option of using --auto-continue adequately address the scenario you gave where a -C continue breakpoint would interfere with stepping, and if so are there other examples where having the breakpoint commands execute when we step onto a breakpoint address would be a problem?

[jimingham]

If you only have one thread, then the PC based definition would be fine. The only way a process can stop at the PC is if the thread hit a trap anyway, so there's really not much difference. The problem came in when you have many threads running in a work function that has a breakpoint. Then on a given stop threads A, B and C have actually executed the trap so they have non-trivial stop reasons, and there are threads D, E and F that stopped when the PC happened to be at the breakpoint PC but haven't executed the trap. So you have to either ignore this stop for them, or following your "PC" rule consider them to have hit the breakpoint. But when they run they are going to trap again, so you also have to tell yourself to ignore the next hit for those threads. It's that accounting that quickly gets hard. Having a simple rule made this more tractable.

[jimingham]

BTW, the one thing I hope we do agree on is "inconsistent" is always bad. So the current bug where we report a breakpoint on the next line as hit if we happen to get out of the current line by running to a breakpoint at the beginning of the next line but not if we happen to step-i out of it is clearly wrong.

We can fix that either by having the "run to breakpoint" one not trigger the breakpoint as I suggested. Or by considering the first stop where the PC happens to be at a breakpoint address to be the breakpoint hit, and ignore the next stop if it happens to trap when we next let that thread run.

When I get a chance, I'm planning to fix the former, because that inconsistency is confusing.

[SLTozer]

BTW, the one thing I hope we do agree on is "inconsistent" is always bad.

Agreed, the inconsistent behaviour is the only thing here that is definitely a bug - the internal decision-making of LLDB ideally shouldn't affect the user experience, and displaying negative internal-only breakpoint IDs is definitely bad - thanks for looking into it!

Regarding the multi-threaded example, it seems to me that if you have threads A-C execute a trap and D-F arrive at the trap but not execute it, either method could work - if we treat threads D-F as having hit the breakpoint, then in my understanding we wouldn't "double hit" the breakpoint as long as we treat them the same as threads A-C. For example, if we only look for executed traps, then for threads A-C we record the hit, and when the user requests to continue we replace the trap with the original instruction, single step threads A-C, replace the instruction with the trap again, and then resume execution for all threads, at which point threads D-F will hit the trap and we'll repeat the same steps for them. If we determine breakpoints hit by checking the PC instead, then for all threads A-F we record the hit, and when the user requests to continue we remove the trap, single step all threads A-F, add the trap, and resume execution for all threads.

The key factor here is that, at least as far as I can tell, once the debuggee has paused execution there's no real difference between whether threads D-F actually executed the trap, or whether the PC is simply at the breakpoint address - the program state should be identical.