[LLDB] Having the status line enabled can cause the program to auto resume

[clayborg]

We debugged an issue we were seeing in a program that has an exec. We noticed sometimes the program would stop at the exec and other times it would ignore it. The issue came down to the GDBRemoteClientBase::ShouldStop(...) and inside of this function, when the status line is enabled, it will return false due to the m_async_count not being zero. The issue stems from the Statusline reading memory while the process is running. When the Statusline object tries to update its execution context ref, it can cause the stack to try and read memory. This starts in frame 25 below.

(lldb) bt
* thread #9, name = 'dbg.evt-handler', stop reason = breakpoint 5.1
  * frame #0: 0x00007fd8596105cb liblldb.so.21.1`lldb_private::process_gdb_remote::GDBRemoteClientBase::Lock::SyncWithContinueThread(this=0x00007fd843342930) at GDBRemoteClientBase.cpp:404:37
    frame #1: 0x00007fd8596103dd liblldb.so.21.1`lldb_private::process_gdb_remote::GDBRemoteClientBase::Lock::Lock(this=0x00007fd843342930, comm=0x000055a3549a9d30, interrupt_timeout=(__r = 20)) at GDBRemoteClientBase.cpp:378:3
    frame #2: 0x00007fd85960fb77 liblldb.so.21.1`lldb_private::process_gdb_remote::GDBRemoteClientBase::SendPacketAndWaitForResponse(this=0x000055a3549a9d30, payload=(Data = "x7fffffffbc00,200", Length = 17), response=0x00007fd843342b18, interrupt_timeout=(__r = 20), sync_on_timeout=true) at GDBRemoteClientBase.cpp:184:8
    frame #3: 0x00007fd85965da57 liblldb.so.21.1`lldb_private::process_gdb_remote::ProcessGDBRemote::DoReadMemory(this=0x000055a3549a9090, addr=140737488337920, buf=0x00007fd834629480, size=512, error=0x00007fd843343148) at ProcessGDBRemote.cpp:3004:18
    frame #4: 0x00007fd858feb9e0 liblldb.so.21.1`lldb_private::Process::ReadMemoryFromInferior(this=0x000055a3549a9090, addr=140737488337920, buf=0x00007fd834629480, size=512, error=0x00007fd843343148) at Process.cpp:2204:9
    frame #5: 0x00007fd8590222a0 liblldb.so.21.1`lldb_private::MemoryCache::GetL2CacheLine(this=0x000055a3549a9948, line_base_addr=140737488337920, error=0x00007fd843343148) at Memory.cpp:138:41
    frame #6: 0x00007fd85902260e liblldb.so.21.1`lldb_private::MemoryCache::Read(this=0x000055a3549a9948, addr=140737488338072, dst=0x00007fd843343030, dst_len=8, error=0x00007fd843343148) at Memory.cpp:209:35
    frame #7: 0x00007fd858feb870 liblldb.so.21.1`lldb_private::Process::ReadMemory(this=0x000055a3549a9090, addr=140737488338072, buf=0x00007fd843343030, size=8, error=0x00007fd843343148) at Process.cpp:1988:27
    frame #8: 0x00007fd85903e327 liblldb.so.21.1`lldb_private::RegisterContext::ReadRegisterValueFromMemory(this=0x00007fd83464d750, reg_info=0x00007fd81800b5d0, src_addr=140737488338072, src_len=8, reg_value=0x00007fd8433432e0) at RegisterContext.cpp:356:25
    frame #9: 0x00007fd85912fa7c liblldb.so.21.1`lldb_private::RegisterContextUnwind::ReadRegisterValueFromRegisterLocation(this=0x00007fd83464d750, regloc=ConcreteRegisterLocation @ 0x00007fd843343248, reg_info=0x00007fd81800b5d0, value=0x00007fd8433432e0) at RegisterContextUnwind.cpp:1131:18
    frame #10: 0x00007fd85912f202 liblldb.so.21.1`lldb_private::RegisterContextUnwind::ReadGPRValue(this=0x00007fd83464d750, register_kind=eRegisterKindGeneric, regnum=0, value=0x00007fd8433439c8) at RegisterContextUnwind.cpp:2188:7
    frame #11: 0x00007fd85912a1e6 liblldb.so.21.1`lldb_private::RegisterContextUnwind::InitializeNonZerothFrame(this=0x00007fd83464d750) at RegisterContextUnwind.cpp:346:8
    frame #12: 0x00007fd8591291d4 liblldb.so.21.1`lldb_private::RegisterContextUnwind::RegisterContextUnwind(this=0x00007fd83464d750, thread=0x00007fd7901022d0, next_frame=ptr = 0x7fd8344b5050, sym_ctx=0x00007fd8340b3100, frame_number=1, unwind_lldb=0x00007fd8344b3a20) at RegisterContextUnwind.cpp:75:5
    frame #13: 0x00007fd859126010 liblldb.so.21.1`lldb_private::UnwindLLDB::GetOneMoreFrame(this=0x00007fd8344b3a20, abi=0x000055a354d1a1c0) at UnwindLLDB.cpp:129:40
    frame #14: 0x00007fd859125a3c liblldb.so.21.1`lldb_private::UnwindLLDB::AddOneMoreFrame(this=0x00007fd8344b3a20, abi=0x000055a354d1a1c0) at UnwindLLDB.cpp:337:17
    frame #15: 0x00007fd859125e16 liblldb.so.21.1`lldb_private::UnwindLLDB::UpdateUnwindPlanForFirstFrameIfInvalid(this=0x00007fd8344b3a20, abi=0x000055a354d1a1c0) at UnwindLLDB.cpp:311:3
    frame #16: 0x00007fd8591258b0 liblldb.so.21.1`lldb_private::UnwindLLDB::AddFirstFrame(this=0x00007fd8344b3a20) at UnwindLLDB.cpp:100:3
    frame #17: 0x00007fd859126e7e liblldb.so.21.1`lldb_private::UnwindLLDB::DoGetFrameInfoAtIndex(this=0x00007fd8344b3a20, idx=0, cfa=0x00007fd843344460, pc=0x00007fd843344468, behaves_like_zeroth_frame=0x00007fd84334445f) at UnwindLLDB.cpp:400:10
    frame #18: 0x00007fd859059cb9 liblldb.so.21.1`lldb_private::Unwind::GetFrameInfoAtIndex(this=0x00007fd8344b3a20, frame_idx=0, cfa=0x00007fd843344460, pc=0x00007fd843344468, behaves_like_zeroth_frame=0x00007fd84334445f) at Unwind.h:53:12
    frame #19: 0x00007fd859057175 liblldb.so.21.1`lldb_private::StackFrameList::FetchFramesUpTo(this=0x00007fd790080c60, end_idx=0, allow_interrupt=AllowInterruption) at StackFrameList.cpp:434:41
    frame #20: 0x00007fd859056ec6 liblldb.so.21.1`lldb_private::StackFrameList::GetFramesUpTo(this=0x00007fd790080c60, end_idx=0, allow_interrupt=AllowInterruption) at StackFrameList.cpp:367:21
    frame #21: 0x00007fd859057cc6 liblldb.so.21.1`lldb_private::StackFrameList::GetFrameAtIndex(this=0x00007fd790080c60, idx=0) at StackFrameList.cpp:632:7
    frame #22: 0x00007fd8590cac1c liblldb.so.21.1`lldb_private::Thread::GetSelectedFrame(this=0x00007fd7901022d0, select_most_relevant=DoNoSelectMostRelevantFrame) at Thread.cpp:274:48
    frame #23: 0x00007fd858fb679c liblldb.so.21.1`lldb_private::ExecutionContextRef::SetTargetPtr(this=0x00007fd843344778, target=0x000055a35484f830, adopt_selected=true) at ExecutionContext.cpp:522:32
    frame #24: 0x00007fd858fb65d4 liblldb.so.21.1`lldb_private::ExecutionContextRef::ExecutionContextRef(this=0x00007fd843344778, target=0x000055a35484f830, adopt_selected=true) at ExecutionContext.cpp:412:3
    frame #25: 0x00007fd858ca469f liblldb.so.21.1`lldb_private::Debugger::GetSelectedExecutionContext(this=0x000055a354b44fe0) at Debugger.cpp:1225:23
    frame #26: 0x00007fd858d778fd liblldb.so.21.1`lldb_private::Statusline::Redraw(this=0x000055a354b45350, update=true) at Statusline.cpp:136:41
    frame #27: 0x00007fd858ca0902 liblldb.so.21.1`lldb_private::Debugger::RedrawStatusline(this=0x000055a354b44fe0, update=true) at Debugger.cpp:1220:19
    frame #28: 0x00007fd858ca7e98 liblldb.so.21.1`lldb_private::Debugger::DefaultEventHandler(this=0x000055a354b44fe0) at Debugger.cpp:2171:7
    frame #29: 0x00007fd858caa0f8 liblldb.so.21.1`lldb_private::Debugger::StartEventHandlerThread()::$_0::operator()(this=0x000055a353dee700) const at Debugger.cpp:2203:42
    frame #30: 0x00007fd858caa0d5 liblldb.so.21.1`void* std::__invoke_impl<void*, lldb_private::Debugger::StartEventHandlerThread()::$_0&>((null)=__invoke_other @ 0x00007fd843344cdf, __f=0x000055a353dee700) at invoke.h:61:14
    frame #31: 0x00007fd858caa085 liblldb.so.21.1`std::enable_if<is_invocable_r_v<void*, lldb_private::Debugger::StartEventHandlerThread()::$_0&>, void*>::type std::__invoke_r<void*, lldb_private::Debugger::StartEventHandlerThread()::$_0&>(__fn=0x000055a353dee700) at invoke.h:114:9
    frame #32: 0x00007fd858ca9fad liblldb.so.21.1`std::_Function_handler<void* (), lldb_private::Debugger::StartEventHandlerThread()::$_0>::_M_invoke(__functor=0x000055a353dee700) at std_function.h:291:9
    frame #33: 0x00007fd858e03b5e liblldb.so.21.1`std::function<void* ()>::operator()(this=0x000055a353dee700) const at std_function.h:560:9
    frame #34: 0x00007fd858e03aba liblldb.so.21.1`lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(arg=0x000055a353dee6e0) at HostNativeThreadBase.cpp:62:10
    frame #35: 0x00007fd85249abc9 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:434:8
    frame #36: 0x00007fd85252ce4c libc.so.6`__clone3 at clone3.S:81

The m_async_count gets set to non-zero in ProcessGDBRemote::DoReadMemory() which calls m_gdb_comm.SendPacketAndWaitForResponse(...), which creates a GDBRemoteClientBase::Lock instance which calls GDBRemoteClientBase::Lock::SyncWithContinueThread which will increment the GDBRemoteClientBase::m_async_count. If any thread then tries to stop during this time when the read memory has the GDBRemoteClientBase::m_async_count set to non-zero, we will end up not stopping and auto resuming due to the GDBRemoteClientBase::ShouldStop(...) returning false.

Disabling the Statusline fixes the issue.

One possible fix is to not use ExecutionContextRef::ExecutionContextRef(Target *target, bool adopt_selected) which is called here:

ExecutionContext Debugger::GetSelectedExecutionContext() {
  bool adopt_selected = true;
  ExecutionContextRef exe_ctx_ref(GetSelectedTarget().get(), adopt_selected);
  return ExecutionContext(exe_ctx_ref);
}

Or we can make ExecutionContextRef::ExecutionContextRef(Target *target, bool adopt_selected) smarter so that it doesn't try to select a thread or frame if the process is running. The issue with this approach is it is racy and would likely need to use the public stop state which might not be reliable due to race conditions.

[JDevlieghere]

Based on the stack trace, I think this is from before #159887. Can you confirm whether this is still an issue after that PR?

[JDevlieghere]

We definitely don't want to compute the execution context unless the process is stopped. That PR should do the right thing by holding onto the stop lock while doing so.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Greg Clayton (clayborg)

We debugged an issue we were seeing in a program that has an exec. We noticed sometimes the program would stop at the exec and other times it would ignore it. The issue came down to the `GDBRemoteClientBase::ShouldStop(...)` and inside of this function, when the status line is enabled, it will return false due to the `m_async_count` not being zero. The issue stems from the `Statusline` reading memory while the process is running. When the `Statusline` object tries to update its execution context ref, it can cause the stack to try and read memory. This starts in frame 25 below.

(lldb) bt
* thread #<!-- -->9, name = 'dbg.evt-handler', stop reason = breakpoint 5.1
  * frame #<!-- -->0: 0x00007fd8596105cb liblldb.so.21.1`lldb_private::process_gdb_remote::GDBRemoteClientBase::Lock::SyncWithContinueThread(this=0x00007fd843342930) at GDBRemoteClientBase.cpp:404:37
    frame #<!-- -->1: 0x00007fd8596103dd liblldb.so.21.1`lldb_private::process_gdb_remote::GDBRemoteClientBase::Lock::Lock(this=0x00007fd843342930, comm=0x000055a3549a9d30, interrupt_timeout=(__r = 20)) at GDBRemoteClientBase.cpp:378:3
    frame #<!-- -->2: 0x00007fd85960fb77 liblldb.so.21.1`lldb_private::process_gdb_remote::GDBRemoteClientBase::SendPacketAndWaitForResponse(this=0x000055a3549a9d30, payload=(Data = "x7fffffffbc00,200", Length = 17), response=0x00007fd843342b18, interrupt_timeout=(__r = 20), sync_on_timeout=true) at GDBRemoteClientBase.cpp:184:8
    frame #<!-- -->3: 0x00007fd85965da57 liblldb.so.21.1`lldb_private::process_gdb_remote::ProcessGDBRemote::DoReadMemory(this=0x000055a3549a9090, addr=140737488337920, buf=0x00007fd834629480, size=512, error=0x00007fd843343148) at ProcessGDBRemote.cpp:3004:18
    frame #<!-- -->4: 0x00007fd858feb9e0 liblldb.so.21.1`lldb_private::Process::ReadMemoryFromInferior(this=0x000055a3549a9090, addr=140737488337920, buf=0x00007fd834629480, size=512, error=0x00007fd843343148) at Process.cpp:2204:9
    frame #<!-- -->5: 0x00007fd8590222a0 liblldb.so.21.1`lldb_private::MemoryCache::GetL2CacheLine(this=0x000055a3549a9948, line_base_addr=140737488337920, error=0x00007fd843343148) at Memory.cpp:138:41
    frame #<!-- -->6: 0x00007fd85902260e liblldb.so.21.1`lldb_private::MemoryCache::Read(this=0x000055a3549a9948, addr=140737488338072, dst=0x00007fd843343030, dst_len=8, error=0x00007fd843343148) at Memory.cpp:209:35
    frame #<!-- -->7: 0x00007fd858feb870 liblldb.so.21.1`lldb_private::Process::ReadMemory(this=0x000055a3549a9090, addr=140737488338072, buf=0x00007fd843343030, size=8, error=0x00007fd843343148) at Process.cpp:1988:27
    frame #<!-- -->8: 0x00007fd85903e327 liblldb.so.21.1`lldb_private::RegisterContext::ReadRegisterValueFromMemory(this=0x00007fd83464d750, reg_info=0x00007fd81800b5d0, src_addr=140737488338072, src_len=8, reg_value=0x00007fd8433432e0) at RegisterContext.cpp:356:25
    frame #<!-- -->9: 0x00007fd85912fa7c liblldb.so.21.1`lldb_private::RegisterContextUnwind::ReadRegisterValueFromRegisterLocation(this=0x00007fd83464d750, regloc=ConcreteRegisterLocation @ 0x00007fd843343248, reg_info=0x00007fd81800b5d0, value=0x00007fd8433432e0) at RegisterContextUnwind.cpp:1131:18
    frame #<!-- -->10: 0x00007fd85912f202 liblldb.so.21.1`lldb_private::RegisterContextUnwind::ReadGPRValue(this=0x00007fd83464d750, register_kind=eRegisterKindGeneric, regnum=0, value=0x00007fd8433439c8) at RegisterContextUnwind.cpp:2188:7
    frame #<!-- -->11: 0x00007fd85912a1e6 liblldb.so.21.1`lldb_private::RegisterContextUnwind::InitializeNonZerothFrame(this=0x00007fd83464d750) at RegisterContextUnwind.cpp:346:8
    frame #<!-- -->12: 0x00007fd8591291d4 liblldb.so.21.1`lldb_private::RegisterContextUnwind::RegisterContextUnwind(this=0x00007fd83464d750, thread=0x00007fd7901022d0, next_frame=ptr = 0x7fd8344b5050, sym_ctx=0x00007fd8340b3100, frame_number=1, unwind_lldb=0x00007fd8344b3a20) at RegisterContextUnwind.cpp:75:5
    frame #<!-- -->13: 0x00007fd859126010 liblldb.so.21.1`lldb_private::UnwindLLDB::GetOneMoreFrame(this=0x00007fd8344b3a20, abi=0x000055a354d1a1c0) at UnwindLLDB.cpp:129:40
    frame #<!-- -->14: 0x00007fd859125a3c liblldb.so.21.1`lldb_private::UnwindLLDB::AddOneMoreFrame(this=0x00007fd8344b3a20, abi=0x000055a354d1a1c0) at UnwindLLDB.cpp:337:17
    frame #<!-- -->15: 0x00007fd859125e16 liblldb.so.21.1`lldb_private::UnwindLLDB::UpdateUnwindPlanForFirstFrameIfInvalid(this=0x00007fd8344b3a20, abi=0x000055a354d1a1c0) at UnwindLLDB.cpp:311:3
    frame #<!-- -->16: 0x00007fd8591258b0 liblldb.so.21.1`lldb_private::UnwindLLDB::AddFirstFrame(this=0x00007fd8344b3a20) at UnwindLLDB.cpp:100:3
    frame #<!-- -->17: 0x00007fd859126e7e liblldb.so.21.1`lldb_private::UnwindLLDB::DoGetFrameInfoAtIndex(this=0x00007fd8344b3a20, idx=0, cfa=0x00007fd843344460, pc=0x00007fd843344468, behaves_like_zeroth_frame=0x00007fd84334445f) at UnwindLLDB.cpp:400:10
    frame #<!-- -->18: 0x00007fd859059cb9 liblldb.so.21.1`lldb_private::Unwind::GetFrameInfoAtIndex(this=0x00007fd8344b3a20, frame_idx=0, cfa=0x00007fd843344460, pc=0x00007fd843344468, behaves_like_zeroth_frame=0x00007fd84334445f) at Unwind.h:53:12
    frame #<!-- -->19: 0x00007fd859057175 liblldb.so.21.1`lldb_private::StackFrameList::FetchFramesUpTo(this=0x00007fd790080c60, end_idx=0, allow_interrupt=AllowInterruption) at StackFrameList.cpp:434:41
    frame #<!-- -->20: 0x00007fd859056ec6 liblldb.so.21.1`lldb_private::StackFrameList::GetFramesUpTo(this=0x00007fd790080c60, end_idx=0, allow_interrupt=AllowInterruption) at StackFrameList.cpp:367:21
    frame #<!-- -->21: 0x00007fd859057cc6 liblldb.so.21.1`lldb_private::StackFrameList::GetFrameAtIndex(this=0x00007fd790080c60, idx=0) at StackFrameList.cpp:632:7
    frame #<!-- -->22: 0x00007fd8590cac1c liblldb.so.21.1`lldb_private::Thread::GetSelectedFrame(this=0x00007fd7901022d0, select_most_relevant=DoNoSelectMostRelevantFrame) at Thread.cpp:274:48
    frame #<!-- -->23: 0x00007fd858fb679c liblldb.so.21.1`lldb_private::ExecutionContextRef::SetTargetPtr(this=0x00007fd843344778, target=0x000055a35484f830, adopt_selected=true) at ExecutionContext.cpp:522:32
    frame #<!-- -->24: 0x00007fd858fb65d4 liblldb.so.21.1`lldb_private::ExecutionContextRef::ExecutionContextRef(this=0x00007fd843344778, target=0x000055a35484f830, adopt_selected=true) at ExecutionContext.cpp:412:3
    frame #<!-- -->25: 0x00007fd858ca469f liblldb.so.21.1`lldb_private::Debugger::GetSelectedExecutionContext(this=0x000055a354b44fe0) at Debugger.cpp:1225:23
    frame #<!-- -->26: 0x00007fd858d778fd liblldb.so.21.1`lldb_private::Statusline::Redraw(this=0x000055a354b45350, update=true) at Statusline.cpp:136:41
    frame #<!-- -->27: 0x00007fd858ca0902 liblldb.so.21.1`lldb_private::Debugger::RedrawStatusline(this=0x000055a354b44fe0, update=true) at Debugger.cpp:1220:19
    frame #<!-- -->28: 0x00007fd858ca7e98 liblldb.so.21.1`lldb_private::Debugger::DefaultEventHandler(this=0x000055a354b44fe0) at Debugger.cpp:2171:7
    frame #<!-- -->29: 0x00007fd858caa0f8 liblldb.so.21.1`lldb_private::Debugger::StartEventHandlerThread()::$_0::operator()(this=0x000055a353dee700) const at Debugger.cpp:2203:42
    frame #<!-- -->30: 0x00007fd858caa0d5 liblldb.so.21.1`void* std::__invoke_impl&lt;void*, lldb_private::Debugger::StartEventHandlerThread()::$_0&amp;&gt;((null)=__invoke_other @ 0x00007fd843344cdf, __f=0x000055a353dee700) at invoke.h:61:14
    frame #<!-- -->31: 0x00007fd858caa085 liblldb.so.21.1`std::enable_if&lt;is_invocable_r_v&lt;void*, lldb_private::Debugger::StartEventHandlerThread()::$_0&amp;&gt;, void*&gt;::type std::__invoke_r&lt;void*, lldb_private::Debugger::StartEventHandlerThread()::$_0&amp;&gt;(__fn=0x000055a353dee700) at invoke.h:114:9
    frame #<!-- -->32: 0x00007fd858ca9fad liblldb.so.21.1`std::_Function_handler&lt;void* (), lldb_private::Debugger::StartEventHandlerThread()::$_0&gt;::_M_invoke(__functor=0x000055a353dee700) at std_function.h:291:9
    frame #<!-- -->33: 0x00007fd858e03b5e liblldb.so.21.1`std::function&lt;void* ()&gt;::operator()(this=0x000055a353dee700) const at std_function.h:560:9
    frame #<!-- -->34: 0x00007fd858e03aba liblldb.so.21.1`lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(arg=0x000055a353dee6e0) at HostNativeThreadBase.cpp:62:10
    frame #<!-- -->35: 0x00007fd85249abc9 libc.so.6`start_thread(arg=&lt;unavailable&gt;) at pthread_create.c:434:8
    frame #<!-- -->36: 0x00007fd85252ce4c libc.so.6`__clone3 at clone3.S:81

The m_async_count gets set to non-zero in ProcessGDBRemote::DoReadMemory() which calls m_gdb_comm.SendPacketAndWaitForResponse(...), which creates a GDBRemoteClientBase::Lock instance which calls GDBRemoteClientBase::Lock::SyncWithContinueThread which will increment the GDBRemoteClientBase::m_async_count. If any thread then tries to stop during this time when the read memory has the GDBRemoteClientBase::m_async_count set to non-zero, we will end up not stopping and auto resuming due to the GDBRemoteClientBase::ShouldStop(...) returning false.

Disabling the Statusline fixes the issue.

One possible fix is to not use ExecutionContextRef::ExecutionContextRef(Target *target, bool adopt_selected) which is called here:

ExecutionContext Debugger::GetSelectedExecutionContext() {
  bool adopt_selected = true;
  ExecutionContextRef exe_ctx_ref(GetSelectedTarget().get(), adopt_selected);
  return ExecutionContext(exe_ctx_ref);
}

Or we can make ExecutionContextRef::ExecutionContextRef(Target *target, bool adopt_selected) smarter so that it doesn't try to select a thread or frame if the process is running. The issue with this approach is it is racy and would likely need to use the public stop state which might not be reliable due to race conditions.

[satyajanga]

Based on the stack trace, I think this is from before #159887. Can you confirm whether this is still an issue after that PR?

Correct. this was on llvm-21.1.0 (2-3 months old). I will verify it with #159887 and update.

[satyajanga]

Closing this issue for now. I will reopen updated information if it is still an issue.