LLDB: `DYLDRendezvous::GetAction()` found two `eAdd` states in a row

[georglauterbach]

Note

I am by no means an LLDB (or LLVM, for that matter) expert. Please assume that I may be missing something obvious...

Prerequisites

I am debugging a program with LLDB that loads many shared objects, more than 100. While the overwhelming majority works flawlessly, I am stuck at one point, and I can't seem to make any progress for two weeks now. I have tried every configuration that seemed to have made sense to me, and I am now here because there may be the possibility that there is something wrong with LLDB itself - possibly.

The code that I am working on closed-source (which is a shame, I know - I cannot do anything about that). But I will share as many details as I can so we can find the problem.

While collecting the information for this report, I uncovered information that I wasn't aware of either; maybe some of the log and information down below are actually the problem's solution?

Environment information

I am debugging inside a container that runs Linux, more specifically: Yocto Linux (Kirkstone). This Linux has glibc 2.35 (no musl) and debug symbols should be available. The glibc source code is not available under /usr/src/debug/glibc/2.35-r0/git, though.

$ /lib/libc.so.6 
GNU C Library (GNU libc) stable release version 2.35.
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 11.3.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.

The program is written in C++ and compiled with g++:

$ g++ --version
g++ (GCC) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.

The C++ standard used is (mostly, except for one or two files) C++17. The CMAKE_CXX_FLAGS are

-fno-omit-frame-pointer -Wl,-z,defs --warn-no-narrowing -Werror=return-type -fvisibility=hidden -fuse-ld=lld -femit-class-debug-always -fvisibility-inlines-hidden -g -fPIC -std=c++17

I am using the command line lldb that ships with vadimcn/codelldb:

$ ./lldb --version
lldb version 19.1.0-custom

I have checked the target's environment, the debugger's environment, but changing the environment does not seem to make a difference here.

ASLR does not seem to make a difference either.

Examining the Problem

The application uses dlopen to load shared objects. The call to dlopen that does not work has a backtrace that looks like this:

(lldb) thread backtrace

error: <SHARED LIBRARY LOADED EARLIER 1>.so 0x027572fd: DW_TAG_member '_M_local_buf' refers to type 0x0000000002866a2e which extends beyond the bounds of 0x027572f3
error: <SHARED LIBRARY LOADED EARLIER 2>.so.so 0x003d9ca2: DW_TAG_member '_M_local_buf' refers to type 0x00000000004713cf which extends beyond the bounds of 0x003d9c98

* thread #42, name = '<SOME SHARED OBJECT>', stop reason = signal SIGSTOP
  * frame #0: 0x00007ffff7fca110 ld-linux-x86-64.so.2`__GI__dl_debug_state at dl-debug.c:116:1
    frame #1: 0x00007ffff7fcfa6c ld-linux-x86-64.so.2`_dl_map_object_from_fd(name=<unavailable>, origname=<unavailable>, fd=-1, fbp=<unavailable>, realname=<unavailable>, loader=<unavailable>, l_type=<unavailable>, mode=<unavailable>, stack_endp=<unavailable>, nsid=<unavailable>) at dl-load.c:1501:7
    frame #2: 0x00007ffff7fd0488 ld-linux-x86-64.so.2`_dl_map_object(loader=0x0000000000000000, name="<BUILD DIR>/<PATH TO LIBRARY THAT DOES NOT WORK>/<LIBRARY THAT DOES NOT WOTRK>.so", type=2, trace_mode=0, mode=<unavailable>, nsid=<unavailable>) at dl-load.c:2309:10
    frame #3: 0x00007ffff7fd4299 ld-linux-x86-64.so.2`dl_open_worker_begin(a=0x00007fff815fe740) at dl-open.c:534:21
    frame #4: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=<unavailable>, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:208:8
    frame #5: 0x00007ffff7fd3a3a ld-linux-x86-64.so.2`dl_open_worker(a=0x00007fff815fe740) at dl-open.c:782:15
    frame #6: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=<unavailable>, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:208:8
    frame #7: 0x00007ffff7fd3e1f ld-linux-x86-64.so.2`_dl_open(file=<unavailable>, mode=-2147483646, caller_dlopen=0x00007fffa4e1f3a1, nsid=-2, argc=5, argv=0x00007fffffffd5e8, env=0x00007fffffffd618) at dl-open.c:886:17
    frame #8: 0x00007ffff6f9a5ac libc.so.6`dlopen_doit(a=0x00007fff815fe9b0) at dlopen.c:56:15
    frame #9: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=0x00007fff815fe910, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:208:8
    frame #10: 0x00007ffff70626e3 libc.so.6`__GI__dl_catch_error(objname=0x00007fff815fe970, errstring=0x00007fff815fe978, mallocedp=0x00007fff815fe96f, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:227:19
    frame #11: 0x00007ffff6f9a0ee libc.so.6`_dlerror_run(operate=<unavailable>, args=<unavailable>) at dlerror.c:138:17
    frame #12: 0x00007ffff6f9a641 libc.so.6`___dlopen [inlined] dlopen_implementation(dl_caller=<unavailable>, mode=<unavailable>, file=<unavailable>) at dlopen.c:71:10
    frame #13: 0x00007ffff6f9a620 libc.so.6`___dlopen(file=<unavailable>, mode=<unavailable>) at dlopen.c:81:12
    frame #14: 0x00007fffa4e1f3a1 <SHARED LIBRARY LOADED EARLIER 1>.so`<SOME NAMESPACE>::<SOME CLASS>::load_dll(this=0x0000555555e4c030, iDllName="<BUILD DIR>/<PATH TO LIBRARY THAT DOES NOT WORK>/<LIBRARY THAT DOES NOT WOTRK>.so") at <SOURCE FILE>.cpp:545:21

<SHARED LIBRARY LOADED EARLIER 1>.so loads another shared library, but I fails. When enabling additional logs via log enable lldb dyld, I see this:

<LOG FROM APPLICATION>

<LOADING A BUNCH OF SHARED LIBRARIES (more than 100) CORRECTLY>

intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state     DYLDRendezvous::Resolve address size: 8, padding 4
intern-state     DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state     m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     m_current  Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 0 (eConsistent), ldbase = 0x00007ffff7fc7000
intern-state     bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eAddModules
intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false

<LOG FROM THE APPLICATION>

Process 50587 stopped and restarted: thread 1 received signal: SIGCHLD

<LOG FROM THE APPLICATION>

Process 50587 stopped and restarted: thread 1 received signal: SIGCHLD

<LOG FROM THE APPLICATION>

intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state     DYLDRendezvous::Resolve address size: 8, padding 4
intern-state     DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state     m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 0 (eConsistent), ldbase = 0x00007ffff7fc7000
intern-state     m_current  Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eNoAction
intern-state     bool DYLDRendezvous::UpdateSOEntries() action = eNoAction
intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false

<AND THEN REPEAT FOREVER:>

intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state     DYLDRendezvous::Resolve address size: 8, padding 4
intern-state     DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state     m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     m_current  Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     DYLDRendezvous::GetAction() found two eAdd states in a row, check process for multiple "_r_debug" symbols. Returning eAddModules to ensure shared libraries get loaded correctly
intern-state     bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eAddModules
intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false

---

I am really out of ideas at this point. Any help is greatly appreciated! Single-stepping does not work - the program continues when executing the command si.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Georg Lauterbach (georglauterbach)

> [!NOTE] > > I am by no means an LLDB (or LLVM, for that matter) expert. Please assume that I may be missing something obvious...

Prerequisites

I am debugging a program with LLDB that loads many shared objects, more than 100. While the overwhelming majority works flawlessly, I am stuck at one point, and I can't seem to make any progress for two weeks now. I have tried every configuration that seemed to have made sense to me, and I am now here because there may be the possibility that there is something wrong with LLDB itself - possibly.

The code that I am working on closed-source (which is a shame, I know - I cannot do anything about that). But I will share as many details as I can so we can find the problem.

While collecting the information for this report, I uncovered information that I wasn't aware of either; maybe some of the log and information down below are actually the problem's solution?

Environment information

I am debugging inside a container that runs Linux, more specifically: Yocto Linux (Kirkstone). This Linux has glibc 2.45 (no musl) and debug symbols should be available. The glibc source code is not available under /usr/src/debug/glibc/2.35-r0/git, though.

$ /lib/libc.so.6 
GNU C Library (GNU libc) stable release version 2.35.
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 11.3.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.

The program is written in C++ and compiled with g++:

$ g++ --version
g++ (GCC) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.

The C++ standard used is (mostly, except for one or two files) C++17. The CMAKE_CXX_FLAGS are

-fno-omit-frame-pointer -Wl,-z,defs --warn-no-narrowing -Werror=return-type -fvisibility=hidden -fuse-ld=lld -femit-class-debug-always -fvisibility-inlines-hidden -g -fPIC -std=c++17

I am using the command line lldb that ships with vadimcn/codelldb:

$ ./lldb --version
lldb version 19.1.0-custom

I have checked the target's environment, the debugger's environment, but changing the environment does not seem to make a difference here.

ASLR does not seem to make a difference either.

Examining the Problem

The application uses dlopen to load shared objects. The call to dlopen that does not work has a backtrace that looks like this:

(lldb) thread backtrace

error: <SHARED LIBRARY LOADED EARLIER 1>.so 0x027572fd: DW_TAG_member '_M_local_buf' refers to type 0x0000000002866a2e which extends beyond the bounds of 0x027572f3
error: <SHARED LIBRARY LOADED EARLIER 2>.so.so 0x003d9ca2: DW_TAG_member '_M_local_buf' refers to type 0x00000000004713cf which extends beyond the bounds of 0x003d9c98

* thread #<!-- -->42, name = 'libref_adv_ld_d', stop reason = signal SIGSTOP
  * frame #<!-- -->0: 0x00007ffff7fca110 ld-linux-x86-64.so.2`__GI__dl_debug_state at dl-debug.c:116:1
    frame #<!-- -->1: 0x00007ffff7fcfa6c ld-linux-x86-64.so.2`_dl_map_object_from_fd(name=&lt;unavailable&gt;, origname=&lt;unavailable&gt;, fd=-1, fbp=&lt;unavailable&gt;, realname=&lt;unavailable&gt;, loader=&lt;unavailable&gt;, l_type=&lt;unavailable&gt;, mode=&lt;unavailable&gt;, stack_endp=&lt;unavailable&gt;, nsid=&lt;unavailable&gt;) at dl-load.c:1501:7
    frame #<!-- -->2: 0x00007ffff7fd0488 ld-linux-x86-64.so.2`_dl_map_object(loader=0x0000000000000000, name="&lt;BUILD DIR&gt;/&lt;PATH TO LIBRARY THAT DOES NOT WORK&gt;/&lt;LIBRARY THAT DOES NOT WOTRK&gt;.so", type=2, trace_mode=0, mode=&lt;unavailable&gt;, nsid=&lt;unavailable&gt;) at dl-load.c:2309:10
    frame #<!-- -->3: 0x00007ffff7fd4299 ld-linux-x86-64.so.2`dl_open_worker_begin(a=0x00007fff815fe740) at dl-open.c:534:21
    frame #<!-- -->4: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=&lt;unavailable&gt;, operate=&lt;unavailable&gt;, args=&lt;unavailable&gt;) at dl-error-skeleton.c:208:8
    frame #<!-- -->5: 0x00007ffff7fd3a3a ld-linux-x86-64.so.2`dl_open_worker(a=0x00007fff815fe740) at dl-open.c:782:15
    frame #<!-- -->6: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=&lt;unavailable&gt;, operate=&lt;unavailable&gt;, args=&lt;unavailable&gt;) at dl-error-skeleton.c:208:8
    frame #<!-- -->7: 0x00007ffff7fd3e1f ld-linux-x86-64.so.2`_dl_open(file=&lt;unavailable&gt;, mode=-2147483646, caller_dlopen=0x00007fffa4e1f3a1, nsid=-2, argc=5, argv=0x00007fffffffd5e8, env=0x00007fffffffd618) at dl-open.c:886:17
    frame #<!-- -->8: 0x00007ffff6f9a5ac libc.so.6`dlopen_doit(a=0x00007fff815fe9b0) at dlopen.c:56:15
    frame #<!-- -->9: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=0x00007fff815fe910, operate=&lt;unavailable&gt;, args=&lt;unavailable&gt;) at dl-error-skeleton.c:208:8
    frame #<!-- -->10: 0x00007ffff70626e3 libc.so.6`__GI__dl_catch_error(objname=0x00007fff815fe970, errstring=0x00007fff815fe978, mallocedp=0x00007fff815fe96f, operate=&lt;unavailable&gt;, args=&lt;unavailable&gt;) at dl-error-skeleton.c:227:19
    frame #<!-- -->11: 0x00007ffff6f9a0ee libc.so.6`_dlerror_run(operate=&lt;unavailable&gt;, args=&lt;unavailable&gt;) at dlerror.c:138:17
    frame #<!-- -->12: 0x00007ffff6f9a641 libc.so.6`___dlopen [inlined] dlopen_implementation(dl_caller=&lt;unavailable&gt;, mode=&lt;unavailable&gt;, file=&lt;unavailable&gt;) at dlopen.c:71:10
    frame #<!-- -->13: 0x00007ffff6f9a620 libc.so.6`___dlopen(file=&lt;unavailable&gt;, mode=&lt;unavailable&gt;) at dlopen.c:81:12
    frame #<!-- -->14: 0x00007fffa4e1f3a1 &lt;SHARED LIBRARY LOADED EARLIER 1&gt;.so`&lt;SOME NAMESPACE&gt;::&lt;SOME CLASS&gt;::load_dll(this=0x0000555555e4c030, iDllName="&lt;BUILD DIR&gt;/&lt;PATH TO LIBRARY THAT DOES NOT WORK&gt;/&lt;LIBRARY THAT DOES NOT WOTRK&gt;.so") at &lt;SOURCE FILE&gt;.cpp:545:21

<SHARED LIBRARY LOADED EARLIER 1>.so loads another shared library, but I fails. When enabling additional logs via log enable lldb dyld, I see this:

<LOG FROM APPLICATION>

<LOADING A BUNCH OF SHARED LIBRARIES (more than 100) CORRECTLY>

intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state     DYLDRendezvous::Resolve address size: 8, padding 4
intern-state     DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state     m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     m_current  Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 0 (eConsistent), ldbase = 0x00007ffff7fc7000
intern-state     bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eAddModules
intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false

<LOG FROM THE APPLICATION>

Process 50587 stopped and restarted: thread 1 received signal: SIGCHLD

<LOG FROM THE APPLICATION>

Process 50587 stopped and restarted: thread 1 received signal: SIGCHLD

<LOG FROM THE APPLICATION>

intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state     DYLDRendezvous::Resolve address size: 8, padding 4
intern-state     DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state     m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 0 (eConsistent), ldbase = 0x00007ffff7fc7000
intern-state     m_current  Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eNoAction
intern-state     bool DYLDRendezvous::UpdateSOEntries() action = eNoAction
intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false

<AND THEN REPEAT FOREVER:>

intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state     DYLDRendezvous::Resolve address size: 8, padding 4
intern-state     DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state     m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     m_current  Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state     DYLDRendezvous::GetAction() found two eAdd states in a row, check process for multiple "_r_debug" symbols. Returning eAddModules to ensure shared libraries get loaded correctly
intern-state     bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eAddModules
intern-state     DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false

---

I am really out of ideas at this point. Any help is greatly appreciated! Single-stepping does not work - the program continues when executing the command si.

[georglauterbach]

The code applicable to my case seems to be located here: https://codebrowser.dev/llvm/lldb/source/Plugins/DynamicLoader/POSIX-DYLD/DYLDRendezvous.cpp.html#255. But I cannot find duplicate _r_debug symbols. How do I correctly check for them, maybe even inside of LLDB?

[georglauterbach]

Is there anyone who has an idea? I'm really struggling with this.

[georglauterbach]

@clayborg I found the old LLVM review that talks about the code that prints the error message that I've seen. Do you have any idea here? I cannot find any symbol called _r_debug as a global (which I think is probably strange). I really do not know how to continue at this point.

[georglauterbach]

Closing this again as no one has even noticed. If anyone comes across this issue, please reach out here. I have not found a solution yet, and had to stop searching (going back to GDB). Post your findings and ideas here, nevertheless, maybe we can re-open and resolve this in the future.

[georglauterbach]

I'm also seeing an error that popped up in a mailing list "DW_TAG_member extends beyond the bounds" (https://lists.llvm.org/pipermail/lldb-dev/2016-March/009947.html).

---

I also tried building LLDB from source on the Yocto Linux machine with these commands:

build.sh

#! /usr/bin/env bash

# ref: https://lldb.llvm.org/resources/build.html
set -eE -u -o pipefail
shopt -s inherit_errexit

sudo apt-get install build-essential swig python3-dev libedit-dev libncurses5-dev libxml2-dev

git clone --depth=1 https://github.com/llvm/llvm-project.git
mkdir -p llvm-project/build
cd llvm-project/build

cmake -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_PROJECTS="clang;lldb" -DLLVM_TARGETS_TO_BUILD=X86 \
  -DLLDB_SKIP_STRIP=OFF -DLLDB_ENABLE_LIBEDIT=OFF -DLLDB_ENABLE_CURSES=OFF -DLLDB_ENABLE_LIBXML2=OFF -DLLDB_ENABLE_LUA=OFF -DLLDB_ENABLE_PYTHON=ON \
  ../llvm

# we use mold to reduce link times significantly
# ref: https://github.com/rui314/mold/releases/
# wget -O /usr/local/bin/mold.tar.gz https://github.com/rui314/mold/releases/download/v2.36.0/mold-2.36.0-x86_64-linux.tar.gz
<PATH TO MOLD> -run ninja lldb lldb-server