lldb: coredump should resolve loaded modules via debuginfod

[aag-norbert-lange]

Hello,

I would like to open a coredump from a different system, and then lldb should be able to fetch the executable + debuginfo for all relevant libraries.

So what should be done:

• get executable + debuginfo for process executable via debuginfod
• get loaded modules in a similar fashion
• get commandline arguments from coredump

What happens:

• lldb seems to only load debuginfo, but not not map the executable memory
• lldb tries to find the opened modules on the host. means it either finds nothing, or worse - map a wrong binary + debuginfo
• this seems to be missing?

GDB works correctly and displays the commandline of the process too.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Norbert Lange (aag-norbert-lange)

Hello,

I would like to open a coredump from a different system, and then lldb should be able to fetch the executable + debuginfo for all relevant libraries.

So what should be done:

• get executable + debuginfo for process executable via debuginfod
• get loaded modules in a similar fashion
• get commandline arguments from coredump

What happens:

• lldb seems to only load debuginfo, but not not map the executable memory
• lldb tries to find the opened modules on the host. means it either finds nothing, or worse - map a wrong binary + debuginfo
• this seems to be missing?

GDB works correctly and displays the commandline of the process too.

[DavidSpickett]

On this point:

get commandline arguments from coredump
this seems to be missing?
GDB works correctly and displays the commandline of the process too.

Are you referring to the line printed when the core file is loaded?

$ gdb /tmp/test.o -c core
<...>
[New LWP 452399]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `/tmp/test.o 1 2 3'.                                     <<<<< This one.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.

lldb can get to the command line arguments but only manually:

$ ./bin/lldb /tmp/test.o --core core
(lldb) target create "/tmp/test.o" --core "core"
Core file '/home/david.spickett/build-llvm-aarch64/core' (aarch64) was loaded.
(lldb) print *((char**)argv + 0)
(char *) 0x0000ffffe429a5a1 "/tmp/test.o"

Just checking because the alternative is LLDB's target.run-args setting, or GDB's show args but both of these only work for live programs. They won't look into the core file.

Perhaps GDB has more ways to inspect them, I'm no expert there.

[aag-norbert-lange]

Are you referring to the line printed when the core file is loaded?

Yes

Perhaps GDB has more ways to inspect them, I'm no expert there.

The corefile has multiple segments, I kinda assumed theres one for some kinda header. I wasn't able to google anything up though, so likely you are right about gdb just printing some global symbols.
I kinda expected target show-launch-environment displaying the env too.

[DavidSpickett]

I think GDB is using the NT_PRPSINFO note. Which is limited to 80 characters, but they have added something using part of NT_AUXV in GDB 16 - https://developers.redhat.com/articles/2025/02/10/3-improvements-gdb-16s-core-file-loading#improved_discovery_of_executable_and_arguments. Maybe that even makes show args work with cores?

For LLDB, yes, we could look for NT_PRPSINFO at least.

Our "command" for dealing with command line arguments is unfortunately just the settings set command, and reading settings back from a core file would be setting a strange precedent to set. Perhaps it could end up under process status or one of those general process/thread info commands.

I would like to open a coredump from a different system, and then lldb should be able to fetch the executable + debuginfo for all relevant libraries.

I know little about debuginfod, but @da-viper does. Should it work with core files? Maybe no one noticed because users tend to debug on a matching system.

[aag-norbert-lange]

I would like to open a coredump from a different system, and then lldb should be able to fetch the executable + debuginfo for all relevant libraries.

I know little about debuginfod, but @da-viper does. Should it work with core files? Maybe no one noticed because users tend to debug on a matching system.

debuginfod already loads the debuginfo of the process executable, it also loads the debuginfo for matching paths.
It does not load the executable.

As end-user, anything but fetching the stuff according to the coredump build-uuid makes little sense. In my case most libraries are missing, but consider you have an older coredump and the used libraries meanwhile were updated, that's a complete mess and I would then prefer them to be not mapped at all instead of a crude mismatch that will definitely send you down wrong paths.

[da-viper]

It seems like multiple issues, It think It would be easier to Identify what is not working and what is valid behaviour with a reproducible example of the issue.

Should it work with core files? Maybe no one noticed because users tend to debug on a matching system.
Loading shared objects should also work on core file (have no confirmed it on external systems).

DebugInfoD artefacts solely depends on the build-id. and if the server has that artifact. If you are moving the file to a different linux distribution, the distribution server may not have the same artefacts. you can set that using DEBUGINFOD_URLS

I have not looked yet at the structure of a core file (I assume it may not include the build id of main executable and shared libraries) hence lldb may have to fallback on the file path to have some debugging.

[aag-norbert-lange]

I have a private debuginfod server which hosts executables and debuginfo.
Trying to demonstrate the issue with debian, I found out that the debian
repos dont host the executables.

1. debugging Coredumps requires all binaries (+debuginfo?) for correct stacktraces (the following example lacks the binaries, because debian unfortunately does not provide them)
2. gdb warns about issue, checks the buildid and does not load binaries that dont match
3. With the private debuginfo server, gdb seems to be able to resolve all binaries + debuginfos (incl. process binary).

I might try setting up a test with fedora, given that an external service is required,
and interesting test-cases require changed binaries its hard to make simple,
reproducable and isolated tests.

So far I compared how gdb / lldb react, and my conlusion is that lldb should verify the buildid
before mapping external binaries (that would be one part of the solution).
Being a bit verbose about it would help too

Creating a test-coredump

mkdir lldb-test && cd lldb-test

cat > test.c << 'EOF'
#include <stdio.h>
#include <assert.h>
#include <string.h>

#include "uv.h"

typedef struct {
  uv_async_t async;
  uv_mutex_t mutex;
  uv_thread_t thread;
} thread_comm_t;


void async_cb(uv_async_t* async) {
  printf("async_cb\n");
  uv_close((uv_handle_t*) async, NULL);
}


void thread_cb(void* arg) {
  thread_comm_t* comm = (thread_comm_t*) arg;

  printf("pre async send\n");
  uv_async_send(&comm->async);
  printf("post async send\n");
}


int main() {
  thread_comm_t comm;
  int r;

  memset(&comm, 0, sizeof(comm));

  r = uv_mutex_init(&comm.mutex);
  assert(r == 0);

  r = uv_async_init(uv_default_loop(), &comm.async, async_cb);
  assert(r == 0);
  comm.async.data = (void*) &comm;

  r = uv_thread_create(&comm.thread, thread_cb, (void*) &comm);
  assert(r == 0);

  uv_run(uv_default_loop(), UV_RUN_DEFAULT);

  return 0;
}
EOF

podman run --rm -v $PWD:/host -it debian:bookworm

In the container:

apt update; apt install --no-install-recommends libuv1-dev gcc libc6-dev gdb ca-certificates
export DEBUGINFOD_URLS="https://debuginfod.debian.net"

gcc -g -O2 /host/test.c -luv -o /tmp/test_core

gdb -batch -iex 'set debuginfod enabled on' -ex 'b async_cb' -ex 'run' -ex 'gcore /tmp/test.coredump' -ex 'bt' /tmp/test_core
cp /tmp/test_core /tmp/test.coredump /host

This is the full and correct stacktrace:

#0  async_cb (async=0x7ffe07873890) at /host/test.c:15
#1  0x00007fe60071809d in uv__async_io (loop=0x7fe600738260 <default_loop_struct>, w=<optimized out>, events=<optimized out>) at ./src/unix/async.c:163
#2  0x00007fe60072be6c in uv__io_poll (loop=loop@entry=0x7fe600738260 <default_loop_struct>, timeout=<optimized out>) at ./src/unix/epoll.c:374
#3  0x00007fe6007189e4 in uv_run (loop=0x7fe600738260 <default_loop_struct>, mode=mode@entry=UV_RUN_DEFAULT) at ./src/unix/core.c:406
#4  0x0000564777fab142 in main () at /host/test.c:45

Starting the test environment:

Using a newer debian version, this one has lldb-19: podman run --rm -v $PWD:/host -it debian:trixie

Setup:

apt update; apt install --no-install-recommends libuv1 lldb gdb ca-certificates
export DEBUGINFOD_URLS="https://debuginfod.debian.net"

cp /host/test.coredump /tmp/

GDB with missing process binary,

rm -f /tmp/test_core
gdb -iex 'set debuginfod enabled on' --core /tmp/test.coredump

This already gives usefull information:

warning: File /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 doesn't match build-id from core-file during file-backed mapping processing

warning: File /usr/lib/x86_64-linux-gnu/libc.so.6 doesn't match build-id from core-file during file-backed mapping processing

warning: File /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0 doesn't match build-id from core-file during file-backed mapping processing

warning: Can't open file /tmp/test_core during file-backed mapping note processing

info symbol async_cb -> Missing

info symbol uv__async_io -> Missing

Backtrace:

#0  0x0000564777fab2a0 in ?? ()
#1  0x00007fe60071809d in ?? ()
#2  0x00007ffe078702c0 in ?? ()
#3  0x00007ffe078702c0 in ?? ()
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000000 in ?? ()

GDB with process binary,

cp /host/test_core /tmp/
gdb -iex 'set debuginfod enabled on' --core /tmp/test.coredump

info symbol async_cb -> OK

info symbol uv__async_io -> Missing

Backtrace:

#0  async_cb (async=0x7ffe07873890) at /host/test.c:15
#1  0x00007fe60071809d in ?? ()
#2  0x00007ffe078702c0 in ?? ()
#3  0x00007ffe078702c0 in ?? ()
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000000 in ?? ()

LLDB with missing process binary,

rm -f /tmp/test_core
lldb --core /tmp/test.coredump

image lookup -s async_cb -> Fail

image lookup -s uv__async_io -> Fail

Backtrace:

* thread #1, name = 'test_core', stop reason = signal SIGTRAP
  * frame #0: 0x0000564777fab2a0

LLDB with process binary,

cp /host/test_core /tmp/
lldb --core /tmp/test.coredump

image lookup -s async_cb -> OK

image lookup -s uv__async_io -> Wrong

Backtrace:

* thread #1, name = 'test_core', stop reason = signal SIGTRAP
  * frame #0: 0x0000564777fab2a0 test_core`async_cb(async=0x00007ffe07873890) at test.c:14:34
    frame #1: 0x00007fe60071809d libuv.so.1`uv_walk at uv-common.c:562:7

[aag-norbert-lange]

DebugInfoD artefacts solely depends on the build-id. and if the server has that artifact. If you are moving the file to a different linux distribution, the distribution server may not have the same artefacts. you can set that using DEBUGINFOD_URLS

there is a special Server which provides (forwards or federates) support for pretty much all popular distros.
And for my usecase, I have one containing all the binaries for embedded systems (tracking back multiple versions over years).

I would argue that's the main idea - you get a report about a crash and don't have to manually piece the filesystem together.
If I want to debug my running system, there aren't many occasions where I cant just attach or start directly under gdb/lldb instead of post-mortem debugging.

[DavidSpickett]

Opened #156427 for the command line bit, so this issue is just for debuginfod now.