LLDB does not see source files from symbols, but GDB does

[gordiig]

Original StackOverflow question

How to reproduce:

• Ubuntu 22.04
• lldb v14.0.0 (from apt)
• gdb v12.1

1. Creating core dump:

$ ulimit -c unlimited
$ sleep 10 & killall -SIGNEGV sleep
$ sudo cp /var/lib/apport/coredump/core._usr_bin_sleep.<...> sleep-core-dump
$ sudo chmod 444 sleep-core-dump

2. Opening core file with gdb. Gdb will also download symbols via debuginfod

$ gdb -q /bin/sleep -c sleep-core-dump
Reading symbols from /bin/sleep...
Reading symbols from /home/gordiig/.cache/debuginfod_client/9605b84bc84dab0a28a772dcebed48b0b98ec3a9/debuginfo...

3. Downloading additional symbol files from Ubuntu's ddeb

$ find-dbgsym-packages --install ./sleep-core-dump

4. Loading core file to lldb and adding symbols to target (from debuginfod or ddeb)

$ lldb /bin/sleep -c sleep-core-dump
(lldb) target symbols add /home/gordiig/.cache/debuginfod_client/9605b84bc84dab0a28a772dcebed48b0b98ec3a9/debuginfo 
# Or from ddeb:
(lldb) target symbols add /usr/lib/debug/.build-id/96/05b84bc84dab0a28a772dcebed48b0b98ec3a9.debug

Gdb bt output:

$ gdb -q /bin/sleep -c sleep-core-dump
Reading symbols from /bin/sleep...
Reading symbols from /home/gordiig/.cache/debuginfod_client/9605b84bc84dab0a28a772dcebed48b0b98ec3a9/debuginfo...
[New LWP 438331]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by 'sleep 10'.
Program terminated with signal SIGSEGV, Segmentation fault.

(gdb) bt
#0  0x00007f4f90fea7fa in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffccf5a9880, rem=rem@entry=0x7ffccf5a9870) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007f4f90fef6e7 in __GI___nanosleep (req=req@entry=0x7ffccf5a9880, rem=rem@entry=0x7ffccf5a9870) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x000055a5b5eb9a00 in rpl_nanosleep (remaining_delay=0x7ffccf5a9870, requested_delay=0x7ffccf5a9870) at lib/nanosleep.c:85
#3  xnanosleep (seconds=<optimized out>) at lib/xnanosleep.c:69
#4  main (argc=<optimized out>, argv=<optimized out>) at src/sleep.c:142

LLDB bt output with debuginfod symbols:

$ lldb /bin/sleep -c sleep-core-dump
(lldb) target create "/bin/sleep" --core "sleep-core-dump"
Core file '/home/gordiig/work/playing-with-dumps/src/dumps-generator/sleep-core-dump' (x86_64) was loaded.

(lldb) target symbols add /home/gordiig/.cache/debuginfod_client/9605b84bc84dab0a28a772dcebed48b0b98ec3a9/debuginfo 
warning: (x86_64) /bin/sleep unsupported DW_FORM values: 0x1f20 0x1f21
warning: failed to set breakpoint site at 0x55a5b5eb9b90 for breakpoint -2.1: error: elf-core does not support enabling breakpoints
symbol file '/home/gordiig/.cache/debuginfod_client/9605b84bc84dab0a28a772dcebed48b0b98ec3a9/debuginfo' has been added to '/bin/sleep'

(lldb) bt
* thread #1, name = 'sleep', stop reason = signal SIGSEGV
  * frame #0: 0x00007f4f90fea7fa libc.so.6`clock_nanosleep + 90
    frame #1: 0x00007f4f90fef6e7 libc.so.6`__nanosleep + 23
    frame #2: 0x000055a5b5eb9a00 sleep`main + 1024
    frame #3: 0x00007f4f90f2ed90 libc.so.6`___lldb_unnamed_symbol3139 + 128
    frame #4: 0x00007f4f90f2ee40 libc.so.6`__libc_start_main + 128
    frame #5: 0x000055a5b5eb9bb5 sleep`_start + 37

(lldb) f 2
frame #2: 0x000055a5b5eb9a00 sleep`main + 1024
sleep`main:
->  0x55a5b5eb9a00 <+1024>: testl  %eax, %eax
    0x55a5b5eb9a02 <+1026>: jne    0x29d6                    ; <+982>
    0x55a5b5eb9a04 <+1028>: movq   0x58(%rsp), %rax
    0x55a5b5eb9a09 <+1033>: subq   %fs:0x28, %rax

LLDB bt output with ddeb symbols:

$ lldb /bin/sleep -c sleep-core-dump                                                                                                                                                                                                                      
(lldb) target create "/bin/sleep" --core "sleep-core-dump"
warning: (x86_64) /bin/sleep unsupported DW_FORM values: 0x1f20 0x1f21
Core file '/home/gordiig/work/playing-with-dumps/src/dumps-generator/sleep-core-dump' (x86_64) was loaded.

(lldb) target symbols add /usr/lib/debug/.build-id/96/05b84bc84dab0a28a772dcebed48b0b98ec3a9.debug
warning: failed to set breakpoint site at 0x55a5b5eb9b90 for breakpoint -2.1: error: elf-core does not support enabling breakpoints
symbol file '/usr/lib/debug/.build-id/96/05b84bc84dab0a28a772dcebed48b0b98ec3a9.debug' has been added to '/bin/sleep'

(lldb) bt
* thread #1, name = 'sleep', stop reason = signal SIGSEGV
  * frame #0: 0x00007f4f90fea7fa libc.so.6`__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007ffccf5a9880, rem=0x00007ffccf5a9870) at clock_nanosleep.c:78:7
    frame #1: 0x00007f4f90fef6e7 libc.so.6`__GI___nanosleep(req=<unavailable>, rem=<unavailable>) at nanosleep.c:25:13
    frame #2: 0x000055a5b5eb9a00 sleep`main + 1024
    frame #3: 0x00007f4f90f2ed90 libc.so.6`__libc_start_call_main(main=(sleep`main), argc=2, argv=0x00007ffccf5a99e8) at libc_start_call_main.h:58:16
    frame #4: 0x00007f4f90f2ee40 libc.so.6`__libc_start_main_impl(main=(sleep`main), argc=2, argv=0x00007ffccf5a99e8, init=(_rtld_global), fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007ffccf5a99d8) at libc-start.c:392:3
    frame #5: 0x000055a5b5eb9bb5 sleep`_start + 37

(lldb) f 2
frame #2: 0x000055a5b5eb9a00 sleep`main + 1024
sleep`main:
->  0x55a5b5eb9a00 <+1024>: testl  %eax, %eax
    0x55a5b5eb9a02 <+1026>: jne    0x29d6                    ; <+982>
    0x55a5b5eb9a04 <+1028>: movq   0x58(%rsp), %rax
    0x55a5b5eb9a09 <+1033>: subq   %fs:0x28, %rax

What may be a bug

GDB sees source file names for frames, so if those files will be placed by those paths, gdb will show me non-asm backtrace.
But LLDB doesn't show me those paths. With ddebs situation is a bit better, but still no path for sleep's main file, which is the most important.

[llvmbot]

@llvm/issue-subscribers-lldb

[tcosprojects]

I am also seeing this, where gdb can load the backtraces from a coredump with file locations but lldb cannot. I have the same warning in lldb as you: warning: (x86_64) ... unsupported DW_FORM values: 0x1f20 0x1f21

It seems this could be related to unsupported dwarf features that lldb doesn't understand. See the following:

• https://fedoraproject.org/wiki/Changes/DebugInfoStandardization#Detailed_Description
• https://lists.llvm.org/pipermail/cfe-users/2019-August/001559.html
• https://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20150518/277186.html

System Info:

• openSUSE Tumbleweed
• lldb 16.0.6 from tumbleweed repo
• gdb 13.2

[borjamunozf]

any update here or solution? I think I'm experiencing the same issue.

[jimingham]

If you debug a running process with the same executable and symbol file, do you see source lines for functions that you weren't seeing when debugging the core file process?

[bytesiz3d]

Ran into the same issue