shared-library-event breakpoint at dlopen does not work in my setup

[georglauterbach]

OS: Yocto Linux (Kirkstone)
VSCode version: 1.93.1
CodeLLDB version: 1.11.0
Compiler: g++ (GCC) 11.3.0
Debuggee: a program that loads a ton of shared objects and runs the code inside them

---

I see a very strange behavior: After most shared object are loaded, which seems to be working fine (and quite fast, honestly 👍🏼), a single dlopen calls wrecks havoc with the debugger. The debugger does not continue, but keeps hitting the breakpoint again and again:

> breakpoint list -i
Current breakpoints:
Kind: shared-library-event
-1: names = {'_dl_debug_state', 'rtld_db_dlactivity', '__dl_rtld_db_dlactivity', 'r_debug_state', '_r_debug_state', '_rtld_debug_state'}, language = c, module = ld-linux-x86-64.so.2, locations = 1, resolved = 1, hit count = 3189

  -1.1: where = ld-linux-x86-64.so.2`__GI__dl_debug_state at dl-debug.c:116:1, address = 0x00007ffff7fca110, resolved, hit count = 3189 

Previous calls to dlopen succeeded. Curiously, when starting the program, halting in main, disabling the shared-library-event breakpoint with breakpoint disable -1 and then continueing makes the program work fine (but the debugger lacks information about shared objects, obviously).

The call to dlopen itself does not seem to be stange either:

dlopen( iDllName.c_str(), RTLD_NOW );

where iDllName = "/workspaces/build-<PROJECT NAME>/kirkstone/<PATH ...>/<SO NAME>.so"

---

• target triple: x86_64--linux

I cannot post all the log because the code I am working with must not be leaked. I am trying to provide as much information here, but I need to be somewhat generic, I am very sorry! Here is the log:

Click to expand

[DEBUG codelldb: :debug_session
] Debug event: 0x7f8c20002960 Event: broadcaster = 0x55eea6bcdbd8 (lldb.process), type = 0x00000001 (state-changed), data = { process = 0x55eea6bcdba0 (pid = 45184), state = stopped
}
[DEBUG codelldb: :debug_session
] Debug event: 0x7f8c200758a0 Event: broadcaster = 0x55eea6bcdbd8 (lldb.process), type = 0x00000001 (state-changed), data = { process = 0x55eea6bcdba0 (pid = 45184), state = running
}
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 294,
    "type": "event",
    "event": "continued",
    "body": {
        "allThreadsContinued": true,
        "threadId": 45184
    }
}
[DEBUG codelldb: :debug_session
] Debug event: 0x7f8ba59fd218 Event: broadcaster = 0x55eea60d4370 (lldb.target), type = 0x00000002 (modules-loaded), data = {<SO NAME>.so
}
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 295,
    "type": "event",
    "event": "module",
    "body": {
        "module": {
            "addressRange": "7FFFD882F000",
            "id": "7FFFD882F000",
            "name": "<SO NAME>.so",
            "path": "/workspaces/build-<PROJECT NAME>/kirkstone/<PATH ...>/<SO NAME>.so",
            "symbolFilePath": "/workspaces/build-<PROJECT NAME>/kirkstone/<PATH ...>/<SO NAME>.so",
            "symbolStatus": "Symbols loaded."
        },
        "reason": "new"
    }
}
[DEBUG codelldb: :dap_codec
] --> {
    "command": "pause",
    "arguments": {
        "threadId": 45184
    },
    "type": "request",
    "seq": 20
}
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 296,
    "type": "response",
    "request_seq": 20,
    "success": true,
    "command": "pause"
}
[DEBUG codelldb: :debug_session
] Debug event: 0x7f8c20002960 Event: broadcaster = 0x55eea6bcdbd8 (lldb.process), type = 0x00000001 (state-changed), data = { process = 0x55eea6bcdba0 (pid = 45184), state = stopped
}
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 297,
    "type": "event",
    "event": "output",
    "body": {
        "category": "stderr",
        "output": "Stop reason: signal SIGSTOP\n"
    }
}
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 298,
    "type": "event",
    "event": "stopped",
    "body": {
        "allThreadsStopped": true,
        "description": "signal SIGSTOP",
        "reason": "exception",
        "threadId": 45336
    }
}
[DEBUG codelldb: :dap_codec
] --> {
    "command": "threads",
    "type": "request",
    "seq": 21
}
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 299,
    "type": "response",
    "request_seq": 21,
    "success": true,
    "command": "threads",
    "body": {
        // list of threads that I cannot show here
    }
}
[DEBUG codelldb: :dap_codec
] --> {
    "command": "stackTrace",
    "arguments": {
        "threadId": 45336,
        "startFrame": 0,
        "levels": 1
    },
    "type": "request",
    "seq": 22
}
[DEBUG codelldb: :disassembly
] ld-linux-x86-64.so.2`__GI__dl_debug_state at dl-debug.c: 116: 1
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 300,
    "type": "response",
    "request_seq": 22,
    "success": true,
    "command": "stackTrace",
    "body": {
        "stackFrames": [
            {
                "column": 0,
                "id": 1004,
                "instructionPointerReference": "0x7FFFF7FCA110",
                "line": 3,
                "name": "__GI__dl_debug_state",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_debug_state",
                    "sourceReference": 1001
                }
            }
        ]
    }
}
[DEBUG codelldb: :dap_codec
] --> {
    "command": "stackTrace",
    "arguments": {
        "threadId": 45336,
        "startFrame": 1,
        "levels": 19
    },
    "type": "request",
    "seq": 23
}
[DEBUG codelldb: :disassembly
] ld-linux-x86-64.so.2`_dl_map_object_from_fd + 3724 at dl-load.c: 1510: 26
[DEBUG codelldb: :disassembly
] ld-linux-x86-64.so.2`_dl_map_object + 504 at dl-load.c: 2309: 10
[DEBUG codelldb: :disassembly
] ld-linux-x86-64.so.2`dl_open_worker_begin + 169 at dl-open.c: 539: 6
[DEBUG codelldb: :disassembly
] libc.so.6`__GI__dl_catch_exception + 113 at dl-error-skeleton.c: 209: 18
[DEBUG codelldb: :disassembly
] ld-linux-x86-64.so.2`dl_open_worker + 58 at dl-open.c: 784: 5
[DEBUG codelldb: :disassembly
] ld-linux-x86-64.so.2`_dl_open + 175 at dl-open.c: 886: 17
[DEBUG codelldb: :disassembly
] libc.so.6`dlopen_doit + 92 at dlopen.c: 56: 13
[DEBUG codelldb: :disassembly
] libc.so.6`__GI__dl_catch_error + 35 at dl-error-skeleton.c: 228: 12
[DEBUG codelldb: :disassembly
] libc.so.6`_dlerror_run + 126 at dlerror.c: 145: 17
[DEBUG codelldb: :disassembly
] libc.so.6`___dlopen + 81 [inlined
] dlopen_implementation + 33 at dlopen.c: 71: 51
    libc.so.6`___dlopen + 48 at dlopen.c: 81: 12
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 301,
    "type": "response",
    "request_seq": 23,
    "success": true,
    "command": "stackTrace",
    "body": {
        "stackFrames": [
            {
                "column": 0,
                "id": 1005,
                "instructionPointerReference": "0x7FFFF7FCFA6C",
                "line": 810,
                "name": "_dl_map_object_from_fd",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_map_object_from_fd",
                    "sourceReference": 1002
                }
            },
            {
                "column": 0,
                "id": 1006,
                "instructionPointerReference": "0x7FFFF7FD0488",
                "line": 134,
                "name": "_dl_map_object",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_map_object",
                    "sourceReference": 1003
                }
            },
            {
                "column": 0,
                "id": 1007,
                "instructionPointerReference": "0x7FFFF7FD4299",
                "line": 49,
                "name": "dl_open_worker_begin",
                "presentationHint": "subtle",
                "source": {
                    "name": "@dl_open_worker_begin",
                    "sourceReference": 1004
                }
            },
            {
                "column": 0,
                "id": 1008,
                "instructionPointerReference": "0x7FFFF7062631",
                "line": 30,
                "name": "__GI__dl_catch_exception",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_catch_exception",
                    "sourceReference": 1005
                }
            },
            {
                "column": 0,
                "id": 1009,
                "instructionPointerReference": "0x7FFFF7FD3A3A",
                "line": 19,
                "name": "dl_open_worker",
                "presentationHint": "subtle",
                "source": {
                    "name": "@dl_open_worker",
                    "sourceReference": 1006
                }
            },
            {
                "column": 0,
                "id": 1010,
                "instructionPointerReference": "0x7FFFF7062631",
                "line": 30,
                "name": "__GI__dl_catch_exception",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_catch_exception",
                    "sourceReference": 1005
                }
            },
            {
                "column": 0,
                "id": 1011,
                "instructionPointerReference": "0x7FFFF7FD3E1F",
                "line": 42,
                "name": "_dl_open",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_open",
                    "sourceReference": 1007
                }
            },
            {
                "column": 0,
                "id": 1012,
                "instructionPointerReference": "0x7FFFF6F9A5AC",
                "line": 25,
                "name": "dlopen_doit",
                "presentationHint": "subtle",
                "source": {
                    "name": "@dlopen_doit",
                    "sourceReference": 1008
                }
            },
            {
                "column": 0,
                "id": 1013,
                "instructionPointerReference": "0x7FFFF7062631",
                "line": 30,
                "name": "__GI__dl_catch_exception",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_catch_exception",
                    "sourceReference": 1005
                }
            },
            {
                "column": 0,
                "id": 1014,
                "instructionPointerReference": "0x7FFFF70626E3",
                "line": 15,
                "name": "__GI__dl_catch_error",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dl_catch_error",
                    "sourceReference": 1009
                }
            },
            {
                "column": 0,
                "id": 1015,
                "instructionPointerReference": "0x7FFFF6F9A0EE",
                "line": 33,
                "name": "_dlerror_run",
                "presentationHint": "subtle",
                "source": {
                    "name": "@_dlerror_run",
                    "sourceReference": 1010
                }
            },
            {
                "column": 0,
                "id": 1016,
                "instructionPointerReference": "0x7FFFF6F9A641",
                "line": 23,
                "name": "dlopen_implementation",
                "presentationHint": "subtle",
                "source": {
                    "name": "@dlopen@GLIBC_2.2.5",
                    "sourceReference": 1011
                }
            },
            {
                "column": 0,
                "id": 1017,
                "instructionPointerReference": "0x7FFFF6F9A620",
                "line": 16,
                "name": "___dlopen",
                "presentationHint": "subtle",
                "source": {
                    "name": "@dlopen@GLIBC_2.2.5",
                    "sourceReference": 1011
                }
            },
            {
                "column": 21,
                "id": 1018,
                "instructionPointerReference": "0x7FFFD1E1F363",
                "line": 545,
                "name": "<SOME MODULE SIGNATURE>::load_dll(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)",
                "source": {
                    "name": "REDACTED",
                    "path": "REDACTED"
                }
            },
            {
                "column": 27,
                "id": 1019,
                "instructionPointerReference": "0x7FFFD1E1E9DE",
                "line": 316,
                "name": "<SOME FUNCTION SIGNATURE>(<SOME TYPE> const&)",
                "source": {
                    "name": "REDACTED",
                    "path": "REDACTED"
                }
            },
            {
                "column": 22,
                "id": 1020,
                "instructionPointerReference": "0x7FFFD1E1CEEC",
                "line": 71,
                "name": "<SOME FUNCTION SIGNATURE>(std::vector<<SOME TYPE>, std::allocator<<SOME TYPE>>> const&)",
                "source": {
                    "name": "REDACTED",
                    "path": "REDACTED"
                }
            },
            {
                "column": 35,
                "id": 1021,
                "instructionPointerReference": "0x7FFFC4716BA4",
                "line": 89,
                "name": "<SOME FUNCTION SIGNATURE>(std::vector<<SOME TYPE>, std::allocator<<SOME TYPE>>> const&)",
                "source": {
                    "name": "REDACTED",
                    "path": "REDACTED"
                }
            },
            {
                "column": 24,
                "id": 1022,
                "instructionPointerReference": "0x7FFFD9012BBA",
                "line": 47,
                "name": "<SOME FUNCTION SIGNATURE>(std::vector<<SOME TYPE>, std::allocator<<SOME TYPE>>> const&)",
                "source": {
                    "name": "REDACTED",
                    "path": "REDACTED"
                }
            },
            {
                "column": 20,
                "id": 1023,
                "instructionPointerReference": "0x7FFFD91992D3",
                "line": 768,
                "name": "<SOME FUNCTION SIGNATURE>()",
                "source": {
                    "name": "REDACTED",
                    "path": "REDACTED"
                }
            }
        ]
    }
}
[DEBUG codelldb: :dap_codec
] --> {
    "command": "scopes",
    "arguments": {
        "frameId": 1018
    },
    "type": "request",
    "seq": 24
}
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 302,
    "type": "response",
    "request_seq": 24,
    "success": true,
    "command": "scopes",
    "body": {
        "scopes": [
            {
                "expensive": false,
                "name": "Local",
                "variablesReference": 1024
            },
            {
                "expensive": false,
                "name": "Static",
                "variablesReference": 1025
            },
            {
                "expensive": false,
                "name": "Global",
                "variablesReference": 1026
            },
            {
                "expensive": false,
                "name": "Registers",
                "variablesReference": 1027
            }
        ]
    }
}
[DEBUG codelldb: :dap_codec
] --> {
    "command": "variables",
    "arguments": {
        "variablesReference": 1024
    },
    "type": "request",
    "seq": 25
}
error: <SO NAME>.so 0x027de6fd: DW_TAG_member '__size' refers to type 0x00000000027de718 which extends beyond the bounds of 0x027de6e3
error: <SO NAME>.so 0x026d7a55: DW_TAG_member '_M_local_buf' refers to type 0x00000000027dea6d which extends beyond the bounds of 0x026d7a4b
[DEBUG codelldb: :dap_codec
] <-- {
    "seq": 303,
    "type": "response",
    "request_seq": 25,
    "success": true,
    "command": "variables",
    "body": {
        "variables": [
            {
                "evaluateName": "this",
                "memoryReference": "0x555555EF6200",
                "name": "this",
                "presentationHint": {
                    "attributes": [
                        "readOnly"
                    ]
                },
                "type": "REDACTED *",
                "value": "{mExclusiveClients:size=0, mRsrcManager:std::__shared_ptr<RsCMW::Rpm::CRsrcManager, __gnu_cxx::_S_atomic>::element_type @ 0x00005555560ea2a0, ...}",
                "variablesReference": 1028
            },
            {
                "evaluateName": "REDACTED",
                "memoryReference": "0x7FFFC15FE9A0",
                "name": "REDACTED",
                "presentationHint": {
                    "attributes": [
                        "readOnly"
                    ]
                },
                "type": "const std::string &",
                "value": "\"/workspaces/build-<PROJECT NAME>/kirkstone/<PATH ...>/<SO NAME>.so\"",
                "variablesReference": 1029
            },
            {
                "evaluateName": "REDACTED",
                "memoryReference": "0x0",
                "name": "REDACTED",
                "presentationHint": {
                    "attributes": [
                        "readOnly"
                    ]
                },
                "type": "void *",
                "value": "<null>",
                "variablesReference": 0
            },
            {
                "evaluateName": "it",
                "memoryReference": "0x7FFFC15FE9D8",
                "name": "it",
                "presentationHint": {
                    "attributes": [
                        "readOnly"
                    ]
                },
                "type": "std::list<REDACTED>::iterator",
                "value": "{_M_node:0x0000555555ef6260}",
                "variablesReference": 1030
            }
        ]
    }
}

Can someone please help me here? I am out of ideas currently... is there an exception thrown in dlopen? It's too late where I am to be sure...

[vadimcn]

https://github.com/vadimcn/codelldb/wiki/Reporting-Bugs

[georglauterbach]

Okay, I see. Do you know how to disable these "internal" breakpoints from LLDB before the debugging starts? I can manually add the shared objects, but I cannot get rid of the breakpoint, doesn't matter what I try.