LLDB TestNetBSDCore.py is flakey on a few bots since main executable detection changes

[DavidSpickett]

Around the time #157170 landed, I started seeing reports of this test failing on at least 2 of the lldb bots, without any clear changes to cause it.

Example 1: x86_64 Windows:
https://lab.llvm.org/buildbot/#/builders/211/builds/2037

PASS: LLDB (C:\buildbot\as-builder-10\lldb-x86-64\build\bin\clang.exe-x86_64) :: test_aarch64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase.test_aarch64_single_threaded)
PASS: LLDB (C:\buildbot\as-builder-10\lldb-x86-64\build\bin\clang.exe-x86_64) :: test_amd64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase.test_amd64_single_threaded)
PASS: LLDB (C:\buildbot\as-builder-10\lldb-x86-64\build\bin\clang.exe-x86_64) :: test_aarch64_process_signaled (TestNetBSDCore.NetBSD2LWPProcessSigCoreTestCase.test_aarch64_process_signaled)
Windows fatal exception: code 0xc0000374
Current thread 0x000034f0 (most recent call first):
  File "C:\buildbot\as-builder-10\lldb-x86-64\build\Lib\site-packages\lldb\__init__.py", line 5596 in DeleteTarget
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\test\API\functionalities\postmortem\netbsd-core\TestNetBSDCore.py", line 133 in do_test
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\test\API\functionalities\postmortem\netbsd-core\TestNetBSDCore.py", line 217 in test_amd64_process_signaled
  File "C:\Python312\Lib\unittest\case.py", line 589 in _callTestMethod
  File "C:\Python312\Lib\unittest\case.py", line 634 in run
  File "C:\Python312\Lib\unittest\case.py", line 690 in __call__
  File "C:\Python312\Lib\unittest\suite.py", line 122 in run
  File "C:\Python312\Lib\unittest\suite.py", line 84 in __call__
  File "C:\Python312\Lib\unittest\suite.py", line 122 in run
  File "C:\Python312\Lib\unittest\suite.py", line 84 in __call__
  File "C:\Python312\Lib\unittest\runner.py", line 240 in run
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\packages\Python\lldbsuite\test\dotest.py", line 1161 in run_suite
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\test\API\dotest.py", line 8 in <module>

Example 2: 32-bit Arm Linux:
https://lab.llvm.org/buildbot/#/builders/18/builds/21208

PASS: LLDB (/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/bin/clang-arm) :: test_aarch64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase)
PASS: LLDB (/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/bin/clang-arm) :: test_amd64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase)
malloc(): unaligned tcache chunk detected
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
malloc(): unaligned tcache chunk detected
Fatal Python error: Aborted

Thread 0xf7ece020 (most recent call first):
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/local/lib/python3.10/dist-packages/lldb/__init__.py", line 12444 in LoadCore
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 121 in do_test
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 212 in test_aarch64_process_signaled
  File "/usr/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
  File "/usr/lib/python3.10/unittest/case.py", line 591 in run
  File "/usr/lib/python3.10/unittest/case.py", line 650 in __call__
  File "/usr/lib/python3.10/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.10/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.10/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.10/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.10/unittest/runner.py", line 184 in run
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 1161 in run_suite
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/dotest.py", line 8 in <module>

I reproduced this in the same environment as the 32-bit Arm build and was able to get it to fail after running it repeatedly. The crash point would sometimes differ. Sometimes it had a backtrace within our code:

 #0 0xdd42db44 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf03b44)
 #1 0xdd42b034 llvm::sys::RunSignalHandlers() (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf01034)
 #2 0xdd42eb30 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0xe831fe50 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0
 #4 0xdcf02504 std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, lldb_private::UUID>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, lldb_private::UUID>>, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&&) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x9d8504)
 #5 0xdd1d5bb4 ProcessElfCore::FindModuleUUID(llvm::StringRef) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xcabbb4)
 #6 0xdf456f4c lldb_private::DynamicLoader::FindModuleViaTarget(lldb_private::FileSpec const&) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x2f2cf4c)
 #7 0xdf457294 lldb_private::DynamicLoader::LoadModuleAtAddress(lldb_private::FileSpec const&, unsigned long long, unsigned long long, bool) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x2f2d294)
 #8 0xdcf238d0 DynamicLoaderPOSIXDYLD::LoadModuleAtAddress(lldb_private::FileSpec const&, unsigned long long, unsigned long long, bool) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x9f98d0)
 #9 0xdcf246ec DynamicLoaderPOSIXDYLD::LoadAllCurrentModules()::$_0::operator()(DYLDRendezvous::SOEntry const&) const DynamicLoaderPOSIXDYLD.cpp:0:0
#10 0xdcca1bb4 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>>::_M_invoke(std::_Any_data const&) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777bb4)
#11 0xdcca1b0c std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777b0c)
#12 0xe8352e0c __pthread_once_slow ./nptl/pthread_once.c:118:7
#13 0xdcca1f04 std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>::_M_complete_async() (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777f04)
#14 0xdcca1fc0 std::_Function_handler<void (), std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()>::_M_invoke(std::_Any_data const&) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777fc0)
#15 0xdd3e5854 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xebb854)
#16 0xdd3e6aa8 void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
Fatal Python error: Segmentation fault

Sometimes it was at a higher level somewhere in glibc:

OK
free(): invalid pointer
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
#0 0xe4dd5b44 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf03b44)
#1 0xe4dd3034 llvm::sys::RunSignalHandlers() (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf01034)
#2 0xe4dd6b30 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
#3 0xefbcde50 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0
#4 0xefbbe6c6 __libc_do_syscall ./csu/../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47:0
#5 0xefbfe69c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
#6 0xefbccfc6 raise ./signal/../sysdeps/posix/raise.c:27:6
Fatal Python error: Aborted

Thread 0xefb83020 (most recent call first):
  File "/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/__init__.py", line 5369 in Terminate
  File "/home/david.spickett/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 747 in exitTestSuite
  File "/home/david.spickett/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 1190 in run_suite
  File "/home/david.spickett/llvm-project/lldb/test/API/dotest.py", line 8 in <module>

With a test that fails sometimes, you can't be 100% sure of anything, but I do not recall seeing these failures in the past, and with #157170 reverted I was able to run the test around 2000 times without issue. Whereas normally somewhere around 200 runs would get a failure.

This may be more common on 32-bit due to the reduced address space, perhaps common on Windows because it's allocator is more strict. Not sure.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: David Spickett (DavidSpickett)

Around the time https://github.com//pull/157170 landed, I started seeing reports of this test failing on at least 2 of the lldb bots, without any clear changes to cause it.

Example 1: x86_64 Windows:
https://lab.llvm.org/buildbot/#/builders/211/builds/2037

PASS: LLDB (C:\buildbot\as-builder-10\lldb-x86-64\build\bin\clang.exe-x86_64) :: test_aarch64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase.test_aarch64_single_threaded)
PASS: LLDB (C:\buildbot\as-builder-10\lldb-x86-64\build\bin\clang.exe-x86_64) :: test_amd64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase.test_amd64_single_threaded)
PASS: LLDB (C:\buildbot\as-builder-10\lldb-x86-64\build\bin\clang.exe-x86_64) :: test_aarch64_process_signaled (TestNetBSDCore.NetBSD2LWPProcessSigCoreTestCase.test_aarch64_process_signaled)
Windows fatal exception: code 0xc0000374
Current thread 0x000034f0 (most recent call first):
  File "C:\buildbot\as-builder-10\lldb-x86-64\build\Lib\site-packages\lldb\__init__.py", line 5596 in DeleteTarget
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\test\API\functionalities\postmortem\netbsd-core\TestNetBSDCore.py", line 133 in do_test
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\test\API\functionalities\postmortem\netbsd-core\TestNetBSDCore.py", line 217 in test_amd64_process_signaled
  File "C:\Python312\Lib\unittest\case.py", line 589 in _callTestMethod
  File "C:\Python312\Lib\unittest\case.py", line 634 in run
  File "C:\Python312\Lib\unittest\case.py", line 690 in __call__
  File "C:\Python312\Lib\unittest\suite.py", line 122 in run
  File "C:\Python312\Lib\unittest\suite.py", line 84 in __call__
  File "C:\Python312\Lib\unittest\suite.py", line 122 in run
  File "C:\Python312\Lib\unittest\suite.py", line 84 in __call__
  File "C:\Python312\Lib\unittest\runner.py", line 240 in run
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\packages\Python\lldbsuite\test\dotest.py", line 1161 in run_suite
  File "C:\buildbot\as-builder-10\lldb-x86-64\llvm-project\lldb\test\API\dotest.py", line 8 in <module>

Example 2: 32-bit Arm Linux:
https://lab.llvm.org/buildbot/#/builders/18/builds/21208

PASS: LLDB (/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/bin/clang-arm) :: test_aarch64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase)
PASS: LLDB (/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/bin/clang-arm) :: test_amd64_single_threaded (TestNetBSDCore.NetBSD1LWPCoreTestCase)
malloc(): unaligned tcache chunk detected
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
malloc(): unaligned tcache chunk detected
Fatal Python error: Aborted

Thread 0xf7ece020 (most recent call first):
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/local/lib/python3.10/dist-packages/lldb/__init__.py", line 12444 in LoadCore
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 121 in do_test
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 212 in test_aarch64_process_signaled
  File "/usr/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
  File "/usr/lib/python3.10/unittest/case.py", line 591 in run
  File "/usr/lib/python3.10/unittest/case.py", line 650 in __call__
  File "/usr/lib/python3.10/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.10/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.10/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.10/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.10/unittest/runner.py", line 184 in run
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 1161 in run_suite
  File "/home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/dotest.py", line 8 in <module>

I reproduced this in the same environment as the 32-bit Arm build and was able to get it to fail after running it repeatedly. The crash point would sometimes differ. Sometimes it had a backtrace within our code:

 #<!-- -->0 0xdd42db44 llvm::sys::PrintStackTrace(llvm::raw_ostream&amp;, int) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf03b44)
 #<!-- -->1 0xdd42b034 llvm::sys::RunSignalHandlers() (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf01034)
 #<!-- -->2 0xdd42eb30 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #<!-- -->3 0xe831fe50 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0
 #<!-- -->4 0xdcf02504 std::__detail::_Map_base&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt; const, lldb_private::UUID&gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt; const, lldb_private::UUID&gt;&gt;, std::__detail::_Select1st, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;&gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;true, false, true&gt;, true&gt;::operator[](std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;&amp;&amp;) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x9d8504)
 #<!-- -->5 0xdd1d5bb4 ProcessElfCore::FindModuleUUID(llvm::StringRef) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xcabbb4)
 #<!-- -->6 0xdf456f4c lldb_private::DynamicLoader::FindModuleViaTarget(lldb_private::FileSpec const&amp;) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x2f2cf4c)
 #<!-- -->7 0xdf457294 lldb_private::DynamicLoader::LoadModuleAtAddress(lldb_private::FileSpec const&amp;, unsigned long long, unsigned long long, bool) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x2f2d294)
 #<!-- -->8 0xdcf238d0 DynamicLoaderPOSIXDYLD::LoadModuleAtAddress(lldb_private::FileSpec const&amp;, unsigned long long, unsigned long long, bool) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x9f98d0)
 #<!-- -->9 0xdcf246ec DynamicLoaderPOSIXDYLD::LoadAllCurrentModules()::$_0::operator()(DYLDRendezvous::SOEntry const&amp;) const DynamicLoaderPOSIXDYLD.cpp:0:0
#<!-- -->10 0xdcca1bb4 std::_Function_handler&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; (), std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;&gt;::_M_invoke(std::_Any_data const&amp;) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777bb4)
#<!-- -->11 0xdcca1b0c std::__future_base::_State_baseV2::_M_do_set(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777b0c)
#<!-- -->12 0xe8352e0c __pthread_once_slow ./nptl/pthread_once.c:118:7
#<!-- -->13 0xdcca1f04 std::__future_base::_Deferred_state&lt;std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;::_M_complete_async() (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777f04)
#<!-- -->14 0xdcca1fc0 std::_Function_handler&lt;void (), std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()&gt;::_M_invoke(std::_Any_data const&amp;) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0x777fc0)
#<!-- -->15 0xdd3e5854 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xebb854)
#<!-- -->16 0xdd3e6aa8 void* llvm::thread::ThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*) ThreadPool.cpp:0:0
Fatal Python error: Segmentation fault

Sometimes it was at a higher level somewhere in glibc:

OK
free(): invalid pointer
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
#<!-- -->0 0xe4dd5b44 llvm::sys::PrintStackTrace(llvm::raw_ostream&amp;, int) (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf03b44)
#<!-- -->1 0xe4dd3034 llvm::sys::RunSignalHandlers() (/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/_lldb.cpython-312-arm-linux-gnueabihf.so+0xf01034)
#<!-- -->2 0xe4dd6b30 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
#<!-- -->3 0xefbcde50 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0
#<!-- -->4 0xefbbe6c6 __libc_do_syscall ./csu/../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47:0
#<!-- -->5 0xefbfe69c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
#<!-- -->6 0xefbccfc6 raise ./signal/../sysdeps/posix/raise.c:27:6
Fatal Python error: Aborted

Thread 0xefb83020 (most recent call first):
  File "/home/david.spickett/build-llvm-arm/local/lib/python3.12/dist-packages/lldb/__init__.py", line 5369 in Terminate
  File "/home/david.spickett/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 747 in exitTestSuite
  File "/home/david.spickett/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 1190 in run_suite
  File "/home/david.spickett/llvm-project/lldb/test/API/dotest.py", line 8 in &lt;module&gt;

With a test that fails sometimes, you can't be 100% sure of anything, but I do not recall seeing these failures in the past, and with #157170 reverted I was able to run the test around 2000 times without issue. Whereas normally somewhere around 200 runs would get a failure.

This may be more common on 32-bit due to the reduced address space, perhaps common on Windows because it's allocator is more strict. Not sure.

[DavidSpickett]

Currently testing the following patch:

diff --git a/lldb/source/Plugins/Process/elf-core/ProcessElfCore.cpp b/lldb/source/Plugins/Process/elf-core/ProcessElfCore.cpp
index 8f5f1242116f..0a57acce4491 100644
--- a/lldb/source/Plugins/Process/elf-core/ProcessElfCore.cpp
+++ b/lldb/source/Plugins/Process/elf-core/ProcessElfCore.cpp
@@ -308,7 +308,9 @@ llvm::StringRef ProcessElfCore::GetMainExecutablePath() {
 }
 
 UUID ProcessElfCore::FindModuleUUID(const llvm::StringRef path) {
-  return m_uuids[std::string(path)];
+  auto it = m_uuids.find(std::string(path));
+  return it == m_uuids.end() ? UUID() : it->second;
+    return it->second;
 }
 
 lldb_private::DynamicLoader *ProcessElfCore::GetDynamicLoader() {

Which seems to be better hygiene overall, but could also be preventing mutation of the unordered_map while other threads try to use it.

I suspect threads because of this from one of the tracebacks:

#16 0xdd3e6aa8 void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0

There is a setting target.parallel-module-load which is default on. I will try hammering the test with it off.

[DavidSpickett]

fyi @GeorgeHuyubo @dmpots

[DavidSpickett]

Which seems to be better hygiene overall, but could also be preventing mutation of the unordered_map while other threads try to use it.

2500+ runs without a problem so this is a "fix" but I suspect iterating a map as it is being modified is also bad, so perhaps we need locking for a proper fix.

[DavidSpickett]

There is a setting target.parallel-module-load which is default on. I will try hammering the test with it off.

I think it is to do with parallel loading because this change also "fixes" things:

diff --git a/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py b/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py
index ff1ef21e02e3..5cb4cc268dd9 100644
--- a/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py
+++ b/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py
@@ -117,6 +117,7 @@ class NetBSDCoreCommonTestCase(TestBase):
                 )
 
     def do_test(self, filename, pid, region_count):
+        self.runCmd("settings set target.parallel-module-load false")
         target = self.dbg.CreateTarget(filename)
         process = target.LoadCore(filename + ".core")

[DavidSpickett]

This is happening on 64-bit Linux as well, x86_64:
https://lab.llvm.org/buildbot/#/builders/162/builds/31245
https://lab.llvm.org/buildbot/#/builders/162/builds/31261

Not very useful traceback, but we would expect that with memory corruption:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe3b965c8
1  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe3b93d1e
2  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe3b973e1
3  libc.so.6                             0x00007f9ff03f4df0
4  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe372395b
5  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe3989547
6  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe57ed653
7  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe57ed8a0
8  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe373d535
9  _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe373e03f
10 _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe352c998
11 _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe352c8f7
12 libc.so.6                             0x00007f9ff044cbc7
13 libc.so.6                             0x00007f9ff044cc39 pthread_once + 25
14 _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe352cccf
15 _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe352cd5c
16 _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe3b52d44
17 _lldb.cpython-313-x86_64-linux-gnu.so 0x00007f9fe3b53e1b
18 libc.so.6                             0x00007f9ff0447b7b
19 libc.so.6                             0x00007f9ff04c55f0 __clone + 64
Fatal Python error: Segmentation fault

Thread 0x00007f9ff03b4100 (most recent call first):
  File "/home/worker/2.0.1/lldb-x86_64-debian/build/local/lib/python3.13/dist-packages/lldb/__init__.py", line 13090 in LoadCore
  File "/home/worker/2.0.1/lldb-x86_64-debian/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 121 in do_test
  File "/home/worker/2.0.1/lldb-x86_64-debian/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 182 in test_aarch64_thread_signaled
  File "/usr/lib/python3.13/unittest/case.py", line 606 in _callTestMethod
  File "/usr/lib/python3.13/unittest/case.py", line 651 in run
  File "/usr/lib/python3.13/unittest/case.py", line 707 in __call__
  File "/usr/lib/python3.13/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.13/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.13/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.13/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.13/unittest/runner.py", line 240 in run
  File "/home/worker/2.0.1/lldb-x86_64-debian/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 1161 in run_suite
  File "/home/worker/2.0.1/lldb-x86_64-debian/llvm-project/lldb/test/API/dotest.py", line 8 in <module>

And AArch64:
https://lab.llvm.org/buildbot/#/builders/59/builds/24279
https://lab.llvm.org/buildbot/#/builders/59/builds/24240

Similar to that seen on 32-bit Arm Linux:

tcache_thread_shutdown(): unaligned tcache chunk detected
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
 #0 0x0000f9ff4e756a40 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) Signals.cpp:0:0
 #1 0x0000f9ff4e75455c llvm::sys::RunSignalHandlers() Signals.cpp:0:0
 #2 0x0000f9ff4e75785c SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x0000f9ff553ea8f8 (linux-vdso.so.1+0x8f8)
 #4 0x0000f9ff5516f1f0 __pthread_kill_implementation ./nptl/./nptl/pthread_kill.c:44:76
 #5 0x0000f9ff5512a67c gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #6 0x0000f9ff55117130 abort ./stdlib/./stdlib/abort.c:81:7
 #7 0x0000f9ff55163300 __libc_message ./libio/../sysdeps/posix/libc_fatal.c:154:6
 #8 0x0000f9ff5517956c ./malloc/./malloc/malloc.c:5668:3
 #9 0x0000f9ff5517dd40 tcache_thread_shutdown ./malloc/./malloc/malloc.c:3218:17
#10 0x0000f9ff5517dd40 __malloc_arena_thread_freeres ./malloc/./malloc/arena.c:1003:3
#11 0x0000f9ff5516d488 start_thread ./nptl/./nptl/pthread_create.c:459:6
#12 0x0000f9ff551d5edc ./misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:82:0
Fatal Python error: Aborted

Thread 0x0000f9ff553dd420 (most recent call first):
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/local/lib/python3.10/dist-packages/lldb/__init__.py", line 5214 in Terminate
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 747 in exitTestSuite
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 1190 in run_suite
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/dotest.py", line 8 in <module>

Extension modules: lldb._lldb (total: 1)

With a variation:

double free or corruption (fasttop)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
 #0 0x0000e3b42768d7ac llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) Signals.cpp:0:0
 #1 0x0000e3b42768b2c8 llvm::sys::RunSignalHandlers() Signals.cpp:0:0
 #2 0x0000e3b42768e5c8 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x0000e3b42e3048f8 (linux-vdso.so.1+0x8f8)
 #4 0x0000e3b42e08f1f0 __pthread_kill_implementation ./nptl/./nptl/pthread_kill.c:44:76
 #5 0x0000e3b42e04a67c gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #6 0x0000e3b42e037130 abort ./stdlib/./stdlib/abort.c:81:7
 #7 0x0000e3b42e083300 __libc_message ./libio/../sysdeps/posix/libc_fatal.c:154:6
 #8 0x0000e3b42e09956c ./malloc/./malloc/malloc.c:5668:3
 #9 0x0000e3b42e09b244 _int_free ./malloc/./malloc/malloc.c:4700:5
#10 0x0000e3b42e09dc74 __libc_free ./malloc/./malloc/malloc.c:3394:3
#11 0x0000e3b427484154 ProcessElfCore::~ProcessElfCore() ProcessElfCore.cpp:0:0
#12 0x0000e3b4270d0b28 lldb_private::Target::DeleteCurrentProcess() Target.cpp:0:0
#13 0x0000e3b4270d2ab4 lldb_private::Target::Destroy() Target.cpp:0:0
#14 0x0000e3b426c7538c lldb::SBDebugger::DeleteTarget(lldb::SBTarget&) (/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/local/lib/python3.10/dist-packages/lldb/_lldb.cpython-310-aarch64-linux-gnu.so+0x578538c)
#15 0x0000e3b426db0da0 _wrap_SBDebugger_DeleteTarget(_object*, _object*) LLDBWrapPython.cpp:0:0
#16 0x0000b7bbf7943c64 (/usr/bin/python3.10+0x103c64)
#17 0x0000b7bbf793a240 _PyObject_MakeTpCall (/usr/bin/python3.10+0xfa240)
#18 0x0000b7bbf7931518 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xf1518)
#19 0x0000b7bbf79448c8 _PyFunction_Vectorcall (/usr/bin/python3.10+0x1048c8)
#20 0x0000b7bbf792c9e0 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xec9e0)
#21 0x0000b7bbf79539a8 (/usr/bin/python3.10+0x1139a8)
#22 0x0000b7bbf792d8fc _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xed8fc)
#23 0x0000b7bbf79539a8 (/usr/bin/python3.10+0x1139a8)
#24 0x0000b7bbf792c8a4 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xec8a4)
#25 0x0000b7bbf79448c8 _PyFunction_Vectorcall (/usr/bin/python3.10+0x1048c8)
#26 0x0000b7bbf792c9e0 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xec9e0)
#27 0x0000b7bbf7953ae8 (/usr/bin/python3.10+0x113ae8)
#28 0x0000b7bbf792e5cc _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xee5cc)
#29 0x0000b7bbf79392a4 _PyObject_FastCallDictTstate (/usr/bin/python3.10+0xf92a4)
#30 0x0000b7bbf7950624 _PyObject_Call_Prepend (/usr/bin/python3.10+0x110624)
#31 0x0000b7bbf7a80660 (/usr/bin/python3.10+0x240660)
#32 0x0000b7bbf793a240 _PyObject_MakeTpCall (/usr/bin/python3.10+0xfa240)
#33 0x0000b7bbf79310c8 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xf10c8)
#34 0x0000b7bbf7953ae8 (/usr/bin/python3.10+0x113ae8)
#35 0x0000b7bbf792e5cc _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xee5cc)
#36 0x0000b7bbf79392a4 _PyObject_FastCallDictTstate (/usr/bin/python3.10+0xf92a4)
#37 0x0000b7bbf7950624 _PyObject_Call_Prepend (/usr/bin/python3.10+0x110624)
#38 0x0000b7bbf7a80660 (/usr/bin/python3.10+0x240660)
#39 0x0000b7bbf793a240 _PyObject_MakeTpCall (/usr/bin/python3.10+0xfa240)
#40 0x0000b7bbf79310c8 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xf10c8)
#41 0x0000b7bbf7953ae8 (/usr/bin/python3.10+0x113ae8)
#42 0x0000b7bbf792e5cc _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xee5cc)
#43 0x0000b7bbf79392a4 _PyObject_FastCallDictTstate (/usr/bin/python3.10+0xf92a4)
#44 0x0000b7bbf7950624 _PyObject_Call_Prepend (/usr/bin/python3.10+0x110624)
#45 0x0000b7bbf7a80660 (/usr/bin/python3.10+0x240660)
#46 0x0000b7bbf793a240 _PyObject_MakeTpCall (/usr/bin/python3.10+0xfa240)
#47 0x0000b7bbf79310c8 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xf10c8)
#48 0x0000b7bbf79448c8 _PyFunction_Vectorcall (/usr/bin/python3.10+0x1048c8)
#49 0x0000b7bbf792c9e0 _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xec9e0)
#50 0x0000b7bbf79448c8 _PyFunction_Vectorcall (/usr/bin/python3.10+0x1048c8)
#51 0x0000b7bbf7930d2c _PyEval_EvalFrameDefault (/usr/bin/python3.10+0xf0d2c)
#52 0x0000b7bbf7a28250 (/usr/bin/python3.10+0x1e8250)
#53 0x0000b7bbf7a280d4 PyEval_EvalCode (/usr/bin/python3.10+0x1e80d4)
#54 0x0000b7bbf7a5b3ac (/usr/bin/python3.10+0x21b3ac)
#55 0x0000b7bbf7a53ab8 (/usr/bin/python3.10+0x213ab8)
#56 0x0000b7bbf7a5b05c (/usr/bin/python3.10+0x21b05c)
#57 0x0000b7bbf7a5a1c4 _PyRun_SimpleFileObject (/usr/bin/python3.10+0x21a1c4)
#58 0x0000b7bbf7a59d90 _PyRun_AnyFileObject (/usr/bin/python3.10+0x219d90)
#59 0x0000b7bbf7a4a7b0 Py_RunMain (/usr/bin/python3.10+0x20a7b0)
#60 0x0000b7bbf7a19208 Py_BytesMain (/usr/bin/python3.10+0x1d9208)
#61 0x0000e3b42e0373fc __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#62 0x0000e3b42e0374cc call_init ./csu/../csu/libc-start.c:128:20
#63 0x0000e3b42e0374cc __libc_start_main ./csu/../csu/libc-start.c:379:5
#64 0x0000b7bbf7a190f0 _start (/usr/bin/python3.10+0x1d90f0)
Fatal Python error: Aborted

Current thread 0x0000e3b42e2f7420 (most recent call first):
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/local/lib/python3.10/dist-packages/lldb/__init__.py", line 5372 in DeleteTarget
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 133 in do_test
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/functionalities/postmortem/netbsd-core/TestNetBSDCore.py", line 157 in test_amd64_single_threaded
  File "/usr/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
  File "/usr/lib/python3.10/unittest/case.py", line 591 in run
  File "/usr/lib/python3.10/unittest/case.py", line 650 in __call__
  File "/usr/lib/python3.10/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.10/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.10/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.10/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.10/unittest/runner.py", line 184 in run
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/dotest.py", line 1161 in run_suite
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/dotest.py", line 8 in <module>

Extension modules: lldb._lldb (total: 1)

This time coming from a destructor.

[DavidSpickett]

For now I've put in a workaround using the setting. Please try to reproduce on a your own machines. It might be the case that it takes many 1000s of tries to fail on some machines, in which case I can test fixes locally where it's much more common.

[dmpots]

Thanks for all the details and the mitigation. I'll take a look now and try to reproduce locally.