lldb test: lldb-api :: functionalities/thread/concurrent_events/TestConcurrentManySignals.py failing on aarch64 CI

[tstellar]

It's not failing all the time, just some times.

Here is the full log: https://github.com/llvm/llvm-project/actions/runs/19988753632/job/57326428935

Relevant part of the log:

FAIL: lldb-api :: functionalities/thread/concurrent_events/TestConcurrentManySignals.py (660 of 2628)
******************** TEST 'lldb-api :: functionalities/thread/concurrent_events/TestConcurrentManySignals.py' FAILED ********************
Script:
-
/usr/bin/python3 /__w/llvm-project/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env LLVM_LIBS_DIR=/__w/llvm-project/llvm-project/build/./lib --env LLVM_INCLUDE_DIR=/__w/llvm-project/llvm-project/build/include --env LLVM_TOOLS_DIR=/__w/llvm-project/llvm-project/build/./bin --libcxx-include-dir /__w/llvm-project/llvm-project/build/include/c++/v1 --libcxx-include-target-dir /__w/llvm-project/llvm-project/build/include/aarch64-unknown-linux-gnu/c++/v1 --libcxx-library-dir /__w/llvm-project/llvm-project/build/./lib/aarch64-unknown-linux-gnu --arch aarch64 --build-dir /__w/llvm-project/llvm-project/build/lldb-test-build.noindex --lldb-module-cache-dir /__w/llvm-project/llvm-project/build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /__w/llvm-project/llvm-project/build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /__w/llvm-project/llvm-project/build/./bin/lldb --compiler /__w/llvm-project/llvm-project/build/./bin/clang --dsymutil /__w/llvm-project/llvm-project/build/./bin/dsymutil --make /usr/bin/gmake --llvm-tools-dir /__w/llvm-project/llvm-project/build/./bin --lldb-obj-root /__w/llvm-project/llvm-project/build/tools/lldb --lldb-libs-dir /__w/llvm-project/llvm-project/build/./lib --cmake-build-type Release /__w/llvm-project/llvm-project/lldb/test/API/functionalities/thread/concurrent_events -p TestConcurrentManySignals.py
--
Exit Code: 1

Command Output (stdout):
--
ldb version 22.0.0git (https://github.com/llvm/llvm-project revision f09c4a3f5c0fd22d60069db47b845a097d1cdc46)
clang revision f09c4a3f5c0fd22d60069db47b845a097d1cdc46
llvm revision f09c4a3f5c0fd22d60069db47b845a097d1cdc46
Skipping the following test categories: msvcstl, dsym, pdb, gmodules, debugserver, objc

--
Command Output (stderr):
--
FAIL: LLDB (/__w/llvm-project/llvm-project/build/bin/clang-aarch64) :: test (TestConcurrentManySignals.ConcurrentManySignals.test)
======================================================================
FAIL: test (TestConcurrentManySignals.ConcurrentManySignals.test)
  Test 100 signals from 100 threads.
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/__w/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 156, in wrapper
   return func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/__w/llvm-project/llvm-project/lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentManySignals.py", line 16, in test
   self.do_thread_actions(num_signal_threads=100)
 File "/__w/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/concurrent_base.py", line 261, in do_thread_actions
   self.assertEqual(
AssertionError: 1 != 0 : Expected main thread (finish) breakpoint to be hit once
Config=aarch64-/__w/llvm-project/llvm-project/build/bin/clang
----------------------------------------------------------------------

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Tom Stellard (tstellar)

It's not failing all the time, just some times.

Here is the full log: https://github.com/llvm/llvm-project/actions/runs/19988753632/job/57326428935

Relevant part of the log:

FAIL: lldb-api :: functionalities/thread/concurrent_events/TestConcurrentManySignals.py (660 of 2628)
******************** TEST 'lldb-api :: functionalities/thread/concurrent_events/TestConcurrentManySignals.py' FAILED ********************
Script:
-
/usr/bin/python3 /__w/llvm-project/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env LLVM_LIBS_DIR=/__w/llvm-project/llvm-project/build/./lib --env LLVM_INCLUDE_DIR=/__w/llvm-project/llvm-project/build/include --env LLVM_TOOLS_DIR=/__w/llvm-project/llvm-project/build/./bin --libcxx-include-dir /__w/llvm-project/llvm-project/build/include/c++/v1 --libcxx-include-target-dir /__w/llvm-project/llvm-project/build/include/aarch64-unknown-linux-gnu/c++/v1 --libcxx-library-dir /__w/llvm-project/llvm-project/build/./lib/aarch64-unknown-linux-gnu --arch aarch64 --build-dir /__w/llvm-project/llvm-project/build/lldb-test-build.noindex --lldb-module-cache-dir /__w/llvm-project/llvm-project/build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /__w/llvm-project/llvm-project/build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /__w/llvm-project/llvm-project/build/./bin/lldb --compiler /__w/llvm-project/llvm-project/build/./bin/clang --dsymutil /__w/llvm-project/llvm-project/build/./bin/dsymutil --make /usr/bin/gmake --llvm-tools-dir /__w/llvm-project/llvm-project/build/./bin --lldb-obj-root /__w/llvm-project/llvm-project/build/tools/lldb --lldb-libs-dir /__w/llvm-project/llvm-project/build/./lib --cmake-build-type Release /__w/llvm-project/llvm-project/lldb/test/API/functionalities/thread/concurrent_events -p TestConcurrentManySignals.py
--
Exit Code: 1

Command Output (stdout):
--
ldb version 22.0.0git (https://github.com/llvm/llvm-project revision f09c4a3f5c0fd22d60069db47b845a097d1cdc46)
clang revision f09c4a3f5c0fd22d60069db47b845a097d1cdc46
llvm revision f09c4a3f5c0fd22d60069db47b845a097d1cdc46
Skipping the following test categories: msvcstl, dsym, pdb, gmodules, debugserver, objc

--
Command Output (stderr):
--
FAIL: LLDB (/__w/llvm-project/llvm-project/build/bin/clang-aarch64) :: test (TestConcurrentManySignals.ConcurrentManySignals.test)
======================================================================
FAIL: test (TestConcurrentManySignals.ConcurrentManySignals.test)
  Test 100 signals from 100 threads.
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/__w/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 156, in wrapper
   return func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/__w/llvm-project/llvm-project/lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentManySignals.py", line 16, in test
   self.do_thread_actions(num_signal_threads=100)
 File "/__w/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/concurrent_base.py", line 261, in do_thread_actions
   self.assertEqual(
AssertionError: 1 != 0 : Expected main thread (finish) breakpoint to be hit once
Config=aarch64-/__w/llvm-project/llvm-project/build/bin/clang
----------------------------------------------------------------------

[jimingham]

That's an odd failure. The code that's getting run up to the test failure is:

        # Run to completion (or crash)
        while not self.inferior_done():
            if self.TraceOn():
                self.runCmd("thread backtrace all")
            self.runCmd("continue")
            self.signal_count += self.count_signaled_threads()
            self.crash_count += len(
                lldbutil.get_crashed_threads(self, self.inferior_process)
            )

        if num_crash_threads > 0 or num_delay_crash_threads > 0:
            # Expecting a crash
            self.assertTrue(
                self.crash_count > 0,
                "Expecting at least one thread to crash. Details: %s"
                % "\t\n".join(self.describe_threads()),
            )

            # Ensure the zombie process is reaped
            self.runCmd("process kill")

        elif num_crash_threads == 0 and num_delay_crash_threads == 0:
            # There should be a single active thread (the main one) which hit
            # the breakpoint after joining
            self.assertEqual(
                1,
                self.finish_breakpoint.GetHitCount(),
                "Expected main thread (finish) breakpoint to be hit once",
            )

The failure is on the last self.assertEqual, with the error being that 0 != 1. So we seem to have hit the finish breakpoint 0 times. The finish breakpoint is set (explicitly by file & line number) on the last line of main here:

int main ()
{
    dotest();
    return 0; // Break here and verify one thread is active.
}

The test code gets to the point where it has created all the threads the test asked for (in this case 100 threads running code that send signals to themselves.) then it gets into the loop shown above and keeps continuing through all the stops till inferior_done is true, which is:

    def inferior_done(self):
        """Returns true if the inferior is done executing all the event threads (and is stopped at self.finish_breakpoint,
        or has terminated execution.
        """
        return (
            self.finish_breakpoint.GetHitCount() > 0
            or self.crash_count > 0
            or self.inferior_process.GetState() == lldb.eStateExited
	)

Since the test failure was that the hit-count of the finish_breakpoint was 0 not 1, and this test sets crash_count to 0, that means the only way "inferior_done" could return true is that the program has exited, though how it managed to do that without hitting that last breakpoint on the return from main is mysterious.