Skip to content

op-reth: print a backtrace when an integration test dies on a signal - #21910

Merged
sebastianst merged 1 commit into
developfrom
seb/eth69-crash-backtrace
Jul 30, 2026
Merged

op-reth: print a backtrace when an integration test dies on a signal#21910
sebastianst merged 1 commit into
developfrom
seb/eth69-crash-backtrace

Conversation

@sebastianst

@sebastianst sebastianst commented Jul 20, 2026

Copy link
Copy Markdown
Member

Stacked on #21909 — review that first; this PR's diff is the second commit only.

Makes the next recurrence of #20973 yield a faulting stack. Instrumentation only: no behavior change to any test, no effect unless a fatal signal fires.

p2p_version::peers_negotiate_eth_69 segfaults during teardown after its body has passed. A binary killed by a signal produces no Rust backtrace — RUST_BACKTRACE only covers panics — so five recurrences over two months have yielded nothing beyond the signal number, and it has never reproduced locally across 570+ attempts. Investigation is stalled for want of a stack.

Core dumps can't supply one. On the CircleCI Docker executor /proc/sys is read-only so core_pattern cannot be set, and the host's pipe pattern makes the kernel write the core outside the container — verified by running an actual segfault in the base image. An in-process handler needs no privileges, no artifacts and no CI config, and works locally.

The faulting thread name is the payload. It discriminates directly between the candidate causes enumerated in this analysis — a database atexit destructor racing its own live threads, background threads outliving process exit, or a runtime teardown race.

Output against a simulated teardown crash:

=== fatal signal 11 in thread 'mdbx-rs-txn-mgr' faulting address 0x0000000000000000 ===
.../deps/flaky-58aa09ba589b158c(+0x362db) [0x560a585772db]
/usr/lib/libc.so.6(+0x3e270) [0x7f6e7d23e270]
...

Two details worth knowing before reviewing the handler:

The stack is emitted in two stages, and the order is load-bearing. backtrace_symbols_fd runs first: it writes straight to the fd and does not call malloc. std::backtrace::Backtrace runs second for names and line numbers — its symbolization allocates heavily and faults a second time when called from a handler, killing the process before anything is printed, since SIGSEGV is blocked inside its own handler. That was the first version of this patch, and it printed nothing at all. Attempting it only after the reliable frames are already on the wire makes that failure free. Expect the symbolized section to be empty in practice.

It re-raises with the default disposition, so the process still dies with the original signal and nextest still records a crash rather than a pass. That composes with #21909: the failed attempt's stderr — this backtrace — is captured into the JUnit <flakyFailure> that #21909 keeps as an artifact and that triggers its Slack alert, so a recurrence yields a durable, retrievable stack even though the retry keeps the job green.

Resolution is module + offset, not function names, because Rust doesn't export symbols to the dynamic table; resolve offline with addr2line, or add -rdynamic later if it proves necessary.

libc is already a workspace dependency; added here as a dev-dependency only.

🤖 Co-created with Claude Opus 4.8 (1M context)

@wiz-0f98cca50a

wiz-0f98cca50a Bot commented Jul 20, 2026

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities 3 High 2 Medium 3 Low
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings -
Software Management Finding Software Management Findings -
Total 3 High 2 Medium 3 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

@sebastianst
sebastianst force-pushed the seb/eth69-flake-mitigation branch from d1cb297 to 76a128a Compare July 21, 2026 11:34
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch from 368a737 to b6d1aa1 Compare July 21, 2026 11:34
@sebastianst
sebastianst force-pushed the seb/eth69-flake-mitigation branch from 76a128a to 6606ce2 Compare July 21, 2026 14:59
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch from b6d1aa1 to de40a75 Compare July 21, 2026 14:59
@sebastianst
sebastianst force-pushed the seb/eth69-flake-mitigation branch from 6606ce2 to 6a9491a Compare July 21, 2026 15:17
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch from de40a75 to 72f81ae Compare July 21, 2026 15:17
@sebastianst
sebastianst force-pushed the seb/eth69-flake-mitigation branch from 6a9491a to 84eb07c Compare July 21, 2026 15:24
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch from 72f81ae to 0b2bdc9 Compare July 21, 2026 15:24
@einar-oplabs einar-oplabs self-assigned this Jul 22, 2026
@einar-oplabs
einar-oplabs self-requested a review July 22, 2026 18:34
@sebastianst
sebastianst force-pushed the seb/eth69-flake-mitigation branch from 84eb07c to 39a461b Compare July 28, 2026 15:11
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch from 0b2bdc9 to 423a6ac Compare July 28, 2026 15:11
@sebastianst
sebastianst force-pushed the seb/eth69-flake-mitigation branch from 39a461b to f86fd6f Compare July 28, 2026 15:37
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch from 423a6ac to 68d3adb Compare July 28, 2026 15:37

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preliminary Claude review.

Comment thread rust/op-reth/crates/node/tests/it/crash_backtrace.rs
Comment thread rust/op-reth/crates/node/tests/it/crash_backtrace.rs
Comment thread rust/op-reth/crates/node/tests/it/crash_backtrace.rs Outdated
Comment thread rust/op-reth/crates/node/tests/it/crash_backtrace.rs Outdated
Comment thread rust/op-reth/crates/node/tests/it/crash_backtrace.rs Outdated
sebastianst added a commit that referenced this pull request Jul 29, 2026
`reth-optimism-node::it p2p_version::peers_negotiate_eth_69` intermittently dies
with SIGSEGV after its body has already passed — the assertions succeed, nextest
prints `test result: ok`, and the binary then segfaults during teardown. The run
fails purely on the signal exit, which has repeatedly ejected unrelated PRs from
the merge queue.

Add a nextest retry scoped to that one test. Retrying is sound because the crash is
teardown-only: every assertion has already succeeded, and nextest reruns in a fresh
process, re-establishing them from scratch rather than papering over a failure. This
is a deliberate exception to the workspace's no-retries policy — a stopgap for an
unresolved crash tracked in #20973.

A retry turns the red job green, which would let recurrences pass unnoticed, and
CircleCI does not parse nextest's `<flakyFailure>` element (so store_test_results and
Insights stay blind to a flaky-pass). So the cargo-test job greps the JUnit report
for `<flakyFailure>` and pings Slack — on every branch, since the crash has never
reproduced locally and each occurrence is a rare chance at a stack. The step never
fails the job, and the JUnit is already kept as an artifact so the failed attempt's
stderr (the crash backtrace, once #21910 lands) is retrievable.

The Slack token comes from a dedicated `slack` CircleCI context, so the two
cargo-test jobs get a context list including it, not the shared `*rust-ci-context`
that 20+ lint/build jobs use and should not hold a bot token. The same gap silently
disabled `rust-deny`'s own `@protocol-oncall` develop-failure ping, so that job is
moved onto the slack context too, restoring the notification its comment promises.

Mitigation only — the crash mechanism is unresolved and tracked in #20973.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sebastianst
sebastianst force-pushed the seb/eth69-flake-mitigation branch from f86fd6f to 5530b90 Compare July 29, 2026 11:26
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch 2 times, most recently from 8968858 to f6a9652 Compare July 30, 2026 09:55
Base automatically changed from seb/eth69-flake-mitigation to develop July 30, 2026 10:21
`p2p_version::peers_negotiate_eth_69` intermittently segfaults during teardown,
after its body has already passed (#20973). A binary killed by a signal produces
no Rust backtrace — `RUST_BACKTRACE` only covers panics — so five recurrences
over two months have yielded nothing beyond the signal number, and the crash has
never reproduced locally across 570+ attempts. Investigation is currently stalled
for want of a faulting stack.

Core dumps are not an option here: on the CircleCI Docker executor `/proc/sys` is
read-only so `core_pattern` cannot be set, and the host's pipe pattern makes the
kernel write the core outside the container. An in-process handler needs no
privileges, no artifacts and no CI configuration, and works locally too.

Install a handler for SIGSEGV/SIGBUS/SIGILL/SIGFPE/SIGABRT that reports the faulting
thread's name and address, then its stack. The thread name alone discriminates
between the candidate causes: a database `atexit` destructor racing its own live
threads, background threads outliving process exit, or a runtime teardown race.

The stack is emitted in two stages, which matters:

  - `backtrace_symbols_fd` first. It writes straight to the fd and does not call
    malloc, so it is safe here. This is the guaranteed output.
  - `std::backtrace::Backtrace` second, for names and line numbers. Its
    symbolization allocates heavily and faults a second time when called from a
    handler — killing the process before anything is printed, since SIGSEGV is
    blocked inside its own handler. Attempting it only after the reliable frames
    are already on the wire means that failure costs nothing.

The handler re-raises with the default disposition, so the process still dies
with the original signal and nextest still records a crash. That composes with
the retry added in the parent commit: nextest prints a failed attempt's stderr
even when a later attempt passes, so a recurrence yields a stack while the job
stays green.

`libc` is already a workspace dependency; this adds it as a dev-dependency only.

Instrumentation only — no behavior change to any test, and no effect unless a
fatal signal fires.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sebastianst
sebastianst force-pushed the seb/eth69-crash-backtrace branch from f6a9652 to 765dd92 Compare July 30, 2026 10:22
@sebastianst
sebastianst marked this pull request as ready for review July 30, 2026 10:46
@sebastianst
sebastianst requested a review from a team as a code owner July 30, 2026 10:46
@sebastianst
sebastianst enabled auto-merge July 30, 2026 10:47
@sebastianst
sebastianst added this pull request to the merge queue Jul 30, 2026
Merged via the queue into develop with commit ae99e76 Jul 30, 2026
124 checks passed
@sebastianst
sebastianst deleted the seb/eth69-crash-backtrace branch July 30, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants