op-reth: print a backtrace when an integration test dies on a signal - #21910
Merged
Conversation
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio. |
sebastianst
force-pushed
the
seb/eth69-flake-mitigation
branch
from
July 21, 2026 11:34
d1cb297 to
76a128a
Compare
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
from
July 21, 2026 11:34
368a737 to
b6d1aa1
Compare
sebastianst
force-pushed
the
seb/eth69-flake-mitigation
branch
from
July 21, 2026 14:59
76a128a to
6606ce2
Compare
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
from
July 21, 2026 14:59
b6d1aa1 to
de40a75
Compare
sebastianst
force-pushed
the
seb/eth69-flake-mitigation
branch
from
July 21, 2026 15:17
6606ce2 to
6a9491a
Compare
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
from
July 21, 2026 15:17
de40a75 to
72f81ae
Compare
sebastianst
force-pushed
the
seb/eth69-flake-mitigation
branch
from
July 21, 2026 15:24
6a9491a to
84eb07c
Compare
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
from
July 21, 2026 15:24
72f81ae to
0b2bdc9
Compare
einar-oplabs
self-requested a review
July 22, 2026 18:34
sebastianst
force-pushed
the
seb/eth69-flake-mitigation
branch
from
July 28, 2026 15:11
84eb07c to
39a461b
Compare
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
from
July 28, 2026 15:11
0b2bdc9 to
423a6ac
Compare
sebastianst
force-pushed
the
seb/eth69-flake-mitigation
branch
from
July 28, 2026 15:37
39a461b to
f86fd6f
Compare
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
from
July 28, 2026 15:37
423a6ac to
68d3adb
Compare
einar-oplabs
approved these changes
Jul 29, 2026
einar-oplabs
left a comment
Contributor
There was a problem hiding this comment.
Preliminary Claude review.
sebastianst
added a commit
that referenced
this pull request
Jul 29, 2026
`reth-optimism-node::it p2p_version::peers_negotiate_eth_69` intermittently dies with SIGSEGV after its body has already passed — the assertions succeed, nextest prints `test result: ok`, and the binary then segfaults during teardown. The run fails purely on the signal exit, which has repeatedly ejected unrelated PRs from the merge queue. Add a nextest retry scoped to that one test. Retrying is sound because the crash is teardown-only: every assertion has already succeeded, and nextest reruns in a fresh process, re-establishing them from scratch rather than papering over a failure. This is a deliberate exception to the workspace's no-retries policy — a stopgap for an unresolved crash tracked in #20973. A retry turns the red job green, which would let recurrences pass unnoticed, and CircleCI does not parse nextest's `<flakyFailure>` element (so store_test_results and Insights stay blind to a flaky-pass). So the cargo-test job greps the JUnit report for `<flakyFailure>` and pings Slack — on every branch, since the crash has never reproduced locally and each occurrence is a rare chance at a stack. The step never fails the job, and the JUnit is already kept as an artifact so the failed attempt's stderr (the crash backtrace, once #21910 lands) is retrievable. The Slack token comes from a dedicated `slack` CircleCI context, so the two cargo-test jobs get a context list including it, not the shared `*rust-ci-context` that 20+ lint/build jobs use and should not hold a bot token. The same gap silently disabled `rust-deny`'s own `@protocol-oncall` develop-failure ping, so that job is moved onto the slack context too, restoring the notification its comment promises. Mitigation only — the crash mechanism is unresolved and tracked in #20973. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sebastianst
force-pushed
the
seb/eth69-flake-mitigation
branch
from
July 29, 2026 11:26
f86fd6f to
5530b90
Compare
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
2 times, most recently
from
July 30, 2026 09:55
8968858 to
f6a9652
Compare
`p2p_version::peers_negotiate_eth_69` intermittently segfaults during teardown, after its body has already passed (#20973). A binary killed by a signal produces no Rust backtrace — `RUST_BACKTRACE` only covers panics — so five recurrences over two months have yielded nothing beyond the signal number, and the crash has never reproduced locally across 570+ attempts. Investigation is currently stalled for want of a faulting stack. Core dumps are not an option here: on the CircleCI Docker executor `/proc/sys` is read-only so `core_pattern` cannot be set, and the host's pipe pattern makes the kernel write the core outside the container. An in-process handler needs no privileges, no artifacts and no CI configuration, and works locally too. Install a handler for SIGSEGV/SIGBUS/SIGILL/SIGFPE/SIGABRT that reports the faulting thread's name and address, then its stack. The thread name alone discriminates between the candidate causes: a database `atexit` destructor racing its own live threads, background threads outliving process exit, or a runtime teardown race. The stack is emitted in two stages, which matters: - `backtrace_symbols_fd` first. It writes straight to the fd and does not call malloc, so it is safe here. This is the guaranteed output. - `std::backtrace::Backtrace` second, for names and line numbers. Its symbolization allocates heavily and faults a second time when called from a handler — killing the process before anything is printed, since SIGSEGV is blocked inside its own handler. Attempting it only after the reliable frames are already on the wire means that failure costs nothing. The handler re-raises with the default disposition, so the process still dies with the original signal and nextest still records a crash. That composes with the retry added in the parent commit: nextest prints a failed attempt's stderr even when a later attempt passes, so a recurrence yields a stack while the job stays green. `libc` is already a workspace dependency; this adds it as a dev-dependency only. Instrumentation only — no behavior change to any test, and no effect unless a fatal signal fires. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sebastianst
force-pushed
the
seb/eth69-crash-backtrace
branch
from
July 30, 2026 10:22
f6a9652 to
765dd92
Compare
sebastianst
marked this pull request as ready for review
July 30, 2026 10:46
sebastianst
enabled auto-merge
July 30, 2026 10:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #21909 — review that first; this PR's diff is the second commit only.
Makes the next recurrence of #20973 yield a faulting stack. Instrumentation only: no behavior change to any test, no effect unless a fatal signal fires.
p2p_version::peers_negotiate_eth_69segfaults during teardown after its body has passed. A binary killed by a signal produces no Rust backtrace —RUST_BACKTRACEonly covers panics — so five recurrences over two months have yielded nothing beyond the signal number, and it has never reproduced locally across 570+ attempts. Investigation is stalled for want of a stack.Core dumps can't supply one. On the CircleCI Docker executor
/proc/sysis read-only socore_patterncannot be set, and the host's pipe pattern makes the kernel write the core outside the container — verified by running an actual segfault in the base image. An in-process handler needs no privileges, no artifacts and no CI config, and works locally.The faulting thread name is the payload. It discriminates directly between the candidate causes enumerated in this analysis — a database
atexitdestructor racing its own live threads, background threads outliving process exit, or a runtime teardown race.Output against a simulated teardown crash:
Two details worth knowing before reviewing the handler:
The stack is emitted in two stages, and the order is load-bearing.
backtrace_symbols_fdruns first: it writes straight to the fd and does not call malloc.std::backtrace::Backtraceruns second for names and line numbers — its symbolization allocates heavily and faults a second time when called from a handler, killing the process before anything is printed, since SIGSEGV is blocked inside its own handler. That was the first version of this patch, and it printed nothing at all. Attempting it only after the reliable frames are already on the wire makes that failure free. Expect the symbolized section to be empty in practice.It re-raises with the default disposition, so the process still dies with the original signal and nextest still records a crash rather than a pass. That composes with #21909: the failed attempt's stderr — this backtrace — is captured into the JUnit
<flakyFailure>that #21909 keeps as an artifact and that triggers its Slack alert, so a recurrence yields a durable, retrievable stack even though the retry keeps the job green.Resolution is module + offset, not function names, because Rust doesn't export symbols to the dynamic table; resolve offline with
addr2line, or add-rdynamiclater if it proves necessary.libcis already a workspace dependency; added here as a dev-dependency only.🤖 Co-created with Claude Opus 4.8 (1M context)