Fix Pectra/EIP-7549 attestation over-counting, add Erigon (Caplin) compatibility, repo & test rework

[stakepeter]

TL;DR

• This service watches a set of Ethereum validators and alerts when they miss attestations or proposals (via Slack + Prometheus). After Ethereum's Pectra hard fork (May 2025), it began inflating "served attestations" by ~34% and firing false "delayed attestation" alerts on healthy validators.
• Root cause: Pectra's EIP-7549 changed how validator votes are packaged into blocks. The same vote can now legitimately appear in multiple blocks, but this service counted every appearance as a separate event. See the deep-dive section below for diagrams.
• Fix-and-cleanup: dedup attestation inclusions (the headline fix), plus add support for our Erigon-based staging endpoint, restructure the repo, replace the test suite with one driven by real captured beacon-chain JSON, add request-level metrics, and apply ~70 small reliability fixes that fell out of the triage.

What this project does (for reviewers unfamiliar with it)

eth2-monitor is a small Go CLI that connects to an Ethereum beacon-chain node (a "consensus layer" RPC endpoint), watches a configured list of validator public keys, and reports:

• missed block proposals,
• missed attestations,
• attestations that were eventually included but later than expected,
• blocks the proposer left empty (no transactions / no blobs / no MEV value),
• proposed blocks that don't match any known MEV-relay bid.

Reports go to a Slack webhook; structured metrics go to Prometheus on port 1337. It runs as a long-lived daemon and is typically deployed alongside a beacon node operated by the same team.

Why this PR exists

The branch began as triage of staging alerts firing on real, healthy validators. Root-causing the alerts revealed two things:

1. Pectra/EIP-7549 broke the original attestation counter. This service was written before Pectra and assumed each vote appeared in exactly one block.
2. An unrelated crash on our Erigon staging endpoint. Erigon's bundled beacon-chain client ("Caplin") returns 404 in a slightly different way than the Prysm/Lighthouse clients the original code was tested against, and the resulting nil-deref crashed the monitor at startup whenever the first slot of an epoch was missed.

Fixing those two things confidently required better test coverage — which made it worth restructuring the monolithic monitoring package and capturing real beacon-chain responses as test fixtures. From there the branch picked up ~70 small reliability and ergonomics fixes. End result: ~165 commits / +10,889 / −1,557 lines across 121 files.

Glossary

One-line definitions of the jargon used below

• Slot / Epoch — A slot is 12 seconds. 32 slots make an epoch (~6.4 minutes). Each slot has at most one block.
• Attestation — A validator's vote on the chain head; broadcast once per epoch, then included by other validators in their block at slot+1 (or later if there's congestion / missed slots).
• Attestation inclusion distance — How many slots elapsed between when the vote was emitted and the block that recorded it. 0 is optimal (next slot), higher = worse.
• Pectra — Ethereum's May-2025 hard fork.
• EIP-7549 — Pectra-era change that moved committee-selection metadata outside the signed portion of an attestation, allowing aggregators to re-package the same votes into different committee groupings. Result: the same vote can legitimately appear in multiple blocks.
• Caplin — Erigon's bundled beacon-chain client (an alternative to running a separate Prysm/Lighthouse/Teku/Nimbus/Lodestar node).
• Hoodi — A public Ethereum testnet, used by our staging environment.
• MEV — Maximal Extractable Value, the surplus a validator collects for ordering transactions in a particular way. Validators get it via "MEV relays" — third-party services that auction blocks and publish bid traces.
• Must() — Internal panic-on-error helper. We're converting "should-never-happen" call sites to detect ctx.Canceled explicitly so a clean shutdown doesn't panic through it.

What changes for operators of this service

Visible in dashboards / alerts (deploy will change graphs):

• Attestation counts drop ~34%. That's the over-counting going away — not a regression. The same validators are doing the same work.
• "Delayed attestation" false-positive alerts stop firing in normal operation. Real delays (e.g. attestations stranded by a long run of missed proposals) still alert.
• Attestation-distance histogram concentrates near 0–1 instead of having a long synthetic tail at 4–7 caused by re-inclusion.
• New top-level metric: ETH2_beaconAPIRequestsTotal / …DurationSeconds — every call to the beacon-chain RPC is now counted by endpoint, HTTP method, and status class. Use this to spot upstream-RPC issues.
• New diagnostics: ETH2_duplicateAttestationsSkipped (the dedup counter — confirms the fix is filtering anything), ETH2_crossEpochAttestations (votes recorded in a later epoch than they were emitted), ETH2_missedSlotsInEpoch (correlates directly with distance spikes), ETH2_rawAttestationDistances (raw distance before correcting for missed slots — compare against the corrected histogram to separate network delay from validator delay), ETH2_totalProposedEmptyBlocks, ETH2_totalMissingBidTraces, ETH2_lastProposedEmptyBlockSlot.

Not visible in dashboards (but matter):

• No longer crashes on Erigon (Caplin) staging. Fixes a real panic from 2026-05-13.
• No longer hangs on shutdown or double-processes the last epoch after a restart.
• Cache writes are crash-durable. Local validator-index cache uses atomic tmpfile + fsync + rename + dir-fsync. 30-minute TTL means newly-active or newly-slashed validators become visible within the window without a restart.
• MEV-relay calls are bounded (4 MiB body cap, per-relay HTTP timeout, ctx-aware retry, capped pagination). Pubkey + blockhash comparisons are now case-insensitive so vanilla-block detection isn't fooled by hex case.

Bigger picture, drill-down

Erigon (Caplin) support

• go.mod redirected to a stakefish fork of go-eth2-client via replace directive: github.com/stakefish/go-eth2-client@feat/erigon-caplin-support (commit 781f0c7f). The fork natively tolerates a few JSON shapes that Caplin emits and the upstream library rejected. Application imports are unchanged. Revertible by removing the replace directive once upstream merges the fix.
• GetValidatorIndexes now probes forward through the slots of an epoch when Caplin returns 404 on the first slot — Caplin resolves "validators at slot N" by walking to the block at that exact slot, which doesn't exist if slot N was missed. Walking forward to the next existing block in the same epoch gives the same answer (validator set is stable within an epoch).
• Regression test TestGetValidatorIndexes_ProbeForwardOnMissedFirstSlot plus the live-beacon E2E test get_validator_indexes_roundtrip cover this.

Repo restructure and monitoring decomposition

• Moved to golang-standards/project-layout. Entrypoint is cmd/eth2-monitor/main.go; library code lives under internal/{beaconchain,monitoring,cli,opts,spec}/. The old pkg/ directory is removed.
• The monolithic pkg/monitoring.go (553 lines, all concerns mixed) is split into per-concern files in internal/monitoring/:
  • monitoring.go — orchestrator loop + SSE head-stream subscription.
  • epoch_context.go — per-epoch state fetch (validator keys, attester duties, proposer duties, committee lookup, blocks, MEV bids).
  • attestations.go — attestation-issue detection (dedup, distance computation, delayed-over-tolerance).
  • proposals.go — proposal-issue detection (missed, empty, vanilla, MEV mismatch).
  • metrics.go — MonitorMetrics struct + factory accepting a prometheus.Registerer (production uses default; tests inject isolated registries).
  • reporting.go — Slack + log fanout helpers.
  • mev.go — MEV relay bid-trace fetching with concurrent paginated retrieval.
  • cache.go — disk-backed JSON cache for validator-index lookups.
  • plus set.go, profiling.go, utilities.go, doc.go.

Test infrastructure rewrite

• New tools/fixturegen/ records real beacon-chain JSON by subscribing to /eth/v1/events?topics=head on the configured RPC endpoint, then tailing new heads and recording every block until a "scenario predicate" matches.
• Captured fixtures land under internal/beaconchain/testdata/beacon/<chain>/<scenario>/ (currently hoodi; new chains = new sub-tree, no code changes needed).
• Six scenarios are captured, each exercising one of the validator-failure-detection paths: happy_path, missed_proposal, empty_block, delayed_attestation, cross_epoch_attestation, plus a _shared/ directory of chain-invariant startup probes.
• Each scenario carries a meta.json recording the anchor slot/epoch + scenario-specific fields (e.g. missed_slot, max_distance, prev_canonical_slot). Tests anchor assertions to these fields instead of hard-coded slot numbers so they stay green across refresh runs. No upstream endpoint identifier is recorded in the committed fixtures.
• Most monitoring + beaconchain tests are now fixture-backed integration tests driving the real BeaconChain client against an httptest server that replays captured JSON / SSE / MEV-relay responses. Synthetic unit tests are retained only for narrow classification edge cases (cross-epoch dedup, AggregationBits offset drift).
• Live-beacon end-to-end tests are gated behind a build tag (e2e) and excluded from CI; run locally with make test-e2e.

CI rework

• golangci-lint bumped to v2.12.2 (was v1) via golangci/golangci-lint-action@v8.
• New test job (go test ./...) gates the release build.
• New coverage job prints per-package coverage to the job log (no artifact, no third-party service).
• Multi-arch build matrix (amd64+arm64; linux/darwin/freebsd/windows).
• Go bumped to 1.25.10 (.tool-versions, go.mod, both workflows).

Reliability / correctness fixes (the long tail)

• ctx.Canceled / context.DeadlineExceeded detected explicitly in SubscribeToEpochs, startup finality fetch, BuildEpochContext, and MEV requests — returned cleanly rather than panicking through Must.
• Defer-order deadlock on metrics-server shutdown fixed; SSE goroutine drains on orchestrator exit; epochs channel closed via defer.
• Off-by-one re-emit of the persisted epoch on restart fixed (counters no longer drift after a restart).
• Cache write uses atomic tmpfile + fsync + rename + dir-fsync, tmpfile co-located with destination to avoid EXDEV. 30-minute TTL on cached validator indices, including the VALIDATOR_INDEX_INVALID sentinel.
• MEV relay: 4 MiB body cap, per-relay HTTP timeout, case-insensitive pubkey + blockhash compare, ctx-aware retry backoff, 64-page pagination cap with underflow guard.
• LoadCache capped at 64 MiB; null-body cache file no longer panics on next save.
• Slack POST bounded by 5-second client timeout; non-2xx responses are logged; nil-deref on transport failure fixed.
• Many nil-guards across attestation / proposal / SSE / cache / MEV paths (block / message / body / data / execution-payload / event-pointer / duty-entry).
• Slashed validators silently excluded from monitoring (now documented as a gotcha — surprising during incident response).
• Log calls migrated from printf-style to zerolog structured fields (Uint64("slot", ...).Msg(...)) for greppability.
• Grafana dashboard rewrite — test-env/grafana/dashboards/eth2-monitor.json (+1890 lines), new panels for the new metrics and a general restyle.
• CLAUDE.md expanded by ~275 lines (architecture, fixtures, gotchas, debugging workflows). Per-package doc comments throughout internal/monitoring/.

Full metric reference (before/after table)

New metrics:

Metric	Type	Purpose
ETH2_duplicateAttestationsSkipped	Counter	(validator, slot) dedup hits — confirms the EIP-7549 fix is filtering re-aggregated inclusions.
ETH2_crossEpochAttestations	Counter	Attestations included in a different epoch than the one they attested to.
ETH2_rawAttestationDistances	Histogram	Raw inclusion distance before the missed-slot adjustment (compare against ETH2_canonicalAttestationDistances to isolate network-vs-validator delay).
ETH2_missedSlotsInEpoch	Gauge	Slots without a proposed block in the most recently processed epoch.
ETH2_totalProposedEmptyBlocks	Counter	Blocks with no execution-layer payload of value (no transactions, no blobs, no Pectra exec requests).
ETH2_totalMissingBidTraces	Counter	Proposed blocks where no tracked MEV relay returned any bid trace.
ETH2_lastProposedEmptyBlockSlot	Gauge	Last empty-block slot.
ETH2_beaconAPIRequestsTotal	CounterVec	Beacon-chain RPC requests by endpoint, method, status_class.
ETH2_beaconAPIRequestDurationSeconds	HistogramVec	Beacon-chain RPC request latency by endpoint, method.

Behavior-corrected metrics (same name, more accurate values):

Metric	Before	After
ETH2_totalServedAttestations	Inflated ~34% by duplicate inclusions	Accurate (deduped)
ETH2_totalDelayedAttestationsOverTolerance	False positives from later re-inclusions of the same vote	Only genuine delays (e.g. attestations stranded by a long run of missed proposals)
ETH2_canonicalAttestationDistances	Polluted with re-inclusion distances (d=4–7)	Concentrated at d=0–1; tail only from real network delay

Removed API calls (no longer needed):

Endpoint	Reason
GET /eth/v1/beacon/states/{state}/committees	Replaced by duty-based committee info built from AttesterDuty responses. Saves ~30s/epoch on networks with large validator sets.
GET /eth/v1/beacon/headers/{block_id}	GetBlockHeader was dead code.

Dedup deep-dive (EIP-7549)

This is the headline fix. Worth understanding even if you skip the rest.

How EIP-7549 changed attestations

Before Pectra (May 2025), each attestation was locked to one committee:

Attestation = signed(slot=100, committee=5, votes=[1,0,1,1])
                              ^^^^^^^^^^^
                              part of signature — can't change

EIP-7549 moved the committee selection outside the signature for ~64× faster BLS verification:

Attestation = {
    signed:  (slot=100, votes=[1,0,1,1])     ← committee NOT signed
    unsigned: CommitteeBits=[0,0,0,0,0,1,0]  ← committee 5, can be re-aggregated
}

The problem

Since committee selection is no longer signed, the same vote can be re-packaged and included in multiple blocks:

sequenceDiagram
    participant V as Validator 42
    participant S100 as Slot 100
    participant B101 as Block 101
    participant B105 as Block 105
    participant B108 as Block 108

    V->>S100: Attests (committee 5)

    Note over B101: Attestation A<br/>committees: [5]
    S100-->>B101: val 42 included (d=0 ✅)

    Note over B105: Attestation B (re-aggregated)<br/>committees: [2, 5, 7]
    S100-->>B105: val 42 included again (d=4 ⚠️)

    Note over B108: Attestation C (re-aggregated)<br/>committees: [5]
    S100-->>B108: val 42 included again (d=7 🚨)

Loading

The Ethereum protocol allows this — re-including an attestation is a no-op for rewards (has_flag() is idempotent in the consensus spec). But this service counted every inclusion as if it were a separate event:

flowchart LR
    subgraph OLD ["Old code (no dedup)"]
        direction TB
        O1["Block 101: d=0"] --> OC["Counted: 3 inclusions"]
        O2["Block 105: d=4"] --> OC
        O3["Block 108: d=7"] --> OC
        OC --> OA["served=3, max d=7<br/>🚨 ALERT fires"]
    end

    subgraph NEW ["Fixed code (with dedup)"]
        direction TB
        N1["Block 101: d=0 ✅ first seen"] --> NC["Counted: 1 inclusion"]
        N2["Block 105: SKIP ↩️ already seen"] -.-> NC
        N3["Block 108: SKIP ↩️ already seen"] -.-> NC
        NC --> NA["served=1, d=0<br/>✅ No alert"]
    end

Loading

The fix

Track which (validator, slot) pairs have been seen. Process blocks in slot order so the earliest (canonical) inclusion wins:

flowchart TD
    START["For each block (sorted by slot)"] --> ATT["For each attestation"]
    ATT --> VAL["For each validator in attestation"]
    VAL --> CHECK{"(validator, slot)<br/>already seen?"}
    CHECK -->|Yes| SKIP["Skip duplicate<br/>DuplicateAttestationsSkipped++"]
    CHECK -->|No| RECORD["Record distance<br/>Mark as seen"]
    RECORD --> DIST{"distance > 2?"}
    DIST -->|Yes| ALERT["DelayedOverTolerance++"]
    DIST -->|No| OK["CanonicalAttestations++"]
    SKIP --> VAL
    ALERT --> VAL
    OK --> VAL

Loading