consensus: standardize slow block JSON output for cross-client metrics

[CPerezz]

Summary

Implement standardized JSON format for slow block logging to enable cross-client performance analysis and protocol research.

This change is part of the Cross-Client Execution Metrics initiative proposed by Gary Rong and CPerezz.

Motivation

Standardized execution metrics are critical for:

• Cross-client performance comparison
• Network health monitoring
• Data-driven protocol research

Real-world example: The EIP-7907 analysis used execution metrics to measure code read latency, per-call overhead scaling, and block execution breakdown. Without standardized metrics across clients, such analysis cannot be validated cross-client.

References

• Original proposal: https://hackmd.io/dg7rizTyTXuCf2LSa2LsyQ (Gary Rong)
• EIP-7907 analysis: https://ethresear.ch/t/data-driven-analysis-on-eip-7907/23850

Configuration

Two new flags in Blocks config section:

Flag	Default	Description
Blocks.SlowBlockThresholdMs	-1 (disabled)	Threshold in ms for logging slow blocks. Set to 1000 to log blocks slower than 1s. Set to 0 to log all blocks.
Blocks.SlowBlockPerTxThresholdMs	-1 (disabled)	Per-transaction threshold in ms. When enabled, individual transactions exceeding this threshold are included in the slow block JSON. Set to 0 to log all transactions.

Both flags default to -1 (disabled). When disabled, there is zero overhead on block processing — no metric captures, no delta computation, no JSON serialization.

When enabled, the overhead is ~31μs per block (~0.08% on a 40ms block) from reading thread-local counters. Per-tx timing adds ~6μs for a 150-tx block when enabled.

Example usage

# Log blocks slower than 1s with all transactions
--Blocks.SlowBlockThresholdMs=1000 --Blocks.SlowBlockPerTxThresholdMs=0

# Log blocks slower than 500ms, only transactions slower than 50ms
--Blocks.SlowBlockThresholdMs=500 --Blocks.SlowBlockPerTxThresholdMs=50

# Log every block, no per-tx breakdown
--Blocks.SlowBlockThresholdMs=0

JSON Output Format

{
  "level": "warn",
  "msg": "Slow block",
  "block": {
    "number": 22075123,
    "hash": "0xabc...",
    "gas_used": 29850000,
    "gas_limit": 30000000,
    "tx_count": 142,
    "blob_count": 6
  },
  "timing": {
    "execution_ms": 52.1,
    "evm_ms": 45.2,
    "blooms_ms": 1.3,
    "receipts_root_ms": 3.8,
    "commit_ms": 2.1,
    "storage_merkle_ms": 18.4,
    "state_root_ms": 5.2,
    "state_hash_ms": 23.6,
    "total_ms": 82.0
  },
  "throughput": {
    "mgas_per_sec": 364.1
  },
  "state_reads": {
    "accounts": 1842,
    "storage_slots": 3201,
    "code": 215,
    "code_bytes": 482310
  },
  "state_writes": {
    "accounts": 312,
    "accounts_deleted": 0,
    "storage_slots": 1805,
    "storage_slots_deleted": 12,
    "code": 3,
    "code_bytes": 8420,
    "eip7702_delegations_set": 0,
    "eip7702_delegations_cleared": 0
  },
  "cache": {
    "account": { "hits": 1520, "misses": 322, "hit_rate": 82.52 },
    "storage": { "hits": 2800, "misses": 401, "hit_rate": 87.48 },
    "code": { "hits": 180, "misses": 35, "hit_rate": 83.72 }
  },
  "evm": {
    "opcodes": 1542000,
    "sload": 3201,
    "sstore": 1805,
    "calls": 2412,
    "empty_calls": 48,
    "creates": 3,
    "self_destructs": 0,
    "contracts_analyzed": 118,
    "cached_contracts_used": 97
  },
  "transactions": [
    {
      "index": 42,
      "hash": "0xdef...",
      "gas_used": 8200000,
      "execution_ms": 12.3,
      "type": "Legacy",
      "to": "0x7a25..."
    },
    {
      "index": 107,
      "hash": "0x123...",
      "gas_used": 5100000,
      "execution_ms": 8.7,
      "type": "EIP1559",
      "to": "0xdead..."
    }
  ]
}

Timing field definitions

Field	Definition
total_ms	Wall-clock block processing time
execution_ms	total - state_hash - commit (backwards-compatible)
evm_ms	total - state_hash - commit - blooms - receipts_root (pure EVM)
blooms_ms	Bloom filter calculation (parallel)
receipts_root_ms	Receipts Patricia trie root computation
commit_ms	State commit without root recalculation
storage_merkle_ms	Storage trie merkleization (Commit(commitRoots: true))
state_root_ms	State root recalculation + commit tree
state_hash_ms	storage_merkle + state_root (total merkleization)

The transactions array is only present when SlowBlockPerTxThresholdMs >= 0.

Implementation

• All metrics use ZeroContentionCounter (lock-free thread-local counters) — no locks, no contention
• Timing uses Stopwatch.GetTimestamp() only at coarse block-phase boundaries (not per state read/write)
• JSON is serialized via Utf8JsonWriter + ArrayBufferWriter<byte> on the ThreadPool, not the block-processing thread
• Per-tx timing uses a thread-static array with one Stopwatch.GetTimestamp() per tx (~20ns)
• Delta computation and JSON output are queued to ThreadPool via UnsafeQueueUserWorkItem

[Copilot]

Pull request overview

This pull request implements standardized JSON format for slow block logging to enable cross-client performance analysis and protocol research. The changes add comprehensive execution metrics tracking and output them in a cross-client standardized JSON format when block processing exceeds a threshold.

Changes:

• Added ThreadLocal accessor methods to Db.Metrics for per-block cache statistics
• Defined new state access and timing metrics in Evm.Metrics (account/storage/code reads and writes)
• Integrated metric tracking into StateProvider and PersistentStorageProvider for state access monitoring
• Updated ProcessingStats to capture metrics and output slow block logs in standardized JSON format

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/Nethermind/Nethermind.State/StateProvider.cs	Adds account read tracking via EvmMetrics.IncrementAccountReads()
src/Nethermind/Nethermind.State/PersistentStorageProvider.cs	Adds storage read tracking via EvmMetrics.IncrementStorageReads()
src/Nethermind/Nethermind.Evm/Metrics.cs	Defines new state access and timing metrics with ThreadLocal accessors
src/Nethermind/Nethermind.Evm/CodeInfoRepository.cs	Adds code read and code bytes tracking
src/Nethermind/Nethermind.Db/Metrics.cs	Adds ThreadLocal accessors for cache hit metrics
src/Nethermind/Nethermind.Core/Threading/ZeroContentionCounter.cs	Adds long value support for metrics tracking
src/Nethermind/Nethermind.Consensus/Processing/ProcessingStats.cs	Implements slow block JSON logging with comprehensive metrics capture

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[LukaszRozmej]

Looks good, added some questions.

Food for thought @benaadams
With so many metrics, maybe it would be useful to have them continuous in memory, so we could make copies by copying memory region and potentially delta's using SIMD?

[benaadams]

Number of additional thread locals bothers me; probably better to have as normal statics but only write on main block (i.e. are these metrics relevent to any other background or parallel activities?)

They are only interesting if mainnet block is slow?

Also they are expensive to write to, so rolling up counts via class field, and then writing batch change would be better

[LukaszRozmej]

Number of additional thread locals bothers me; probably better to have as normal statics but only write on main block (i.e. are these metrics relevent to any other background or parallel activities?)

They are only interesting if mainnet block is slow?

Also they are expensive to write to, so rolling up counts via class field, and then writing batch change would be better

I'm kind of thinking it shouldn't be a static, or the only threadstatic should be an object (struct?). That would require bigger changes.

[flcl42]

Can user just Prague.Instance or so instead of getting it from the spec provider

[CPerezz]

Yes but this will need to be maintained always. In fact, we would have to use Osaka here to be sure we are fwd. compatible no?

I updated anyways, but I'm not fully convinced..

[flcl42]

It's just so using mainnet timestamps for single spec provider does not look perfect to me. Btw you can init a _releaseSpec some where with value like Prague.Instance or Osaka.Instance and reuse it.

If you mean that you would better use just the top mainnet spec, we might try to introduce Mainnet.Instance that points to the most recent fork and we change it when time comes. Can't recall of something existing like this

[flcl42]

@benaadams @LukaszRozmej maybe we can make some generic argument trick to avoid calling metric if don't need to(if MetricsOff is passed metrics are excluded due to unconditional if)?

[CPerezz]

@LukaszRozmej @benaadams Can yoou look at ead23b9 and see if that cuts it for you?

[LukaszRozmej]

@CPerezz I think it is fine for now, but it will break when parallel execution comes in. Regardless if it is Block-STM or BAL based.

[asdacap]

Maybe this can help. https://github.com/NethermindEth/nethermind/blob/master/src/Nethermind/Nethermind.Init/Modules/MainProcessingContext.cs#L39. There is a HintGet for get that is cached by prewarmer.

[CPerezz]

@CPerezz I think it is fine for now, but it will break when parallel execution comes in. Regardless if it is Block-STM or BAL based.

Hey @LukaszRozmej thanks! Indeed the plan is to also start metering based on BALs via: https://notes.ethereum.org/@ethpandaops/bal-otel

[CPerezz]

@benaadams @LukaszRozmej any way we can merge this this week?

So far Reth and Besu are almost done, so you're the last team we are waiting on such that the stateful tests come with more insights.

LMK if you need any help 🙏