Add stateful EST generator and executor + first generated XEN test

[kamilchodola]

Summary

This PR adds two cooperating tools:

1. eest_stateful_generator.py — a one-shot orchestrator that:

   • boots a Nethermind node from a snapshot using OverlayFS (ephemeral upper),
   • pre-seeds a few engine payloads,
   • launches a mitmdump reverse proxy for JSON-RPC,
   • runs Execution Spec Tests (EST) remotely against the proxy,
   • saves produced payloads and cleans up.

2. mitm_addon.py — a mitmproxy addon that groups intercepted eth_sendRawTransaction calls by test metadata and produces engine payloads without ever triggering a reorg (reorg is fully disabled for now).

Both tools save artifacts under payloads/ and extensive logs under /root/mitm_addon.log (proxy) and /root/nethermind.log (container logs).

---

Why

• Create deterministic, reproducible execution payloads while running EST suites against a local node.
• Capture payloads grouped by test/phase for later replay.
• Keep the environment ephemeral via OverlayFS to avoid mutating the base snapshot.
• Disable reorg handling for now to simplify flows and avoid accidental reorg paths during early testing.

---

Important engine behavior note (custom Nethermind image)

This setup uses a custom Nethermind image:
nethermindeth/nethermind:gp-hacked

That image intentionally implements a non-standard variant of engine_getPayloadV4:

• ✅ It accepts additional parameters (e.g., our generator calls it like
  engine_getPayloadV4([txrlp_list_or_None, rpc_address_or_EMPTY])), which is not per the EL Engine API spec.
• ✅ It directly returns a block with the desired payload attributes without requiring a prior engine_forkchoiceUpdatedV3 carrying payload attributes.
  In other words, the block is produced immediately on getPayload, rather than first configuring payload attributes via FCU and then calling getPayload.

What’s in this PR

• eest_stateful_generator.py

  • Snapshot bootstrap (ethpandaops) with optional refresh/skip.
  • OverlayFS mount (lower=snapshot, upper/work ephemeral, merged mounted).
  • Nethermind container startup with Engine+RPC exposed.
  • Health checks for ports and JSON-RPC.
  • Chain ID detection (CLI > map > RPC fallback).
  • Engine pre-seeding (“gas-bump” and “funding”) to create initial payload(s).
  • Writes mitm_config.json for the addon, including a dynamic finalized_block.
  • Starts mitmdump on :8549 with the addon and runs EST via uv.
  • Graceful teardown and cleanup (unless --keep).

• mitm_addon.py

  • Buffers eth_sendRawTransaction by (file_base, test_name, phase) derived from request metadata (id JSON or X-EEST-ID header).
  • Quiet-period flush (1s) or forced flush on group switch.
  • Calls engine_getPayloadV4([txrlps, "EMPTY"]), saves payload JSON to payloads/<file>/<test>/<phase>/<n>.json, then submits engine_newPayloadV4 and engine_forkchoiceUpdatedV3 with head/safe/finalized anchored.
  • JWT auth for engine requests; verbose, redacted logging.

---

Step-by-step: eest_stateful_generator.py runtime flow

1. Parse arguments
   Reads chain, test path, fork, RPC seed info, snapshot flags, --keep, etc.

2. Bootstrap tools & repo

   • Ensures requests is installed.
   • Clones execution-spec-tests repo if missing.
   • Ensures uv is available, pins Python 3.11 for the repo, and runs uv sync --all-extras.

3. Snapshot prep

   • If not --no-snapshot, downloads the latest Nethermind snapshot for the selected chain (via Docker/alpine) to execution-data/, unless it already exists and --refresh-snapshot isn’t set.
   • If --no-snapshot, ensures execution-data/ exists (empty).

4. JWT setup
   Writes a random engine-jwt/jwt.hex if missing.

5. OverlayFS mount

   • Prepares overlay-upper/, overlay-work/, overlay-merged/.
   • Mounts OverlayFS with lowerdir=execution-data, upperdir=overlay-upper, workdir=overlay-work into overlay-merged/ (uses sudo if not root).

6. Nethermind container

   • Removes any existing eest-nethermind.
   • Starts nethermindeth/nethermind:gp-hacked with:
     • DB at /db (overlay-merged bind),
     • Engine on :8551, RPC on :8545, JWT at /jwt/jwt.hex,
     • generous txpool and gas limit settings,
     • dev/unsecure RPC auth disabled (for local use).

7. Health checks

   • Waits for 127.0.0.1:8545 to accept TCP.
   • Polls eth_blockNumber until RPC is responsive or fails with logs and cleanup.

8. Chain ID

   • Chooses rpc_chain_id from CLI; else mapping; else eth_chainId; else defaults to 1.

9. Mitmproxy install
   Ensures mitmproxy is available.

10. Engine pre-seeding

    • Tries to generate “gas-bump” payloads (saved under payloads/gas-bump-*.json) to increase gas limit to desired 1 TeraGas.
    • Generates a “funding” payload targeting --rpc-address; captures the resulting blockHash and uses it as finalized_block. Funding long.Max ETH amount so it is enough for any testing.

11. Add-on config

    • Writes mitm_config.json with:
      • rpc_direct = http://127.0.0.1:8545
      • engine_url = http://127.0.0.1:8551
      • jwt_hex_path = engine-jwt/jwt.hex
      • finalized_block = <block hash from funding prep>

12. Start mitmdump + addon

    • Verifies mitm_addon.py is present.
    • Launches mitmdump on :8549 in reverse mode to :8545, with addon and conservative connection settings.
    • Waits for :8549 to bind.

13. Run EST

    • Runs uv run execute remote -v --fork=<fork> --rpc-seed-key=<key> --rpc-chain-id=<id> --rpc-endpoint=http://127.0.0.1:8549 <tests_path> -- -m benchmark -n 1 in the EST repo.
    • All test transactions go through the proxy and are captured/produced by the addon.

14. Teardown

    • Unless --keep:
      • Saves Nethermind logs to /root/nethermind.log.
      • Stops/removes the container.
      • Unmounts OverlayFS.
      • Removes overlay-* dirs.
    • Always attempts to terminate mitmdump.
    • Prints Done.

Artifacts produced

• payloads/:
  • gas-bump-*.json and funding-*.json (pre-seed),
  • Per-test engine payloads at payloads/<file>/<test>/<phase>/<n>.json.
• Logs:
  • /root/mitm_addon.log (addon & proxy),
  • /root/nethermind.log (container logs).

---

Step-by-step: mitm_addon.py runtime flow

1. Load config (MITM_ADDON_CONFIG env or mitm_config.json).
2. Start background monitor thread for quiet-period flush.
3. Intercept requests:
   • Logs all POSTs.
   • For eth_sendRawTransaction, derives (file_base, test_name, phase) from id or X-EEST-ID.
   • Buffers txs until quiet period or group switch.
4. Flush (produce payload):
   • Logs txpool summary.
   • Calls engine_getPayloadV4([txrlps, "EMPTY"]) (custom image returns block immediately).
   • Saves payload JSON under payloads/<file>/<test>/<phase>/<n>.json.
   • Calls engine_newPayloadV4 + engine_forkchoiceUpdatedV3.
   • Updates finalized anchor if in global-setup.
   • Reorg is fully disabled.
5. Shutdown gracefully stops monitor thread.

---

Configuration & flags

Generator

• --chain (mainnet|sepolia|holesky|goerli|ethereum; default mainnet)
• --test-path (path inside EST repo; default tests)
• --fork (e.g., Prague; default Prague)
• --rpc-endpoint (defaults to proxy http://127.0.0.1:8549)
• --rpc-seed-key (required) seed private key for EST funding
• --rpc-address (required) address to pre-fund
• --rpc-chain-id (override chain id)
• --no-snapshot, --refresh-snapshot
• --keep (preserve container/mounts/logs)

Addon

• Env MITM_ADDON_CONFIG → path to mitm_config.json:

  {
    "rpc_direct": "http://127.0.0.1:8545",
    "engine_url": "http://127.0.0.1:8551",
    "jwt_hex_path": "engine-jwt/jwt.hex",
    "finalized_block": "0x..."
  }

[jochem-brouwer]

I need some help here, I likely have something setup wrong or have the wrong gp-hacked version. I am getting

[WARN] Gas bump failed: Engine error: {'code': -38005, 'message': 'unsupported fork'}
[WARN] Funding prep failed: Engine error: {'code': -38005, 'message': 'unsupported fork'}

I think the config of gp-hacked might be wrong here also, these unsupported fork errors are from Nethermind's engine API as far as I can see and I believe the config might think we are at block 0 and not at a mainnet block. I am using this to test mainnet scenarios and specifically researching the impact of raising gas limit to 60M in context of practical scenarios like the XEN transactions (heavy state edits) to see how all 5 clients handle this. CC @dmitriy-b. My TG handle is @jochembrouwer (no dash 😉 ). This tool looks fantastic and is already many steps ahead of me (OverlayFS on top of the snapshot), would love to use this 😄 👍

[jochem-brouwer]

Ok, it's definitely something wrong loading the snapshot:

/home/jochem-brouwer/gas-benchmarks/overlay-merged# ls
badBlocks  blobTransactions  blockInfos  blockNumbers  blocks  bloom  code  discoveryNodes  headers  keystore  logs  metadata  nethermind_db  peers  receipts  state

This is the snapshot downloaded from EthPandaOps.

Logs: (I executed the gp-hacked with the exact same commands as from the relevant python file here)

-----------------------------  Initialization Completed  -----------------------------

This node    : <REDACTED>
RPC modules  : Admin, Debug, Eth, Flashbots, Net, Trace, TxPool, Web3
Public id    : Nethermind/v1.33.0-unstable+ac781ead/linux-x64/dotnet9.0.8
Node address : <REDACTED>
JSON RPC     : http://0.0.0.0:8545 ; http://0.0.0.0:8551
Gaslimit     : 1,000,000,000,000
ExtraData    : Nethermind v1.33.0a
External IP  : <REDACTED>
Ethereum     : <REDACTED>
Discovery    : <REDACTED>
Client id    : Nethermind/v1.33.0-unstable+ac781ead/linux-x64/dotnet9.0.8
Chainspec    : chainspec/foundation.json
Chain head   : 0 (0xd4e567...cb8fa3)
Chain ID     : Mainnet
-------------------------------------------------------------------------------------- 

So it's somehow at chain head 0 🤔

EDIT: just discovered I can also read the logs from /root/nethermind.log. But it's at block 0 there also.

[jochem-brouwer]

Ok, I have moved the mainnet dir inside my snapshot inside a new directory nethermind_db. This seems to be the directory which Nethermind tries to load. Now it timeouts, but it is loading the chain, so this is progress 😄 👍

[kamilchodola]

OK so two things which are noted:

1. Script works only for root user - to be adjusted but not a huge issue for now
2. Snapshots downloaded for Nethermind are not having the exact expected structure by nethermind - Nethermind expects nethermind_db/mainnet/db_content while in snapshot there is only mainnet/db_content - so just needs to take an account for that