lokahi: serve the experimental opstack block-building namespace on chain routes

[claude]

Requested by Sebastian Stammler · Slack thread

Before

A lokahi chain route refused every opstack_* call with -32601 Method not found. op-supernode's virtual op-nodes serve the experimental block-building namespace on every chain's route (op-node/node/node.go registerAPIs, gated on ExperimentalOPStackAPI, which the devstack's makeNodeCfg always sets), and the op-test-sequencer's standard builder/committer/publisher drive block building through it: opstack_openBlockV1, opstack_cancelBlockV1, opstack_sealBlockV1, opstack_commitBlockV1, opstack_publishBlockV1. Under lokahi, every acceptance test that sequences through the test sequencer (dsl/sequencer.go New()/Next()) failed at the first opstack_openBlockV1 — the 18-test failure bucket of CircleCI job 5510201 (e.g. unsafe_head_test.go:82).

After

Each chain's route serves the namespace with op-node's semantics, exactly when the chain's configuration asks for it, and the devstack asks for it the way it does on op-supernode.

How

op-node is the spec, and each lokahi chain is a kona node, so the namespace is implemented in kona and registered where op-node registers its own — in the node's RPC module set, which lokahi's launcher routes under /<chain-id>:

• kona-rpc — opstack jsonrpsee namespace with wire types matching the Go types field for field (eth.BlockID, eth.PayloadInfo, opsigner.SignedExecutionPayloadEnvelope; serde tests pin Go-shaped JSON), and op-node's -401xx build error codes (op-service/apis/opstack.go). RpcBuilder gains experimental_opstack, default off.
• kona-node-service — OpStackRpc (actors/rpc/opstack.rs) delegates as op-node's does:
  • openBlockV1 = EngineController.OpenBlock: parent-exists check, then a direct engine_forkchoiceUpdated with head = parent and the given attributes (safe/finalized labels from the engine-state watch), with startPayload's error mapping code for code.
  • sealBlockV1/cancelBlockV1 = SealBlock/CancelBlock: a direct engine_getPayload versioned by the job's timestamp; sealing does not canonicalize. An unknown payload id is -40120.
  • commitBlockV1 = CommitBlock: engine_newPayload + unsafe-head move + canonicalizing FCU. Those writes move state the ChainController owns, so the request goes through its queue as a new ChainControllerRequest::CommitBlock.
  • publishBlockV1 = OpNode.PublishBlock: the signed envelope goes out on the chain's gossip topic with the caller's signature as given (the publish channel item can now carry a pre-made signature; the sequencer's own unsigned path is unchanged).
• kona-engine — new CommitTask: an InsertTask with a result channel, following SealTask's always-succeeds-with-channel pattern, so the caller hears the newPayload verdict once instead of the queue retrying (or silently dropping an invalid payload) behind their back.
• kona-node — --rpc.experimental-opstack-api, so a standalone kona-node can serve the namespace too.
• lokahi — experimental-opstack-api in the TOML ([defaults] and per chain, default off), passed into each chain's RpcBuilder.
• op-devstack — the generated lokahi TOML sets experimental-opstack-api = true in [defaults], mirroring makeNodeCfg's ExperimentalOPStackAPI: true for op-supernode's virtual nodes.

Deliberate, documented divergences from op-node:

• publishBlockV1 schedules the payload onto the network actor's publish queue and answers; op-node awaits the gossipsub publish. A gossip failure is logged by the network actor rather than returned (the devstack dsl ignores publish errors either way).
• commitBlockV1 refuses a payload that does not descend from the local-safe head (kona's unsafe-head admission rule); op-node has no counterpart. The refusal is returned to the caller, not silently dropped like the gossip path drops it.
• open/seal/cancel are direct engine calls, as in op-node, but kona serializes them against the queue's own engine traffic only through the execution layer, not through op-node's controller mutex; the payload build job lives in the EL, so a concurrent queue FCU does not invalidate it.

Testing

• rust/lokahi/tests/opstack_route.rs (two_chains.rs style, real binary against stubbed L1/EL): fail-first verified — with the namespace unregistered, the flag-on test fails with -32601 where op-node behaviour is expected; with it, both tests pass. Gate off → -32601; gate on → openBlockV1 refuses a missing parent with op-node's message and code, sealBlockV1 fails as the -40199 build error, commitBlockV1 reaches the controller and returns the engine's refusal, publishBlockV1 schedules and answers, and the namespace stays off the supernode root.
• kona-engine CommitTask unit tests (refusal answered + no retry; invalid payload reaches the caller once); kona-rpc serde parity tests against Go JSON; lokahi config-resolution test; devstack TOML test.
• cargo test -p lokahi -p lokahi-interop -p kona-engine -p kona-rpc -p kona-node-service -p kona-node, scoped clippy --all-features --all-targets -- -D warnings, nightly fmt.
• A full local acceptance-test run does not fit this machine's disk (contracts + release Rust builds); the 18-test bucket is measured via the combined branch's CI run after merging this into lokahi-interop: combined measurement branch for the interop suites (DO NOT MERGE) #22634.

Measured (CircleCI job 5511460, head b9383a3 on #22634)

The RPC seam works end to end: zero Method not found in the whole run. In TestReorgInitExecMsg's log the test sequencer's forced reorg goes through this namespace and sticks — committer.go:54 "Committed block to op-stack" block=a3727f..:13, op-reth canonicalizes the sibling 13–15, lokahi's own sequencer continues on the new fork at 16+ (so CommitTask's state update mirrors op-node's SetUnsafeHead + FCU correctly), and the DSL confirms reorg on divergence block on that chain. TestReorgUnsafeHead (the seam bucket's pure-reorg test) went fail → pass.

The other former Method-not-found tests now progress past sequencing and stop at pre-existing lokahi gaps that this PR does not claim to implement:

1. Interop invalidation is not applied. The verifier detects the invalid block and reaches Decision::Invalidate, but the phase deliberately holds instead of applying it — lokahi-interop/src/verifier.rs progress(): Decision::Invalidate | Decision::Rewind => Self::hold(..), logging Interop verification reached a decision this phase does not apply every 500 ms round. The deposits-only replacement never happens, so the dependent chain's head never reorgs (expected head to reorg … after 30–50 attempts). Affects TestReorgInitExecMsg, TestReorgInvalidExecMsgs/*, and the supernode reorg/replacement tests; TestL2ReorgAfterL1Reorg (failing identically before this PR) sits on the same phase's unapplied Rewind decision. op-supernode's counterpart is the supervisor-driven invalidate → deposits-only replacement flow; in lokahi that is the verifier's decision-application phase, a separate feature.
2. cross-unsafe is not tracked. kona has no cross-unsafe head; lokahi/src/query/wire.rs documents cross_unsafe_l2 decoding as zero Go-side. TestPostInteropUpgradeComprehensive waits for cross-unsafe ≥ 3 and reads 0 forever.
3. Supernode query gaps. TestInteropFaultProofs / TestPreinteropFaultProofs* reach no optimistic output for chain N at timestamp T: QueryChain::optimistic_at answers None on ChainAt::NotYet, which ChainAt::from_snapshot also returns for a local-safe head whose L1 pairing is Unpaired; lokahi also has no invalidated-output archive (documented in lokahi/src/query/chain.rs). TestFPP fails with no longer records which L1 block made its block at timestamp T safe — safe-db early-history, failing identically before this PR (job 5510201).
4. TestSequencingWindowExpiry: lokahi exits with Error: chain 901: EngineTask(Consolidate(SealTaskFailed(UnsafeHeadChangedSinceBuild))) — the consolidation deposit-replacement path's seal (kona-engine seal/task.rs, parent-vs-unsafe-head check) has no result channel, so the error escalates to Critical (seal/error.rs) and halts the node; the identical crash appears in job 5510201, before this PR.
5. interop/proofs/serial 30 m TestMain timeout is downstream of 1 and 3: TestInteropFaultProofs_IntraBlock's subtests burn 300–663 s retry budgets serially and the package alarm fires 21 m 30 s in; several IntraBlock subtests newly pass inside.

The one pass→fail signal, interop/message TestInitExecMsgWithDSL (16 s pass → 30 m silent hang, empty JUnit body), does not touch this PR's code paths: the test never drives the test sequencer, and the test sequencer has no auto-loop (fullseq.Sequencer.forceStart returns ErrNotImplemented), so no opstack_* method is ever called in that test; the namespace registration is inert without calls. The same head passed the op-node matrix job (5511459), and 12 of 13 tests in the same package passed on the lokahi run. A rerun of the lokahi job at the same head (job 5511920) confirms it: TestInitExecMsgWithDSL passes in 17.4 s. Against the pre-opstack baseline (5510201), the rerun's only stable delta is TestReorgUnsafeHead fail → pass; the rerun's one new one-off, TestPreinteropSingleChainFaultProofs, fails with the pre-existing safe-db signature (no longer records which L1 block …, gap 3 above) and passed at the same head in 5511460.

Deliberately unchanged after the measurement: commitBlockV1's local-safe-descendant refusal. None of the measured failures implicate it — the reorg tests stop the batchers before forking, so every commit landed above local-safe and succeeded — and the flow that would commit at or below local-safe (invalid-block replacement) rewinds derivation first in op-node too, so its commits also land above the rewound local-safe.

Review notes: the repo's rust-code-reviewer/go-code-reviewer agents could not be dispatched from this session; the diff was self-reviewed against docs/ai/rust-dev.md and docs/ai/go-dev.md instead.

Part of #22547

---

Generated by Claude Code