lokahi: serve superroot_atTimestamp on each chain's route, as op-supernode does

Every route of the Go op-supernode answers superroot_atTimestamp, because
op-node registers the superroot namespace on its RPC unconditionally
(op-node/node/node.go, registerAPIs): the root answers over the whole chain
set, and each /<chainID> route answers op-node's single-chain implementation
(op-node/node/superroot_api.go). kona-node has no such namespace, so a lokahi
chain route built from kona's method set alone refused the call with "Method
not found".

The consumers that dial a chain route for it are real. The devstack's
per-chain proposers are pointed at the chain CL's own URL
(op-devstack/sysgo/singlechain_runtime.go, SuperRootRpcs = l2CL.UserRPC()),
and the anchor of a newly added game type is read the same way
(op-devstack/sysgo/add_game_type.go). In the first full lokahi acceptance run
(CircleCI job 5508932) this was the 19-test failure bucket: those proposers
retried superroot_atTimestamp against kona's per-chain module to end of log
while the challenger and the super proposer, dialling the same socket's root,
were answered.

The route's answer is composed exactly as the root composes its
pre-verification branch, specialised to one chain: the same QueryChain reads,
the same Aggregate arithmetic, the same from_handoff composition — so the
wire shape and the commitment are bit-identical to what the root states for a
set of one, which is also what op-node serves (it consults no verifier; a
single-chain node has none). supernode_syncStatus stays off the chain routes,
as it is off op-node's RPC.

kona registers a chain's route while the chain composes, so the route's extra
methods are deposited before composition and hold a OnceLock handle filled
once the chain exists — the same binding-before-composition the root's query
API uses; a call in that window is answered "still starting". A deposited
method that collides with one kona serves fails the chain's launch loudly
rather than shadowing either.

The two_chains integration test now asserts each chain's route answers
superroot_atTimestamp for exactly its own chain and refuses
supernode_syncStatus. It failed before this change with the same error the
acceptance run recorded, and passes after.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEDeDWcHt2X1mZYhvHmnZ5

Author: claude

rust/lokahi/src/query/mod.rs

@@ -194,6 +194,100 @@ impl QueryHandle {
     }
 }
 
+/// The handle a chain's own route holds on query state that does not exist when the route's
+/// methods are declared: `superroot_atTimestamp` for that one chain.
+///
+/// This is the Go stack's addressing, mirrored. op-node registers the `superroot` namespace on
+/// its RPC unconditionally (`op-node/node/node.go`, `registerAPIs`), so every route of the Go
+/// op-supernode answers `superroot_atTimestamp`: the root serves the whole set's super root, and
+/// each chain's route serves that chain's own — op-node's single-chain implementation
+/// (`op-node/node/superroot_api.go`), which is what dispute infrastructure pointed at one chain
+/// reads. kona-node has no such namespace, so a lokahi route built from kona's methods alone
+/// refused the call with "Method not found" — and the consumers that dial a chain route for it
+/// are real: the devstack's per-chain proposers (`op-devstack/sysgo/singlechain_runtime.go` sets
+/// `SuperRootRpcs` to the chain CL's own URL) and the anchor read when a game type is added
+/// (`op-devstack/sysgo/add_game_type.go`).
+///
+/// A [`OnceLock`] behind the module for the same reason [`QueryHandle`] is one behind the root's:
+/// kona registers the chain's route while the chain composes, so the module must be deposited
+/// before composition — before the queues it answers from exist. A call landing in that window is
+/// answered with [`QueryError::Starting`] rather than with an unregistered method.
+///
+/// The set-wide `supernode_syncStatus` is deliberately *not* served here: op-node's RPC has no
+/// `supernode` namespace, so a chain route that answered it would be a surface the Go supernode
+/// does not have.
+#[derive(Debug, Clone, Default)]
+pub(crate) struct ChainRouteQueries(Arc<OnceLock<QueryChain>>);
+
+impl ChainRouteQueries {
+    /// Publishes the composed chain's reads to the route's RPC methods.
+    pub(crate) fn compose(&self, chain: QueryChain) {
+        if self.0.set(chain).is_err() {
+            // Unreachable: each chain composes once. Reported rather than panicked on, because a
+            // route that is already answering correctly should not be stopped by it.
+            warn!(target: "lokahi", "A chain route's query API was composed twice; keeping the first");
+        }
+    }
+
+    /// Returns the composed chain, or says the supernode is still starting.
+    fn state(&self) -> Result<&QueryChain, QueryError> {
+        self.0.get().ok_or(QueryError::Starting)
+    }
+
+    /// Builds the RPC module serving the chain route's `superroot` namespace from this handle.
+    pub(crate) fn into_rpc_module(
+        self,
+    ) -> Result<RpcModule<()>, jsonrpsee::core::RegisterMethodError> {
+        let mut module = RpcModule::new(());
+        module.merge(SuperrootQueryApiServer::into_rpc(ChainRouteQuery { handle: self }))?;
+        Ok(module)
+    }
+}
+
+/// The server behind one chain route's `superroot` namespace.
+#[derive(Debug)]
+struct ChainRouteQuery {
+    /// The chain this route answers for.
+    handle: ChainRouteQueries,
+}
+
+impl ChainRouteQuery {
+    /// Answers `superroot_atTimestamp` for the one chain, as op-node answers it.
+    ///
+    /// op-node's `superrootAPI.atTimestamp` consults no verifier — a single-chain node has none —
+    /// so its answer is always the optimistic single-chain super root: the chain's output at the
+    /// timestamp, paired with the L1 block its safe-head history says made it safe, or an
+    /// omit-chain response when the chain has not derived that far. That is exactly the root's
+    /// handoff branch specialised to one chain, so it is composed from the same pieces: the same
+    /// [`QueryChain`] reads, the same [`Aggregate`] arithmetic (a minimum over one), and the same
+    /// [`WireSuperRootData::from_handoff`] composition, which keeps the wire shape and the
+    /// commitment bit-identical to what the root would state for a set of one.
+    async fn at_timestamp(&self, timestamp: u64) -> Result<WireSuperRootAtTimestamp, QueryError> {
+        let chain = self.handle.state()?;
+        let mut statuses = BTreeMap::new();
+        statuses.insert(chain.chain_id(), chain.sync_status().await?);
+        let aggregate = Aggregate::of(statuses);
+
+        let mut optimistic = BTreeMap::new();
+        if let Some(output) = chain.optimistic_at(timestamp).await? {
+            optimistic.insert(chain.chain_id(), output);
+        }
+
+        let data = WireSuperRootData::from_handoff(timestamp, &aggregate, &optimistic);
+        Ok(WireSuperRootAtTimestamp::new(&aggregate, &optimistic, aggregate.current_l1, data))
+    }
+}
+
+#[async_trait::async_trait]
+impl SuperrootQueryApiServer for ChainRouteQuery {
+    async fn superroot_at_timestamp(
+        &self,
+        timestamp: WireU64,
+    ) -> RpcResult<WireSuperRootAtTimestamp> {
+        Ok(self.at_timestamp(timestamp.0).await?)
+    }
+}
+
 /// The server behind both namespaces.
 #[derive(Debug, Clone)]
 struct QueryRpc {

rust/lokahi/src/rpc.rs

@@ -16,8 +16,11 @@
 //! Each chain's method set is built by kona, inside [`RollupNode::compose`], from the channels
 //! that chain's actors read. lokahi cannot rebuild it and does not try: it hands kona a launcher
 //! ([`SharedRpcServerLauncher`]) which takes the finished [`RpcModule`] and registers it as a
-//! route instead of binding it to a socket. So the per-chain method set stays kona's, byte for
-//! byte, including the HTTP middleware a standalone kona-node serves it behind.
+//! route instead of binding it to a socket. So kona's methods reach the route as kona built them,
+//! including the HTTP middleware a standalone kona-node serves them behind. What a route serves
+//! *beyond* kona's set — Go op-supernode's chain routes answer `superroot_atTimestamp`, because
+//! each virtual op-node registers that namespace itself — is deposited per chain through
+//! [`SupernodeRpc::add_chain_methods`] and merged at registration.
 //!
 //! [`RollupNode::compose`]: kona_node_service::RollupNode::compose
 
@@ -89,6 +92,14 @@ type Handler = Arc<
 struct Routes {
     /// Route by chain id's decimal string: the path segment a caller writes.
     chains: RwLock<HashMap<String, Option<Handler>>>,
+    /// Supernode-owned methods merged into a chain's route when it registers, by the same key.
+    ///
+    /// This is how a chain's route comes to serve more than kona's method set, the way the Go
+    /// op-supernode's routes do: op-node registers a `superroot` namespace of its own, so every
+    /// virtual node op-supernode serves under `/<chainID>` answers `superroot_atTimestamp`.
+    /// kona builds its module set from its actors and lokahi does not reopen it; what lokahi
+    /// serves per chain beyond it is deposited here and merged at registration.
+    extras: RwLock<HashMap<String, Methods>>,
     /// The handler for `/`: the supernode's own namespaces.
     root: RwLock<Option<Handler>>,
     /// The stop handle every service built from this table holds.
@@ -234,6 +245,7 @@ impl SupernodeRpc {
         let (stop_handle, server_handle) = stop_channel();
         let routes = Arc::new(Routes {
             chains: RwLock::new(chain_ids.into_iter().map(|id| (id.to_string(), None)).collect()),
+            extras: RwLock::new(HashMap::new()),
             root: RwLock::new(None),
             stop_handle,
             _server_handle: server_handle,
@@ -268,6 +280,25 @@ impl SupernodeRpc {
     pub(crate) fn launcher(&self, chain_id: u64) -> SharedRpcServerLauncher {
         Arc::new(ChainLauncher { routes: Arc::clone(&self.routes), chain_id: chain_id.to_string() })
     }
+
+    /// Adds supernode-owned methods to chain `chain_id`'s route, merged when the route registers.
+    ///
+    /// Must be called before the chain is composed, because kona registers the route *while*
+    /// composing: the supernode deposits each chain's extra methods first and only then composes
+    /// it, so a route never registers before its extras are here. The methods therefore exist
+    /// before the state they answer from — the same ordering the root's query API lives with —
+    /// and hold a handle that is filled once the chain exists. A method that collides with one
+    /// kona already serves fails the chain's launch loudly rather than serving one of the two
+    /// quietly — if kona grows one of these methods natively, the right move is to stop supplying
+    /// it here, not to shadow it.
+    pub(crate) fn add_chain_methods(&self, chain_id: u64, methods: impl Into<Methods>) {
+        let _ = self
+            .routes
+            .extras
+            .write()
+            .expect("the extras lock is never held across a panic")
+            .insert(chain_id.to_string(), methods.into());
+    }
 }
 
 impl Drop for SupernodeRpc {
@@ -411,7 +442,26 @@ struct ChainLauncher {
 impl RpcServerLauncher for ChainLauncher {
     type Handle = Route;
 
-    async fn launch(&self, modules: RpcModule<()>) -> Result<Self::Handle, std::io::Error> {
+    async fn launch(&self, mut modules: RpcModule<()>) -> Result<Self::Handle, std::io::Error> {
+        // The supernode's own additions to this chain's route, deposited before the actors
+        // started. Merged into kona's set rather than replacing anything in it; a collision is a
+        // real conflict — two implementations of one method on one route — and fails the launch
+        // with the method's name rather than picking one silently.
+        let extra = self
+            .routes
+            .extras
+            .read()
+            .expect("the extras lock is never held across a panic")
+            .get(&self.chain_id)
+            .cloned();
+        if let Some(extra) = extra {
+            modules.merge(extra).map_err(|err| {
+                std::io::Error::other(format!(
+                    "chain {} route: a supernode-supplied method collides with kona's: {err}",
+                    self.chain_id
+                ))
+            })?;
+        }
         self.routes.register(&self.chain_id, modules.into());
         Ok(Route::new(Arc::clone(&self.routes), self.chain_id.clone()))
     }

rust/lokahi/src/supernode.rs

@@ -4,7 +4,7 @@ use crate::{
     admin,
     config::{L1Settings, ResolvedChain, ResolvedConfig, SequencerSettings},
     interop::{ChainInterop, HostedChain, InteropActor, InteropTestHandle},
-    query::{QueryChain, QueryHandle},
+    query::{ChainRouteQueries, QueryChain, QueryHandle},
     rpc::SupernodeRpc,
 };
 use alloy_primitives::{Address, B256};
@@ -250,6 +250,24 @@ impl Supernode {
             let datadir = chain.settings.datadir.clone();
             let rollup_config = chain.rollup_config.clone();
             let interop_state = chain.interop.clone();
+
+            // The chain's own route serves `superroot_atTimestamp` for that one chain, over the
+            // same reads as the set-wide answer at the root. This mirrors the Go op-supernode's
+            // routes, whose virtual op-nodes each register a `superroot` namespace of their own;
+            // the devstack's per-chain proposers and game-anchor reads dial the chain route for
+            // it. Deposited before composing, because kona registers the route *while* the chain
+            // composes; like the root's query API, the methods exist first and are handed their
+            // chain below, once it exists.
+            let route_queries = ChainRouteQueries::default();
+            if let Some(rpc) = rpc.as_ref() {
+                rpc.add_chain_methods(
+                    chain_id,
+                    route_queries.clone().into_rpc_module().with_context(|| {
+                        format!("failed to build the route query API of chain {chain_id}")
+                    })?,
+                );
+            }
+
             let ComposedChain {
                 actors: chain_actors,
                 l1_watcher_ports,
@@ -268,16 +286,24 @@ impl Supernode {
             // local-safe head itself and reports a timestamp behind it as unavailable history —
             // which is what it is. Interop turns that recording on, and every consumer of these
             // methods runs against an interop cluster.
-            query_chains.push(QueryChain::new(
-                chain_id,
-                Arc::new(rollup_config.clone()),
-                QueuedEngineRpcClient::new(controller_rpc_request_tx.clone()),
-                l1_query_tx,
-                interop_state.as_ref().map_or_else(
-                    || Arc::new(DisabledDatabase) as SharedSafeDb,
-                    |state| state.safe_db.clone(),
-                ),
-            ));
+            let shared_rollup_config = Arc::new(rollup_config.clone());
+            let safe_db = interop_state.as_ref().map_or_else(
+                || Arc::new(DisabledDatabase) as SharedSafeDb,
+                |state| state.safe_db.clone(),
+            );
+            let query_chain = || {
+                QueryChain::new(
+                    chain_id,
+                    Arc::clone(&shared_rollup_config),
+                    QueuedEngineRpcClient::new(controller_rpc_request_tx.clone()),
+                    l1_query_tx.clone(),
+                    safe_db.clone(),
+                )
+            };
+            query_chains.push(query_chain());
+
+            // Past this point the chain exists, so its route's query API can answer for it.
+            route_queries.compose(query_chain());
 
             // A promoter exists exactly when the chain was composed with an externally fed
             // cross-safe head, which is exactly when interop is on. Taking it here is what makes

rust/lokahi/tests/two_chains.rs

@@ -120,6 +120,50 @@ async fn two_chains_answer_on_one_socket_under_their_own_routes() {
         "an absent super root omits `data` rather than sending null: {superroot}"
     );
     assert_eq!(superroot["chain_ids"], json!([CHAIN_A.to_string(), CHAIN_B.to_string()]));
+
+    // Each chain's route answers `superroot_atTimestamp` for that one chain. This is the Go
+    // stack's addressing: op-node registers the `superroot` namespace on its RPC unconditionally
+    // (`op-node/node/node.go`, `registerAPIs`), so every route of the Go op-supernode serves the
+    // method — the root over the whole set, each chain's route over that chain alone. Real
+    // consumers dial the chain route for it: the devstack's per-chain proposers
+    // (`op-devstack/sysgo/singlechain_runtime.go`, `SuperRootRpcs = l2CL.UserRPC()`) and the
+    // anchor read when a game type is added (`op-devstack/sysgo/add_game_type.go`). A route built
+    // from kona's method set alone refuses the call with "Method not found", which is how those
+    // consumers failed against lokahi while the root answered.
+    for chain_id in [CHAIN_A, CHAIN_B] {
+        let client = HttpClientBuilder::default()
+            .build(node.chain_url(chain_id))
+            .expect("build a chain rpc client");
+        let superroot = client
+            .request::<Value, _>("superroot_atTimestamp", rpc_params!["0x7fffffff"])
+            .await
+            .unwrap_or_else(|err| {
+                panic!("the route of chain {chain_id} did not answer superroot_atTimestamp: {err}")
+            });
+        assert_eq!(
+            superroot["chain_ids"],
+            json!([chain_id.to_string()]),
+            "the route of chain {chain_id} must answer for exactly that chain: {superroot}"
+        );
+        assert_eq!(
+            superroot["optimistic_at_timestamp"],
+            json!({}),
+            "the chain has not derived that timestamp: {superroot}"
+        );
+        assert!(
+            superroot.get("data").is_none(),
+            "an absent super root omits `data` on a chain route too: {superroot}"
+        );
+
+        // The set-wide method stays off the chain route, as it is off op-node's RPC: what the
+        // whole chain set has derived is the root's answer, and a chain route that answered it
+        // would be a surface the Go supernode does not have.
+        let err = client.request::<Value, _>("supernode_syncStatus", rpc_params![]).await;
+        assert!(
+            err.is_err(),
+            "supernode_syncStatus must not answer on chain {chain_id}'s route: it is a root method"
+        );
+    }
 }
 
 /// One chain's execution layer being unreachable is that chain's problem.