Gloas serve envelope rpc by eserilev · Pull Request #8896 · sigp/lighthouse

eserilev · 2026-02-24T09:14:59Z

Issue Addressed

Serves envelope by range and by root requests. Added PayloadEnvelopeStreamer so that we dont need to alter upstream code when we introduce blinded payload envelopes.

dapplion · 2026-02-25T19:34:59Z

beacon_node/network/src/network_beacon_processor/rpc_methods.rs

+        let requested_envelopes = request.beacon_block_roots.len();
+        let mut envelope_stream = match self
+            .chain
+            .get_payload_envelopes_checking_caches(request.beacon_block_roots.to_vec())


Why is it necessary to have this stream gadget instead of a simple for loop?

for block_root in block_roots { if Some(payload) = chain.get_payload_checking_caches(block_root) { send_to_peer(payload) } }

We're going to use getPayloadBodiesByRange which allows us to fetch a batch of payloads in one engine api request. If we did the simple for loop we'd have to make an engine api request for each payload

…oas-serve-envelope-rpc

jimmygchen

The new protocols need to be added to this function:

lighthouse/beacon_node/lighthouse_network/src/rpc/protocol.rs

Line 389 in ffc2b97

fn currently_supported(fork_context: &ForkContext) -> Vec<ProtocolId> {

Added some tests to catch this kind of bug:

#8920

beacon_node/lighthouse_network/src/rpc/codec.rs

beacon_node/lighthouse_network/src/rpc/methods.rs

jimmygchen · 2026-03-02T01:34:18Z

beacon_node/lighthouse_network/src/rpc/protocol.rs

+                rpc_block_limits_by_fork(fork_context.current_fork_name())
+            }
+            Protocol::PayloadEnvelopesByRoot => {
+                rpc_block_limits_by_fork(fork_context.current_fork_name())


we may need a rpc_payload_envelope_limits function and update the rpc_block_limits_by_fork to include Gloas block size limit.

updated rpc_block_limits_by_fork and introduced rpc_payload_envelope_limits

We still need to set rpc_block_limits_by_fork max to be bellatrix_max for historical sync compatibility

beacon_node/lighthouse_network/src/rpc/protocol.rs

beacon_node/network/src/network_beacon_processor/tests.rs

jimmygchen · 2026-03-02T02:03:22Z

beacon_node/beacon_chain/src/execution_payload_envelope_streamer.rs

+                // TODO(gloas) we'll want to use the execution layer directly to call
+                //  the engine api method eth_getBlockByHash()
+                self.store
+                    .get_payload_envelope(&beacon_block_root)


synchronous db read inside async task here, use a blocking task (spawn_blocking)

added spawn_blocking. I had to introduce a new span_blocking_and_await fn inside task executor. its the same spawn_blocking wrapper we have in beacon chain. I did this because I didnt want envelope streamer to rely on beacon chain.

beacon_node/beacon_chain/src/execution_payload_envelope_streamer.rs

…oas-serve-envelope-rpc

jimmygchen · 2026-03-06T11:42:37Z

beacon_node/lighthouse_network/src/rpc/codec.rs

+            RequestType::PayloadEnvelopesByRoot(PayloadEnvelopesByRootRequest {
+                beacon_block_roots: RuntimeVariableList::from_ssz_bytes(
+                    decoded_buffer,
+                    spec.max_request_blocks(current_fork),


This should be spec.max_request_payloads() to be consistent with the other fixes

jimmygchen

The new protocols need count validation here too, otherwise a peer can make us iterate the full chain instead of the expected max 128 slots:

lighthouse/beacon_node/lighthouse_network/src/rpc/handler.rs

Line 957 in e1d3dcc

_ => {}

jimmygchen · 2026-03-06T12:04:49Z

beacon_node/beacon_chain/src/execution_payload_envelope_streamer.rs

+        let results = match self.load_envelopes(&beacon_block_roots).await {
+            Ok(results) => results,
+            Err(e) => {
+                send_errors(beacon_block_roots, sender, e).await;


may be worth logging some errors/warn here?

added warn log

…oas-serve-envelope-rpc

eserilev · 2026-03-14T08:47:26Z

This is ready for a review. We're skipping envelope verification here, since we dont have fork choice yet.

…conChain

dapplion · 2026-03-16T00:37:21Z

Suggestion: avoid wrapping canonical_head in Arc — BeaconChain is already behind an Arc everywhere, so the streamer can hold Arc<BeaconChain<T>> instead.

Implement CanonicalHeadReader on BeaconChain<T> (delegating to self.canonical_head) and pass self.clone() to the streamer. This avoids the structural change to the canonical_head field.

See: dapplion@3e6afae31

dapplion · 2026-03-16T01:39:16Z

Also, BeaconBlockGloas::full() is dead code — it's defined but never called (SignedExecutionPayloadEnvelope::max_size() computes the size arithmetically instead). Beyond being unused, the impl is fragile: it starts from empty(spec) then mutates fields with push loops, so adding a new field compiles fine but silently leaves it empty. A direct struct construction would catch missing fields at compile time.

Removed it along with 9 unused imports it pulled in: dapplion@ed8c5495f

…oas-serve-envelope-rpc

mergify · 2026-03-16T06:11:15Z

Some required checks have failed. Could you please take a look @eserilev? 🙏

eserilev · 2026-03-16T06:14:34Z

@dapplion your changes make sense, i went ahead and merged them. thanks!

pawanjay176 · 2026-03-13T23:59:29Z

consensus/types/src/execution/execution_payload_envelope.rs

 use tree_hash_derive::TreeHash;

-#[derive(Debug, Clone, Serialize, Encode, Decode, Deserialize, TestRandom, TreeHash, Educe)]
+#[derive(


Can we create an explicit empty() method for the default case?

pawanjay176 · 2026-03-14T00:01:25Z

consensus/types/src/execution/signed_execution_payload_envelope.rs

+
+    /// Returns the maximum SSZ-encoded size.
+    #[allow(clippy::arithmetic_side_effects)]
+    pub fn max_size() -> usize {


might be better to have a max_size on the ExecutionPayloadEnvelope as well and use it here instead so we don't forget to update it if the spec changes

pawanjay176 · 2026-03-14T00:20:33Z

common/task_executor/src/lib.rs


+/// Error type for spawning a blocking task and awaiting its result.
+#[derive(Debug)]
+pub enum SpawnBlockingError {


I'm not a fan of doing this without refactoring the spawn_blocking_handle function in beacon_chain.rs to also use this.
We'll just end up having multiple functions that do the same thing for no reason.

this is no longer needed due to the adapter pattern change and has been removed

pawanjay176 · 2026-03-14T00:22:35Z

beacon_node/beacon_chain/src/execution_payload_envelope_streamer.rs

+    // TODO(gloas) remove expect when execution layer field
+    // is no longer dead.
+    #[expect(dead_code)]
+    execution_layer: ExecutionLayer<T::EthSpec>,


Why don't we start with using the beacon chain here instead and do beacon_chain.execution_layer when we need to access it? that way we get to keep pretty similar logic to the beaocn block streamer.
We need the beacon_chain to access the db anyway

If testing is the concern, we can take a similar approach to the fetch_blobs adapter.

Here, it seems like we need ExecutionLayer, CanonicalHead and the store all of which are present in the BeaconChain

ive went with the adapter pattern

pawanjay176 · 2026-03-14T00:25:44Z

beacon_node/beacon_chain/src/execution_payload_envelope_streamer.rs

+            Ok(Some(cached_envelope))
+        } else {
+            // TODO(gloas) we'll want to use the execution layer directly to call
+            //  the engine api method eth_getBlockByHash()


do you mean getPayloadBodiesByRange or some equivalent thing that returns in the right format for the CL?

yes, i've updated the comment

…oas-serve-envelope-rpc

eserilev · 2026-03-17T05:16:48Z

went ahead with the adapter pattern, i think its the cleaner approach. thanks pawan for suggesting it. theres a bit of code churn from those changes, sorry about that... this should be ready for another round of reviews

mergify · 2026-03-17T07:35:43Z

This pull request has merge conflicts. Could you please resolve them @eserilev? 🙏

dapplion · 2026-03-18T22:53:23Z

🤖 Note on memory amplification (inherited pattern, not new to this PR):

load_envelopes() materializes all results into a Vec before streaming, and launch_stream() uses an unbounded_channel(). With max_request_payloads = 128 and practical envelope sizes ~2MB (gas-limited), one request buffers ~256MB. Per-peer limits allow 4 concurrent requests (2 by-range + 2 by-root via MAX_CONCURRENT_REQUESTS=2), so ~1GB per peer. With 220 max peers, theoretical worst case is ~220GB.

(lion) However in practice the Beacon processor has limited worker threads, so not all 1024 queued items run simultaneously, on high CPU machine it may be a slight concern.

The BlockStreamer has the exact same pattern and has been running in production, so this isn't a regression — just worth noting as the envelope payloads can be larger than blocks. Rate limiting (128 items/10s per peer) and peer scoring provide some mitigation.

eserilev added 3 commits February 24, 2026 00:55

Serve rpc by range and by root:

ffc2b97

fix spelling issues

e966428

Delete dead code

45912fb

eserilev requested a review from jxs as a code owner February 24, 2026 09:15

eserilev added gloas ready-for-review The code is ready for review labels Feb 24, 2026

dapplion reviewed Feb 25, 2026

View reviewed changes

Merge branch 'unstable' of https://github.com/sigp/lighthouse into gl…

5127054

…oas-serve-envelope-rpc

jimmygchen requested changes Mar 2, 2026

View reviewed changes

jimmygchen reviewed Mar 2, 2026

View reviewed changes

beacon_node/lighthouse_network/src/rpc/codec.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Mar 2, 2026

View reviewed changes

beacon_node/lighthouse_network/src/rpc/methods.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Mar 2, 2026

View reviewed changes

beacon_node/lighthouse_network/src/rpc/protocol.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Mar 2, 2026

View reviewed changes

beacon_node/network/src/network_beacon_processor/tests.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Mar 2, 2026

View reviewed changes

jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Mar 2, 2026

jimmygchen mentioned this pull request Mar 2, 2026

Add test for protocol registration completeness #8920

Open

ethDreamer reviewed Mar 2, 2026

View reviewed changes

beacon_node/beacon_chain/src/execution_payload_envelope_streamer.rs Outdated Show resolved Hide resolved

eserilev added 4 commits March 2, 2026 23:28

Merge branch 'unstable' of https://github.com/sigp/lighthouse into gl…

ff0c05e

…oas-serve-envelope-rpc

Use max request payloads

0212e84

Fixes based on feedback

8378f7d

more fixes

81e105d

eserilev added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Mar 4, 2026

jimmygchen reviewed Mar 6, 2026

View reviewed changes

jimmygchen requested changes Mar 6, 2026

View reviewed changes

jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Mar 6, 2026

jimmygchen reviewed Mar 6, 2026

View reviewed changes

eserilev added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Mar 14, 2026

Merge branch 'unstable' of https://github.com/sigp/lighthouse into gl…

df931fd

…oas-serve-envelope-rpc

avoid wrapping canonical_head in Arc, impl CanonicalHeadReader on Bea…

3e6afae

…conChain

remove dead BeaconBlockGloas::full() impl

ed8c549

Merge branch 'unstable' of https://github.com/sigp/lighthouse into gl…

9c2ff0d

…oas-serve-envelope-rpc

mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Mar 16, 2026

cleanup

be9e32f

eserilev added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Mar 16, 2026

pawanjay176 reviewed Mar 16, 2026

View reviewed changes

eserilev added 4 commits March 16, 2026 21:30

Reffactor to use adapter pattern

0769ffc

Merge branch 'unstable' of https://github.com/sigp/lighthouse into gl…

54c062d

…oas-serve-envelope-rpc

Fix comments

a9bc8a1

ssz size calc refactor

54dfa0c

mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Mar 17, 2026

Resolve merge conflicts

12e86c3

eserilev added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Mar 18, 2026

Merge branch 'unstable' into gloas-serve-envelope-rpc

9f72751

Conversation

eserilev commented Feb 24, 2026

Issue Addressed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmygchen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jimmygchen Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmygchen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eserilev commented Mar 14, 2026

Uh oh!

dapplion commented Mar 16, 2026

Uh oh!

dapplion commented Mar 16, 2026

Uh oh!

mergify bot commented Mar 16, 2026

Uh oh!

eserilev commented Mar 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eserilev commented Mar 17, 2026

Uh oh!

mergify bot commented Mar 17, 2026

Uh oh!

dapplion commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

jimmygchen left a comment •

edited

Loading

jimmygchen Mar 2, 2026 •

edited

Loading

dapplion commented Mar 18, 2026 •

edited

Loading