engine_getBlobsV2: Don't prohibit partial responses

[MarcoPolo]

Reverts to previous behavior of supporting partial responses. This enables optimizations that require these partial responses.

Replaces #669 (as raul is OOO, and it would be good to do this sooner rather than later)

Discussion from ACDT 41 showed consensus around supporting partial responses and going back to the old text.

Context from that issue still applies:

Background

• engine_getBlobsV1 currently achieves a 70-80% hit rate, expected to remain high even as blob volume increases.
• EIP-4844 introduced full blob sidecar broadcasts at the CL layer; with PeerDAS, the CL now gossips column sidecars.
• Because columns sidecars require all cells to be present, support for partial hits was previously dropped in this commit.

Motivation

• The P2P networking team is actively working on backwards-compatible optimizations that can be deployed in-between forks.
• For instance, we're prototyping cell-level deltas to facilitate column reconciliation based on local availability.
• This would reduce redundant full-column transmissions and reclaim useful bandwith to further scale blobs.

Remarks

• We're aware this change might be procedurally late, but its ease of implementation and its ability to unlock an entire class of interim improvements makes it worthwhile to consider (and we only noticed this opportunity in Berlin).
• We'll happily submit PRs to implement this in EL and CL clients if needed. Just let us know!

[pawanjay176]

@MarcoPolo Is the intention of returning partial responses to test the cell level deltas approach?
Right now, we do not return a partial response to avoid the encoding/decoding of blobs unnecessarily. If the cell level deltas approach doesn't actually give us significant savings, then this change is just a net negative for fusaka.

I would prefer not to merge spec changes to facilitate optimisations until we can validate that they actually give us the savings we expect.

[MarcoPolo]

I think a better title to this PR is "engine_getBlobsV2: Don't prohibit partial responses."

@MarcoPolo Is the intention of returning partial responses to test the cell level deltas approach?

Less about testing it, and more about leaving the door open for this optimization after Fusaka launches.

Right now, we do not return a partial response to avoid the encoding/decoding of blobs unnecessarily. If the cell level deltas approach doesn't actually give us significant savings, then this change is just a net negative for fusaka.

Understood. Note that it is perfectly valid for an EL implementation with this proposed change to retain the current behavior. From:

4. Client software MUST return null if syncing or otherwise unable to serve blob pool data.

The implementation may just decide to return null if any blobs are missing. There is no guarantee that the EL side has any blobs. The proposed change is to not prohibit partial responses, in case it's useful.

An implementation may want to use an environment variable or compile flag to return partial responses, and leave that flag off until the optimization exists. In this case, there would be no extra cause and it would be at worst a net zero change.

If the optimization works, using today's data, it would reduce data column message size by (blob_count-1)/blob_count. We'd only need to transmit the one missing cell. This would buy us more scaling room without a network upgrade.

I would prefer not to merge spec changes to facilitate optimisations until we can validate that they actually give us the savings we expect.

Ideally we'd have this optimization implemented already. We don't yet, but we know it is possible if we allow partial responses (or equivalently don't prohibit partial response).

Removing the prohibition doesn't cost us anything (as mentioned above), but leaves the door open to reusing work across the CL <-> EL. Keeping the prohibition forces the CL to start from scratch even if 99% of the work has already been done by the EL.

I'm happy to dive deeper into the proposed optimization in case it would be helpful.

[flcl42]

Strictly returning null in case of partially available data was added because partial response figured out to be completely useless specifically in case of Osaka fork. Which can change in the next fork. If you are saying partial responses can be still useful in Osaka - it's a great reason to return partial responses. Otherwise we can still enable it as an optional alternative response, at least because some clients do not want to care. In Nethermind we want to return null just because partial is useless and it saves time to return null. In the next fork we will change this behavior if partial responses will be useful and it will not require a flag. We also still return partial responses in Cancun

[MarcoPolo]

If you are saying partial responses can be still useful in Osaka - it's a great reason to return partial responses.

Yes, this is what I'm saying.

[mkalinin]

Looks good to me. Happy to merge once there are approvals from CL and EL devs

[kasey]

I fully support partial messages.

[tbenr]

it makes sense to me. EL can decide what to do (and even change behavior via a CLI param) remaining compatible with the API.

[nflaig]

I prefer this over adding a v3 later, there is not much downside to allowing partial responses

[pawanjay176]

LGTM.