BatchResponseMaxSize does not account for error responses in batch handling

[handfuzzer]

System information

Geth version: Geth 1.17.0-unstable (go1.24.0/linux-amd64)
CL client & version: N/A
OS & Version: Ubuntu 24.04.3 LTS (Linux x86_64, kernel 6.14.8-2-pve)
Commit hash : 15a9e92bbd310eca6da2e0f49c34b2e503dfbc00

Expected behaviour

When Server.SetBatchLimits(itemLimit, maxResponseSize) (or WithBatchResponseSizeLimit) is configured, the server should stop processing additional items in a batch once the cumulative response size exceeds maxResponseSize.

This limit should apply uniformly to both:

• Successful responses (result)
• Error responses (error, including error.data via rpc.DataError)

Once the limit is exceeded, subsequent batch items should return -32003 (response too large).

Actual behaviour

In rpc/handler.go, (*handler).handleBatch only accounts for len(resp.Result) when accumulating responseBytes.

For error responses:

• resp.Result is nil
• Large error payloads (especially large error.data) are not counted toward BatchResponseMaxSize

As a result, batch responses can significantly exceed maxResponseSize, and -32003 (response too large) is not reliably returned for later batch items.

Steps to reproduce the behaviour

1. Configure an RPC server with a small batch response size limit, e.g.:

   srv.SetBatchLimits(1000, 200)

2. Register an RPC method that always returns an error implementing rpc.DataError with a large ErrorData() payload (e.g. a long string).

3. Send a JSON-RPC batch containing multiple calls to that method.

4. Observe that:

   • The total batch response exceeds maxResponseSize
   • All (or too many) batch items are processed
   • Subsequent items do not consistently return -32003

[TanayK07]

I've opened a fix for this. The issue is in rpc/handler.go where handleBatch only counts len(resp.Result) for the response size accumulator. Since resp.Result is nil for error responses, error payloads (including large error.data) are never counted toward BatchResponseMaxSize.

The fix marshals resp.Error to JSON and adds its size to responseBytes when the error field is non-nil. I've also added a test case that sends a batch of error-returning calls to verify the size limit is enforced.

PR: coming up