Limit response sizes from query-frontends #13829

charleskorn · 2025-12-15T06:08:12Z

What this PR does

This PR introduces a limit on the response sizes produced by query-frontends.

If a response is larger than the configured limit, the request is aborted and an error is returned.

The default value, 128MB, is based on the response sizes we see in the real world for Grafana Cloud Metrics customers.

I've switched to a fork of json-iterator/go containing the changes from json-iterator/go#721.

Which issue(s) this PR fixes or relates to

(none)

Checklist

Tests updated.
Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]. If changelog entry is not needed, please add the changelog-not-needed label to the PR.
about-versioning.md updated with experimental features.

Note

Adds a configurable max query response size to query-frontends (default 128MB) and enforces it for JSON and Protobuf responses, updating docs, defaults, tests, and JSON library.

Query-frontend:
- Response size limit: Enforce max response size for query results (default 128MB) via -query-frontend.max-response-size-bytes/query-frontend.max_response_size_bytes.
- Codec: Enforce limits for both json and protobuf encodings; propagate size-limit errors using globalerror.MaxResponseSizeBytes and apierror.TypeTooLargeEntry.
- Wiring: Pass limit through config and module init; update defaults in operations/mimir/mimir-flags-defaults.json and help output.
Docs:
- Add CHANGELOG entry; document flag in configuration parameters and versioning pages; update help-all.txt.
Dependencies:
- Replace github.com/json-iterator/go with a fork adding MaxMarshalledBytes; vendor updates to enforce output size.
Tests:
- Add size-limit tests for JSON/labels/series and Protobuf; adapt existing tests to new NewCodec signature.

^{Written by Cursor Bugbot for commit 9e702fe. This will update automatically on new commits. Configure here.}

github-actions · 2025-12-15T06:10:07Z

💻 Deploy preview available (Limit response sizes from query-frontends):

cursor · 2025-12-15T06:18:45Z

pkg/frontend/querymiddleware/codec_protobuf.go

+	size := payload.Size()
+	if uint64(size) > f.maxEncodedSize {
+		return nil, apierror.Newf(apierror.TypeTooLargeEntry, "Protobuf response (%d bytes) is too large: "+responseSizeTooLargeErrorFormat, size, f.maxEncodedSize)
+	}


Bug: Inconsistent zero-value handling between JSON and Protobuf formatters

The response size limit behaves inconsistently when MaxResponseSizeBytes is set to 0. In the JSON formatter (via the forked json-iterator), enforceMarshalledBytesLimit is set to config.maxMarshalledBytes > 0, meaning a value of 0 disables limit enforcement entirely (allowing unlimited sizes). However, in ProtobufFormatter, the check uint64(size) > f.maxEncodedSize is always true when maxEncodedSize is 0 and size > 0, causing all non-empty Protobuf responses to be rejected. This inconsistency could cause confusing behavior if someone sets the limit to 0.

Additional Locations (1)

vendor/github.com/json-iterator/go/stream.go#L35-L36

cursor · 2025-12-15T06:18:46Z

pkg/frontend/querymiddleware/codec_protobuf.go

+	size := payload.Size()
+	if uint64(size) > f.maxEncodedSize {
+		return nil, apierror.Newf(apierror.TypeTooLargeEntry, "Protobuf response (%d bytes) is too large: "+responseSizeTooLargeErrorFormat, size, f.maxEncodedSize)
+	}


Bug: Inconsistent zero-value handling between JSON and Protobuf formatters

The response size limit behaves inconsistently when MaxResponseSizeBytes is set to 0. In the JSON formatter (via the forked json-iterator), enforceMarshalledBytesLimit is set to config.maxMarshalledBytes > 0, meaning a value of 0 disables limit enforcement entirely (allowing unlimited sizes). However, in ProtobufFormatter, the check uint64(size) > f.maxEncodedSize is always true when maxEncodedSize is 0 and size > 0, causing all non-empty Protobuf responses to be rejected. This inconsistency could cause confusing behavior if someone sets the limit to 0.

Additional Locations (1)

vendor/github.com/json-iterator/go/stream.go#L35-L36

cursor

Bug: Missing limit check in WriteUint16 early return path

In the forked json-iterator library, WriteUint16 is missing an enforceMaxBytes() call on its early return path when the value is less than 1000. While WriteUint32 correctly calls enforceMaxBytes() on all return paths (lines 82, 90, 104), WriteUint16 only calls it at line 61, not at the early return on line 56. This means when marshalling uint16/int16 values less than 1000, those specific bytes bypass the limit check, potentially allowing responses to slightly exceed the configured MaxResponseSizeBytes limit before the next write operation catches it.

vendor/github.com/json-iterator/go/stream_int.go#L53-L56

mimir/vendor/github.com/json-iterator/go/stream_int.go

Lines 53 to 56 in 9e702fe

    
           q1 := val / 1000 
        
           if q1 == 0 { 
        
           	stream.buf = writeFirstBuf(stream.buf, digits[val]) 
        
           	return

tcp13equals2 · 2025-12-15T08:07:42Z

vendor/github.com/json-iterator/go/stream.go

 // WriteByte writes a single byte.
 func (stream *Stream) writeByte(c byte) {
 	stream.buf = append(stream.buf, c)
+	stream.enforceMaxBytes()


Rather then call stream.enforceMaxBytes() everywhere, would it make sense to create an append() func on the Stream which does the buf append and calls this enforceMaxBytes()?

func (stream *Stream) write(obj ...byte) { stream.buf = append(stream.buf, obj...) stream.enforceMaxBytes() }

Could also be a writeNoCheck() which does the append but does not do the max bytes checks - in the case of a string where we don't want to check the max bytes on every char appended.

tcp13equals2 · 2025-12-15T08:12:39Z

vendor/github.com/json-iterator/go/stream_str.go

 	}
 	if i == valLen {
 		stream.buf = append(stream.buf, '"')
+		stream.enforceMaxBytes()


For strings, is it worth looking at the len(s) and peeking at the current accumulated buf len and making a preemptive test if the string is likely to exceed the max len?

This might provide against any really big strings coming into this.

tcp13equals2 · 2025-12-15T08:30:24Z

vendor/github.com/json-iterator/go/stream.go

+	stream.enforceMaxBytes()
+}
+
+func (stream *Stream) enforceMaxBytes() {


Does the benchmark tool pick up any noticeable difference serializing large documents?

tcp13equals2

I left a couple of general comments - but overall this makes sense and looks good.

56quarters

I see that we already use quite a few replace directives but I don't like adding another one for something as fundamental as JSON encoding. Looks like the json-iterator repo was just archived. Is there any equivalent functionality in v2 of the stdlib JSON encoding?

pkg/frontend/querymiddleware/codec.go

56quarters · 2025-12-15T14:18:57Z

vendor/github.com/json-iterator/go/stream.go

+		return
+	}
+
+	if uint64(len(stream.buf)) > stream.marshalledBytesLimitRemaining {


Instead of panicking like this, what if we used the same approach that we use when reading/writing TSDB indexes and set an err member for the stream. This would allow callers to continue to call methods that don't return errors. Would that negate the purpose of the limit?

56quarters · 2025-12-15T14:31:28Z

pkg/frontend/querymiddleware/roundtrip.go

 	f.BoolVar(&cfg.ShardActiveSeriesQueries, "query-frontend.shard-active-series-queries", false, "True to enable sharding of active series queries.")
 	f.BoolVar(&cfg.UseActiveSeriesDecoder, "query-frontend.use-active-series-decoder", false, "Set to true to use the zero-allocation response decoder for active series queries.")
 	f.BoolVar(&cfg.CacheSamplesProcessedStats, "query-frontend.cache-samples-processed-stats", false, "Cache statistics of processed samples on results cache. Deprecated: has no effect.")
+	f.Uint64Var(&cfg.MaxResponseSizeBytes, maxResponseSizeBytesFlag, 128*1024*1024, "Maximum allowed response size for query responses, in bytes.")


Can you clarify if this limit can be disabled and how? It doesn't seem like 0 will disable the limit.

56quarters · 2025-12-15T14:39:26Z

pkg/frontend/querymiddleware/codec.go

 	_, sp := tracer.Start(ctx, "APIResponse.ToHTTPResponse")
 	defer sp.End()

 	a, ok := res.GetPrometheusResponse()


The protobuf response object here has a method to compute its size. Maybe using the protobuf size as an estimate for how big the JSON response would be is good enough? It wouldn't require forking the JSON encoder.

tacole02

Docs look good! Thank you!

tacole02 · 2025-12-15T23:46:24Z

CHANGELOG.md

 * [CHANGE] Query-frontend: Removed support for calculating 'cache-adjusted samples processed' query statistic. The `-query-frontend.cache-samples-processed-stats` CLI flag has been deprecated and will be removed in a future release. Setting it has now no effect. #13582
 * [CHANGE] Querier: Renamed experimental flag `-querier.prefer-availability-zone` to `-querier.prefer-availability-zones` and changed it to accept a comma-separated list of availability zones. All zones in the list are given equal priority when querying ingesters and store-gateways. #13756 #13758
 * [CHANGE] Ingester: Stabilize experimental flag `-ingest-storage.write-logs-fsync-before-kafka-commit-concurrency` to fsync write logs before the offset is committed to Kafka. Remove `-ingest-storage.write-logs-fsync-before-kafka-commit-enabled` since this is always enabled now. #13591
+* [CHANGE] Query-frontend: Enforce a limit on the size of responses returned by query-frontends. Defaults to 128MB and can be configured with `-query-frontend.max-response-size-bytes`. #13829 


Suggested change

* [CHANGE] Query-frontend: Enforce a limit on the size of responses returned by query-frontends. Defaults to 128MB and can be configured with `-query-frontend.max-response-size-bytes`. #13829

* [CHANGE] Query-frontend: Enforce a limit on the size of responses returned by query-frontends. Defaults to 128MB. You can configure this limit with `-query-frontend.max-response-size-bytes`. #13829

charleskorn · 2025-12-16T04:59:10Z

Is there any equivalent functionality in v2 of the stdlib JSON encoding?

golang/go#56733 proposes a WithByteLimit method for encoding/json/v2 that would achieve what we're after, but it has not yet been implemented.

encoding/json has no equivalent option and suffers from the same problem as json-iterator/go where it doesn't write to the underlying io.Writer until the entire value has been buffered in memory, so we couldn't use a io.Writer that places a limit on the size of the written value to enforce a limit like this.

We could close this and wait for a) this marshalling code to switch to using encoding/json/v2 and b) for golang/go#56733 to be implemented - what do you think?

56quarters · 2025-12-16T21:30:54Z

golang/go#56733 proposes a WithByteLimit method for encoding/json/v2 that would achieve what we're after, but it has not yet been implemented.

encoding/json has no equivalent option and suffers from the same problem as json-iterator/go where it doesn't write to the underlying io.Writer until the entire value has been buffered in memory, so we couldn't use a io.Writer that places a limit on the size of the written value to enforce a limit like this.

We could close this and wait for a) this marshalling code to switch to using encoding/json/v2 and b) for golang/go#56733 to be implemented - what do you think?

My preference is wait for this to (hopefully) be added to encoding/json/v2. I would rather not use a fork of the json-iterator project in the mean time. I think if this (large JSON responses in the query-frontend) becomes a common stability issue we have options that don't require using a forked project.

charleskorn added 7 commits December 11, 2025 15:01

Add CLI flag

d0e4860

Pass payload size to formatters

eb9387f

Enforce maximum payload size when encoding Protobuf payloads

e2c9c6b

Use json-iterator/go fork

b5382e8

Enforce limit for JSON responses

d44906f

Use a clearer error message with the name of the associated limit

fab7168

Add changelog entry, and add flag to about-versioning.md.

9e702fe

charleskorn marked this pull request as ready for review December 15, 2025 06:11

charleskorn requested review from a team, stevesg and tacole02 as code owners December 15, 2025 06:11

cursor bot reviewed Dec 15, 2025

View reviewed changes

tcp13equals2 reviewed Dec 15, 2025

View reviewed changes

tcp13equals2 approved these changes Dec 15, 2025

View reviewed changes

56quarters reviewed Dec 15, 2025

View reviewed changes

tacole02 reviewed Dec 15, 2025

View reviewed changes

charleskorn closed this Dec 16, 2025

	q1 := val / 1000
	if q1 == 0 {
	stream.buf = writeFirstBuf(stream.buf, digits[val])
	return

	* [CHANGE] Query-frontend: Enforce a limit on the size of responses returned by query-frontends. Defaults to 128MB and can be configured with `-query-frontend.max-response-size-bytes`. #13829
	* [CHANGE] Query-frontend: Enforce a limit on the size of responses returned by query-frontends. Defaults to 128MB. You can configure this limit with `-query-frontend.max-response-size-bytes`. #13829

Limit response sizes from query-frontends #13829

Limit response sizes from query-frontends #13829

Uh oh!

Conversation

charleskorn commented Dec 15, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Which issue(s) this PR fixes or relates to

Checklist

Uh oh!

github-actions bot commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor bot Dec 15, 2025

Choose a reason for hiding this comment

Bug: Inconsistent zero-value handling between JSON and Protobuf formatters

Uh oh!

cursor bot Dec 15, 2025

Choose a reason for hiding this comment

Bug: Inconsistent zero-value handling between JSON and Protobuf formatters

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Missing limit check in WriteUint16 early return path

Uh oh!

tcp13equals2 Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tcp13equals2 Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

tcp13equals2 Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

tcp13equals2 left a comment

Choose a reason for hiding this comment

Uh oh!

56quarters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

56quarters Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

56quarters Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

56quarters Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

tacole02 left a comment

Choose a reason for hiding this comment

Uh oh!

tacole02 Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

charleskorn commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

56quarters commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

charleskorn commented Dec 15, 2025 •

edited by cursor bot

Loading

github-actions bot commented Dec 15, 2025 •

edited

Loading

tcp13equals2 Dec 15, 2025 •

edited

Loading

charleskorn commented Dec 16, 2025 •

edited

Loading