Add HTTP compression support (via 'Accept-Encoding') for metrics (e.g. OTLP, Prometheus, ...)

[frittentheke]

Prometheus metrics exposed via https://docs.spring.io/spring-boot/reference/actuator/metrics.html#actuator.metrics.export.prometheus, and very likely all served metrics currently don't support HTTP compressing even if scraper does send appropriate Accept-Encoding headers due to the Spring webserver not being configured appropriately?

Even for push metrics (e.g. OTLP) compression has to be explicitly enabled ...

• https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters
• Provide auto-configuration for OpenTelemetry metrics #36546

Metrics are usually presented as Content-Type: text/plain, but Prometheus / OTEL also support protobuf and openmetrics (https://prometheus.io/docs/instrumenting/content_negotiation/#protocol-headers)
OpenTelemetry (https://opentelemetry.io/docs/specs/otlp/#otlphttp) standardized this to:

• application/openmetrics-text
• application/openmetrics-protobuf

in the end this type of data is VERY compressible, >90%. Depending on the application and the amount of metrics, each scrape is hundreds of KBs so this adds up to substantial bandwidth savings, money even when running on public cloud infrastructure which changes for the amount of data transferred (across AZs).

I know SpringBoot already optionally supports HTTP compression via https://docs.spring.io/spring-boot/how-to/webserver.html#howto.webserver.enable-response-compression. This is something that has to be switched on explicitly, so looking at articles like https://dev.to/rock_win_c053fa5fb2399067/enabling-gzip-compression-in-spring-boot-for-faster-web-apps-3abe this is likely not universally known.

If I may quote Prometheus' @SuperQ (again) ... prometheus/prometheus#13166 (comment)

Compression is cheap on modern CPUs and saves 90% of bandwith - this should really be enabled by default and not something people find by accident or the hard when looking at their $Hyperscaler bill.

[wilkinsona]

Thanks for the suggestion.

Enabling compression via Accept-Encoding is only appropriate for metrics that are scraped. That limits things to Prometheus and its scrape endpoint. For other formats, such as OTLP, where the metrics are pushed out by the various Micrometer registries, there's no opportunity for content negotiation.

I'm not sure that it makes sense to apply special treatment to the Prometheus scrape endpoint. If a user hasn't enabled compression then they may not want it at all for whatever reason. If we offered Prometheus-specific support for compression, we'd have to provide a property that controls it. This then makes it harder to enable or disable compression as there are potentially two properties that need to be set.

Rather than Prometheus-specific compression support, we could consider changing the list of mime types that are compressed to include application/openmetrics-text and perhaps application/vnd.google.protobuf (the content type produced by io.prometheus.metrics.expositionformats.PrometheusProtobufWriter) as well. I'll discuss this with the team.

Beyond that, I'm not sure that we should do anything else. If we're looking at options that aren't specific to Prometheus, I don't think we should default server.compression.enabled to true. We typically defer to the expertise of the web server maintainers and Tomcat, for example, does not enable compression by default.

[wilkinsona]

The default value for the mime types that may be compressed is aligned with Tomcat's and enforced by a test. We'd like to keep that alignment so a change to the default mime types isn't an option.

As an alternative, we've opened #48927 to make it easier to add to the mime types that will be compressed.

[frittentheke]

Thanks @wilkinsona for the lightning fast response. While I understand (maybe not fully) that you follow Apache Tomcat defaults, you skipped my remarks about the "sane defaults" approach.

Spring Boot builds on top of Tomcat (and other webservers) and provides a good out of the box experience.
(Pre-)Configuration of the used components with "sane defaults" is one of the ways to achieve this awesome experience.

Having to know that a) compression needs to be explicitly turned on and b) a list of content-types has to be added and maintained across a all usages of Spring Boot is the various applications that are out there today does not, in my opinion, provide a good service in this regard.

I hope you give this some more thought and discussion. I explicitly suggested to turn on compression "just" for the Prometheus metrics (which is part of the integration that SpringBoot provides) by default and not the whole webserver was about not messing with the current state of things in regards to application traffic. I'd even argue that oportunistic and fast compression should actually be on (and with the application/openmetrics-* content types included) by default. But that certainly is a bigger discussion - thus I pointed only at the wasteful metrics responses.

[wilkinsona]

We spent half of our meeting time yesterday discussing this. I don't think there was any lack of thought or discussion. We consider the current defaults to be sane. Aligning with Tomcat and the expertise of the Tomcat maintainers is our preferred approach.

[frittentheke]

We spent half of our meeting time yesterday discussing this. I don't think there was any lack of thought or discussion. We consider the current defaults to be sane. Aligning with Tomcat and the expertise of the Tomcat maintainers is our preferred approach.

Thanks for the insight @wilkinsona and sorry about not considering there was more (thought) to the fast response than it appeared.

My argument was not that Tomcat's defaults are not "sane", but that SpringBoot is not just a bunch of best-of-breed tools and libraries glued together, but also about the way they are all configured. And a sane configuration of a tool or component like Tomcat might very much differ in the context of this conglomerate of tools when compared to them individually and stand-alone.

Tomcat in itself does not provide Prometheus metrics. So their defaults cannot accommodate for this, or any particular, use cases you might build WITH Tomcat or on top of Tomcat.

Edit:

And just leaving a good default out for everyone to find in the documentation (or some blog post somewhere) to me seems like a lost opportunity to be of service to your user base in this particular regard. Most folks don't have this aspect on their agenda when starting off writing with Spring Boot applications ... they rely on the defaults to be good until they are not. And that is what I argue about - they are not good ... or let's say "the best" defaults IMHO.

[jonatan-ivanov]

Since OTLP was also mixed in here, I just wanted to mention that we merged in compression support for OTLP into Micrometer very recently: micrometer-metrics/micrometer#7012

[wilkinsona]

@frittentheke we need to agree to disagree here.

Enabling compression is not without its downsides which is why it is disabled by default in Tomcat. There's nothing in particular about a Spring Boot app that changes this so we choose to align with Tomcat.

Enabling compression for the Prometheus scrape endpoint alone isn't something that we want to do. It would require the endpoint to re-implement content negotiation and handle things like the Accept-Encoding header and the Transfer-Encoding header. Both of these should be dealt with lower down the stack in the HTTP server, particularly when it comes to supporting a range of different compression algorithms that may be acceptable to the client.