v2.10.3

• [SECURITY] S3 SSE-C encryption_key is now treated as a secret to prevent it from being exposed in plaintext. Resolves CVE-2026-28377. #6711 (@mattdurham)

v2.10.2

[BUGFIX] Apply exemplars hint end-to-end and fix safety cap bypass in metrics queries. #6559 (@zhxiaogg)
[ENHANCEMENT] Used frontend MaxExemplars config as single source of truth for exemplar limits. Added a safety cap at the traceql engine entry points. #6515 (@zhxiaogg)
[CHANGE] Set default max_result_limit for search to 256*1024 #6525 (@zhxiaogg)

v2.10.1

Changes

• [CHANGE] Upgrade Tempo to Go 1.25.7 #6449 (@zalegrala)

Bugfixes

• [BUGFIX] Correct avg_over_time calculation #6252 (@ruslan-mikhailov)
• [BUGFIX] Correct instant query calculation for rate() #6205 (@ruslan-mikhailov)
• [BUGFIX] Fix query-frontend unable to convert dedicated column blob option #6377 (@stoewer)
• [BUGFIX] Fix issue with additional matched spans in queries with only resource attributes #6432 (@stoewer)
• [BUGFIX] Fix live-store deadlock occurring after a complete block failure #6338 (@ruslan-mikhailov)
• [BUGFIX] Fix query_end_cutoff param for query range #6360 (@ruslan-mikhailov)
• [BUGFIX] Fix dimension_mappings being unconditionally overwritten by overrides, causing them to be set to nil when set through static config but not in overrides #6390 (@carles-grafana)
• [BUGFIX] Fix metrics generator panic when write_relabel_configs is set, caused by uninitialized NameValidationScheme after Prometheus dependency upgrade #6399 (@carles-grafana)

v2.10.0

Breaking Changes

• [CHANGE] BREAKING CHANGE Validate tenant ID in frontend and distributor #5786 (@carles-grafana)
• [CHANGE] Remove vParquet2 encoding #6071 (@mdisibio)
  BREAKING CHANGE In the last release vParquet2 was deprecated and blocked from writes. Now, it's
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet2 blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet2 must run a previous release configured for vParquet3 or higher until all existing vParquet2 blocks
  have expired and been deleted from the backend, or else it will encounter read errors after upgrading to this release.
• [CHANGE] BREAKING CHANGE Added scope and query params while rewriting integration tests to the Tempo HTTP client SearchTagsV2WithRange function. #6088 (@joe-elliott)
• [CHANGE] BREAKING CHANGE TempoDB schemas vParquet3 and v2 are deprecated and will be removed in 3.0. #6198 (@joe-elliott)

Changes

• [CHANGE] Add stricter validation to user-configurable overrides API, such as refusing duplicate labels #6008 #6104 (@carles-grafana, @mdisibio)
• [CHANGE] Expose metrics_generator.dimension_mappings to user-configurable overrides API #5989 (@carles-grafana)
• [CHANGE] Expose metrics_generator.intrinsic_dimensions to user-configurable overrides API #5974 (@carles-grafana)
• [CHANGE] Expose metrics_generator.trace_id_label_name to user-configurable overrides API #5972 (@carles-grafana)
• [CHANGE] Expose metrics_generator.ingestion_time_range_slack to user-configurable overrides API #5958 (@carles-grafana)
• [CHANGE] Expose metrics_generator.native_histogram_bucket_factor and native_histogram_min_reset_duration to user-configurable overrides API #5973 (@carles-grafana)
• [CHANGE] Remove remaining aws-sdk-go references and migrate tests to MinIO #5856 (@anglerfishlyy)
• [CHANGE] Remove busybox from Tempo image to make it more minimal and prevent future vulnerabilities #5717 (@carles-grafana)
• [CHANGE] Upgrade Tempo to Go 1.25.4 #5939 #6001 (@ruslan-mikhailov)
• [CHANGE] Upgrade Tempo to Go 1.25.5 #6096 #6089 (@joe-elliott)

Features

• [FEATURE] New block encoding vParquet5 with dedicated attributes for integers and events, improved performance on high-cardinality/high-length attributes, faster TraceQL queries, and childCount intrinsic. See the vParquet5 section below for full details. #6219 (@ie-pham @mdisibio @stoewer)
• [FEATURE] TraceQL: Add minInt/maxInt. #5982 (@Syedowais312)
• [FEATURE] TraceQL: Add support for attribute = nil #4905 (@ie-pham)
• [FEATURE] Add support for Accept: application/vnd.grafana.llm to Tempo endpoints to improve MCP responses. Currently supported directly by trace by id and tag values #5962 (@joe-elliott)
  This response is subject to change and should not be relied on. It is intended for LLM consumption only. Even a fundamental change to its representation (yaml? markdown?) would not be considered breaking.
• [FEATURE] Add entity-based limiting mode for metrics-generator as an alternative to series-based limiting. #5788 (@Logiraptor)
• [FEATURE] Add tempo_metrics_generator_registry_active_series_demand_estimate that estimates metrics-generator active series demand even when the active series limit is reached #5710 (@carles-grafana)
• [FEATURE] Metrics generator will now produce overflow series to capture new data once limits are hit. These series have the label metric_overflow="true". #5954 (@Logiraptor)
• [FEATURE] Add support for external storage to trace by id endpoint #6185 (@Logiraptor)

Enhancements

• [ENHANCEMENT] add database_name_attributes config to service graph processor #5398 (@KyriosGN0)
• [ENHANCEMENT] Added validation mode and tests for tempo-vulture #5605 (@davidham)
• [ENHANCEMENT] Add SSE-C encryption support to S3 backend #5789 (@steffsas)
• [ENHANCEMENT] docs: Add explicit notes about authentication #5735 (@electron0zero)
• [ENHANCEMENT] Add secure connection support to tempo-cli #5692 (@TheoBrigitte)
• [ENHANCEMENT] Add config to enable instance label for spanmetrics series #5706 (@ie-pham)
• [ENHANCEMENT] Unsafe search hints for frontend performance tests #5723 (@ruslan-mikhailov)
• [ENHANCEMENT] TraceQL query performance improvement #5773 (@mdisibio)
• [ENHANCEMENT] Performance improvement in quantile_over_time() #5996 (@mdisibio)
• [ENHANCEMENT] Improve shutdown time in the first 30 seconds #5725 (@ldufr)
• [ENHANCEMENT] Add metric for tracking added latency to write requests #5781 (@mapno)
• [ENHANCEMENT] Update jsonnet vendor dependencies #6202 (@javiermolinar)
• [ENHANCEMENT] Improve error message when overrides fail to parse #5787 (@mapno)
• [ENHANCEMENT] Add default_spans_per_span_set configuration option to control default spans returned per trace in search results #5858 (@iamrajiv)
• [ENHANCEMENT] Increase weight for heavy TraceQL queries #5782 (@ruslan-mikhailov)
• [ENHANCEMENT] Improve observability of collection failures in the metrics generator with error categorization #5936 (@javiermolinar)
• [ENHANCEMENT] Update tempo-mixin dashboards to use timeseries panels with explicit units #6210 (@javiermolinar)
• [ENHANCEMENT] Update list of intrinsics returned by search tags endpoint #5857 (@andreasgerstmayr)
• [ENHANCEMENT] Add "Requests Executed" panel for querier metrics in the operational dashboard. #5848 (@anglerfishlyy)
• [ENHANCEMENT] Drop and metric invalid utf8 for all metrics in metrics generator instead of sending invalid data. #5980 (@joe-elliott)
• [ENHANCEMENT] Add support for application/protobuf in frontend endpoints #5865 (@oleg-kozliuk-grafana)
• [ENHANCEMENT] Validate metrics-generator histogram buckets #5991 (@carles-grafana)
• [ENHANCEMENT] TraceQL: Add support for with(exemplars=false) hint to disable exemplars in metrics queries #6204(@joe-elliott)
• [ENHANCEMENT] Removed MustNewConstMetric to prevent panic and added validation for usage tracker config. Added tempo_distributor_usage_tracker_errors_total to surface errors in usage tracker. #5981 (@electron0zero)
• [ENHANCEMENT] Simplify block-builder partition assignment with config parameter partitions_per_instance #6022 (@mapno)
• [ENHANCEMENT] Allow RetryInfo to be disabled in per tenant overrides #5741 (@electron0zero)
• [ENHANCEMENT] Set maxKeys to 1 for S3 confirm list operation #6114 (@cancub)

Bugfixes

• [BUGFIX] Fix leading zero handling in native histograms implementations #6033 (@zalegrala)
• [BUGFIX] Prevent slice panic when truncating series after topk() by adding bounds check in metrics query-range combiner #6010 (@Syedowais312)
• [BUGFIX] Fix compactor to properly consider SSE-KMS information during metadata copy #5774 (@steffsas)
• [BUGFIX] Fix spss=0 parameter to properly mean unlimited spans instead of being rejected, and respect max_spans_per_span_set=0 configuration #5858 (@iamrajiv)
• [BUGFIX] Fix incorrect results in TraceQL compare() caused by potential hash collision of string array attributes #5835 (@mdisibio)
• [BUGFIX] Fix != nil checks broken in TraceQL Metrics #5998 (@mdisibio)
• [BUGFIX] Fix = nil handling with intrins...

v2.8.3

• [CHANGE] Upgrade Tempo to Go 1.25.5 #6096 #6089 (@joe-elliott)
• [BUGFIX] Deadlock on invalid query to api/v2/search/tags (SearchTagsV2) #5607 (@ruslan-mikhailov)
• [BUGFIX] Fix Tempo configuration options that were always overridden by runtime overrides. #5202 (@KyriosGN0)
• [BUGFIX] Fix trace idle period in ingesters to be correctly applied. Add max live trace period to limit how long traces remain in memory. #5346 (@joe-elliott)
• [BUGFIX] Return 400 Bad Request instead of 500 when an invalid tag name is provided to the SearchTagValuesV2 endpoint. #5493 (@carles-grafana)
• [BUGFIX] Fix panic in query_range http handling that could be triggered by cancellations or other errors #5667 (@mdisibio)

v2.9.1

• [CHANGE] Upgrade Tempo to Go 1.25.5 #6096 #6089 (@joe-elliott)

v2.10.0-rc.0

Breaking Changes

• [CHANGE] BREAKING CHANGE Validate tenant ID in frontend and distributor #5786 (@carles-grafana)
• [CHANGE] Remove vParquet2 encoding #6071 (@mdisibio)
  BREAKING CHANGE In the last release vParquet2 was deprecated and blocked from writes. Now, it's
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet2 blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet2 must run a previous release configured for vParquet3 or higher until all existing vParquet2 blocks
  have expired and been deleted from the backend, or else it will encounter read errors after upgrading to this release.
• [CHANGE] BREAKING CHANGE Added scope and query params while rewriting integration tests to the Tempo HTTP client SearchTagsV2WithRange function. #6088 (@joe-elliott)
• [CHANGE] BREAKING CHANGE TempoDB schemas vParquet3 and v2 are deprecated and will be removed in 3.0. #6198 (@joe-elliott)

Changes

• [CHANGE] Add stricter validation to user-configurable overrides API, such as refusing duplicate labels #6008 #6104 (@carles-grafana, @mdisibio)
• [CHANGE] Expose metrics_generator.dimension_mappings to user-configurable overrides API #5989 (@carles-grafana)
• [CHANGE] Expose metrics_generator.intrinsic_dimensions to user-configurable overrides API #5974 (@carles-grafana)
• [CHANGE] Expose metrics_generator.trace_id_label_name to user-configurable overrides API #5972 (@carles-grafana)
• [CHANGE] Expose metrics_generator.ingestion_time_range_slack to user-configurable overrides API #5958 (@carles-grafana)
• [CHANGE] Expose metrics_generator.native_histogram_bucket_factor and native_histogram_min_reset_duration to user-configurable overrides API #5973 (@carles-grafana)
• [CHANGE] Remove remaining aws-sdk-go references and migrate tests to MinIO #5856 (@anglerfishlyy)
• [CHANGE] Remove busybox from Tempo image to make it more minimal and prevent future vulnerabilities #5717 (@carles-grafana)
• [CHANGE] Upgrade Tempo to Go 1.25.4 #5939 #6001 (@ruslan-mikhailov)
• [CHANGE] Upgrade Tempo to Go 1.25.5 #6096 #6089 (@joe-elliott)

Features

• [FEATURE] New block encoding vParquet5 with dedicated attributes for integers and events, improved performance on high-cardinality/high-length attributes, faster TraceQL queries, and childCount intrinsic. See the vParquet5 section below for full details. #6219 (@ie-pham @mdisibio @stoewer)
• [FEATURE] TraceQL: Add minInt/maxInt. #5982 (@Syedowais312)
• [FEATURE] TraceQL: Add support for attribute = nil #4905 (@ie-pham)
• [FEATURE] Add support for Accept: application/vnd.grafana.llm to Tempo endpoints to improve MCP responses. Currently supported directly by trace by id and tag values #5962 (@joe-elliott)
  This response is subject to change and should not be relied on. It is intended for LLM consumption only. Even a fundamental change to its representation (yaml? markdown?) would not be considered breaking.
• [FEATURE] Add entity-based limiting mode for metrics-generator as an alternative to series-based limiting. #5788 (@Logiraptor)
• [FEATURE] Add tempo_metrics_generator_registry_active_series_demand_estimate that estimates metrics-generator active series demand even when the active series limit is reached #5710 (@carles-grafana)
• [FEATURE] Metrics generator will now produce overflow series to capture new data once limits are hit. These series have the label metric_overflow="true". #5954 (@Logiraptor)
• [FEATURE] Add support for external storage to trace by id endpoint #6185 (@Logiraptor)

Enhancements

• [ENHANCEMENT] add database_name_attributes config to service graph processor #5398 (@KyriosGN0)
• [ENHANCEMENT] Added validation mode and tests for tempo-vulture #5605 (@davidham)
• [ENHANCEMENT] Add SSE-C encryption support to S3 backend #5789 (@steffsas)
• [ENHANCEMENT] docs: Add explicit notes about authentication #5735 (@electron0zero)
• [ENHANCEMENT] Add secure connection support to tempo-cli #5692 (@TheoBrigitte)
• [ENHANCEMENT] Add config to enable instance label for spanmetrics series #5706 (@ie-pham)
• [ENHANCEMENT] Unsafe search hints for frontend performance tests #5723 (@ruslan-mikhailov)
• [ENHANCEMENT] TraceQL query performance improvement #5773 (@mdisibio)
• [ENHANCEMENT] Performance improvement in quantile_over_time() #5996 (@mdisibio)
• [ENHANCEMENT] Improve shutdown time in the first 30 seconds #5725 (@ldufr)
• [ENHANCEMENT] Add metric for tracking added latency to write requests #5781 (@mapno)
• [ENHANCEMENT] Update jsonnet vendor dependencies #6202 (@javiermolinar)
• [ENHANCEMENT] Improve error message when overrides fail to parse #5787 (@mapno)
• [ENHANCEMENT] Add default_spans_per_span_set configuration option to control default spans returned per trace in search results #5858 (@iamrajiv)
• [ENHANCEMENT] Increase weight for heavy TraceQL queries #5782 (@ruslan-mikhailov)
• [ENHANCEMENT] Improve observability of collection failures in the metrics generator with error categorization #5936 (@javiermolinar)
• [ENHANCEMENT] Update tempo-mixin dashboards to use timeseries panels with explicit units #6210 (@javiermolinar)
• [ENHANCEMENT] Update list of intrinsics returned by search tags endpoint #5857 (@andreasgerstmayr)
• [ENHANCEMENT] Add "Requests Executed" panel for querier metrics in the operational dashboard. #5848 (@anglerfishlyy)
• [ENHANCEMENT] Drop and metric invalid utf8 for all metrics in metrics generator instead of sending invalid data. #5980 (@joe-elliott)
• [ENHANCEMENT] Add support for application/protobuf in frontend endpoints #5865 (@oleg-kozliuk-grafana)
• [ENHANCEMENT] Validate metrics-generator histogram buckets #5991 (@carles-grafana)
• [ENHANCEMENT] TraceQL: Add support for with(exemplars=false) hint to disable exemplars in metrics queries #6204(@joe-elliott)
• [ENHANCEMENT] Removed MustNewConstMetric to prevent panic and added validation for usage tracker config. Added tempo_distributor_usage_tracker_errors_total to surface errors in usage tracker. #5981 (@electron0zero)
• [ENHANCEMENT] Simplify block-builder partition assignment with config parameter partitions_per_instance #6022 (@mapno)
• [ENHANCEMENT] Allow RetryInfo to be disabled in per tenant overrides #5741 (@electron0zero)
• [ENHANCEMENT] Set maxKeys to 1 for S3 confirm list operation #6114 (@cancub)

Bugfixes

• [BUGFIX] Fix leading zero handling in native histograms implementations #6033 (@zalegrala)
• [BUGFIX] Prevent slice panic when truncating series after topk() by adding bounds check in metrics query-range combiner #6010 (@Syedowais312)
• [BUGFIX] Fix compactor to properly consider SSE-KMS information during metadata copy #5774 (@steffsas)
• [BUGFIX] Fix spss=0 parameter to properly mean unlimited spans instead of being rejected, and respect max_spans_per_span_set=0 configuration #5858 (@iamrajiv)
• [BUGFIX] Fix incorrect results in TraceQL compare() caused by potential hash collision of string array attributes #5835 (@mdisibio)
• [BUGFIX] Fix != nil checks broken in TraceQL Metrics #5998 (@mdisibio)
• [BUGFIX] Fix = nil handling with intrins...

v2.9.0

Breaking Changes

• [CHANGE] BREAKING CHANGE Migrated Tempo Vulture and Integration Tests from the deprecated Jaeger agent/exporter to the standard OTLP exporter. Vulture now pushes traces to the Tempo OTLP GRCP endpoint. #5058 (@iamrajiv, @javiermolinar)
• [CHANGE] BREAKING CHANGE TraceQL Metrics buckets are calculated based on data in past. #5366 (@ruslan-mikhailov)
• [CHANGE] BREAKING CHANGE Fix incorrect TraceQL metrics results when series labels include strings and integers with same textural representation.
  This also changes the TraceQL metrics responses of /api/metrics/query_range and /api/metrics/query to remove the redundant
  prom_labels field which was the error source. There may be an interruption to TraceQL metrics queries during rollout while components are running the previous version. #5659 (@mdisibio)
• [CHANGE] BREAKING CHANGE Deprecating vParquet2 block format #5688 (@ie-pham)

Changes

• [CHANGE] Return Bad Request from all frontend endpoints if the tenant can't be extracted. #5480 (@carles-grafana)
• [CHANGE] Do not count cached querier responses for SLO metrics such as inspected bytes. #5185 (@carles-grafana)
• [CHANGE] Adjust the definition of tempo_metrics_generator_processor_service_graphs_expired_edges to exclude edges that are counted in the service graph. #5319 (@joe-elliott)
• [CHANGE] Command tempo-cli analyse block(s) excludes attributes with array values. #5380 (@stoewer)
• [CHANGE] Remove .005s and add a 1.5s bucket to all request duration histograms. #5492 (@joe-elliott)
• [CHANGE] Improve tempo writes dashboard. #5500 (@javiermolinar)
• [CHANGE] Upgrade Tempo to go 1.25.0. #5548 (@javiermolinar)
• [CHANGE] Upgrade Tempo to go 1.25.1 #5685 (@electron0zero)
• [CHANGE] Drop tracing bridges in favor of OTEL only tracing. #5594 (@zalegrala)
• [CHANGE] Enable HTTP writes in the multi-tenant example. #5297 (@carles-grafana)

Features

• [FEATURE] Add MCP Server support. #5212 (@joe-elliott)
• [FEATURE] Add query hints sample=true and sample=0.xx which can speed up TraceQL metrics queries by sampling a subset of the data to provide an approximate result. #5469 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview1 with low-resolution timestamp columns for better TraceQL metrics performance. This format is in development and breaking changes are expected before final release. #5495 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview2 with dedicated attribute columns for integers. This format is in development and breaking changes are expected before final release. #5639 (@stoewer)

Enhancement

• [ENHANCEMENT] Add counter query_frontend_bytes_inspected_total, which shows the total number of bytes read from disk and object storage #5310 (@carles-grafana)
• [ENHANCEMENT] Add histograms spans_distance_in_future_seconds / spans_distance_in_past_seconds that count spans with end timestamp in the future / past. While spans in the future are accepted, they are invalid and may not be found using the Search API. #4936 (@carles-grafana)
• [ENHANCEMENT] Add support for scope in cost-attribution usage tracker. #5646 (@electron0zero)
• [ENHANCEMENT] Add alert for high error rate reported by vulture. #5206 (@ruslan-mikhailov)
• [ENHANCEMENT] Support the new db.namespace attribute for service-graph DB nodes. #5602 (@gouthamve)
• [ENHANCEMENT] TraceQL metrics performance increase for simple queries. #5247 (@mdisibio)
• [ENHANCEMENT] TraceQL search and metrics performance increase. #5280 (@mdisibio)
• [ENHANCEMENT] TraceQL performance improvement. #5218 (@mdisibio)
• [ENHANCEMENT] TraceQL compare() performance improvement. #5419 (@mdisibio)
• [ENHANCEMENT] Align traceql attribute struct for better performance. #5240 (@mdisibio)
• [ENHANCEMENT] Drop invalid prometheus label names in the spanmetrics processor. #5122 (@KyriosGN0)
• [ENHANCEMENT] Improve logging and tracing in the write path to include tenant info. #5436 (@javiermolinar)
• [ENHANCEMENT] Added usage tracker example. #5356 (@javiermolinar)
• [ENHANCEMENT] Add Stop method. #5293 (@stephanos)
• [ENHANCEMENT] Use peer attributes to determine the name of a client service virtual node in the service graph. #5381 (@MartenM)
• [ENHANCEMENT] Put actual size for writing to backend. #5413 (@ruslan-mikhailov)
• [ENHANCEMENT] Upgrade Azurite and Fake-gcs-server to latest version. #5512 (@javiermolinar)
• [ENHANCEMENT] Make block ordering deterministic. #5411 (@rajiv-singh)
• [ENHANCEMENT] Improve exemplar selection in quantile_over_time(). #5278 (@zalegrala)
• [ENHANCEMENT] Measure bytes received before limits and publish it as tempo_distributor_ingress_bytes_total. #5601 (@mapno)
• [ENHANCEMENT] Add total size logging functionality to track trace #5625(@sienna011022)

Bugfix

• [BUGFIX] Fix Tempo configuration options that are always overrided with config overrides section. #5202 (@KyriosGN0)
• [BUGFIX] Correctly apply trace idle period in ingesters and add the concept of trace live period. #5346 (@joe-elliott)
• [BUGFIX] Fix invalid YAML output from /status/runtime_config endpoint by adding document separator. #5371 (@iamrajiv)
• [BUGFIX] Fix panic in query_range HTTP handling that could be triggered by cancellations or other errors. #5667 (@mdisibio)
• [BUGFIX] Do not allow very small steps. #5441 (@ruslan-mikhailov)
• [BUGFIX] Fix incorrect TraceQL string comparison of strings starting with numbers. #5658 (@mdisibio)
• [BUGFIX] Fix incorrect results in TraceQL compare() for spans with array attributes #5519 (@ruslan-mikhailov)
• [BUGFIX] Fix cache collision for incomplete query in SearchTagValuesV2 #5549 (@ruslan-mikhailov)
• [BUGFIX] Fix for structural operator with empty left-hand spanset. #5578 (@ruslan-mikhailov)
• [BUGFIX] Deadlock on invalid query to api/v2/search/tags. (SearchTagsV2) #5607 (@ruslan-mikhailov)
• [BUGFIX] Fixed incorrect root span detection when spans have a child_of link but no parent. #3634 (@mexirica)
• [BUGFIX] Prevent metrics-generator WAL deletion when tenant is empty. #5586 (@sienna011022)
• [BUGFIX] Fix docker-compose port configuration for Alloy gRPC (4319 → 4317). #5536
• [BUGFIX] Fix panic error from empty span id. #5464
• [BUGFIX] Return Bad Request from frontend if the provided tag is invalid in SearchTagValuesV2 endpoint. #5493 (@carles-grafana)

Tempo Rearchitecture [EXPERIMENTAL]

• [CHANGE] BREAKING CHANGE Drop unused backend_scheduler.tenant_measurement_interval, use backend_scheduler.compaction.measure_interval instead. #5328 (@zalegrala)
• [CHANGE] Allow configuration of min/max input blocks for compaction provider. #5373 (@zalegrala)
• [CHANGE] BREAKING CHANGE Add require minimum time between tenant sorting in backend-scheduler. #5410 (@zalegrala)
  The configuration for backend_scheduler.provider.compaction.backoff has been removed.
  Additionally, the compaction_tenant_backoff_total metric has been renamed to compaction_empty_tenant_cycle_total for clarity.
• [CHANGE] Shard backend-scheduler work files locally and modify backend work structure to accommodate sharding approach. #5412 (@zalegrala)
• [CHANGE] Change worker to shutdown after the current job, waiting 30s by default. [#5460](ht...

v2.9.0-rc.0

Breaking Changes

• [CHANGE] BREAKING CHANGE We are no longer publishing rpm and deb packages due to an internal change to the handling of signing keys. This can be restored if we find that folks are actually using these packages. #5684 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Migrated Tempo Vulture and Integration Tests from the deprecated Jaeger agent/exporter to the standard OTLP exporter. Vulture now pushes traces to the Tempo OTLP GRCP endpoint. #5058 (@iamrajiv, @javiermolinar)
• [CHANGE] BREAKING CHANGE TraceQL Metrics buckets are calculated based on data in past. #5366 (@ruslan-mikhailov)
• [CHANGE] BREAKING CHANGE Fix incorrect TraceQL metrics results when series labels include strings and integers with same textural representation.
  This also changes the TraceQL metrics responses of /api/metrics/query_range and /api/metrics/query to remove the redundant
  prom_labels field which was the error source. There may be an interruption to TraceQL metrics queries during rollout while components are running the previous version. #5659 (@mdisibio)

Changes

• [CHANGE] Return Bad Request from all frontend endpoints if the tenant can't be extracted. #5480 (@carles-grafana)
• [CHANGE] Do not count cached querier responses for SLO metrics such as inspected bytes. #5185 (@carles-grafana)
• [CHANGE] Adjust the definition of tempo_metrics_generator_processor_service_graphs_expired_edges to exclude edges that are counted in the service graph. #5319 (@joe-elliott)
• [CHANGE] Command tempo-cli analyse block(s) excludes attributes with array values. #5380 (@stoewer)
• [CHANGE] Remove .005s and add a 1.5s bucket to all request duration histograms. #5492 (@joe-elliott)
• [CHANGE] Improve tempo writes dashboard. #5500 (@javiermolinar)
• [CHANGE] Upgrade Tempo to go 1.25.0. #5548 (@javiermolinar)
• [CHANGE] Drop tracing bridges in favor of OTEL only tracing. #5594 (@zalegrala)
• [CHANGE] Enable HTTP writes in the multi-tenant example. #5297 (@carles-grafana)
• [CHANGE] Upgrade Tempo to go 1.25.1 #5685 (@electron0zero)

Features

• [FEATURE] Add MCP Server support. #5212 (@joe-elliott)
• [FEATURE] Add query hints sample=true and sample=0.xx which can speed up TraceQL metrics queries by sampling a subset of the data to provide an approximate result. #5469 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview1 with low-resolution timestamp columns for better TraceQL metrics performance. This format is in development and breaking changes are expected before final release. #5495 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview2 with dedicated attribute columns for integers. This format is in development and breaking changes are expected before final release. #5639 (@stoewer)

Enhancement

• [ENHANCEMENT] Add counter query_frontend_bytes_inspected_total, which shows the total number of bytes read from disk and object storage #5310 (@carles-grafana)
• [ENHANCEMENT] Add histograms spans_distance_in_future_seconds / spans_distance_in_past_seconds that count spans with end timestamp in the future / past. While spans in the future are accepted, they are invalid and may not be found using the Search API. #4936 (@carles-grafana)
• [ENHANCEMENT] Add support for scope in cost-attribution usage tracker. #5646 (@electron0zero)
• [ENHANCEMENT] Add alert for high error rate reported by vulture. #5206 (@ruslan-mikhailov)
• [ENHANCEMENT] Support the new db.namespace attribute for service-graph DB nodes. #5602 (@gouthamve)
• [ENHANCEMENT] TraceQL metrics performance increase for simple queries. #5247 (@mdisibio)
• [ENHANCEMENT] TraceQL search and metrics performance increase. #5280 (@mdisibio)
• [ENHANCEMENT] TraceQL performance improvement. #5218 (@mdisibio)
• [ENHANCEMENT] TraceQL compare() performance improvement. #5419 (@mdisibio)
• [ENHANCEMENT] Align traceql attribute struct for better performance. #5240 (@mdisibio)
• [ENHANCEMENT] Drop invalid prometheus label names in the spanmetrics processor. #5122 (@KyriosGN0)
• [ENHANCEMENT] Improve logging and tracing in the write path to include tenant info. #5436 (@javiermolinar)
• [ENHANCEMENT] Added usage tracker example. #5356 (@javiermolinar)
• [ENHANCEMENT] Add Stop method. #5293 (@stephanos)
• [ENHANCEMENT] Use peer attributes to determine the name of a client service virtual node in the service graph. #5381 (@MartenM)
• [ENHANCEMENT] Put actual size for writing to backend. #5413 (@ruslan-mikhailov)
• [ENHANCEMENT] Upgrade Azurite and Fake-gcs-server to latest version. #5512 (@javiermolinar)
• [ENHANCEMENT] Make block ordering deterministic. #5411 (@rajiv-singh)
• [ENHANCEMENT] Improve exemplar selection in quantile_over_time(). #5278 (@zalegrala)
• [ENHANCEMENT] Measure bytes received before limits and publish it as tempo_distributor_ingress_bytes_total. #5601 (@mapno)
• [ENHANCEMENT] Add total size logging functionality to track trace #5625(@sienna011022)

Bugfix

• [BUGFIX] Fix Tempo configuration options that are always overrided with config overrides section. #5202 (@KyriosGN0)
• [BUGFIX] Correctly apply trace idle period in ingesters and add the concept of trace live period. #5346 (@joe-elliott)
• [BUGFIX] Fix invalid YAML output from /status/runtime_config endpoint by adding document separator. #5371 (@iamrajiv)
• [BUGFIX] Fix panic in query_range HTTP handling that could be triggered by cancellations or other errors. #5667 (@mdisibio)
• [BUGFIX] Do not allow very small steps. #5441 (@ruslan-mikhailov)
• [BUGFIX] Fix incorrect TraceQL string comparison of strings starting with numbers. #5658 (@mdisibio)
• [BUGFIX] Fix incorrect results in TraceQL compare() for spans with array attributes #5519 (@ruslan-mikhailov)
• [BUGFIX] Fix cache collision for incomplete query in SearchTagValuesV2 #5549 (@ruslan-mikhailov)
• [BUGFIX] Fix for structural operator with empty left-hand spanset. #5578 (@ruslan-mikhailov)
• [BUGFIX] Deadlock on invalid query to api/v2/search/tags. (SearchTagsV2) #5607 (@ruslan-mikhailov)
• [BUGFIX] Fixed incorrect root span detection when spans have a child_of link but no parent. #3634 (@mexirica)
• [BUGFIX] Prevent metrics-generator WAL deletion when tenant is empty. #5586 (@sienna011022)
• [BUGFIX] Fix docker-compose port configuration for Alloy gRPC (4319 → 4317). #5536
• [BUGFIX] Fix panic error from empty span id. #5464
• [BUGFIX] Return Bad Request from frontend if the provided tag is invalid in SearchTagValuesV2 endpoint. #5493 (@carles-grafana)

Tempo Rearchitecture [EXPERIMENTAL]

• [CHANGE] BREAKING CHANGE Drop unused backend_scheduler.tenant_measurement_interval, use backend_scheduler.compaction.measure_interval instead. #5328 (@zalegrala)
• [CHANGE] Allow configuration of min/max input blocks for compaction provider. #5373 (@zalegrala)
• [CHANGE] BREAKING CHANGE Add require minimum time between tenant sorting in backend-scheduler. #5410 (@zalegrala)
  The configuration for backend_scheduler.provider.compaction.backoff has been removed.
  Additionally, the compaction_tenant_backoff_total metric has been renamed to compaction_empty_tenant_cycle_total for clarity.
• [CHANGE] Shard backend-scheduler work files locally and modify backend work structure to accommodate sharding approach. [#5...

v2.8.2

• [CHANGE] Update Go to version 1.24.4. #5323 (@stoewer)
• [BUGFIX] Add nil check to partitionAssignmentVar. #5198 (@mapno)
• [BUGFIX] Correct instant query calculation. #5252 (@ruslan-mikhailov)
• [BUGFIX] Fix tracing context propagation in distributor HTTP write requests. #5312 (@mapno)
• [BUGFIX] Fix search by trace:id with short trace ID. #5331 (@ruslan-mikhailov)
• [BUGFIX] Fix bug where most_recent=true wouldn't return most recent results when query overlapped ingesters and few other blocks. #5438 (@joe-elliott)
• [BUGFIX] Fix panic when counter series is missing during avg_over_time aggregation. #5300 (@ie-pham)