Histogram field support in ES|QL #128622
Description
Histogram fields don't track the histogram type currently, so percentile aggregations and merging can be very inaccurate. To fix this, we plan to introduce separate field types per histogram type, matching what's needed for OTel data. Once available, these field types will be wired in ES|QL, with support for percentile aggs and bucketing operations.
Discussion: https://github.com/elastic/metrics-program/discussions/72
Tasks:
Quick outline of the work we expect to need for this. Each of these top level items is expected to be several PRs worth of work.
Exponential Histogram Support
- Add Exponential histogram library to elasticsearch libs (with circuit breaker integration)
- Add Exponential Histogram field type
- Add support for min/max/sum/count storage in field type and support in library
- Add Exponential Histogram DataType for ESQL, including function and aggregation support
- If we decide to use exponential histograms for explicit bucket boundary histograms, we might want to add special buckets for positive and negative infinity values
-
The algorithm currently is a bit wasteful if the merged histograms are much smaller than the accumulator and can be optimized by first buffering their buckets. As we expect many histograms received via OTLP to have only few populated buckets, we should try this optimization and benchmark it's effectiveness.Benchmarks have shown that this is currently not worth it, see Remove Todo for exponential histogram merging optimization. #135345
T-Digest Support
- T-digest field type
- T-Digest ESQL Data type, including function and aggregation support
HDR Histogram Support
- Fork HDR Histogram library to add circuit breaker support (Maybe? We've avoided this so far...)
- HDR Histogram Field type
- HDR Histogram DataType, including funciton and aggregation support