Add support for storing and querying T-Digest sketches #137649
Description
Main meta issue for tracking the T-Digest field support
Field Mapper
- Basic field mapper, largely copied form
Histogram(T digest field mapper #137546) - Store Min/Max/Sum/Count on ingest. (Add summary metrics for tdigest fields #137982)
- Compress large input sets (probably after tech preview)
- Input validation, especially for summary data fields (after tech preview)
- Improved randomized testing, yaml tests (Add summary metrics for tdigest fields #137982)
- Review behavior for sum sub-field on empty histogram (null vs 0) (Sum of empty t-digest should be null #138378)
- Documentation
ESQL Support
Since we want this to have parity with the Exponential Histogram field, this is the same list as #137549
- Add a blockloader (Block loader and compute interface for t-digest field #138246)
- Add an ESQL data and block type
- Wire up CSV test data loading for T-digest field (T digest csv test support #138391)
- Robust test data for CSV tests (probably after tech preview)
- Add merge aggregation (or support this in the exponential histogram merge agg)
- Add support for PERCENTILES aggregation
- Add support for MEDIAN aggregation
- Add support for AVG aggregation
- Add support for MIN aggregation
- Add support for MAX aggregation
- Add support for SUM aggregation
- Add support for CASE ( as return value)
- Add support for COALESCE
- Add support for COUNT aggregation
- Maybe add a new VALUE_COUNT aggregation (depends on the semantics of COUNT)
- Add support for VARIANCE aggregation
- Add support for STD_DEV aggregation
- Add support for MEDIAN_ABSOLUTE_DEVIATION aggregation
- Add support for == operator
- Add support for string conversion (TO_STRING, TO_TDIGEST)
ESQL Support for Legacy Histograms
One of the goals for creating a t-digest data type in ESQL is support for legacy histogram field data. We intend to do this by creating a Histogram field type which can then be converted into a t-digest.
- Histogram block loader
- Histogram Data type
- Histogram to T-Digest conversion function
- Implicit conversion planner rule?
Classic Aggregation support
- T-digest values source
This is the list of aggregations we support on legacy histograms (except for boxplot, but we should support that)
- Add support for value_count
- Add support for sum
- Add support for average
- Add support for min
- Add support for max
- Add support for percentiles
- Add support for percentile rank
- Add support for range
- Add support for histogram
- Add support for boxplot