-
Notifications
You must be signed in to change notification settings - Fork 25.7k
Description
We want to allow switching to the exponential_histogram field type for OTLP ingestion use cases without breaking backwards compatibility.
This can cause two kinds of problems:
- exponential-histogram JSON documents are attempted to be ingested into indices which use the
histogramtype: This can happen e.g. if the index template for a data stream has been adjusted, but a rollover has not happened yet - histogram JSON documents are attempted to be ingested into indices using the
exponential_histogramtype: Happens for example due to older OpenTelemetry collector versions running in parallel with newer ones, which already adjusted the field type
We decided on adding support for coerce to the histogram and exponential_histogram field mappers to cover these use cases:
- accept exponential histogram fields on
histogramfields and convert them to T-Digest forcoerceAdd coerce setting to histogram field mapperย #137191 - accept histogram fields (assuming T-Digest content) on
exponential_histogramfields and convert them to exponential histograms forcoerceImplement coerce for exponential_histogram field typeย #137944
The histogram field type content can be either T-Digests or HDR histograms. We decided on assuming T-Digest to implement this feature for the following reasons:
- T-Digest and exponential histograms support negative values, while HDR histograms do not
- T-Digest is the kind used in classic Elastic APM and the current EDOT Collector (OTLP ingest)
For the conversion from exponential histogram to T-Digest we'll use the same algorithm already implemented by the elasticsearch OTLP metrics endpoint. For the conversion from T-Digest to exponential histograms, we'll use an algorithm which ideally inverts this conversion, introducing as little error as possible.