Skip to content

Commit d9f36b3

Browse files
authored
New execution model for rates (#134267)
Our current execution path for rate aggregation, particularly for calculating counter resets, requires data in each bucket to be strictly ordered by timestamp. This necessitates a specialized execution path for time-series data, which I believe is unnecessarily complex. I've explored an alternative model: instead of pre-sorting data in the source operator, we buffer data in each bucket and perform a merge-sort just before emitting the output. This would eliminate the need for specialized time-series code and allow us to leverage existing ES|QL optimizations. The main downside is the memory usage for buffering rate points. Each data point requires about 16 bytes; typical queries over a few million points would use less than 100MB, but worst-case scenarios could consume up to 32GB, potentially causing circuit breaking errors. We can mitigate this with the following enhancements: 1. Execute segments in descending max_timestamp order: By processing segments this way, the source operator can provide a "high-water mark" (the maximum timestamp that may appear in the current or subsequent segments). This allows the rate aggregator to safely flush any buffered data that is more recent than this mark, keeping the buffer size minimal or avoid buffering data points. 2. Dynamically split shards by time interval: For large time ranges with interleaved data, we can partition execution into smaller time intervals based on min and max timestamps. This limits buffer size and improves parallelism. This PR is the first step, it cut over from the current to the new rate with buffer. This new rate still delegates to the old rate after merging the buffer. I benchmarked this change for the below query, the executiton time reduced from 405ms -> 270ms. ``` TS my* | WHERE `metrics.system.cpu.time` IS NOT NULL AND @timestamp >= \"2025-07-25T14:55:59.000Z\" AND @timestamp <= \"2025-07-25T16:25:59.000Z\" | STATS AVG(RATE(`metrics.system.cpu.time`)) BY host.name, BUCKET(@timestamp, 1h) | LIMIT 10000 ``` I expect more to come with `high-water mark`. Relates #134324
1 parent 0d92bb2 commit d9f36b3

File tree

43 files changed

+2746
-4892
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+2746
-4892
lines changed

x-pack/plugin/esql/compute/build.gradle

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -628,28 +628,6 @@ tasks.named('stringTemplates').configure {
628628
it.outputFile = "org/elasticsearch/compute/aggregation/ValuesBytesRefAggregator.java"
629629
}
630630

631-
File rateAggregatorInputFile = file("src/main/java/org/elasticsearch/compute/aggregation/X-RateAggregator.java.st")
632-
template {
633-
it.properties = intProperties
634-
it.inputFile = rateAggregatorInputFile
635-
it.outputFile = "org/elasticsearch/compute/aggregation/RateIntAggregator.java"
636-
}
637-
template {
638-
it.properties = longProperties
639-
it.inputFile = rateAggregatorInputFile
640-
it.outputFile = "org/elasticsearch/compute/aggregation/RateLongAggregator.java"
641-
}
642-
template {
643-
it.properties = floatProperties
644-
it.inputFile = rateAggregatorInputFile
645-
it.outputFile = "org/elasticsearch/compute/aggregation/RateFloatAggregator.java"
646-
}
647-
template {
648-
it.properties = doubleProperties
649-
it.inputFile = rateAggregatorInputFile
650-
it.outputFile = "org/elasticsearch/compute/aggregation/RateDoubleAggregator.java"
651-
}
652-
653631
File stdDevAggregatorInputFile = file("src/main/java/org/elasticsearch/compute/aggregation/X-StdDevAggregator.java.st")
654632
template {
655633
it.properties = intProperties

x-pack/plugin/esql/compute/src/main/generated-src/org/elasticsearch/compute/aggregation/RateFloatAggregator.java

Lines changed: 0 additions & 376 deletions
This file was deleted.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/RateDoubleAggregatorFunctionSupplier.java

Lines changed: 0 additions & 46 deletions
This file was deleted.

0 commit comments

Comments
 (0)