[ES|QL] Consider min/max from predicates when transform date_trunc/bucket to round_to #131341

fang-xing-esql · 2025-07-15T23:39:42Z

This is a follow up enhancement to #128639, aimed at improving the date_histogram related ES|QL performance regression tests.

The nyc_taxis dataset is used in the date_histogram and date_trunc related ES|QL performance regression tests. It contains a wide date range, from the year 1900 to 2253. If relying only on Lucene statistics to calculate the fixed rounding points for the dropoff_datetime field, the fixed rounding points could not be calculated due to this wide range. As a result, the date_trunc function in the existing performance regression tests is not rewritten to round_to, even though the query below only processes two months of data.

FROM nyc_taxis 
| where dropoff_datetime < "2015-03-01T00:00:00" AND dropoff_datetime >= "2015-01-01T00:00:00" 
| eval dropoffs_over_time=date_trunc(1 week, dropoff_datetime) 
| stats c = count(dropoff_datetime) by dropoffs_over_time 
| sort dropoffs_over_time

This PR takes the query predicates into account and calculates the fixed rounding points based on the intersection of the date range from SearchStats (1900–2253) and the query's filter range (2015-01-01 to 2025-03-01). By using this narrowed intersection (2015-01-01 to 2025-03-01), the date_trunc function can be safely optimized to round_to.

Performance Observations
The three queries below are measured against main and this branch, with different calendar intervals and various data volume within each date range. The observation is that different RoundTo evaluators perform differently comparing with DateTruncDatetimeEvaluator. The

The elapsed time below are the average of 10 runs of each query on main and branch

Q1 - weekly interval on 3 months of data, RoundToLongLinearSearchEvaluator
FROM nyc_taxis 
| where dropoff_datetime < "2015-03-01" AND dropoff_datetime >= "2015-01-01" 
| eval dropoffs_over_time=date_trunc(1 week, dropoff_datetime) 
| stats c = count(dropoff_datetime) by dropoffs_over_time 
| sort dropoffs_over_time

main: 449.7 ms
branch: 499.8 ms, 11% slower

Q2 - monthly interval on 3 months of data, RoundToLong3Evaluator
FROM nyc_taxis 
| where dropoff_datetime < "2015-03-01" AND dropoff_datetime >= "2015-01-01" 
| eval dropoffs_over_time=date_trunc(1 month, dropoff_datetime) 
| stats c = count(dropoff_datetime) by dropoffs_over_time 
| sort dropoffs_over_time

main: 543.8 ms
branch: 464.3 ms, 14.6% improvement

Q3 - monthly interval on 12 months of data, RoundToLongBinarySearchEvaluator
FROM nyc_taxis 
| where dropoff_datetime < "2016-01-01" AND dropoff_datetime >= "2015-01-01" 
| eval dropoffs_over_time=date_trunc(1 month, dropoff_datetime) 
| stats c = count(dropoff_datetime) by dropoffs_over_time 
| sort dropoffs_over_time

main: 2209.6 ms
branch: 1877.3 ms, 14% improvement

Looking more closely at Q1, where a slight performance degradation was observed on the 3-month dataset using 9 weekly buckets with RoundToLongLinearSearchEvaluator, I noticed that the query behaves differently depending on the evaluator used. Among the three RoundTo evaluators, the manual RoundToLong9Evaluator performs the best, and it also outperforms DateTruncDatetimeEvaluator.

{
            "operator" : "EvalOperator[evaluator=RoundToLongBinarySearchEvaluator[field=Attribute[channel=1]]]",
            "status" : {
              "process_nanos" : 47546640,
              "pages_processed" : 718,
              "rows_received" : 5131188,
              "rows_emitted" : 5131188
            }
},
{
            "operator" : "EvalOperator[evaluator=RoundToLongLinearSearchEvaluator[field=Attribute[channel=1]]]",
            "status" : {
              "process_nanos" : 36383473,
              "pages_processed" : 718,
              "rows_received" : 5131188,
              "rows_emitted" : 5131188
            }
}
{
            "operator" : "EvalOperator[evaluator=RoundToLong9Evaluator[field=Attribute[channel=1], p0=1419811200000, p1=1420416000000, p2=1421020800000, p3=1421625600000, p4=1422230400000, p5=1422835200000, p6=1423440000000, p7=1424044800000, p8=1424649600000]]",
            "status" : {
              "process_nanos" : 24817116,
              "pages_processed" : 718,
              "rows_received" : 5131188,
              "rows_emitted" : 5131188
            }
 }
{
            "operator" : "EvalOperator[evaluator=DateTruncDatetimeEvaluator[fieldVal=Attribute[channel=1], rounding=Rounding[WEEK_OF_WEEKYEAR in Z][fixed to midnight]]]",
            "status" : {
              "process_nanos" : 33711389,
              "pages_processed" : 761,
              "rows_received" : 5699770,
              "rows_emitted" : 5699770
            }
}

There are a couple of follow-up tasks for the performance-related work:

Open a separate PR to replace RoundToLongLinearSearchEvaluator with manual evaluators, as experiments have shown that the manual versions perform better. [ES|QL] Replace RoundTo linear search evaluator with manual evaluators #131733
Add more queries with various calendar intervals to the performance regression tests, so we can better track changes and ensure consistent performance across different scenarios. [ES|QL] Add new queries into nyc_taxis track to measure DateTrunc to RoundTo transformation rally-tracks#824

…o round_to

elasticsearchmachine · 2025-07-15T23:40:09Z

Hi @fang-xing-esql, I've created a changelog YAML for you.

nik9000 · 2025-07-18T17:15:39Z

...sticsearch/xpack/esql/optimizer/rules/logical/local/LocalSubstituteSurrogateExpressions.java

+
+    private List<EsqlBinaryComparison> predicates(Eval eval, Expression field) {
+        List<EsqlBinaryComparison> binaryComparisons = new ArrayList<>();
+        eval.forEachUp(Filter.class, filter -> {


This scans up from the root of the query, right? Is there a way to put something between the eval and the filter that makes the scan invalid? I know if we weren't dealing with NameId you could rename stuff. But that shouldn't be possible now. But is there anything that is?

Like, maybe you can filter on timestamp and then change it. Or is that never possible because of NameId too?

I think this is fine actually. But I don't know enough!

I’m going to add some more queries to the functional tests, there will definitely be cases, like the eval, rename commands, that can complicate things. I implemented the rewrite on the data node rather than the coordinator node to avoid transport version changes. My thinking was that the coordinator node might be able to simplify some of the more complex scenarios before the plan reaches the data node.

I haven’t specifically handled NameIds yet, but I’ll keep an eye on them as I test with more complex queries.

I added some more queries with Eval, Rename that change the field reference by DateTrunc and Bucket. We have a couple of rules that are smart enough to find the real field, even though they are changed by Eval or Rename earlier.

...sticsearch/xpack/esql/optimizer/rules/logical/local/LocalSubstituteSurrogateExpressions.java

elasticsearchmachine · 2025-07-24T01:14:00Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

fang-xing-esql · 2025-07-24T15:25:54Z

@astefan @bpintea Could you help review the changes to LocalSubstituteSurrogateExpressions in this PR? It updates the rule to extract predicates related to the field referenced by DateTrunc and Bucket, and consider the high/low bound from predicates when rewriting them into RoundTo.

I added some new tests covering various kinds of predicates on the field, including overlaps or non-overlaps with Lucene statistics, and also some queries involving eval or rename, where the field is changed to a reference before calling DateTrunc and Bucket.

I'd really appreciate your thoughts, if you can think of any other way to break/test it please let me know, thanks in advance!

astefan

The way the LocalSubstituteSurrogateExpressions looks like in main is a generic rule that deals with local surrogate expressions of any kind and applies the surrogate logic in functions found in eval.

This PR though, leaks the logic of specific surrogate expressions (like the ones of DateTrunc and Bucket) in LocalSubstituteSurrogateExpressions like finding all the EsqlBinaryComparisons, checking if they act on a foldable value etc.

Maybe these two functions (bucket and date_trunc) should not use the surrogate expressions approach to have them changed to round_to depending on things that are not only the ones from SearchStats.

astefan · 2025-07-28T14:32:55Z

...ql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/DateTrunc.java

+        Holder<Boolean> foundMaxValue = new Holder<>(false);
+        for (EsqlBinaryComparison binaryComparison : binaryComparisons) {
+            if (binaryComparison.right() instanceof Literal l) {
+                long value = Long.parseLong(l.value().toString());


Why are we doing here the parsing of a long from String? The dataType of the Literal is not enough?

Make sense, I'll refactor this.

astefan · 2025-07-28T14:34:30Z

...ql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/date/DateTrunc.java

+            Long minFromPredicates = minMaxFromPredicates.v1();
+            Long maxFromPredicates = minMaxFromPredicates.v2();
+            // Consolidate min/max from SearchStats and query
+            if (minFromPredicates instanceof Long minValue) {


This value here should already be a Long. Why the instanceof?

We need a check for null, not instanceof here, I'll change it.

fang-xing-esql · 2025-07-30T02:53:14Z

The way the LocalSubstituteSurrogateExpressions looks like in main is a generic rule that deals with local surrogate expressions of any kind and applies the surrogate logic in functions found in eval.

This PR though, leaks the logic of specific surrogate expressions (like the ones of DateTrunc and Bucket) in LocalSubstituteSurrogateExpressions like finding all the EsqlBinaryComparisons, checking if they act on a foldable value etc.

Maybe these two functions (bucket and date_trunc) should not use the surrogate expressions approach to have them changed to round_to depending on things that are not only the ones from SearchStats.

That is a good point. Actually at the beginning, I was debating between two implementations - a generalized transformation leveraging the surrogate expression mechanism, or a dedicated rewrite rule specifically targeting bucket and date_trunc.

This PR is the first approach, and #132143 is the second. They both can achieve the goal, I'm open to either way. It is true that the predicates analysis can happen on the coordinator node, the main reason that I choose to evaluate them together with SearchStats on the data node is to avoid carrying over new attributes(min/max from predicates) from coordinator node to data node, and no new TransportVersion is required, and simplifies the backward compatibility.

fang-xing-esql · 2025-08-06T18:49:05Z

Close this as we decide to take this approach - #132143

consider min/max from predicates when transform date_trunc/bucket int…

caa2f73

…o round_to

fang-xing-esql added >enhancement :Analytics/ES|QL AKA ESQL v9.2.0 labels Jul 15, 2025

Update docs/changelog/131341.yaml

93c3297

fang-xing-esql mentioned this pull request Jul 18, 2025

[ES|QL] Substitute date_trunc with round_to when the pre-calculated rounding points are available #128639

Merged

nik9000 reviewed Jul 18, 2025

View reviewed changes

This was referenced Jul 22, 2025

[ES|QL] Add new queries into nyc_taxis track to measure DateTrunc to RoundTo transformation elastic/rally-tracks#824

Merged

[ES|QL] Replace RoundTo linear search evaluator with manual evaluators #131733

Merged

fang-xing-esql and others added 5 commits July 23, 2025 00:49

more tests

3563dcb

Merge branch 'main' into datetrunc-roundto-with-min-max-from-predicates

2ed82f4

refactor

de99e19

Merge branch 'main' into datetrunc-roundto-with-min-max-from-predicates

41a493d

add more comments

5ba54a5

fang-xing-esql marked this pull request as ready for review July 24, 2025 01:13

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jul 24, 2025

fang-xing-esql requested review from astefan and bpintea July 24, 2025 15:12

fang-xing-esql and others added 2 commits July 25, 2025 12:51

refactor tests

def5d8d

Merge branch 'main' into datetrunc-roundto-with-min-max-from-predicates

9f66fa6

astefan requested changes Jul 28, 2025

View reviewed changes

fang-xing-esql mentioned this pull request Jul 30, 2025

[ES|QL] Consider min/max from predicates when transform date_trunc/bucket to round_to option 2 #132143

Merged

fang-xing-esql and others added 2 commits July 30, 2025 20:10

update according to review comments

edcde73

Merge branch 'main' into datetrunc-roundto-with-min-max-from-predicates

6c296d0

fang-xing-esql closed this Aug 6, 2025

[ES|QL] Consider min/max from predicates when transform date_trunc/bucket to round_to #131341

[ES|QL] Consider min/max from predicates when transform date_trunc/bucket to round_to #131341

Uh oh!

Conversation

fang-xing-esql commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 15, 2025

Uh oh!

nik9000 Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 24, 2025

Uh oh!

fang-xing-esql commented Jul 24, 2025

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

astefan Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

astefan Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql commented Jul 30, 2025

Uh oh!

fang-xing-esql commented Aug 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fang-xing-esql commented Jul 15, 2025 •

edited

Loading