Skip to content

Esql - Fix lucene push down behavior when a range contains nanos and millis #125595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 25, 2025
Merged

Esql - Fix lucene push down behavior when a range contains nanos and millis #125595

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ad78aff
Test and fix for mixed type range
not-napoleon Mar 25, 2025
675c85c
Merge branch 'main' into esql-date-nanos-range-mixed-type-bug
not-napoleon Mar 25, 2025
c400cba
Update docs/changelog/125595.yaml
not-napoleon Mar 25, 2025
91fea37
[CI] Auto commit changes from spotless
Mar 25, 2025
a73b8e8
add capability for the fix
not-napoleon Mar 25, 2025
21acf40
right, I wrote two tests for that bug, they both need a required capa…
not-napoleon Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/125595.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 125595
summary: Esql - Fix lucene push down behavior when a range contains nanos and millis
area: ES|QL
type: bug
issues: []
106 changes: 106 additions & 0 deletions x-pack/plugin/esql/qa/testFixtures/src/main/resources/date_nanos.csv-spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1762,3 +1762,109 @@ nanos:date_nanos
2023-10-23T12:15:03.360103847Z
2023-10-23T12:15:03.360103847Z
;

Range Without Included Endpoints
required_capability: to_date_nanos
required_capability: fix_date_nanos_lucene_pushdown_bug

FROM date_nanos
| WHERE millis > "2020-01-01"
| WHERE nanos > to_datenanos("2023-10-23T12:15:03.360103847") AND nanos < to_datenanos("2023-10-23T13:53:55.832987654Z")
| KEEP nanos;
ignoreOrder:true
warningRegex:Line 3:67: evaluation of \[nanos < to_datenanos\(\\\"2023-10-23T13:53:55\.832987654Z\\\"\)\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:67: java.lang.IllegalArgumentException: single-value function encountered multi-value
warningRegex:Line 3:9: evaluation of \[nanos > to_datenanos\(\\\"2023-10-23T12:15:03\.360103847\\\"\)\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:9: java.lang.IllegalArgumentException: single-value function encountered multi-value

nanos:date_nanos
2023-10-23T13:52:55.015787878Z
2023-10-23T13:51:54.732102837Z
2023-10-23T13:33:34.937193000Z
2023-10-23T12:27:28.948000000Z
;

Range Without Included Endpoints with implicit casting
required_capability: to_date_nanos
required_capability: fix_date_nanos_lucene_pushdown_bug

FROM date_nanos
| WHERE millis > "2020-01-01"
| WHERE nanos > "2023-10-23T12:15:03.360103847" AND nanos < "2023-10-23T13:53:55.832987654Z"
| KEEP nanos;
ignoreOrder:true
warningRegex:Line 3:53: evaluation of \[nanos < \\\"2023-10-23T13:53:55\.832987654Z\\\"\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:53: java.lang.IllegalArgumentException: single-value function encountered multi-value
warningRegex:Line 3:9: evaluation of \[nanos > \\\"2023-10-23T12:15:03\.360103847\\\"\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:9: java.lang.IllegalArgumentException: single-value function encountered multi-value

nanos:date_nanos
2023-10-23T13:52:55.015787878Z
2023-10-23T13:51:54.732102837Z
2023-10-23T13:33:34.937193000Z
2023-10-23T12:27:28.948000000Z
;

Range With Now date math
required_capability: to_date_nanos
required_capability: fix_date_nanos_lucene_pushdown_bug

FROM date_nanos
| WHERE millis > "2020-01-01"
| WHERE nanos > TO_DATETIME("2023-10-23T12:27:28.948") AND nanos < now() - 1d
| KEEP nanos;
ignoreOrder:true
warningRegex:Line 3:60: evaluation of \[nanos < now\(\) - 1d\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:60: java.lang.IllegalArgumentException: single-value function encountered multi-value
warningRegex:Line 3:9: evaluation of \[nanos > TO_DATETIME\(\\\"2023-10-23T12:27:28\.948\\\"\)\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:9: java.lang.IllegalArgumentException: single-value function encountered multi-value

nanos:date_nanos
2023-10-23T13:55:01.543123456Z
2023-10-23T13:53:55.832987654Z
2023-10-23T13:52:55.015787878Z
2023-10-23T13:51:54.732102837Z
2023-10-23T13:33:34.937193000Z
;

Mixed Nanos Millis Range Compare
required_capability: to_date_nanos
required_capability: fix_date_nanos_lucene_pushdown_bug

FROM date_nanos
| WHERE millis > "2020-01-01"
| WHERE nanos > to_datenanos("2023-10-23T12:15:03.360103847") AND nanos < to_datetime("2023-10-23T13:53:55.832")
| KEEP nanos;
ignoreOrder:true
warningRegex:Line 3:67: evaluation of \[nanos < to_datetime\(\\\"2023-10-23T13:53:55\.832\\\"\)\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:67: java.lang.IllegalArgumentException: single-value function encountered multi-value
warningRegex:Line 3:9: evaluation of \[nanos > to_datenanos\(\\\"2023-10-23T12:15:03\.360103847\\\"\)\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:9: java.lang.IllegalArgumentException: single-value function encountered multi-value

nanos:date_nanos
2023-10-23T13:52:55.015787878Z
2023-10-23T13:51:54.732102837Z
2023-10-23T13:33:34.937193000Z
2023-10-23T12:27:28.948000000Z
;

Mixed Nanos Millis Range Compare, millis field
required_capability: to_date_nanos
required_capability: fix_date_nanos_lucene_pushdown_bug
required_capability: fix_date_nanos_mixed_range_pushdown_bug

FROM date_nanos
| WHERE millis > to_datenanos("2023-10-23T12:15:03.360103847") AND millis < to_datetime("2023-10-23T13:53:55.832")
| KEEP nanos;
ignoreOrder:true
warningRegex:Line 3:67: evaluation of \[nanos < to_datetime\(\\\"2023-10-23T13:53:55\.832Z\\\"\)\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:67: java.lang.IllegalArgumentException: single-value function encountered multi-value
warningRegex:Line 3:9: evaluation of \[nanos > to_datenanos\(\\\"2023-10-23T12:15:03\.360103847\\\"\)\] failed, treating result as null\. Only first 20 failures recorded\.
warningRegex:Line 3:9: java.lang.IllegalArgumentException: single-value function encountered multi-value

nanos:date_nanos
2023-10-23T13:52:55.015787878Z
2023-10-23T13:51:54.732102837Z
2023-10-23T13:33:34.937193000Z
2023-10-23T12:27:28.948000000Z
;
5 changes: 5 additions & 0 deletions x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -554,6 +554,11 @@ public enum Cap {
* Indicates that https://github.com/elastic/elasticsearch/issues/125439 (incorrect lucene push down for date nanos) is fixed
*/
FIX_DATE_NANOS_LUCENE_PUSHDOWN_BUG(),
/**
* Fixes a bug where dates are incorrectly formatted if a where clause compares nanoseconds to both milliseconds and nanoseconds,
* e.g. {@code WHERE millis > to_datenanos("2023-10-23T12:15:03.360103847") AND millis < to_datetime("2023-10-23T13:53:55.832")}
*/
FIX_DATE_NANOS_MIXED_RANGE_PUSHDOWN_BUG(),
/**
* DATE_PARSE supports reading timezones
*/
Expand Down
26 changes: 16 additions & 10 deletions ...ck/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/predicate/Range.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@

import static java.util.Arrays.asList;
import static org.elasticsearch.xpack.esql.core.expression.Foldables.valueOf;
import static org.elasticsearch.xpack.esql.core.type.DataType.DATETIME;
import static org.elasticsearch.xpack.esql.core.type.DataType.DATE_NANOS;
import static org.elasticsearch.xpack.esql.core.type.DataType.IP;
import static org.elasticsearch.xpack.esql.core.type.DataType.UNSIGNED_LONG;
Expand All @@ -42,9 +41,8 @@
import static org.elasticsearch.xpack.esql.core.util.NumericUtils.unsignedLongAsNumber;
import static org.elasticsearch.xpack.esql.type.EsqlDataTypeConverter.DEFAULT_DATE_NANOS_FORMATTER;
import static org.elasticsearch.xpack.esql.type.EsqlDataTypeConverter.DEFAULT_DATE_TIME_FORMATTER;
import static org.elasticsearch.xpack.esql.type.EsqlDataTypeConverter.dateTimeToString;
import static org.elasticsearch.xpack.esql.type.EsqlDataTypeConverter.dateWithTypeToString;
import static org.elasticsearch.xpack.esql.type.EsqlDataTypeConverter.ipToString;
import static org.elasticsearch.xpack.esql.type.EsqlDataTypeConverter.nanoTimeToString;
import static org.elasticsearch.xpack.esql.type.EsqlDataTypeConverter.versionToString;

// BETWEEN or range - is a mix of gt(e) AND lt(e)
Expand Down Expand Up @@ -217,16 +215,23 @@ private RangeQuery translate(TranslatorHandler handler) {
String format = null;

DataType dataType = value.dataType();
logger.trace("Translating Range into lucene query. dataType is [{}] upper is [{}] lower is [{}]", dataType, lower, upper);
if (dataType == DataType.DATETIME && lower.dataType() == DATETIME && upper.dataType() == DATETIME) {
l = dateTimeToString((Long) l);
u = dateTimeToString((Long) u);
logger.warn(
"Translating Range into lucene query. dataType is [{}] upper is [{}<{}>] lower is [{}<{}>]",
dataType,
lower,
lower.dataType(),
upper,
upper.dataType()
);
if (dataType == DataType.DATETIME) {
l = dateWithTypeToString((Long) l, lower.dataType());
u = dateWithTypeToString((Long) u, upper.dataType());
format = DEFAULT_DATE_TIME_FORMATTER.pattern();
}

if (dataType == DATE_NANOS && lower.dataType() == DATE_NANOS && upper.dataType() == DATE_NANOS) {
l = nanoTimeToString((Long) l);
u = nanoTimeToString((Long) u);
if (dataType == DATE_NANOS) {
l = dateWithTypeToString((Long) l, lower.dataType());
u = dateWithTypeToString((Long) u, upper.dataType());
format = DEFAULT_DATE_NANOS_FORMATTER.pattern();
}

Expand Down Expand Up @@ -258,6 +263,7 @@ private RangeQuery translate(TranslatorHandler handler) {
u = unsignedLongAsNumber(ul);
}
}
logger.warn("Building range query with format string [{}]", format);
return new RangeQuery(source(), handler.nameOf(value), l, includeLower(), u, includeUpper(), format, zoneId);
}

Expand Down
Loading