Skip to content

Commit 190a3f1

Browse files
fang-xing-esqlfang-xing-esql
authored
[ES|QL] Take date date_nanos implicit casting out of snapshot (#133369)
* Enable date date_nanos autocasting
1 parent 5e7159e commit 190a3f1

File tree

9 files changed

+87
-44
lines changed

9 files changed

+87
-44
lines changed

‎docs/changelog/133369.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 133369
2+
summary: Enable `date` `date_nanos` implicit casting
3+
area: ES|QL
4+
type: enhancement
5+
issues: []

‎docs/reference/query-languages/esql/esql-multi-index.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,62 @@ FROM events_*
135135
| 2023-10-23T12:27:28.948Z | 172.21.2.113 | 2764889 | Connected to 10.1.0.2 |
136136
| 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233 | Connected to 10.1.0.3 |
137137

138+
### Date and date_nanos union type [esql-multi-index-date-date-nanos-union]
139+
```{applies_to}
140+
stack: ga 9.2.0
141+
```
142+
When the type of an {{esql}} field is a *union* of `date` and `date_nanos` across different indices, {{esql}} automatically casts all values to the `date_nanos` type during query execution. This implicit casting ensures that all values are handled with nanosecond precision, regardless of their original type. As a result, users can write queries against such fields without needing to perform explicit type conversions, and the query engine will seamlessly align the types for consistent and precise results.
143+
144+
`date_nanos` fields offer higher precision but have a narrower range of valid values compared to `date` fields. This limits their representable dates roughly from 1970 to 2262. This is because dates are stored as a `long` representing nanoseconds since the epoch. When a field is mapped as both `date` and `date_nanos` across different indices, {{esql}} defaults to the more precise `date_nanos` type. This behavior ensures that no precision is lost when querying multiple indices with differing date field types. For dates that fall outside the valid range of `date_nanos` in fields that are mapped to both `date` and `date_nanos` across different indices, {{esql}} returns null by default. However, users can explicitly cast these fields to the `date` type to obtain a valid value, with precision limited to milliseconds.
145+
146+
For example, if the `@timestamp` field is mapped as `date` in one index and `date_nanos` in another, {{esql}} will automatically treat all `@timestamp` values as `date_nanos` during query execution. This allows users to write queries that utilize the `@timestamp` field without encountering type mismatch errors, ensuring accurate time-based operations and comparisons across the combined dataset.
147+
148+
**index: events_date**
149+
150+
```
151+
{
152+
"mappings": {
153+
"properties": {
154+
"@timestamp": { "type": "date" },
155+
"client_ip": { "type": "ip" },
156+
"event_duration": { "type": "long" },
157+
"message": { "type": "keyword" }
158+
}
159+
}
160+
}
161+
```
162+
163+
**index: events_date_nanos**
164+
165+
```
166+
{
167+
"mappings": {
168+
"properties": {
169+
"@timestamp": { "type": "date_nanos" },
170+
"client_ip": { "type": "ip" },
171+
"event_duration": { "type": "long" },
172+
"message": { "type": "keyword" }
173+
}
174+
}
175+
}
176+
```
177+
178+
```esql
179+
FROM events_date*
180+
| EVAL date = @timestamp::date
181+
| KEEP @timestamp, date, client_ip, event_duration, message
182+
| SORT date
183+
```
184+
185+
| @timestamp:date_nanos | date:date | client_ip:ip | event_duration:long | message:keyword |
186+
|--------------------------| --- |--------------|---------| --- |
187+
| null |1969-10-23T13:33:34.937Z| 172.21.0.5 | 1232382 |Disconnected|
188+
| 2023-10-23T12:15:03.360Z |2023-10-23T12:15:03.360Z| 172.21.2.162 | 3450233 |Connected to 10.1.0.3|
189+
| 2023-10-23T12:15:03.360103847Z|2023-10-23T12:15:03.360Z| 172.22.2.162 | 3450233 |Connected to 10.1.0.3|
190+
| 2023-10-23T12:27:28.948Z |2023-10-23T12:27:28.948Z| 172.22.2.113 | 2764889 |Connected to 10.1.0.2|
191+
| 2023-10-23T12:27:28.948Z |2023-10-23T12:27:28.948Z| 172.21.2.113 | 2764889 |Connected to 10.1.0.2|
192+
| 2023-10-23T13:33:34.937193Z |2023-10-23T13:33:34.937Z| 172.22.0.5 | 1232382 |Disconnected|
193+
| null |2263-10-23T13:51:54.732Z| 172.21.3.15 | 725448 |Connection error|
138194

139195
## Index metadata [esql-multi-index-index-metadata]
140196

‎x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,6 @@
5858
import static org.elasticsearch.test.ListMatcher.matchesList;
5959
import static org.elasticsearch.test.MapMatcher.assertMap;
6060
import static org.elasticsearch.test.MapMatcher.matchesMap;
61-
import static org.elasticsearch.xpack.esql.action.EsqlCapabilities.Cap.IMPLICIT_CASTING_DATE_AND_DATE_NANOS;
6261
import static org.elasticsearch.xpack.esql.core.type.DataType.isMillisOrNanos;
6362
import static org.elasticsearch.xpack.esql.qa.rest.RestEsqlTestCase.Mode.SYNC;
6463
import static org.elasticsearch.xpack.esql.tools.ProfileParser.parseProfile;
@@ -715,9 +714,7 @@ public void testSuggestedCast() throws IOException {
715714
Map<String, Object> results = entityAsMap(resp);
716715
List<?> columns = (List<?>) results.get("columns");
717716
DataType suggestedCast = DataType.suggestedCast(Set.of(listOfTypes.get(i), listOfTypes.get(j)));
718-
if (IMPLICIT_CASTING_DATE_AND_DATE_NANOS.isEnabled()
719-
&& isMillisOrNanos(listOfTypes.get(i))
720-
&& isMillisOrNanos(listOfTypes.get(j))) {
717+
if (isMillisOrNanos(listOfTypes.get(i)) && isMillisOrNanos(listOfTypes.get(j))) {
721718
// datetime and date_nanos are casted to date_nanos implicitly
722719
assertThat(columns, equalTo(List.of(Map.ofEntries(Map.entry("name", "my_field"), Map.entry("type", "date_nanos")))));
723720
} else {

‎x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -353,7 +353,7 @@ public enum Cap {
353353
/**
354354
* Support implicit casting for union typed fields that are mixed with date and date_nanos type.
355355
*/
356-
IMPLICIT_CASTING_DATE_AND_DATE_NANOS(Build.current().isSnapshot()),
356+
IMPLICIT_CASTING_DATE_AND_DATE_NANOS,
357357

358358
/**
359359
* Support for named or positional parameters in EsqlQueryRequest.

‎x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java‎

Lines changed: 23 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,6 @@
149149
import static java.util.Collections.emptyList;
150150
import static java.util.Collections.singletonList;
151151
import static org.elasticsearch.xpack.core.enrich.EnrichPolicy.GEO_MATCH_TYPE;
152-
import static org.elasticsearch.xpack.esql.action.EsqlCapabilities.Cap.IMPLICIT_CASTING_DATE_AND_DATE_NANOS;
153152
import static org.elasticsearch.xpack.esql.core.type.DataType.AGGREGATE_METRIC_DOUBLE;
154153
import static org.elasticsearch.xpack.esql.core.type.DataType.BOOLEAN;
155154
import static org.elasticsearch.xpack.esql.core.type.DataType.DATETIME;
@@ -192,7 +191,7 @@ public class Analyzer extends ParameterizedRuleExecutor<LogicalPlan, AnalyzerCon
192191
new ResolveLookupTables(),
193192
new ResolveFunctions(),
194193
new ResolveInference(),
195-
new DateMillisToNanosInEsRelation(IMPLICIT_CASTING_DATE_AND_DATE_NANOS.isEnabled())
194+
new DateMillisToNanosInEsRelation()
196195
),
197196
new Batch<>(
198197
"Resolution",
@@ -1975,42 +1974,32 @@ private static LogicalPlan planWithoutSyntheticAttributes(LogicalPlan plan) {
19751974
*/
19761975
private static class DateMillisToNanosInEsRelation extends Rule<LogicalPlan, LogicalPlan> {
19771976

1978-
private final boolean isSnapshot;
1979-
1980-
DateMillisToNanosInEsRelation(boolean isSnapshot) {
1981-
this.isSnapshot = isSnapshot;
1982-
}
1983-
19841977
@Override
19851978
public LogicalPlan apply(LogicalPlan plan) {
1986-
if (isSnapshot) {
1987-
return plan.transformUp(EsRelation.class, relation -> {
1988-
if (relation.indexMode() == IndexMode.LOOKUP) {
1989-
return relation;
1979+
return plan.transformUp(EsRelation.class, relation -> {
1980+
if (relation.indexMode() == IndexMode.LOOKUP) {
1981+
return relation;
1982+
}
1983+
return relation.transformExpressionsUp(FieldAttribute.class, f -> {
1984+
if (f.field() instanceof InvalidMappedField imf && imf.types().stream().allMatch(DataType::isDate)) {
1985+
HashMap<ResolveUnionTypes.TypeResolutionKey, Expression> typeResolutions = new HashMap<>();
1986+
var convert = new ToDateNanos(f.source(), f);
1987+
imf.types().forEach(type -> typeResolutions(f, convert, type, imf, typeResolutions));
1988+
var resolvedField = ResolveUnionTypes.resolvedMultiTypeEsField(f, typeResolutions);
1989+
return new FieldAttribute(
1990+
f.source(),
1991+
f.parentName(),
1992+
f.qualifier(),
1993+
f.name(),
1994+
resolvedField,
1995+
f.nullable(),
1996+
f.id(),
1997+
f.synthetic()
1998+
);
19901999
}
1991-
return relation.transformExpressionsUp(FieldAttribute.class, f -> {
1992-
if (f.field() instanceof InvalidMappedField imf && imf.types().stream().allMatch(DataType::isDate)) {
1993-
HashMap<ResolveUnionTypes.TypeResolutionKey, Expression> typeResolutions = new HashMap<>();
1994-
var convert = new ToDateNanos(f.source(), f);
1995-
imf.types().forEach(type -> typeResolutions(f, convert, type, imf, typeResolutions));
1996-
var resolvedField = ResolveUnionTypes.resolvedMultiTypeEsField(f, typeResolutions);
1997-
return new FieldAttribute(
1998-
f.source(),
1999-
f.parentName(),
2000-
f.qualifier(),
2001-
f.name(),
2002-
resolvedField,
2003-
f.nullable(),
2004-
f.id(),
2005-
f.synthetic()
2006-
);
2007-
}
2008-
return f;
2009-
});
2000+
return f;
20102001
});
2011-
} else {
2012-
return plan;
2013-
}
2002+
});
20142003
}
20152004
}
20162005

‎x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4141,7 +4141,6 @@ public void testBucketWithIntervalInStringInGroupingReferencedInAggregation() {
41414141
}
41424142

41434143
public void testImplicitCastingForDateAndDateNanosFields() {
4144-
assumeTrue("requires snapshot", EsqlCapabilities.Cap.IMPLICIT_CASTING_DATE_AND_DATE_NANOS.isEnabled());
41454144
IndexResolution indexWithUnionTypedFields = indexWithDateDateNanosUnionType();
41464145
Analyzer analyzer = AnalyzerTestUtils.analyzer(indexWithUnionTypedFields);
41474146

‎x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2393,7 +2393,6 @@ public void testMatchFunctionStatisWithNonPushableCondition() {
23932393
}
23942394

23952395
public void testToDateNanosPushDown() {
2396-
assumeTrue("requires snapshot", EsqlCapabilities.Cap.IMPLICIT_CASTING_DATE_AND_DATE_NANOS.isEnabled());
23972396
IndexResolution indexWithUnionTypedFields = indexWithDateDateNanosUnionType();
23982397
plannerOptimizerDateDateNanosUnionTypes = new TestPlannerOptimizer(EsqlTestUtils.TEST_CFG, makeAnalyzer(indexWithUnionTypedFields));
23992398
var stats = EsqlTestUtils.statsForExistingField("date_and_date_nanos", "date_and_date_nanos_and_long");

‎x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/QueryTranslatorTests.java‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@
1111
import org.elasticsearch.license.XPackLicenseState;
1212
import org.elasticsearch.test.ESTestCase;
1313
import org.elasticsearch.xpack.esql.EsqlTestUtils;
14-
import org.elasticsearch.xpack.esql.action.EsqlCapabilities;
1514
import org.elasticsearch.xpack.esql.analysis.Analyzer;
1615
import org.elasticsearch.xpack.esql.analysis.AnalyzerContext;
1716
import org.elasticsearch.xpack.esql.analysis.Verifier;
@@ -326,7 +325,6 @@ OR CIDR_MATCH(ip0, "fe80::cae2:65ff:fece:feb9") OR host == "beta\"""", matchesRe
326325
}
327326

328327
public void testToDateNanos() {
329-
assumeTrue("requires snapshot", EsqlCapabilities.Cap.IMPLICIT_CASTING_DATE_AND_DATE_NANOS.isEnabled());
330328
IndexResolution indexWithUnionTypedFields = indexWithDateDateNanosUnionType();
331329
plannerOptimizerDateDateNanosUnionTypes = new TestPlannerOptimizer(EsqlTestUtils.TEST_CFG, makeAnalyzer(indexWithUnionTypedFields));
332330
var stats = EsqlTestUtils.statsForExistingField("date_and_date_nanos", "date_and_date_nanos_and_long");

‎x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/160_union_types.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -913,7 +913,7 @@ suggested_type:
913913
- method: POST
914914
path: /_query
915915
parameters: []
916-
capabilities: [suggested_cast, implicit_casting_date_and_date_nanos]
916+
capabilities: [suggested_cast, implicit_casting_date_and_date_nanos, aggregate_metric_double_rendering]
917917
reason: "date and date_nanos should no longer produce suggested_cast column"
918918

919919
- do:

0 commit comments

Comments
 (0)