Skip to content

Commit 3716b03

Browse files
jdconradjdconrad
committed
Merge branch 'main' into mtv4
2 parents 9475f29 + 3718184 commit 3716b03

File tree

142 files changed

+3061
-907
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

142 files changed

+3061
-907
lines changed

‎docs/changelog/133166.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 133166
2+
summary: Improve Expanding Lookup Join performance by pushing a filter to the right
3+
side of the lookup join
4+
area: ES|QL
5+
type: enhancement
6+
issues: []

‎docs/changelog/133369.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 133369
2+
summary: Enable `date` `date_nanos` implicit casting
3+
area: ES|QL
4+
type: enhancement
5+
issues: []

‎docs/reference/query-languages/esql/esql-multi-index.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,62 @@ FROM events_*
135135
| 2023-10-23T12:27:28.948Z | 172.21.2.113 | 2764889 | Connected to 10.1.0.2 |
136136
| 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233 | Connected to 10.1.0.3 |
137137

138+
### Date and date_nanos union type [esql-multi-index-date-date-nanos-union]
139+
```{applies_to}
140+
stack: ga 9.2.0
141+
```
142+
When the type of an {{esql}} field is a *union* of `date` and `date_nanos` across different indices, {{esql}} automatically casts all values to the `date_nanos` type during query execution. This implicit casting ensures that all values are handled with nanosecond precision, regardless of their original type. As a result, users can write queries against such fields without needing to perform explicit type conversions, and the query engine will seamlessly align the types for consistent and precise results.
143+
144+
`date_nanos` fields offer higher precision but have a narrower range of valid values compared to `date` fields. This limits their representable dates roughly from 1970 to 2262. This is because dates are stored as a `long` representing nanoseconds since the epoch. When a field is mapped as both `date` and `date_nanos` across different indices, {{esql}} defaults to the more precise `date_nanos` type. This behavior ensures that no precision is lost when querying multiple indices with differing date field types. For dates that fall outside the valid range of `date_nanos` in fields that are mapped to both `date` and `date_nanos` across different indices, {{esql}} returns null by default. However, users can explicitly cast these fields to the `date` type to obtain a valid value, with precision limited to milliseconds.
145+
146+
For example, if the `@timestamp` field is mapped as `date` in one index and `date_nanos` in another, {{esql}} will automatically treat all `@timestamp` values as `date_nanos` during query execution. This allows users to write queries that utilize the `@timestamp` field without encountering type mismatch errors, ensuring accurate time-based operations and comparisons across the combined dataset.
147+
148+
**index: events_date**
149+
150+
```
151+
{
152+
"mappings": {
153+
"properties": {
154+
"@timestamp": { "type": "date" },
155+
"client_ip": { "type": "ip" },
156+
"event_duration": { "type": "long" },
157+
"message": { "type": "keyword" }
158+
}
159+
}
160+
}
161+
```
162+
163+
**index: events_date_nanos**
164+
165+
```
166+
{
167+
"mappings": {
168+
"properties": {
169+
"@timestamp": { "type": "date_nanos" },
170+
"client_ip": { "type": "ip" },
171+
"event_duration": { "type": "long" },
172+
"message": { "type": "keyword" }
173+
}
174+
}
175+
}
176+
```
177+
178+
```esql
179+
FROM events_date*
180+
| EVAL date = @timestamp::date
181+
| KEEP @timestamp, date, client_ip, event_duration, message
182+
| SORT date
183+
```
184+
185+
| @timestamp:date_nanos | date:date | client_ip:ip | event_duration:long | message:keyword |
186+
|--------------------------| --- |--------------|---------| --- |
187+
| null |1969-10-23T13:33:34.937Z| 172.21.0.5 | 1232382 |Disconnected|
188+
| 2023-10-23T12:15:03.360Z |2023-10-23T12:15:03.360Z| 172.21.2.162 | 3450233 |Connected to 10.1.0.3|
189+
| 2023-10-23T12:15:03.360103847Z|2023-10-23T12:15:03.360Z| 172.22.2.162 | 3450233 |Connected to 10.1.0.3|
190+
| 2023-10-23T12:27:28.948Z |2023-10-23T12:27:28.948Z| 172.22.2.113 | 2764889 |Connected to 10.1.0.2|
191+
| 2023-10-23T12:27:28.948Z |2023-10-23T12:27:28.948Z| 172.21.2.113 | 2764889 |Connected to 10.1.0.2|
192+
| 2023-10-23T13:33:34.937193Z |2023-10-23T13:33:34.937Z| 172.22.0.5 | 1232382 |Disconnected|
193+
| null |2263-10-23T13:51:54.732Z| 172.21.3.15 | 725448 |Connection error|
138194

139195
## Index metadata [esql-multi-index-index-metadata]
140196

‎libs/core/src/main/java/org/elasticsearch/core/TimeValue.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -340,7 +340,7 @@ private static String formatDecimal(double value, int fractionPieces) {
340340
}
341341

342342
public String getStringRep() {
343-
if (duration < 0) {
343+
if (duration < 0 && TimeUnit.MILLISECONDS == timeUnit) {
344344
return Long.toString(duration);
345345
}
346346
return switch (timeUnit) {

‎libs/core/src/test/java/org/elasticsearch/common/unit/TimeValueTests.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ public void testRoundTrip() {
109109
assertThat(TimeValue.parseTimeValue(s, null, "test").getStringRep(), equalTo(s));
110110
final TimeValue t = new TimeValue(randomIntBetween(1, 128), randomFrom(TimeUnit.values()));
111111
assertThat(TimeValue.parseTimeValue(t.getStringRep(), null, "test"), equalTo(t));
112+
assertThat(TimeValue.timeValueSeconds(-1), equalTo(TimeValue.parseTimeValue(TimeValue.timeValueSeconds(-1).getStringRep(), "foo")));
112113
}
113114

114115
private static final String FRACTIONAL_TIME_VALUES_ARE_NOT_SUPPORTED = "fractional time values are not supported";

‎modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamIT.java‎

Lines changed: 27 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -2429,24 +2429,20 @@ public void testGetEffectiveMappings() throws Exception {
24292429
{
24302430
ComponentTemplate ct1 = new ComponentTemplate(new Template(null, new CompressedXContent("""
24312431
{
2432-
"_doc":{
2433-
"dynamic":"strict",
2434-
"properties":{
2435-
"field1":{
2436-
"type":"text"
2437-
}
2432+
"dynamic":"strict",
2433+
"properties":{
2434+
"field1":{
2435+
"type":"text"
24382436
}
24392437
}
24402438
}
24412439
"""), null), 3L, null);
24422440
ComponentTemplate ct2 = new ComponentTemplate(new Template(null, new CompressedXContent("""
24432441
{
2444-
"_doc":{
2445-
"dynamic":"strict",
2446-
"properties":{
2447-
"field2":{
2448-
"type":"text"
2449-
}
2442+
"dynamic":"strict",
2443+
"properties":{
2444+
"field2":{
2445+
"type":"text"
24502446
}
24512447
}
24522448
}
@@ -2464,12 +2460,10 @@ public void testGetEffectiveMappings() throws Exception {
24642460
.indexPatterns(List.of("effective-*"))
24652461
.template(Template.builder().mappings(CompressedXContent.fromJSON("""
24662462
{
2467-
"_doc":{
2468-
"dynamic":"strict",
2469-
"properties":{
2470-
"field3":{
2471-
"type":"text"
2472-
}
2463+
"dynamic":"strict",
2464+
"properties":{
2465+
"field3":{
2466+
"type":"text"
24732467
}
24742468
}
24752469
}
@@ -2530,25 +2524,22 @@ public void testGetEffectiveMappings() throws Exception {
25302524
Map<String, Object> effectiveMappingMap = XContentHelper.convertToMap(effectiveMappings.uncompressed(), true, XContentType.JSON)
25312525
.v2();
25322526
Map<String, Object> expectedEffectiveMappingMap = Map.of(
2533-
"_doc",
2527+
"dynamic",
2528+
"strict",
2529+
"_data_stream_timestamp",
2530+
Map.of("enabled", true),
2531+
"properties",
25342532
Map.of(
2535-
"dynamic",
2536-
"strict",
2537-
"_data_stream_timestamp",
2538-
Map.of("enabled", true),
2539-
"properties",
2540-
Map.of(
2541-
"@timestamp",
2542-
Map.of("type", "date"),
2543-
"field1",
2544-
Map.of("type", "keyword"),
2545-
"field2",
2546-
Map.of("type", "text"),
2547-
"field3",
2548-
Map.of("type", "text"),
2549-
"field4",
2550-
Map.of("type", "keyword")
2551-
)
2533+
"@timestamp",
2534+
Map.of("type", "date"),
2535+
"field1",
2536+
Map.of("type", "keyword"),
2537+
"field2",
2538+
Map.of("type", "text"),
2539+
"field3",
2540+
Map.of("type", "text"),
2541+
"field4",
2542+
Map.of("type", "keyword")
25522543
)
25532544
);
25542545
assertThat(effectiveMappingMap, equalTo(expectedEffectiveMappingMap));

‎modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TransportUpdateDataStreamMappingsActionIT.java‎

Lines changed: 26 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -55,38 +55,32 @@ public void testGetAndUpdateMappings() throws IOException {
5555
createDataStream(dataStreamName);
5656

5757
Map<String, Object> originalMappings = Map.of(
58-
"_doc",
59-
Map.of(
60-
"dynamic",
61-
"strict",
62-
"_data_stream_timestamp",
63-
Map.of("enabled", true),
64-
"properties",
65-
Map.of("@timestamp", Map.of("type", "date"), "foo1", Map.of("type", "text"), "foo2", Map.of("type", "text"))
66-
)
58+
"dynamic",
59+
"strict",
60+
"_data_stream_timestamp",
61+
Map.of("enabled", true),
62+
"properties",
63+
Map.of("@timestamp", Map.of("type", "date"), "foo1", Map.of("type", "text"), "foo2", Map.of("type", "text"))
6764
);
6865
Map<String, Object> mappingOverrides = Map.of(
6966
"properties",
7067
Map.of("foo2", Map.of("type", "keyword"), "foo3", Map.of("type", "text"))
7168
);
7269
Map<String, Object> expectedEffectiveMappings = Map.of(
73-
"_doc",
70+
"dynamic",
71+
"strict",
72+
"_data_stream_timestamp",
73+
Map.of("enabled", true),
74+
"properties",
7475
Map.of(
75-
"dynamic",
76-
"strict",
77-
"_data_stream_timestamp",
78-
Map.of("enabled", true),
79-
"properties",
80-
Map.of(
81-
"@timestamp",
82-
Map.of("type", "date"),
83-
"foo1",
84-
Map.of("type", "text"),
85-
"foo2",
86-
Map.of("type", "keyword"),
87-
"foo3",
88-
Map.of("type", "text")
89-
)
76+
"@timestamp",
77+
Map.of("type", "date"),
78+
"foo1",
79+
Map.of("type", "text"),
80+
"foo2",
81+
Map.of("type", "keyword"),
82+
"foo3",
83+
Map.of("type", "text")
9084
)
9185
);
9286
assertExpectedMappings(dataStreamName, Map.of(), originalMappings);
@@ -137,15 +131,13 @@ public void testGetAndUpdateMappings() throws IOException {
137131
private void createDataStream(String dataStreamName) throws IOException {
138132
String mappingString = """
139133
{
140-
"_doc":{
141-
"dynamic":"strict",
142-
"properties":{
143-
"foo1":{
144-
"type":"text"
145-
},
146-
"foo2":{
147-
"type":"text"
148-
}
134+
"dynamic":"strict",
135+
"properties":{
136+
"foo1":{
137+
"type":"text"
138+
},
139+
"foo2":{
140+
"type":"text"
149141
}
150142
}
151143
}

‎modules/data-streams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/250_data_stream_mappings.yml‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ setup:
3434
name: my-data-stream-1
3535
- match: { data_streams.0.name: my-data-stream-1 }
3636
- match: { data_streams.0.mappings: {} }
37-
- length: { data_streams.0.effective_mappings._doc.properties: 2 }
37+
- length: { data_streams.0.effective_mappings.properties: 2 }
3838

3939
- do:
4040
indices.get_data_stream:
@@ -56,7 +56,7 @@ setup:
5656
- match: { data_streams.0.name: my-data-stream-1 }
5757
- match: { data_streams.0.applied_to_data_stream: true }
5858
- match: { data_streams.0.mappings.properties.name.type: "keyword" }
59-
- match: { data_streams.0.effective_mappings._doc.properties.name.type: "keyword" }
59+
- match: { data_streams.0.effective_mappings.properties.name.type: "keyword" }
6060

6161
- do:
6262
indices.rollover:
@@ -71,9 +71,9 @@ setup:
7171
indices.get_data_stream_mappings:
7272
name: my-data-stream-1
7373
- match: { data_streams.0.name: my-data-stream-1 }
74-
- length: { data_streams.0.effective_mappings._doc.properties: 3 }
74+
- length: { data_streams.0.effective_mappings.properties: 3 }
7575
- match: { data_streams.0.mappings.properties.name.type: "keyword" }
76-
- match: { data_streams.0.effective_mappings._doc.properties.name.type: "keyword" }
76+
- match: { data_streams.0.effective_mappings.properties.name.type: "keyword" }
7777

7878
- do:
7979
indices.get_data_stream:
@@ -149,9 +149,9 @@ setup:
149149
- match: { data_streams.0.mappings.properties.field1.type: "text" }
150150
- match: { data_streams.0.mappings.properties.field2: null }
151151
- match: { data_streams.0.mappings.properties.field3.type: "text" }
152-
- match: { data_streams.0.effective_mappings._doc.properties.field1.type: "text" }
153-
- match: { data_streams.0.effective_mappings._doc.properties.field2.type: "keyword" }
154-
- match: { data_streams.0.effective_mappings._doc.properties.field3.type: "text" }
152+
- match: { data_streams.0.effective_mappings.properties.field1.type: "text" }
153+
- match: { data_streams.0.effective_mappings.properties.field2.type: "keyword" }
154+
- match: { data_streams.0.effective_mappings.properties.field3.type: "text" }
155155

156156
- do:
157157
cluster.put_component_template:
@@ -180,10 +180,10 @@ setup:
180180
indices.get_data_stream_mappings:
181181
name: my-component-only-data-stream-1
182182
- match: { data_streams.0.name: my-component-only-data-stream-1 }
183-
- length: { data_streams.0.effective_mappings._doc.properties: 5 }
183+
- length: { data_streams.0.effective_mappings.properties: 5 }
184184
- match: { data_streams.0.mappings.properties.field1.type: "text" }
185-
- match: { data_streams.0.effective_mappings._doc.properties.field3.type: "text" }
186-
- match: { data_streams.0.effective_mappings._doc.properties.field4.type: "keyword" }
185+
- match: { data_streams.0.effective_mappings.properties.field3.type: "text" }
186+
- match: { data_streams.0.effective_mappings.properties.field4.type: "keyword" }
187187

188188
- do:
189189
indices.get_data_stream:
@@ -212,7 +212,7 @@ setup:
212212
- match: { data_streams.0.name: my-component-only-data-stream-1 }
213213
- match: { data_streams.0.applied_to_data_stream: true }
214214
- match: { data_streams.0.mappings null }
215-
- match: { data_streams.0.effective_mappings._doc.properties.field1.type: "keyword" }
216-
- match: { data_streams.0.effective_mappings._doc.properties.field2.type: "keyword" }
217-
- match: { data_streams.0.effective_mappings._doc.properties.field3: null }
218-
- match: { data_streams.0.effective_mappings._doc.properties.field4.type: "keyword" }
215+
- match: { data_streams.0.effective_mappings.properties.field1.type: "keyword" }
216+
- match: { data_streams.0.effective_mappings.properties.field2.type: "keyword" }
217+
- match: { data_streams.0.effective_mappings.properties.field3: null }
218+
- match: { data_streams.0.effective_mappings.properties.field4.type: "keyword" }

‎muted-tests.yml‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -495,24 +495,9 @@ tests:
495495
- class: org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryStopIT
496496
method: testStopQueryLocal
497497
issue: https://github.com/elastic/elasticsearch/issues/133481
498-
- class: org.elasticsearch.compute.data.BasicBlockTests
499-
method: testIntBlock
500-
issue: https://github.com/elastic/elasticsearch/issues/133596
501-
- class: org.elasticsearch.compute.data.BasicBlockTests
502-
method: testDoubleBlock
503-
issue: https://github.com/elastic/elasticsearch/issues/133606
504-
- class: org.elasticsearch.compute.data.BasicBlockTests
505-
method: testBooleanBlock
506-
issue: https://github.com/elastic/elasticsearch/issues/133608
507-
- class: org.elasticsearch.compute.data.BasicBlockTests
508-
method: testLongBlock
509-
issue: https://github.com/elastic/elasticsearch/issues/133618
510498
- class: org.elasticsearch.xpack.esql.inference.rerank.RerankOperatorTests
511499
method: testSimpleCircuitBreaking
512500
issue: https://github.com/elastic/elasticsearch/issues/133619
513-
- class: org.elasticsearch.compute.data.BasicBlockTests
514-
method: testFloatBlock
515-
issue: https://github.com/elastic/elasticsearch/issues/133621
516501
- class: org.elasticsearch.xpack.esql.qa.single_node.GenerativeIT
517502
method: test
518503
issue: https://github.com/elastic/elasticsearch/issues/133077
@@ -528,12 +513,6 @@ tests:
528513
- class: org.elasticsearch.xpack.ml.integration.InferenceIT
529514
method: testInferClassificationModel
530515
issue: https://github.com/elastic/elasticsearch/issues/133448
531-
- class: org.elasticsearch.xpack.logsdb.LogsIndexingIT
532-
method: testRouteOnSortFields
533-
issue: https://github.com/elastic/elasticsearch/issues/133993
534-
- class: org.elasticsearch.xpack.logsdb.LogsIndexingIT
535-
method: testShrink
536-
issue: https://github.com/elastic/elasticsearch/issues/133875
537516
- class: org.elasticsearch.xpack.esql.action.CrossClusterQueryWithPartialResultsIT
538517
method: testPartialResults
539518
issue: https://github.com/elastic/elasticsearch/issues/131481
@@ -549,10 +528,6 @@ tests:
549528
- class: org.elasticsearch.xpack.cluster.routing.allocation.DataTierAllocationDeciderIT
550529
method: testShardsAreKeptInPreferredTierUntilTheNextTierIsInItsFinalState
551530
issue: https://github.com/elastic/elasticsearch/issues/134050
552-
- class: org.elasticsearch.xpack.esql.expression.function.aggregate.ValuesTests
553-
issue: https://github.com/elastic/elasticsearch/issues/134072
554-
- class: org.elasticsearch.xpack.esql.expression.function.aggregate.SampleTests
555-
issue: https://github.com/elastic/elasticsearch/issues/134072
556531

557532
# Examples:
558533
#

‎server/src/main/java/org/elasticsearch/TransportVersions.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -357,6 +357,8 @@ static TransportVersion def(int id) {
357357
public static final TransportVersion STREAMS_ENDPOINT_PARAM_RESTRICTIONS = def(9_148_0_00);
358358
public static final TransportVersion RESOLVE_INDEX_MODE_FILTER = def(9_149_0_00);
359359
public static final TransportVersion SEMANTIC_QUERY_MULTIPLE_INFERENCE_IDS = def(9_150_0_00);
360+
public static final TransportVersion ESQL_LOOKUP_JOIN_PRE_JOIN_FILTER = def(9_151_0_00);
361+
public static final TransportVersion INFERENCE_API_DISABLE_EIS_RATE_LIMITING = def(9_152_0_00);
360362

361363
/*
362364
* STOP! READ THIS FIRST! No, really,

0 commit comments

Comments
 (0)