Skip to content

Commit 0727669

Browse files
martijnvgmartijnvg
committed
Merge remote-tracking branch 'es/main' into runtime_fields_synthetic_source
2 parents e4f158a + 2bc5498 commit 0727669

File tree

166 files changed

+3368
-953
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

166 files changed

+3368
-953
lines changed

‎TESTING.asciidoc‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -681,12 +681,15 @@ There are multiple base classes for tests:
681681
directly by unit tests.
682682
* **`ESSingleNodeTestCase`**: This test case sets up a cluster that has a
683683
single node.
684-
* **`ESIntegTestCase`**: An integration test case that creates a cluster that
685-
might have multiple nodes.
686-
* **`ESRestTestCase`**: An integration tests that interacts with an external
687-
cluster via the REST API. This is used for Java based REST tests.
688-
* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run
689-
YAML based REST tests.
684+
* **`ESIntegTestCase`**: An internal integration test that starts nodes within the same JVM as the test.
685+
These allow you to test functionality that is not exposed via the REST API, or for verifying a certain internal state.
686+
Additionally, you can easily simulate tricky distributed setups that are difficult to do in REST tests.
687+
If you only need to start one node, use `ESSingleNodeTestCase` instead, which is a much lighter test setup.
688+
* **`ESRestTestCase`**: An integration test that interacts with an external
689+
cluster via the REST API. This is used for Java based REST tests. This should
690+
be the first choice for writing integration tests as these tests run in a much more
691+
realistic setup.
692+
* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run YAML based REST tests.
690693

691694
=== Good practices
692695

‎docs/changelog/133166.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 133166
2+
summary: Improve Expanding Lookup Join performance by pushing a filter to the right
3+
side of the lookup join
4+
area: ES|QL
5+
type: enhancement
6+
issues: []

‎docs/changelog/133369.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 133369
2+
summary: Enable `date` `date_nanos` implicit casting
3+
area: ES|QL
4+
type: enhancement
5+
issues: []

‎docs/reference/query-languages/esql/esql-multi-index.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,62 @@ FROM events_*
135135
| 2023-10-23T12:27:28.948Z | 172.21.2.113 | 2764889 | Connected to 10.1.0.2 |
136136
| 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233 | Connected to 10.1.0.3 |
137137

138+
### Date and date_nanos union type [esql-multi-index-date-date-nanos-union]
139+
```{applies_to}
140+
stack: ga 9.2.0
141+
```
142+
When the type of an {{esql}} field is a *union* of `date` and `date_nanos` across different indices, {{esql}} automatically casts all values to the `date_nanos` type during query execution. This implicit casting ensures that all values are handled with nanosecond precision, regardless of their original type. As a result, users can write queries against such fields without needing to perform explicit type conversions, and the query engine will seamlessly align the types for consistent and precise results.
143+
144+
`date_nanos` fields offer higher precision but have a narrower range of valid values compared to `date` fields. This limits their representable dates roughly from 1970 to 2262. This is because dates are stored as a `long` representing nanoseconds since the epoch. When a field is mapped as both `date` and `date_nanos` across different indices, {{esql}} defaults to the more precise `date_nanos` type. This behavior ensures that no precision is lost when querying multiple indices with differing date field types. For dates that fall outside the valid range of `date_nanos` in fields that are mapped to both `date` and `date_nanos` across different indices, {{esql}} returns null by default. However, users can explicitly cast these fields to the `date` type to obtain a valid value, with precision limited to milliseconds.
145+
146+
For example, if the `@timestamp` field is mapped as `date` in one index and `date_nanos` in another, {{esql}} will automatically treat all `@timestamp` values as `date_nanos` during query execution. This allows users to write queries that utilize the `@timestamp` field without encountering type mismatch errors, ensuring accurate time-based operations and comparisons across the combined dataset.
147+
148+
**index: events_date**
149+
150+
```
151+
{
152+
"mappings": {
153+
"properties": {
154+
"@timestamp": { "type": "date" },
155+
"client_ip": { "type": "ip" },
156+
"event_duration": { "type": "long" },
157+
"message": { "type": "keyword" }
158+
}
159+
}
160+
}
161+
```
162+
163+
**index: events_date_nanos**
164+
165+
```
166+
{
167+
"mappings": {
168+
"properties": {
169+
"@timestamp": { "type": "date_nanos" },
170+
"client_ip": { "type": "ip" },
171+
"event_duration": { "type": "long" },
172+
"message": { "type": "keyword" }
173+
}
174+
}
175+
}
176+
```
177+
178+
```esql
179+
FROM events_date*
180+
| EVAL date = @timestamp::date
181+
| KEEP @timestamp, date, client_ip, event_duration, message
182+
| SORT date
183+
```
184+
185+
| @timestamp:date_nanos | date:date | client_ip:ip | event_duration:long | message:keyword |
186+
|--------------------------| --- |--------------|---------| --- |
187+
| null |1969-10-23T13:33:34.937Z| 172.21.0.5 | 1232382 |Disconnected|
188+
| 2023-10-23T12:15:03.360Z |2023-10-23T12:15:03.360Z| 172.21.2.162 | 3450233 |Connected to 10.1.0.3|
189+
| 2023-10-23T12:15:03.360103847Z|2023-10-23T12:15:03.360Z| 172.22.2.162 | 3450233 |Connected to 10.1.0.3|
190+
| 2023-10-23T12:27:28.948Z |2023-10-23T12:27:28.948Z| 172.22.2.113 | 2764889 |Connected to 10.1.0.2|
191+
| 2023-10-23T12:27:28.948Z |2023-10-23T12:27:28.948Z| 172.21.2.113 | 2764889 |Connected to 10.1.0.2|
192+
| 2023-10-23T13:33:34.937193Z |2023-10-23T13:33:34.937Z| 172.22.0.5 | 1232382 |Disconnected|
193+
| null |2263-10-23T13:51:54.732Z| 172.21.3.15 | 725448 |Connection error|
138194

139195
## Index metadata [esql-multi-index-index-metadata]
140196

‎libs/core/src/main/java/org/elasticsearch/core/TimeValue.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -340,7 +340,7 @@ private static String formatDecimal(double value, int fractionPieces) {
340340
}
341341

342342
public String getStringRep() {
343-
if (duration < 0) {
343+
if (duration < 0 && TimeUnit.MILLISECONDS == timeUnit) {
344344
return Long.toString(duration);
345345
}
346346
return switch (timeUnit) {

‎libs/core/src/test/java/org/elasticsearch/common/unit/TimeValueTests.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ public void testRoundTrip() {
109109
assertThat(TimeValue.parseTimeValue(s, null, "test").getStringRep(), equalTo(s));
110110
final TimeValue t = new TimeValue(randomIntBetween(1, 128), randomFrom(TimeUnit.values()));
111111
assertThat(TimeValue.parseTimeValue(t.getStringRep(), null, "test"), equalTo(t));
112+
assertThat(TimeValue.timeValueSeconds(-1), equalTo(TimeValue.parseTimeValue(TimeValue.timeValueSeconds(-1).getStringRep(), "foo")));
112113
}
113114

114115
private static final String FRACTIONAL_TIME_VALUES_ARE_NOT_SUPPORTED = "fractional time values are not supported";

‎modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamIT.java‎

Lines changed: 27 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -2429,24 +2429,20 @@ public void testGetEffectiveMappings() throws Exception {
24292429
{
24302430
ComponentTemplate ct1 = new ComponentTemplate(new Template(null, new CompressedXContent("""
24312431
{
2432-
"_doc":{
2433-
"dynamic":"strict",
2434-
"properties":{
2435-
"field1":{
2436-
"type":"text"
2437-
}
2432+
"dynamic":"strict",
2433+
"properties":{
2434+
"field1":{
2435+
"type":"text"
24382436
}
24392437
}
24402438
}
24412439
"""), null), 3L, null);
24422440
ComponentTemplate ct2 = new ComponentTemplate(new Template(null, new CompressedXContent("""
24432441
{
2444-
"_doc":{
2445-
"dynamic":"strict",
2446-
"properties":{
2447-
"field2":{
2448-
"type":"text"
2449-
}
2442+
"dynamic":"strict",
2443+
"properties":{
2444+
"field2":{
2445+
"type":"text"
24502446
}
24512447
}
24522448
}
@@ -2464,12 +2460,10 @@ public void testGetEffectiveMappings() throws Exception {
24642460
.indexPatterns(List.of("effective-*"))
24652461
.template(Template.builder().mappings(CompressedXContent.fromJSON("""
24662462
{
2467-
"_doc":{
2468-
"dynamic":"strict",
2469-
"properties":{
2470-
"field3":{
2471-
"type":"text"
2472-
}
2463+
"dynamic":"strict",
2464+
"properties":{
2465+
"field3":{
2466+
"type":"text"
24732467
}
24742468
}
24752469
}
@@ -2530,25 +2524,22 @@ public void testGetEffectiveMappings() throws Exception {
25302524
Map<String, Object> effectiveMappingMap = XContentHelper.convertToMap(effectiveMappings.uncompressed(), true, XContentType.JSON)
25312525
.v2();
25322526
Map<String, Object> expectedEffectiveMappingMap = Map.of(
2533-
"_doc",
2527+
"dynamic",
2528+
"strict",
2529+
"_data_stream_timestamp",
2530+
Map.of("enabled", true),
2531+
"properties",
25342532
Map.of(
2535-
"dynamic",
2536-
"strict",
2537-
"_data_stream_timestamp",
2538-
Map.of("enabled", true),
2539-
"properties",
2540-
Map.of(
2541-
"@timestamp",
2542-
Map.of("type", "date"),
2543-
"field1",
2544-
Map.of("type", "keyword"),
2545-
"field2",
2546-
Map.of("type", "text"),
2547-
"field3",
2548-
Map.of("type", "text"),
2549-
"field4",
2550-
Map.of("type", "keyword")
2551-
)
2533+
"@timestamp",
2534+
Map.of("type", "date"),
2535+
"field1",
2536+
Map.of("type", "keyword"),
2537+
"field2",
2538+
Map.of("type", "text"),
2539+
"field3",
2540+
Map.of("type", "text"),
2541+
"field4",
2542+
Map.of("type", "keyword")
25522543
)
25532544
);
25542545
assertThat(effectiveMappingMap, equalTo(expectedEffectiveMappingMap));

‎modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TransportUpdateDataStreamMappingsActionIT.java‎

Lines changed: 26 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -55,38 +55,32 @@ public void testGetAndUpdateMappings() throws IOException {
5555
createDataStream(dataStreamName);
5656

5757
Map<String, Object> originalMappings = Map.of(
58-
"_doc",
59-
Map.of(
60-
"dynamic",
61-
"strict",
62-
"_data_stream_timestamp",
63-
Map.of("enabled", true),
64-
"properties",
65-
Map.of("@timestamp", Map.of("type", "date"), "foo1", Map.of("type", "text"), "foo2", Map.of("type", "text"))
66-
)
58+
"dynamic",
59+
"strict",
60+
"_data_stream_timestamp",
61+
Map.of("enabled", true),
62+
"properties",
63+
Map.of("@timestamp", Map.of("type", "date"), "foo1", Map.of("type", "text"), "foo2", Map.of("type", "text"))
6764
);
6865
Map<String, Object> mappingOverrides = Map.of(
6966
"properties",
7067
Map.of("foo2", Map.of("type", "keyword"), "foo3", Map.of("type", "text"))
7168
);
7269
Map<String, Object> expectedEffectiveMappings = Map.of(
73-
"_doc",
70+
"dynamic",
71+
"strict",
72+
"_data_stream_timestamp",
73+
Map.of("enabled", true),
74+
"properties",
7475
Map.of(
75-
"dynamic",
76-
"strict",
77-
"_data_stream_timestamp",
78-
Map.of("enabled", true),
79-
"properties",
80-
Map.of(
81-
"@timestamp",
82-
Map.of("type", "date"),
83-
"foo1",
84-
Map.of("type", "text"),
85-
"foo2",
86-
Map.of("type", "keyword"),
87-
"foo3",
88-
Map.of("type", "text")
89-
)
76+
"@timestamp",
77+
Map.of("type", "date"),
78+
"foo1",
79+
Map.of("type", "text"),
80+
"foo2",
81+
Map.of("type", "keyword"),
82+
"foo3",
83+
Map.of("type", "text")
9084
)
9185
);
9286
assertExpectedMappings(dataStreamName, Map.of(), originalMappings);
@@ -137,15 +131,13 @@ public void testGetAndUpdateMappings() throws IOException {
137131
private void createDataStream(String dataStreamName) throws IOException {
138132
String mappingString = """
139133
{
140-
"_doc":{
141-
"dynamic":"strict",
142-
"properties":{
143-
"foo1":{
144-
"type":"text"
145-
},
146-
"foo2":{
147-
"type":"text"
148-
}
134+
"dynamic":"strict",
135+
"properties":{
136+
"foo1":{
137+
"type":"text"
138+
},
139+
"foo2":{
140+
"type":"text"
149141
}
150142
}
151143
}

‎modules/data-streams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/250_data_stream_mappings.yml‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ setup:
3434
name: my-data-stream-1
3535
- match: { data_streams.0.name: my-data-stream-1 }
3636
- match: { data_streams.0.mappings: {} }
37-
- length: { data_streams.0.effective_mappings._doc.properties: 2 }
37+
- length: { data_streams.0.effective_mappings.properties: 2 }
3838

3939
- do:
4040
indices.get_data_stream:
@@ -56,7 +56,7 @@ setup:
5656
- match: { data_streams.0.name: my-data-stream-1 }
5757
- match: { data_streams.0.applied_to_data_stream: true }
5858
- match: { data_streams.0.mappings.properties.name.type: "keyword" }
59-
- match: { data_streams.0.effective_mappings._doc.properties.name.type: "keyword" }
59+
- match: { data_streams.0.effective_mappings.properties.name.type: "keyword" }
6060

6161
- do:
6262
indices.rollover:
@@ -71,9 +71,9 @@ setup:
7171
indices.get_data_stream_mappings:
7272
name: my-data-stream-1
7373
- match: { data_streams.0.name: my-data-stream-1 }
74-
- length: { data_streams.0.effective_mappings._doc.properties: 3 }
74+
- length: { data_streams.0.effective_mappings.properties: 3 }
7575
- match: { data_streams.0.mappings.properties.name.type: "keyword" }
76-
- match: { data_streams.0.effective_mappings._doc.properties.name.type: "keyword" }
76+
- match: { data_streams.0.effective_mappings.properties.name.type: "keyword" }
7777

7878
- do:
7979
indices.get_data_stream:
@@ -149,9 +149,9 @@ setup:
149149
- match: { data_streams.0.mappings.properties.field1.type: "text" }
150150
- match: { data_streams.0.mappings.properties.field2: null }
151151
- match: { data_streams.0.mappings.properties.field3.type: "text" }
152-
- match: { data_streams.0.effective_mappings._doc.properties.field1.type: "text" }
153-
- match: { data_streams.0.effective_mappings._doc.properties.field2.type: "keyword" }
154-
- match: { data_streams.0.effective_mappings._doc.properties.field3.type: "text" }
152+
- match: { data_streams.0.effective_mappings.properties.field1.type: "text" }
153+
- match: { data_streams.0.effective_mappings.properties.field2.type: "keyword" }
154+
- match: { data_streams.0.effective_mappings.properties.field3.type: "text" }
155155

156156
- do:
157157
cluster.put_component_template:
@@ -180,10 +180,10 @@ setup:
180180
indices.get_data_stream_mappings:
181181
name: my-component-only-data-stream-1
182182
- match: { data_streams.0.name: my-component-only-data-stream-1 }
183-
- length: { data_streams.0.effective_mappings._doc.properties: 5 }
183+
- length: { data_streams.0.effective_mappings.properties: 5 }
184184
- match: { data_streams.0.mappings.properties.field1.type: "text" }
185-
- match: { data_streams.0.effective_mappings._doc.properties.field3.type: "text" }
186-
- match: { data_streams.0.effective_mappings._doc.properties.field4.type: "keyword" }
185+
- match: { data_streams.0.effective_mappings.properties.field3.type: "text" }
186+
- match: { data_streams.0.effective_mappings.properties.field4.type: "keyword" }
187187

188188
- do:
189189
indices.get_data_stream:
@@ -212,7 +212,7 @@ setup:
212212
- match: { data_streams.0.name: my-component-only-data-stream-1 }
213213
- match: { data_streams.0.applied_to_data_stream: true }
214214
- match: { data_streams.0.mappings null }
215-
- match: { data_streams.0.effective_mappings._doc.properties.field1.type: "keyword" }
216-
- match: { data_streams.0.effective_mappings._doc.properties.field2.type: "keyword" }
217-
- match: { data_streams.0.effective_mappings._doc.properties.field3: null }
218-
- match: { data_streams.0.effective_mappings._doc.properties.field4.type: "keyword" }
215+
- match: { data_streams.0.effective_mappings.properties.field1.type: "keyword" }
216+
- match: { data_streams.0.effective_mappings.properties.field2.type: "keyword" }
217+
- match: { data_streams.0.effective_mappings.properties.field3: null }
218+
- match: { data_streams.0.effective_mappings.properties.field4.type: "keyword" }

0 commit comments

Comments
 (0)