Skip to content

Commit e28965e

Browse files
Merge branch 'main' into remove-unused-threadpooltypes
2 parents 6008f78 + c699af2 commit e28965e

File tree

143 files changed

+2929
-1730
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

143 files changed

+2929
-1730
lines changed

docs/changelog/115091.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
pr: 115091
2+
summary: Added stricter range type checks and runtime warnings for ENRICH
3+
area: ES|QL
4+
type: bug
5+
issues:
6+
- 107357
7+
- 116799

docs/changelog/116944.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
pr: 116944
2+
summary: "Remove support for type, fields, `copy_to` and boost in metadata field definition"
3+
area: Mapping
4+
type: breaking
5+
issues: []
6+
breaking:
7+
title: "Remove support for type, fields, copy_to and boost in metadata field definition"
8+
area: Mapping
9+
details: The type, fields, copy_to and boost parameters are no longer supported in metadata field definition
10+
impact: Users providing type, fields, copy_to or boost as part of metadata field definition should remove them from their mappings.
11+
notable: false

docs/changelog/90529.yaml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
pr: 90529
2+
summary: Output a consistent format when generating error json
3+
area: Infra/REST API
4+
type: "breaking"
5+
issues:
6+
- 89387
7+
breaking:
8+
title: Error JSON structure has changed when detailed errors are disabled
9+
area: REST API
10+
details: |-
11+
This change modifies the JSON format of error messages returned to REST clients
12+
when detailed messages are turned off.
13+
Previously, JSON returned when an exception occurred, and `http.detailed_errors.enabled: false` was set,
14+
just consisted of a single `"error"` text field with some basic information.
15+
Setting `http.detailed_errors.enabled: true` (the default) changed this field
16+
to an object with more detailed information.
17+
With this change, non-detailed errors now have the same structure as detailed errors. `"error"` will now always
18+
be an object with, at a minimum, a `"type"` and `"reason"` field. Additional fields are included when detailed
19+
errors are enabled.
20+
To use the previous structure for non-detailed errors, use the v8 REST API.
21+
impact: |-
22+
If you have set `http.detailed_errors.enabled: false` (the default is `true`)
23+
the structure of JSON when any exceptions occur now matches the structure when
24+
detailed errors are enabled.
25+
To use the previous structure for non-detailed errors, use the v8 REST API.
26+
notable: false

docs/reference/esql/esql-enrich-data.asciidoc

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -138,8 +138,33 @@ include::{es-ref-dir}/ingest/apis/enrich/execute-enrich-policy.asciidoc[tag=upda
138138

139139
include::../ingest/enrich.asciidoc[tag=update-enrich-policy]
140140

141-
==== Limitations
141+
==== Enrich Policy Types and Limitations
142+
The {esql} `ENRICH` command supports all three enrich policy types:
143+
144+
`geo_match`::
145+
Matches enrich data to incoming documents based on a <<query-dsl-geo-shape-query,`geo_shape` query>>.
146+
For an example, see <<geo-match-enrich-policy-type>>.
147+
148+
`match`::
149+
Matches enrich data to incoming documents based on a <<query-dsl-term-query,`term` query>>.
150+
For an example, see <<match-enrich-policy-type>>.
151+
152+
`range`::
153+
Matches a number, date, or IP address in incoming documents to a range in the
154+
enrich index based on a <<query-dsl-term-query,`term` query>>. For an example,
155+
see <<range-enrich-policy-type>>.
156+
142157
// tag::limitations[]
143-
The {esql} `ENRICH` command only supports enrich policies of type `match`.
144-
Furthermore, `ENRICH` only supports enriching on a column of type `keyword`.
158+
While all three enrich policy types are supported, there are some limitations to be aware of:
159+
160+
* The `geo_match` enrich policy type only supports the `intersects` spatial relation.
161+
* It is required that the `match_field` in the `ENRICH` command is of the correct type.
162+
For example, if the enrich policy is of type `geo_match`, the `match_field` in the `ENRICH`
163+
command must be of type `geo_point` or `geo_shape`.
164+
Likewise, a `range` enrich policy requires a `match_field` of type `integer`, `long`, `date`, or `ip`,
165+
depending on the type of the range field in the original enrich index.
166+
* However, this constraint is relaxed for `range` policies when the `match_field` is of type `KEYWORD`.
167+
In this case the field values will be parsed during query execution, row by row.
168+
If any value fails to parse, the output values for that row will be set to `null`,
169+
an appropriate warning will be produced and the query will continue to execute.
145170
// end::limitations[]

docs/reference/esql/esql-language.asciidoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ Detailed reference documentation for the {esql} language:
1414
* <<esql-enrich-data>>
1515
* <<esql-process-data-with-dissect-and-grok>>
1616
* <<esql-implicit-casting>>
17+
* <<esql-time-spans>>
1718

1819
include::esql-syntax.asciidoc[]
1920
include::esql-commands.asciidoc[]
@@ -23,3 +24,4 @@ include::multivalued-fields.asciidoc[]
2324
include::esql-process-data-with-dissect-grok.asciidoc[]
2425
include::esql-enrich-data.asciidoc[]
2526
include::implicit-casting.asciidoc[]
27+
include::time-spans.asciidoc[]

docs/reference/esql/esql-syntax.asciidoc

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -157,21 +157,15 @@ FROM employees
157157
==== Timespan literals
158158

159159
Datetime intervals and timespans can be expressed using timespan literals.
160-
Timespan literals are a combination of a number and a qualifier. These
161-
qualifiers are supported:
162-
163-
* `millisecond`/`milliseconds`/`ms`
164-
* `second`/`seconds`/`sec`/`s`
165-
* `minute`/`minutes`/`min`
166-
* `hour`/`hours`/`h`
167-
* `day`/`days`/`d`
168-
* `week`/`weeks`/`w`
169-
* `month`/`months`/`mo`
170-
* `quarter`/`quarters`/`q`
171-
* `year`/`years`/`yr`/`y`
160+
Timespan literals are a combination of a number and a temporal unit. The
161+
supported temporal units are listed in <<esql-time-spans-table, time span unit>>.
162+
More examples of the usages of time spans can be found in
163+
<<esql-time-spans, Use time spans in ES|QL>>.
164+
172165

173166
Timespan literals are not whitespace sensitive. These expressions are all valid:
174167

175168
* `1day`
176169
* `1 day`
177170
* `1 day`
171+

docs/reference/esql/functions/binary.asciidoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ Supported types:
8787

8888
include::types/greater_than_or_equal.asciidoc[]
8989

90+
[[esql-add]]
9091
==== Add `+`
9192
[.text-center]
9293
image::esql/functions/signature/add.svg[Embedded,opts=inline]
@@ -98,6 +99,7 @@ Supported types:
9899

99100
include::types/add.asciidoc[]
100101

102+
[[esql-subtract]]
101103
==== Subtract `-`
102104
[.text-center]
103105
image::esql/functions/signature/sub.svg[Embedded,opts=inline]

docs/reference/esql/implicit-casting.asciidoc

Lines changed: 22 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
<titleabbrev>Implicit casting</titleabbrev>
66
++++
77

8-
Often users will input `date`, `ip`, `version`, `date_period` or `time_duration` as simple strings in their queries for use in predicates, functions, or expressions. {esql} provides <<esql-type-conversion-functions, type conversion functions>> to explicitly convert these strings into the desired data types.
8+
Often users will input `date`, `date_period`, `time_duration`, `ip` or `version` as simple strings in their queries for use in predicates, functions, or expressions. {esql} provides <<esql-type-conversion-functions, type conversion functions>> to explicitly convert these strings into the desired data types.
99

1010
Without implicit casting users must explicitly code these `to_X` functions in their queries, when string literals don't match the target data types they are assigned or compared to. Here is an example of using `to_datetime` to explicitly perform a data type conversion.
1111

@@ -18,7 +18,10 @@ FROM employees
1818
| LIMIT 1
1919
----
2020

21-
Implicit casting improves usability, by automatically converting string literals to the target data type. This is most useful when the target data type is `date`, `ip`, `version`, `date_period` or `time_duration`. It is natural to specify these as a string in queries.
21+
[discrete]
22+
[[esql-implicit-casting-example]]
23+
==== Implicit casting example
24+
Implicit casting automatically converts string literals to the target data type. This allows users to specify string values for types like `date`, `date_period`, `time_duration`, `ip` and `version` in their queries.
2225

2326
The first query can be coded without calling the `to_datetime` function, as follows:
2427

@@ -31,35 +34,36 @@ FROM employees
3134
| LIMIT 1
3235
----
3336

34-
[float]
35-
=== Implicit casting support
37+
[discrete]
38+
[[esql-implicit-casting-supported-operations]]
39+
==== Operations that support implicit casting
3640

3741
The following table details which {esql} operations support implicit casting for different data types.
3842

3943
[%header.monospaced.styled,format=dsv,separator=|]
4044
|===
41-
||ScalarFunction*|Operator*|<<esql-group-functions, GroupingFunction>>|<<esql-agg-functions, AggregateFunction>>
42-
|DATE|Y|Y|Y|N
43-
|IP|Y|Y|Y|N
44-
|VERSION|Y|Y|Y|N
45-
|BOOLEAN|Y|Y|Y|N
46-
|DATE_PERIOD/TIME_DURATION|Y|N|Y|N
45+
|ScalarFunctions|Operators|<<esql-group-functions, GroupingFunctions>>|<<esql-agg-functions, AggregateFunctions>>
46+
DATE|Y|Y|Y|N
47+
DATE_PERIOD/TIME_DURATION|Y|N|Y|N
48+
IP|Y|Y|Y|N
49+
VERSION|Y|Y|Y|N
50+
BOOLEAN|Y|Y|Y|N
4751
|===
4852

49-
ScalarFunction* includes:
53+
ScalarFunctions includes:
5054

51-
<<esql-conditional-functions-and-expressions, Conditional Functions and Expressions>>
55+
* <<esql-conditional-functions-and-expressions, Conditional Functions and Expressions>>
5256

53-
<<esql-date-time-functions, Date and Time Functions>>
57+
* <<esql-date-time-functions, Date and Time Functions>>
5458

55-
<<esql-ip-functions, IP Functions>>
59+
* <<esql-ip-functions, IP Functions>>
5660

5761

58-
Operator* includes:
62+
Operators includes:
5963

60-
<<esql-binary-operators, Binary Operators>>
64+
* <<esql-binary-operators, Binary Operators>>
6165

62-
<<esql-unary-operators, Unary Operator>>
66+
* <<esql-unary-operators, Unary Operator>>
6367

64-
<<esql-in-operator, IN>>
68+
* <<esql-in-operator, IN>>
6569

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
[[esql-time-spans]]
2+
=== {esql} time spans
3+
4+
++++
5+
<titleabbrev>Time spans</titleabbrev>
6+
++++
7+
8+
Time spans represent intervals between two datetime values. There are currently two supported types of time spans:
9+
10+
* `DATE_PERIOD` specifies intervals in years, quarters, months, weeks and days
11+
* `TIME_DURATION` specifies intervals in hours, minutes, seconds and milliseconds
12+
13+
A time span requires two elements: an integer value and a temporal unit.
14+
15+
Time spans work with grouping functions such as <<esql-bucket, BUCKET>>, scalar functions such as <<esql-date_trunc, DATE_TRUNC>> and arithmetic operators such as <<esql-add, `+`>> and <<esql-subtract, `-`>>. Convert strings to time spans using <<esql-to_dateperiod, TO_DATEPERIOD>>, <<esql-to_timeduration, TO_TIMEDURATION>>, or the cast operators `::DATE_PERIOD`, `::TIME_DURATION`.
16+
17+
[discrete]
18+
[[esql-time-spans-examples]]
19+
==== Examples of using time spans in {esql}
20+
21+
22+
With `BUCKET`:
23+
[source.merge.styled,esql]
24+
----
25+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketWeeklyHistogramWithSpan]
26+
----
27+
[%header.monospaced.styled,format=dsv,separator=|]
28+
|===
29+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketWeeklyHistogramWithSpan-result]
30+
|===
31+
32+
33+
With `DATE_TRUNC`:
34+
[source.merge.styled,esql]
35+
----
36+
include::{esql-specs}/date.csv-spec[tag=docsDateTrunc]
37+
----
38+
[%header.monospaced.styled,format=dsv,separator=|]
39+
|===
40+
include::{esql-specs}/date.csv-spec[tag=docsDateTrunc-result]
41+
|===
42+
43+
44+
With `+` and/or `-`:
45+
[source.merge.styled,esql]
46+
----
47+
include::{esql-specs}/date.csv-spec[tag=docsNowWhere]
48+
----
49+
[%header.monospaced.styled,format=dsv,separator=|]
50+
|===
51+
include::{esql-specs}/date.csv-spec[tag=docsNowWhere-result]
52+
|===
53+
54+
55+
When a time span is provided as a named parameter in string format, `TO_DATEPERIOD`, `::DATE_PERIOD`, `TO_TIMEDURATION` or `::TIME_DURATION` can be used to convert to its corresponding time span value for arithmetic operations like `+` and/or `-`.
56+
[source, esql]
57+
----
58+
POST /_query
59+
{
60+
"query": """
61+
FROM employees
62+
| EVAL x = hire_date + ?timespan::DATE_PERIOD, y = hire_date - TO_DATEPERIOD(?timespan)
63+
""",
64+
"params": [{"timespan" : "1 day"}]
65+
}
66+
----
67+
68+
When a time span is provided as a named parameter in string format, it can be automatically converted to its corresponding time span value in grouping functions and scalar functions, like `BUCKET` and `DATE_TRUNC`.
69+
[source, esql]
70+
----
71+
POST /_query
72+
{
73+
"query": """
74+
FROM employees
75+
| WHERE hire_date >= "1985-01-01T00:00:00Z" AND hire_date < "1986-01-01T00:00:00Z"
76+
| STATS hires_per_week = COUNT(*) BY week = BUCKET(hire_date, ?timespan)
77+
| SORT week
78+
""",
79+
"params": [{"timespan" : "1 week"}]
80+
}
81+
----
82+
83+
[source, esql]
84+
----
85+
POST /_query
86+
{
87+
"query": """
88+
FROM employees
89+
| KEEP first_name, last_name, hire_date
90+
| EVAL year_hired = DATE_TRUNC(?timespan, hire_date)
91+
""",
92+
"params": [{"timespan" : "1 year"}]
93+
}
94+
----
95+
96+
[discrete]
97+
[[esql-time-spans-table]]
98+
==== Supported temporal units
99+
[%header.monospaced.styled,format=dsv,separator=|]
100+
|===
101+
Temporal Units|Valid Abbreviations
102+
year|y, yr, years
103+
quarter|q, quarters
104+
month|mo, months
105+
week|w, weeks
106+
day|d, days
107+
hour|h, hours
108+
minute|min, minutes
109+
second|s, sec, seconds
110+
millisecond|ms, milliseconds
111+
|===

docs/reference/how-to.asciidoc

Lines changed: 12 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,21 @@
11
[[how-to]]
2-
= How to
2+
= Optimizations
33

4-
[partintro]
5-
--
6-
Elasticsearch ships with defaults which are intended to give a good out of
7-
the box experience. Full text search, highlighting, aggregations, and indexing
8-
should all just work without the user having to change anything.
4+
Elasticsearch's default settings provide a good out-of-box experience for basic operations like full text search, highlighting, aggregations, and indexing.
95

10-
Once you better understand how you want to use Elasticsearch, however,
11-
there are a number of optimizations you can make to improve performance
12-
for your use case.
6+
However, there are a number of optimizations you can make to improve performance for your use case.
137

14-
This section provides guidance about which changes should and shouldn't be
15-
made.
16-
--
8+
This section provides recommendations for various use cases.
179

18-
include::how-to/general.asciidoc[]
10+
* <<general-recommendations>>
11+
* <<tune-for-indexing-speed>>
12+
* <<tune-for-search-speed>>
13+
* <<tune-knn-search>>
14+
* <<tune-for-disk-usage>>
15+
* <<size-your-shards>>
16+
* <<use-elasticsearch-for-time-series-data>>
1917

20-
include::how-to/recipes.asciidoc[]
18+
include::how-to/general.asciidoc[]
2119

2220
include::how-to/indexing-speed.asciidoc[]
2321

0 commit comments

Comments
 (0)