Skip to content

Commit c20d53c

Browse files
idegtiarenkoidegtiarenko
authored
Merge branch 'main' into retry_shard_movements_during_query
2 parents 811d8fc + 061c8bb commit c20d53c

39 files changed

+2619
-2114
lines changed

‎docs/reference/elasticsearch/rest-apis/sort-search-results.md‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,16 +36,27 @@ GET /my-index-000001/_search
3636
{
3737
"sort" : [
3838
{ "post_date" : {"order" : "asc", "format": "strict_date_optional_time_nanos"}},
39-
"user",
4039
{ "name" : "desc" },
4140
{ "age" : "desc" },
41+
"user",
4242
"_score"
4343
],
4444
"query" : {
4545
"term" : { "user" : "kimchy" }
4646
}
4747
}
4848
```
49+
Order matters when defining multiple sort fields, because {{es}} attempts to sort on the first field in the array. Each subsequent field in the array is only used if the previous fields result in a tie. If documents have identical values across all specified sort fields, {{es}} uses the document ID as the final tie-breaker.
50+
51+
Here's how the example query attempts to sort results:
52+
53+
- First by `post_date`
54+
- If `post_date` values are identical, sorts by `name`
55+
- If both `post_date` and `name` are identical, sorts by `age`
56+
- If the first three fields are identical, sorts by `user`
57+
- If all previous fields are identical, sorts by `_score`
58+
59+
By default, Elasticsearch sorts `numeric` fields in descending order and `string` fields in ascending order unless you explicitly specify otherwise. All three formats shown in the example above are valid. Some make the sort order explicit, while others rely on Elasticsearch’s default behavior.
4960

5061
::::{note}
5162
`_doc` has no real use-case besides being the most efficient sort order. So if you don’t care about the order in which documents are returned, then you should sort by `_doc`. This especially helps when [scrolling](/reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results).

‎docs/reference/query-languages/esql/README.md‎

Lines changed: 143 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,25 @@
11
The ES|QL documentation is composed of static content and generated content.
22
The static content exists in this directory and can be edited by hand.
3-
However, the sub-directories `_snippets`, `images` and `kibana` contain mostly
4-
generated content.
3+
However, the subdirectories `_snippets`, `images` and `kibana` contain a mix
4+
of static and generated content, and so updating them is a bit more involved.
5+
6+
## Static content
7+
8+
The root `esql` directory and the following two subdirectories contain static content:
9+
* `commands` - contains the static content for the ES|QL commands.
10+
This content will typically contain mostly `include` directives for content in the `_snippets` or `images` directories.
11+
* `functions-operators` - contains the static content for the ES|QL functions and operators.
12+
Again this will contain mostly `include` directives for content in the `_snippets` or `images` directories.
13+
14+
## Mixed static and generated content
15+
16+
Generated content is created by running the ESQL tests in the `x-pack/plugin/esql` module.
17+
It will be written into three subdirectories of the `esql` directory:
518

619
### _snippets
720

8-
In `_snippets` there are files that can be included within other files
9-
using the [File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
21+
In `_snippets` there are files that can be included within other files using the
22+
[File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
1023
feature of the Elastic Docs V3 system.
1124
Most, but not all, files in this directory are generated.
1225
In particular the directories `_snippets/functions/*` and `_snippets/operators/*`
@@ -21,7 +34,7 @@ contain subdirectories that are mostly generated:
2134

2235
Most functions can use the generated docs generated in the `layout` directory.
2336
If we need something more custom for the function we can make a file in this
24-
directory that can `include::` any parts of the files above.
37+
directory that can `include` any parts of the files above.
2538

2639
To regenerate the files for a function run its tests using gradle.
2740
For example to generate docs for the `CASE` function:
@@ -34,17 +47,140 @@ To regenerate the files for all functions run all of ESQL's tests using gradle:
3447
./gradlew :x-pack:plugin:esql:test
3548
```
3649

50+
#### Lists
51+
52+
The `_snippets/lists` directory contains re-usable content for lists of commands, functions or operators.
53+
Whenever adding a command, function or operator, you usually need to add it to one of these lists.
54+
The lists should also match natural groupings of the commands, functions or operators.
55+
For example, when adding an aggregation function, add to the `aggregation-functions.md` file.
56+
57+
#### Commands
58+
59+
The `_snippets/commands` directory contains the content for the ES|QL commands.
60+
There are two subdirectories, one static and one generated:
61+
* `layout` - contains the static content for the ES|QL commands.
62+
The files in this directory are the main content for the documentation for the commands.
63+
They are not generated, and so this is the primary place to edit the content, or add new commands.
64+
* `examples` - contains the generated content for the ES|QL commands.
65+
The files in this directory are generated from the test `CommandDocsTests` in the `x-pack/plugin/esql` module.
66+
The structure of the subdirectories mimics the csv-spec files and test tags used in the tests.
67+
68+
Including generated examples in the command documentation is done by using the include directive.
69+
3770
### images
3871

39-
The `images` directory contains `functions` and `operators` sub-directories with
72+
The `images` directory contains `functions` and `operators` subdirectories with
4073
the `*.svg` files used to describe the syntax of each function or operator.
4174
These are all generated by the same tests that generate the functions and operators docs above.
4275

4376
### kibana
4477

45-
The `kibana` directory contains `definition` and `docs` sub-directories that are generated:
78+
The `kibana` directory contains `definition` and `docs` subdirectories that are generated:
4679

4780
* `kibana/definition` - function definitions for kibana's ESQL editor
4881
* `kibana/docs` - the inline docs for kibana
4982

5083
These are also generated as part of the unit tests described above.
84+
85+
## Under the hood
86+
87+
There are three overlapping mechanisms for generating the content:
88+
* The `AbstractFunctionTestCase` class generates the content for all the functions and most operators.
89+
This class makes use of the `DocsV3Support` class to generate the content.
90+
It uses the `@FunctionInfo` and `@Param` annotations on function and operator classes to know what content should be generated.
91+
All tests that extend this class will automatically generate the content for the functions they test.
92+
* Some operators do not have a clear class or test class, and so the content is generated by custom
93+
tests that do not extend the `AbstractOperatorTestCase` class. See, for example, operators such as `Cast ::`,
94+
which uses `CastOperatorTests` to call directly into the `DocsV3Support` class to generate the content.
95+
* Commands do not have dedicated classes or test classes with annotation that can be used.
96+
For this reason, the command documentation is generated by the `CommandDocsTests` class.
97+
Currently, this only covers tested examples used in the documentation, and all other commands
98+
content is static.
99+
Since there are no annotations to mark which examples to use, the command documentation
100+
relies on the docs author providing the knowledge of which examples to use by creating subdirectories
101+
and examples files that match the csv-spec files and tags to include.
102+
103+
To help differentiate between the static and generated content, the generated content is prefixed with a comment:
104+
```
105+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
106+
```
107+
108+
## Tutorials
109+
110+
### Adding a new command
111+
112+
When adding a new command, for example adding the `CHANGE_POINT` command, do the following:
113+
1. Create a new file in the `_snippets/commands/layout` directory with the name of the command, for example `change_point.md`.
114+
2. Add the content for the command to the file. See other files in this directory for examples.
115+
3. Add the command to the list in `_snippets/lists/processing-commands.md`.
116+
4. Add an include directive to the `commands/processing-commands.md` file to include the new command.
117+
5. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
118+
119+
### Adding examples to commands
120+
121+
When adding tested examples to a command, for example adding an example to the `CHANGE_POINT` command, do the following:
122+
* Make sure you have an example in an appropriate csv-spec file in the `x-pack/plugin/esql/qa/testFixtures/src/main/resources/` directory.
123+
* Make sure the example has a tag that is unique in that file, and matches the intent of the test, or the docs reason for including that test.
124+
* If you only want to show the query, and no results, then do not tag the results table,
125+
otherwise tag the results table with a tag that has the same name as the query tag, but with the suffix `-result`.
126+
* Create a file with the name of the tag in a subdirectory with the name of the csv-spec file
127+
in the `_snippets/commands/examples` directory. While you could add the content to that file, it is not necessary, merely that the file exists
128+
* Run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
129+
130+
For example, we tag the following test in change_point.csv-spec:
131+
132+
```
133+
example for docs
134+
required_capability: change_point
135+
136+
// tag::changePointForDocs[]
137+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
138+
| MV_EXPAND key
139+
| EVAL value = CASE(key<13, 0, 42)
140+
| CHANGE_POINT value ON key
141+
| WHERE type IS NOT NULL
142+
// end::changePointForDocs[]
143+
;
144+
145+
// tag::changePointForDocs-result[]
146+
key:integer | value:integer | type:keyword | pvalue:double
147+
13 | 42 | step_change | 0.0
148+
// end::changePointForDocs-result[]
149+
;
150+
```
151+
152+
Then we create the file `_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md` with the content:
153+
```
154+
This should be overwritten
155+
```
156+
157+
Then we run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
158+
159+
Now the content of the changePointForDocs.md file should have been updated:
160+
161+
```
162+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
163+
164+
\```esql
165+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
166+
| MV_EXPAND key
167+
| EVAL value = CASE(key<13, 0, 42)
168+
| CHANGE_POINT value ON key
169+
| WHERE type IS NOT NULL
170+
\```
171+
172+
| key:integer | value:integer | type:keyword | pvalue:double |
173+
| --- | --- | --- | --- |
174+
| 13 | 42 | step_change | 0.0 |
175+
```
176+
177+
Finally include this file in the `CHANGE_POINT` command file `_snippets/commands/layout/change_point.md`:
178+
179+
```
180+
**Examples**
181+
182+
The following example shows the detection of a step change:
183+
184+
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
185+
:::
186+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
5+
| MV_EXPAND key
6+
| EVAL value = CASE(key<13, 0, 42)
7+
| CHANGE_POINT value ON key
8+
| WHERE type IS NOT NULL
9+
```
10+
11+
| key:integer | value:integer | type:keyword | pvalue:double |
12+
| --- | --- | --- | --- |
13+
| 13 | 42 | step_change | 0.0 |

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinHostNameTwice.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM system_metrics
5+
| LOOKUP JOIN host_inventory ON host.name
6+
| LOOKUP JOIN ownerships ON host.name
7+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinServiceId.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM app_logs
5+
| LOOKUP JOIN service_owners ON service_id
6+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinSourceIp.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM firewall_logs
5+
| LOOKUP JOIN threat_list ON source.IP
6+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinSourceIpWhere.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM firewall_logs
5+
| LOOKUP JOIN threat_list ON source.IP
6+
| WHERE threat_level IS NOT NULL
7+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/lookup-join.csv-spec/filterOnLeftSide.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM employees
5+
| EVAL language_code = languages
6+
| WHERE emp_no >= 10091 AND emp_no < 10094
7+
| LOOKUP JOIN languages_lookup ON language_code
8+
```
9+
10+
| emp_no:integer | language_code:integer | language_name:keyword |
11+
| --- | --- | --- |
12+
| 10091 | 3 | Spanish |
13+
| 10092 | 1 | English |
14+
| 10093 | 3 | Spanish |

‎docs/reference/query-languages/esql/_snippets/commands/examples/lookup-join.csv-spec/filterOnRightSide.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM employees
5+
| EVAL language_code = languages
6+
| LOOKUP JOIN languages_lookup ON language_code
7+
| WHERE emp_no >= 10091 AND emp_no < 10094
8+
```
9+
10+
| emp_no:integer | language_code:integer | language_name:keyword |
11+
| --- | --- | --- |
12+
| 10091 | 3 | Spanish |
13+
| 10092 | 1 | English |
14+
| 10093 | 3 | Spanish |

‎docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## `CHANGE_POINT` [esql-change_point]
2+
3+
:::{note}
4+
The `CHANGE_POINT` command requires a [platinum license](https://www.elastic.co/subscriptions).
5+
:::
6+
7+
::::{warning}
8+
This functionality is in technical preview and may be
9+
changed or removed in a future release. Elastic will work to fix any
10+
issues, but features in technical preview are not subject to the support
11+
SLA of official GA features.
12+
::::
13+
14+
`CHANGE_POINT` detects spikes, dips, and change points in a metric.
15+
16+
**Syntax**
17+
18+
```esql
19+
CHANGE_POINT value [ON key] [AS type_name, pvalue_name]
20+
```
21+
22+
**Parameters**
23+
24+
`value`
25+
: The column with the metric in which you want to detect a change point.
26+
27+
`key`
28+
: The column with the key to order the values by. If not specified, `@timestamp` is used.
29+
30+
`type_name`
31+
: The name of the output column with the change point type. If not specified, `type` is used.
32+
33+
`pvalue_name`
34+
: The name of the output column with the p-value that indicates how extreme the change point is. If not specified, `pvalue` is used.
35+
36+
**Description**
37+
38+
`CHANGE_POINT` detects spikes, dips, and change points in a metric. The command adds columns to
39+
the table with the change point type and p-value, that indicates how extreme the change point is
40+
(lower values indicate greater changes).
41+
42+
The possible change point types are:
43+
* `dip`: a significant dip occurs at this change point
44+
* `distribution_change`: the overall distribution of the values has changed significantly
45+
* `spike`: a significant spike occurs at this point
46+
* `step_change`: the change indicates a statistically significant step up or down in value distribution
47+
* `trend_change`: there is an overall trend change occurring at this point
48+
49+
::::{note}
50+
There must be at least 22 values for change point detection. Fewer than 1,000 is preferred.
51+
::::
52+
53+
**Examples**
54+
55+
The following example shows the detection of a step change:
56+
57+
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
58+
:::

0 commit comments

Comments
 (0)