Skip to content

Commit 3f2f5ee

Browse files
jan-elasticcraigtaverner
jan-elastic
and
craigtaverner
authored
ES|QL change_point docs and tech preview (#126407)
* ES|QL change point docs * Move ES|QL change_point to tech preview * Update docs/reference/query-languages/esql/esql-commands.md Co-authored-by: Craig Taverner <[email protected]> * different example + add it the csv tests * Restructure change_point docs to new structure * Added generated test examples to change_point docs * Fixed a few README.md text mistakes and added more details * fix grammar * License check * regen parser * Update docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md Co-authored-by: Craig Taverner <[email protected]> --------- Co-authored-by: Craig Taverner <[email protected]>
1 parent 411a946 commit 3f2f5ee

File tree

17 files changed

+2286
-2058
lines changed

17 files changed

+2286
-2058
lines changed

‎docs/reference/query-languages/esql/README.md‎

Lines changed: 143 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,25 @@
11
The ES|QL documentation is composed of static content and generated content.
22
The static content exists in this directory and can be edited by hand.
3-
However, the sub-directories `_snippets`, `images` and `kibana` contain mostly
4-
generated content.
3+
However, the subdirectories `_snippets`, `images` and `kibana` contain a mix
4+
of static and generated content, and so updating them is a bit more involved.
5+
6+
## Static content
7+
8+
The root `esql` directory and the following two subdirectories contain static content:
9+
* `commands` - contains the static content for the ES|QL commands.
10+
This content will typically contain mostly `include` directives for content in the `_snippets` or `images` directories.
11+
* `functions-operators` - contains the static content for the ES|QL functions and operators.
12+
Again this will contain mostly `include` directives for content in the `_snippets` or `images` directories.
13+
14+
## Mixed static and generated content
15+
16+
Generated content is created by running the ESQL tests in the `x-pack/plugin/esql` module.
17+
It will be written into three subdirectories of the `esql` directory:
518

619
### _snippets
720

8-
In `_snippets` there are files that can be included within other files
9-
using the [File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
21+
In `_snippets` there are files that can be included within other files using the
22+
[File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
1023
feature of the Elastic Docs V3 system.
1124
Most, but not all, files in this directory are generated.
1225
In particular the directories `_snippets/functions/*` and `_snippets/operators/*`
@@ -21,7 +34,7 @@ contain subdirectories that are mostly generated:
2134

2235
Most functions can use the generated docs generated in the `layout` directory.
2336
If we need something more custom for the function we can make a file in this
24-
directory that can `include::` any parts of the files above.
37+
directory that can `include` any parts of the files above.
2538

2639
To regenerate the files for a function run its tests using gradle.
2740
For example to generate docs for the `CASE` function:
@@ -34,17 +47,140 @@ To regenerate the files for all functions run all of ESQL's tests using gradle:
3447
./gradlew :x-pack:plugin:esql:test
3548
```
3649

50+
#### Lists
51+
52+
The `_snippets/lists` directory contains re-usable content for lists of commands, functions or operators.
53+
Whenever adding a command, function or operator, you usually need to add it to one of these lists.
54+
The lists should also match natural groupings of the commands, functions or operators.
55+
For example, when adding an aggregation function, add to the `aggregation-functions.md` file.
56+
57+
#### Commands
58+
59+
The `_snippets/commands` directory contains the content for the ES|QL commands.
60+
There are two subdirectories, one static and one generated:
61+
* `layout` - contains the static content for the ES|QL commands.
62+
The files in this directory are the main content for the documentation for the commands.
63+
They are not generated, and so this is the primary place to edit the content, or add new commands.
64+
* `examples` - contains the generated content for the ES|QL commands.
65+
The files in this directory are generated from the test `CommandDocsTests` in the `x-pack/plugin/esql` module.
66+
The structure of the subdirectories mimics the csv-spec files and test tags used in the tests.
67+
68+
Including generated examples in the command documentation is done by using the include directive.
69+
3770
### images
3871

39-
The `images` directory contains `functions` and `operators` sub-directories with
72+
The `images` directory contains `functions` and `operators` subdirectories with
4073
the `*.svg` files used to describe the syntax of each function or operator.
4174
These are all generated by the same tests that generate the functions and operators docs above.
4275

4376
### kibana
4477

45-
The `kibana` directory contains `definition` and `docs` sub-directories that are generated:
78+
The `kibana` directory contains `definition` and `docs` subdirectories that are generated:
4679

4780
* `kibana/definition` - function definitions for kibana's ESQL editor
4881
* `kibana/docs` - the inline docs for kibana
4982

5083
These are also generated as part of the unit tests described above.
84+
85+
## Under the hood
86+
87+
There are three overlapping mechanisms for generating the content:
88+
* The `AbstractFunctionTestCase` class generates the content for all the functions and most operators.
89+
This class makes use of the `DocsV3Support` class to generate the content.
90+
It uses the `@FunctionInfo` and `@Param` annotations on function and operator classes to know what content should be generated.
91+
All tests that extend this class will automatically generate the content for the functions they test.
92+
* Some operators do not have a clear class or test class, and so the content is generated by custom
93+
tests that do not extend the `AbstractOperatorTestCase` class. See, for example, operators such as `Cast ::`,
94+
which uses `CastOperatorTests` to call directly into the `DocsV3Support` class to generate the content.
95+
* Commands do not have dedicated classes or test classes with annotation that can be used.
96+
For this reason, the command documentation is generated by the `CommandDocsTests` class.
97+
Currently, this only covers tested examples used in the documentation, and all other commands
98+
content is static.
99+
Since there are no annotations to mark which examples to use, the command documentation
100+
relies on the docs author providing the knowledge of which examples to use by creating subdirectories
101+
and examples files that match the csv-spec files and tags to include.
102+
103+
To help differentiate between the static and generated content, the generated content is prefixed with a comment:
104+
```
105+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
106+
```
107+
108+
## Tutorials
109+
110+
### Adding a new command
111+
112+
When adding a new command, for example adding the `CHANGE_POINT` command, do the following:
113+
1. Create a new file in the `_snippets/commands/layout` directory with the name of the command, for example `change_point.md`.
114+
2. Add the content for the command to the file. See other files in this directory for examples.
115+
3. Add the command to the list in `_snippets/lists/processing-commands.md`.
116+
4. Add an include directive to the `commands/processing-commands.md` file to include the new command.
117+
5. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
118+
119+
### Adding examples to commands
120+
121+
When adding tested examples to a command, for example adding an example to the `CHANGE_POINT` command, do the following:
122+
* Make sure you have an example in an appropriate csv-spec file in the `x-pack/plugin/esql/qa/testFixtures/src/main/resources/` directory.
123+
* Make sure the example has a tag that is unique in that file, and matches the intent of the test, or the docs reason for including that test.
124+
* If you only want to show the query, and no results, then do not tag the results table,
125+
otherwise tag the results table with a tag that has the same name as the query tag, but with the suffix `-result`.
126+
* Create a file with the name of the tag in a subdirectory with the name of the csv-spec file
127+
in the `_snippets/commands/examples` directory. While you could add the content to that file, it is not necessary, merely that the file exists
128+
* Run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
129+
130+
For example, we tag the following test in change_point.csv-spec:
131+
132+
```
133+
example for docs
134+
required_capability: change_point
135+
136+
// tag::changePointForDocs[]
137+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
138+
| MV_EXPAND key
139+
| EVAL value = CASE(key<13, 0, 42)
140+
| CHANGE_POINT value ON key
141+
| WHERE type IS NOT NULL
142+
// end::changePointForDocs[]
143+
;
144+
145+
// tag::changePointForDocs-result[]
146+
key:integer | value:integer | type:keyword | pvalue:double
147+
13 | 42 | step_change | 0.0
148+
// end::changePointForDocs-result[]
149+
;
150+
```
151+
152+
Then we create the file `_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md` with the content:
153+
```
154+
This should be overwritten
155+
```
156+
157+
Then we run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
158+
159+
Now the content of the changePointForDocs.md file should have been updated:
160+
161+
```
162+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
163+
164+
\```esql
165+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
166+
| MV_EXPAND key
167+
| EVAL value = CASE(key<13, 0, 42)
168+
| CHANGE_POINT value ON key
169+
| WHERE type IS NOT NULL
170+
\```
171+
172+
| key:integer | value:integer | type:keyword | pvalue:double |
173+
| --- | --- | --- | --- |
174+
| 13 | 42 | step_change | 0.0 |
175+
```
176+
177+
Finally include this file in the `CHANGE_POINT` command file `_snippets/commands/layout/change_point.md`:
178+
179+
```
180+
**Examples**
181+
182+
The following example shows the detection of a step change:
183+
184+
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
185+
:::
186+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
5+
| MV_EXPAND key
6+
| EVAL value = CASE(key<13, 0, 42)
7+
| CHANGE_POINT value ON key
8+
| WHERE type IS NOT NULL
9+
```
10+
11+
| key:integer | value:integer | type:keyword | pvalue:double |
12+
| --- | --- | --- | --- |
13+
| 13 | 42 | step_change | 0.0 |

‎docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## `CHANGE_POINT` [esql-change_point]
2+
3+
:::{note}
4+
The `CHANGE_POINT` command requires a [platinum license](https://www.elastic.co/subscriptions).
5+
:::
6+
7+
::::{warning}
8+
This functionality is in technical preview and may be
9+
changed or removed in a future release. Elastic will work to fix any
10+
issues, but features in technical preview are not subject to the support
11+
SLA of official GA features.
12+
::::
13+
14+
`CHANGE_POINT` detects spikes, dips, and change points in a metric.
15+
16+
**Syntax**
17+
18+
```esql
19+
CHANGE_POINT value [ON key] [AS type_name, pvalue_name]
20+
```
21+
22+
**Parameters**
23+
24+
`value`
25+
: The column with the metric in which you want to detect a change point.
26+
27+
`key`
28+
: The column with the key to order the values by. If not specified, `@timestamp` is used.
29+
30+
`type_name`
31+
: The name of the output column with the change point type. If not specified, `type` is used.
32+
33+
`pvalue_name`
34+
: The name of the output column with the p-value that indicates how extreme the change point is. If not specified, `pvalue` is used.
35+
36+
**Description**
37+
38+
`CHANGE_POINT` detects spikes, dips, and change points in a metric. The command adds columns to
39+
the table with the change point type and p-value, that indicates how extreme the change point is
40+
(lower values indicate greater changes).
41+
42+
The possible change point types are:
43+
* `dip`: a significant dip occurs at this change point
44+
* `distribution_change`: the overall distribution of the values has changed significantly
45+
* `spike`: a significant spike occurs at this point
46+
* `step_change`: the change indicates a statistically significant step up or down in value distribution
47+
* `trend_change`: there is an overall trend change occurring at this point
48+
49+
::::{note}
50+
There must be at least 22 values for change point detection. Fewer than 1,000 is preferred.
51+
::::
52+
53+
**Examples**
54+
55+
The following example shows the detection of a step change:
56+
57+
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
58+
:::

‎docs/reference/query-languages/esql/_snippets/lists/processing-commands.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
* [`CHANGE_POINT`](../../commands/processing-commands.md#esql-change_point)
12
* [`DISSECT`](../../commands/processing-commands.md#esql-dissect)
23
* [`DROP`](../../commands/processing-commands.md#esql-drop)
34
* [`ENRICH`](../../commands/processing-commands.md#esql-enrich)

‎docs/reference/query-languages/esql/commands/processing-commands.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ mapped_pages:
1717
:::{include} ../_snippets/lists/processing-commands.md
1818
:::
1919

20+
:::{include} ../_snippets/commands/layout/change_point.md
21+
:::
22+
2023
:::{include} ../_snippets/commands/layout/dissect.md
2124
:::
2225

‎x-pack/plugin/esql/qa/testFixtures/src/main/resources/change_point.csv-spec‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1198,3 +1198,22 @@ true | 1 | null | null
11981198
true | 1 | null | null
11991199
true | 1 | null | null
12001200
;
1201+
1202+
1203+
example for docs
1204+
required_capability: change_point
1205+
1206+
// tag::changePointForDocs[]
1207+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
1208+
| MV_EXPAND key
1209+
| EVAL value = CASE(key<13, 0, 42)
1210+
| CHANGE_POINT value ON key
1211+
| WHERE type IS NOT NULL
1212+
// end::changePointForDocs[]
1213+
;
1214+
1215+
// tag::changePointForDocs-result[]
1216+
key:integer | value:integer | type:keyword | pvalue:double
1217+
13 | 42 | step_change | 0.0
1218+
// end::changePointForDocs-result[]
1219+
;

‎x-pack/plugin/esql/src/main/antlr/EsqlBaseLexer.tokens‎

Lines changed: 2 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,10 +55,10 @@ processingCommand
5555
| enrichCommand
5656
| mvExpandCommand
5757
| joinCommand
58+
| changePointCommand
5859
// in development
5960
| {this.isDevVersion()}? inlinestatsCommand
6061
| {this.isDevVersion()}? lookupCommand
61-
| {this.isDevVersion()}? changePointCommand
6262
| {this.isDevVersion()}? completionCommand
6363
| {this.isDevVersion()}? insistCommand
6464
| {this.isDevVersion()}? forkCommand
@@ -257,7 +257,7 @@ inlinestatsCommand
257257
;
258258

259259
changePointCommand
260-
: DEV_CHANGE_POINT value=qualifiedName (ON key=qualifiedName)? (AS targetType=qualifiedName COMMA targetPvalue=qualifiedName)?
260+
: CHANGE_POINT value=qualifiedName (ON key=qualifiedName)? (AS targetType=qualifiedName COMMA targetPvalue=qualifiedName)?
261261
;
262262

263263
insistCommand

‎x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.tokens‎

Lines changed: 2 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎x-pack/plugin/esql/src/main/antlr/lexer/ChangePoint.g4‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ lexer grammar ChangePoint;
99
//
1010
// | CHANGE_POINT command
1111
//
12-
DEV_CHANGE_POINT : {this.isDevVersion()}? 'change_point' -> pushMode(CHANGE_POINT_MODE);
12+
CHANGE_POINT : 'change_point' -> pushMode(CHANGE_POINT_MODE);
1313

1414
mode CHANGE_POINT_MODE;
1515
CHANGE_POINT_PIPE : PIPE -> type(PIPE), popMode;

0 commit comments

Comments
 (0)