Skip to content

Commit 063e73f

Browse files
andreidanandreidan
authored
Merge branch 'main' into cache-metrics-online-prewarming-type
2 parents e6a010c + ec495e9 commit 063e73f

38 files changed

+2611
-2108
lines changed

‎docs/reference/query-languages/esql/README.md‎

Lines changed: 143 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,25 @@
11
The ES|QL documentation is composed of static content and generated content.
22
The static content exists in this directory and can be edited by hand.
3-
However, the sub-directories `_snippets`, `images` and `kibana` contain mostly
4-
generated content.
3+
However, the subdirectories `_snippets`, `images` and `kibana` contain a mix
4+
of static and generated content, and so updating them is a bit more involved.
5+
6+
## Static content
7+
8+
The root `esql` directory and the following two subdirectories contain static content:
9+
* `commands` - contains the static content for the ES|QL commands.
10+
This content will typically contain mostly `include` directives for content in the `_snippets` or `images` directories.
11+
* `functions-operators` - contains the static content for the ES|QL functions and operators.
12+
Again this will contain mostly `include` directives for content in the `_snippets` or `images` directories.
13+
14+
## Mixed static and generated content
15+
16+
Generated content is created by running the ESQL tests in the `x-pack/plugin/esql` module.
17+
It will be written into three subdirectories of the `esql` directory:
518

619
### _snippets
720

8-
In `_snippets` there are files that can be included within other files
9-
using the [File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
21+
In `_snippets` there are files that can be included within other files using the
22+
[File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
1023
feature of the Elastic Docs V3 system.
1124
Most, but not all, files in this directory are generated.
1225
In particular the directories `_snippets/functions/*` and `_snippets/operators/*`
@@ -21,7 +34,7 @@ contain subdirectories that are mostly generated:
2134

2235
Most functions can use the generated docs generated in the `layout` directory.
2336
If we need something more custom for the function we can make a file in this
24-
directory that can `include::` any parts of the files above.
37+
directory that can `include` any parts of the files above.
2538

2639
To regenerate the files for a function run its tests using gradle.
2740
For example to generate docs for the `CASE` function:
@@ -34,17 +47,140 @@ To regenerate the files for all functions run all of ESQL's tests using gradle:
3447
./gradlew :x-pack:plugin:esql:test
3548
```
3649

50+
#### Lists
51+
52+
The `_snippets/lists` directory contains re-usable content for lists of commands, functions or operators.
53+
Whenever adding a command, function or operator, you usually need to add it to one of these lists.
54+
The lists should also match natural groupings of the commands, functions or operators.
55+
For example, when adding an aggregation function, add to the `aggregation-functions.md` file.
56+
57+
#### Commands
58+
59+
The `_snippets/commands` directory contains the content for the ES|QL commands.
60+
There are two subdirectories, one static and one generated:
61+
* `layout` - contains the static content for the ES|QL commands.
62+
The files in this directory are the main content for the documentation for the commands.
63+
They are not generated, and so this is the primary place to edit the content, or add new commands.
64+
* `examples` - contains the generated content for the ES|QL commands.
65+
The files in this directory are generated from the test `CommandDocsTests` in the `x-pack/plugin/esql` module.
66+
The structure of the subdirectories mimics the csv-spec files and test tags used in the tests.
67+
68+
Including generated examples in the command documentation is done by using the include directive.
69+
3770
### images
3871

39-
The `images` directory contains `functions` and `operators` sub-directories with
72+
The `images` directory contains `functions` and `operators` subdirectories with
4073
the `*.svg` files used to describe the syntax of each function or operator.
4174
These are all generated by the same tests that generate the functions and operators docs above.
4275

4376
### kibana
4477

45-
The `kibana` directory contains `definition` and `docs` sub-directories that are generated:
78+
The `kibana` directory contains `definition` and `docs` subdirectories that are generated:
4679

4780
* `kibana/definition` - function definitions for kibana's ESQL editor
4881
* `kibana/docs` - the inline docs for kibana
4982

5083
These are also generated as part of the unit tests described above.
84+
85+
## Under the hood
86+
87+
There are three overlapping mechanisms for generating the content:
88+
* The `AbstractFunctionTestCase` class generates the content for all the functions and most operators.
89+
This class makes use of the `DocsV3Support` class to generate the content.
90+
It uses the `@FunctionInfo` and `@Param` annotations on function and operator classes to know what content should be generated.
91+
All tests that extend this class will automatically generate the content for the functions they test.
92+
* Some operators do not have a clear class or test class, and so the content is generated by custom
93+
tests that do not extend the `AbstractOperatorTestCase` class. See, for example, operators such as `Cast ::`,
94+
which uses `CastOperatorTests` to call directly into the `DocsV3Support` class to generate the content.
95+
* Commands do not have dedicated classes or test classes with annotation that can be used.
96+
For this reason, the command documentation is generated by the `CommandDocsTests` class.
97+
Currently, this only covers tested examples used in the documentation, and all other commands
98+
content is static.
99+
Since there are no annotations to mark which examples to use, the command documentation
100+
relies on the docs author providing the knowledge of which examples to use by creating subdirectories
101+
and examples files that match the csv-spec files and tags to include.
102+
103+
To help differentiate between the static and generated content, the generated content is prefixed with a comment:
104+
```
105+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
106+
```
107+
108+
## Tutorials
109+
110+
### Adding a new command
111+
112+
When adding a new command, for example adding the `CHANGE_POINT` command, do the following:
113+
1. Create a new file in the `_snippets/commands/layout` directory with the name of the command, for example `change_point.md`.
114+
2. Add the content for the command to the file. See other files in this directory for examples.
115+
3. Add the command to the list in `_snippets/lists/processing-commands.md`.
116+
4. Add an include directive to the `commands/processing-commands.md` file to include the new command.
117+
5. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
118+
119+
### Adding examples to commands
120+
121+
When adding tested examples to a command, for example adding an example to the `CHANGE_POINT` command, do the following:
122+
* Make sure you have an example in an appropriate csv-spec file in the `x-pack/plugin/esql/qa/testFixtures/src/main/resources/` directory.
123+
* Make sure the example has a tag that is unique in that file, and matches the intent of the test, or the docs reason for including that test.
124+
* If you only want to show the query, and no results, then do not tag the results table,
125+
otherwise tag the results table with a tag that has the same name as the query tag, but with the suffix `-result`.
126+
* Create a file with the name of the tag in a subdirectory with the name of the csv-spec file
127+
in the `_snippets/commands/examples` directory. While you could add the content to that file, it is not necessary, merely that the file exists
128+
* Run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
129+
130+
For example, we tag the following test in change_point.csv-spec:
131+
132+
```
133+
example for docs
134+
required_capability: change_point
135+
136+
// tag::changePointForDocs[]
137+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
138+
| MV_EXPAND key
139+
| EVAL value = CASE(key<13, 0, 42)
140+
| CHANGE_POINT value ON key
141+
| WHERE type IS NOT NULL
142+
// end::changePointForDocs[]
143+
;
144+
145+
// tag::changePointForDocs-result[]
146+
key:integer | value:integer | type:keyword | pvalue:double
147+
13 | 42 | step_change | 0.0
148+
// end::changePointForDocs-result[]
149+
;
150+
```
151+
152+
Then we create the file `_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md` with the content:
153+
```
154+
This should be overwritten
155+
```
156+
157+
Then we run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
158+
159+
Now the content of the changePointForDocs.md file should have been updated:
160+
161+
```
162+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
163+
164+
\```esql
165+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
166+
| MV_EXPAND key
167+
| EVAL value = CASE(key<13, 0, 42)
168+
| CHANGE_POINT value ON key
169+
| WHERE type IS NOT NULL
170+
\```
171+
172+
| key:integer | value:integer | type:keyword | pvalue:double |
173+
| --- | --- | --- | --- |
174+
| 13 | 42 | step_change | 0.0 |
175+
```
176+
177+
Finally include this file in the `CHANGE_POINT` command file `_snippets/commands/layout/change_point.md`:
178+
179+
```
180+
**Examples**
181+
182+
The following example shows the detection of a step change:
183+
184+
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
185+
:::
186+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
5+
| MV_EXPAND key
6+
| EVAL value = CASE(key<13, 0, 42)
7+
| CHANGE_POINT value ON key
8+
| WHERE type IS NOT NULL
9+
```
10+
11+
| key:integer | value:integer | type:keyword | pvalue:double |
12+
| --- | --- | --- | --- |
13+
| 13 | 42 | step_change | 0.0 |

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinHostNameTwice.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM system_metrics
5+
| LOOKUP JOIN host_inventory ON host.name
6+
| LOOKUP JOIN ownerships ON host.name
7+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinServiceId.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM app_logs
5+
| LOOKUP JOIN service_owners ON service_id
6+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinSourceIp.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM firewall_logs
5+
| LOOKUP JOIN threat_list ON source.IP
6+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/docs-lookup-join.csv-spec/lookupJoinSourceIpWhere.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM firewall_logs
5+
| LOOKUP JOIN threat_list ON source.IP
6+
| WHERE threat_level IS NOT NULL
7+
```

‎docs/reference/query-languages/esql/_snippets/commands/examples/lookup-join.csv-spec/filterOnLeftSide.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM employees
5+
| EVAL language_code = languages
6+
| WHERE emp_no >= 10091 AND emp_no < 10094
7+
| LOOKUP JOIN languages_lookup ON language_code
8+
```
9+
10+
| emp_no:integer | language_code:integer | language_name:keyword |
11+
| --- | --- | --- |
12+
| 10091 | 3 | Spanish |
13+
| 10092 | 1 | English |
14+
| 10093 | 3 | Spanish |

‎docs/reference/query-languages/esql/_snippets/commands/examples/lookup-join.csv-spec/filterOnRightSide.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM employees
5+
| EVAL language_code = languages
6+
| LOOKUP JOIN languages_lookup ON language_code
7+
| WHERE emp_no >= 10091 AND emp_no < 10094
8+
```
9+
10+
| emp_no:integer | language_code:integer | language_name:keyword |
11+
| --- | --- | --- |
12+
| 10091 | 3 | Spanish |
13+
| 10092 | 1 | English |
14+
| 10093 | 3 | Spanish |

‎docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## `CHANGE_POINT` [esql-change_point]
2+
3+
:::{note}
4+
The `CHANGE_POINT` command requires a [platinum license](https://www.elastic.co/subscriptions).
5+
:::
6+
7+
::::{warning}
8+
This functionality is in technical preview and may be
9+
changed or removed in a future release. Elastic will work to fix any
10+
issues, but features in technical preview are not subject to the support
11+
SLA of official GA features.
12+
::::
13+
14+
`CHANGE_POINT` detects spikes, dips, and change points in a metric.
15+
16+
**Syntax**
17+
18+
```esql
19+
CHANGE_POINT value [ON key] [AS type_name, pvalue_name]
20+
```
21+
22+
**Parameters**
23+
24+
`value`
25+
: The column with the metric in which you want to detect a change point.
26+
27+
`key`
28+
: The column with the key to order the values by. If not specified, `@timestamp` is used.
29+
30+
`type_name`
31+
: The name of the output column with the change point type. If not specified, `type` is used.
32+
33+
`pvalue_name`
34+
: The name of the output column with the p-value that indicates how extreme the change point is. If not specified, `pvalue` is used.
35+
36+
**Description**
37+
38+
`CHANGE_POINT` detects spikes, dips, and change points in a metric. The command adds columns to
39+
the table with the change point type and p-value, that indicates how extreme the change point is
40+
(lower values indicate greater changes).
41+
42+
The possible change point types are:
43+
* `dip`: a significant dip occurs at this change point
44+
* `distribution_change`: the overall distribution of the values has changed significantly
45+
* `spike`: a significant spike occurs at this point
46+
* `step_change`: the change indicates a statistically significant step up or down in value distribution
47+
* `trend_change`: there is an overall trend change occurring at this point
48+
49+
::::{note}
50+
There must be at least 22 values for change point detection. Fewer than 1,000 is preferred.
51+
::::
52+
53+
**Examples**
54+
55+
The following example shows the detection of a step change:
56+
57+
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
58+
:::

‎docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md‎

Lines changed: 14 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -52,53 +52,37 @@ In case of name collisions, the newly created columns will override existing col
5252
**IP Threat correlation**: This query would allow you to see if any source
5353
IPs match known malicious addresses.
5454

55-
```esql
56-
FROM firewall_logs
57-
| LOOKUP JOIN threat_list ON source.IP
58-
```
55+
:::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinSourceIp.md
56+
:::
5957

6058
To filter only for those rows that have a matching `threat_list` entry, use `WHERE ... IS NOT NULL` with a field from the lookup index:
6159

62-
```esql
63-
FROM firewall_logs
64-
| LOOKUP JOIN threat_list ON source.IP
65-
| WHERE threat_level IS NOT NULL
66-
```
60+
:::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinSourceIpWhere.md
61+
:::
6762

6863
**Host metadata correlation**: This query pulls in environment or
6964
ownership details for each host to correlate with your metrics data.
7065

71-
```esql
72-
FROM system_metrics
73-
| LOOKUP JOIN host_inventory ON host.name
74-
| LOOKUP JOIN employees ON host.name
75-
```
66+
:::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinHostNameTwice.md
67+
:::
7668

7769
**Service ownership mapping**: This query would show logs with the owning
7870
team or escalation information for faster triage and incident response.
7971

80-
```esql
81-
FROM app_logs
82-
| LOOKUP JOIN service_owners ON service_id
83-
```
72+
:::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinServiceId.md
73+
:::
8474

8575
`LOOKUP JOIN` is generally faster when there are fewer rows to join
8676
with. {{esql}} will try and perform any `WHERE` clause before the
8777
`LOOKUP JOIN` where possible.
8878

89-
The two following examples will have the same results. The two examples
90-
have the `WHERE` clause before and after the `LOOKUP JOIN`. It does not
79+
The following two examples will have the same results. One has the
80+
`WHERE` clause before and the other after the `LOOKUP JOIN`. It does not
9181
matter how you write your query, our optimizer will move the filter
9282
before the lookup when possible.
9383

94-
```esql
95-
FROM Left
96-
| WHERE Language IS NOT NULL
97-
| LOOKUP JOIN Right ON Key
98-
```
84+
:::{include} ../examples/lookup-join.csv-spec/filterOnLeftSide.md
85+
:::
9986

100-
```esql
101-
FROM Left
102-
| LOOKUP JOIN Right ON Key
103-
| WHERE Language IS NOT NULL
104-
```
87+
:::{include} ../examples/lookup-join.csv-spec/filterOnRightSide.md
88+
:::

0 commit comments

Comments
 (0)