You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: x-pack/plugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/EsqlQueryGenerator.java
Copy file name to clipboardExpand all lines: x-pack/plugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java
These tests generate random queries and execute them.
4
+
5
+
The intention is not to test the single commands, but rather to test how ES|QL query engine
6
+
(parser, optimizers, query layout, compute) manages very complex queries.
7
+
8
+
The test workflow is the following:
9
+
10
+
1. Generate a source command (eg. `FROM idx`)
11
+
2. Execute it
12
+
3. Check the result
13
+
4. Based on the previous query output, generate a pipe command (eg. `| EVAL foo = to_lower(bar))`
14
+
5. Append the command to the query and execute it
15
+
6. Check the result
16
+
7. If the query is less than N commands (see `GenerativeRestTest.MAX_DEPTH)`, go to point `4`
17
+
18
+
This workflow is executed M times (see `GenerativeRestTest.ITERATIONS`)
19
+
20
+
The result check happens at two levels:
21
+
22
+
* query success/failure - If the query fails:
23
+
** If the error is in `GenerativeRestTest.ALLOWED_ERRORS`, ignore it and start with next iteration.
24
+
** Otherwise throw an assertion error
25
+
* check result correctness - this is delegated to last executed command generator
26
+
27
+
== Implementing your own command generator
28
+
29
+
If you implement a new command, and you want it to be tested by the generative tests, you can add a command generator here.
30
+
31
+
All you have to do is:
32
+
33
+
* add a class in `org.elasticsearch.xpack.esql.qa.rest.generative.command.source` (if it's a source command) or in `org.elasticsearch.xpack.esql.qa.rest.generative.command.pipe` (if it's a pipe command)
34
+
* Implement `CommandGenerator` interface (see its javadoc, it should be explicative. Or just have a look at one of the existing commands, eg. `SortGenerator`)
35
+
** Implement `CommandGenerator.generate()` method, that will return the command.
36
+
*** Have a look at `EsqlQueryGenerator`, it contains many utility methods that will help you generate random expressions.
37
+
** Implement `CommandGenerator.validateOutput()` to validate the output of the query.
38
+
* Add your class to `EsqlQueryGenerator.SOURCE_COMMANDS` (if it's a source command) or `EsqlQueryGenerator.PIPE_COMMANDS` (if it's a pipe command).
39
+
* Run `GenerativeIT` at least a couple of times: these tests can be pretty noisy.
40
+
* If you get unexpected errors (real bugs in ES|QL), please open an issue and add the error to `GenerativeRestTest.ALLOWED_ERRORS`. Run tests again until everything works fine.
41
+
42
+
43
+
IMPORTANT: be careful when validating the output (Eg. the row count), as ES|QL can be quite non-deterministic when there are no SORTs
* Implement this method to generate a command, that will be appended to an existing query and then executed.
68
+
* See also {@link CommandDescription}
69
+
*
70
+
* @param previousCommands the list of the previous commands in the query
71
+
* @param previousOutput the output returned by the query so far.
72
+
* @param schema The columns returned by the query so far. It contains name and type information for each column.
73
+
* @return All the details about the generated command. See {@link CommandDescription}.
74
+
* If something goes wrong and for some reason you can't generate a command, you should return {@link CommandGenerator#EMPTY_DESCRIPTION}
75
+
*/
76
+
CommandDescriptiongenerate(
77
+
List<CommandDescription> previousCommands,
78
+
List<EsqlQueryGenerator.Column> previousOutput,
79
+
QuerySchemaschema
80
+
);
81
+
82
+
/**
83
+
* This will be invoked after the query execution.
84
+
* You are expected to put validation logic in here.
85
+
*
86
+
* @param previousCommands The list of commands before the last generated one.
87
+
* @param command The description of the command you just generated.
88
+
* It also contains the context information you stored during command generation.
89
+
* @param previousColumns The output schema of the original query (without last generated command).
90
+
* It contains name and type information for each column, see {@link EsqlQueryGenerator.Column}
91
+
* @param previousOutput The output of the original query (without last generated command), as a list (rows) of lists (columns) of values
92
+
* @param columns The output schema of the full query (WITH last generated command).
93
+
* @param output The output of the full query (WITH last generated command), as a list (rows) of lists (columns) of values
94
+
* @return The result of the output validation. If the validation succeeds, you should return {@link CommandGenerator#VALIDATION_OK}.
95
+
* Also, if for some reason you can't validate the output, just return {@link CommandGenerator#VALIDATION_OK}; for a command, having a generator without
96
+
* validation is much better than having no generator at all.
97
+
*/
98
+
ValidationResultvalidateOutput(
99
+
List<CommandDescription> previousCommands,
100
+
CommandDescriptioncommand,
101
+
List<EsqlQueryGenerator.Column> previousColumns,
102
+
List<List<Object>> previousOutput,
103
+
List<EsqlQueryGenerator.Column> columns,
104
+
List<List<Object>> output
105
+
);
106
+
107
+
staticValidationResultexpectSameRowCount(
108
+
List<CommandDescription> previousCommands,
109
+
List<List<Object>> previousOutput,
110
+
List<List<Object>> output
111
+
) {
112
+
113
+
// ES|QL is quite non-deterministic in this sense, we can't guarantee it for now
114
+
// if (output.size() != previousOutput.size()) {
115
+
// return new ValidationResult(false, "Expecting [" + previousOutput.size() + "] rows, but got [" + output.size() + "]");
0 commit comments