Skip to content

Commit ba59768

Browse files
benwtrentbenwtrent
authored
Merge branch 'main' into ivf-bulk-writer-refactor
2 parents 4d387b6 + 4164058 commit ba59768

File tree

118 files changed

+2868
-914
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

118 files changed

+2868
-914
lines changed

‎docs/changelog/127223.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 127223
2+
summary: Wrap ES KNN queries with PatienceKNN query
3+
area: Vector Search
4+
type: feature
5+
issues: []

‎docs/changelog/128866.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 128866
2+
summary: Add `age_in_millis` to ILM Explain Response
3+
area: ILM+SLM
4+
type: enhancement
5+
issues:
6+
- 103659

‎docs/changelog/130251.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 130251
2+
summary: Speed up (filtered) KNN queries for flat vector fields
3+
area: Vector Search
4+
type: enhancement
5+
issues: []

‎docs/changelog/130344.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 130344
2+
summary: "Fix queries with missing index, `skip_unavailable` and filters"
3+
area: ES|QL
4+
type: bug
5+
issues: []

‎docs/reference/elasticsearch/index-settings/index-modules.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,3 +259,6 @@ $$$index-esql-stored-fields-sequential-proportion$$$
259259

260260
`index.esql.stored_fields_sequential_proportion`
261261
: Tuning parameter for deciding when {{esql}} will load [Stored fields](/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#stored-fields) using a strategy tuned for loading dense sequence of documents. Allows values between 0.0 and 1.0 and defaults to 0.2. Indices with documents smaller than 10kb may see speed improvements loading `text` fields by setting this lower.
262+
263+
$$$index-dense-vector-hnsw-early-termination$$$ `index.dense_vector.hnsw_early_termination`
264+
: Whether to apply _patience_ based early termination strategy to knn queries over HNSW graphs (see [paper](https://cs.uwaterloo.ca/~jimmylin/publications/Teofili_Lin_ECIR2025.pdf)). This is only applicable to `dense_vector` fields with `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types. Defaults to `false`.

‎docs/reference/query-languages/esql/README.md‎

Lines changed: 83 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -105,16 +105,95 @@ To help differentiate between the static and generated content, the generated co
105105
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
106106
```
107107

108+
## Version differentiation in Docs V3
109+
110+
> [!IMPORTANT]
111+
> Starting with 9.0, we no longer publish separate documentation branches for every minor release (`9.0`, `9.1`, `9.2`, etc.).
112+
> This means there won't be a different page for `9.1`, `9.2`, and so on. Instead, all changes landing in subsequent minor releases **will appear on the same page**.
113+
114+
Because we now publish just one docs set off of the `main` branch, we use the [`applies_to` metadata](https://elastic.github.io/docs-builder/syntax/applies/) to differentiate features and their availability across different versions. This is a [cumulative approach](https://elastic.github.io/docs-builder/contribute/#cumulative-docs): instead of creating separate pages for each product and release, we update a **single page** with product- and version-specific details over time.
115+
116+
`applies_to` allows us to clearly communicate when features are introduced, when they transition from preview to GA, and which versions support specific functionality.
117+
118+
This metadata accepts a lifecycle and an optional version.
119+
120+
### Functions and operators
121+
122+
Use the `@FunctionAppliesTo` annotation within the `@FunctionInfo` annotation on function and operator classes to specify the lifecycle and version for functions and operators.
123+
124+
For example, to indicate that a function is in technical preview and applies to version 9.0.0, you would use:
125+
126+
```java
127+
@FunctionInfo(
128+
returnType = "boolean",
129+
appliesTo = {
130+
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0")
131+
},
132+
...
133+
)
134+
```
135+
136+
When a feature evolves from preview in `9.0` to GA in `9.2`, add a new entry alongside the existing preview entry and remove the `preview = true` boolean:
137+
138+
```java
139+
@FunctionInfo(
140+
returnType = "boolean",
141+
preview = false, // the preview boolean can be removed (or flipped to false) when the function becomes GA
142+
appliesTo = {
143+
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0"),
144+
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2.0")
145+
},
146+
...
147+
)
148+
```
149+
150+
We updated [`DocsV3Support.java`](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3Support.java) to generate the `applies_to` metadata correctly for functions and operators.
151+
152+
### Inline `applies_to` metadata
153+
154+
Use [inline annotations](https://elastic.github.io/docs-builder/syntax/applies/#inline-annotations) to specify `applies_to` metadata in descriptions, parameter lists, etc.
155+
156+
For example, the second item in this list is in technical preview as of version 9.2:
157+
158+
```markdown
159+
- Item 1
160+
- Item 2 {applies_to}`stack: preview 9.2.`
161+
```
162+
163+
### Key rules
164+
165+
1. **Use the `preview = true` boolean** for any tech preview feature - this is required for the Kibana inline docs
166+
- **Remove `preview = true`** only when the feature becomes GA on serverless and is _definitely_ going GA in the next minor release
167+
2. **Never delete `appliesTo` entries** - only add new ones as features evolve from preview to GA
168+
3. **Use specific versions** (`9.0.0`, `9.1.0`) when known, or just `PREVIEW` without a version if timing is uncertain
169+
4. **Add `applies_to` to examples** where necessary
170+
171+
> [!IMPORTANT]
172+
> We don't use `applies_to` in the legacy asciidoc system for 8.x and earlier versions.
173+
174+
### Supported lifecycles
175+
176+
- `PREVIEW` - Feature is in technical preview
177+
- `GA` - Feature is generally available
178+
- `DEPRECATED` - Feature is deprecated and will be removed in a future release
179+
- `UNAVAILABLE` - Feature is not available in the current version, but may be available in future releases
180+
181+
> [!NOTE]
182+
> Unreleased version information is automatically sanitized in the docs build output. For example, say you specify `preview 9.3.0`:
183+
> - Before `9.3.0` is released, the live documentation will display "Planned for a future release" instead of the specific version number.
184+
> - This will be updated automatically when the version is released.
185+
108186
## Tutorials
109187

110188
### Adding a new command
111189

112190
When adding a new command, for example adding the `CHANGE_POINT` command, do the following:
113191
1. Create a new file in the `_snippets/commands/layout` directory with the name of the command, for example `change_point.md`.
114-
2. Add the content for the command to the file. See other files in this directory for examples.
115-
3. Add the command to the list in `_snippets/lists/processing-commands.md`.
116-
4. Add an include directive to the `commands/processing-commands.md` file to include the new command.
117-
5. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
192+
2. Ensure to specify what versions the command applies to. See [Version differentiation in Docs V3](#version-differentiation-in-docs-v3) for details. [Example PR](https://github.com/elastic/elasticsearch/pull/130314/files#diff-0ab90b6202c5d9eeea75dc95a7cb71dc4d720230342718bff887816771a5a803R3-R6).
193+
3. Add the content for the command to the file. See other files in this directory for examples.
194+
4. Add the command to the list in `_snippets/lists/processing-commands.md`.
195+
5. Add an include directive to the `commands/processing-commands.md` file to include the new command.
196+
6. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
118197

119198
### Adding examples to commands
120199

‎docs/reference/query-languages/esql/_snippets/commands/examples/fork.csv-spec/simpleFork.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM employees
5+
| FORK ( WHERE emp_no == 10001 )
6+
( WHERE emp_no == 10002 )
7+
| KEEP emp_no, _fork
8+
| SORT emp_no
9+
```
10+
11+
| emp_no:integer | _fork:keyword |
12+
| --- | --- |
13+
| 10001 | fork1 |
14+
| 10002 | fork2 |

‎docs/reference/query-languages/esql/_snippets/commands/examples/fork.csv-spec/simpleForkWithStats.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM books METADATA _score
5+
| WHERE author:"Faulkner"
6+
| EVAL score = round(_score, 2)
7+
| FORK (SORT score DESC, author | LIMIT 5 | KEEP author, score)
8+
(STATS total = COUNT(*))
9+
| SORT _fork, score DESC, author
10+
```
11+
12+
| author:text | score:double | _fork:keyword | total:long |
13+
| --- | --- | --- | --- |
14+
| William Faulkner | 2.39 | fork1 | null |
15+
| William Faulkner | 2.39 | fork1 | null |
16+
| Colleen Faulkner | 1.59 | fork1 | null |
17+
| Danny Faulkner | 1.59 | fork1 | null |
18+
| Keith Faulkner | 1.59 | fork1 | null |
19+
| null | null | fork2 | 18 |

‎docs/reference/query-languages/esql/_snippets/commands/layout/fork.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
## `FORK` [esql-fork]
2+
3+
```yaml {applies_to}
4+
serverless: preview
5+
stack: preview 9.1.0
6+
```
7+
8+
The `FORK` processing command creates multiple execution branches to operate
9+
on the same input data and combines the results in a single output table.
10+
11+
**Syntax**
12+
13+
```esql
14+
FORK ( <processing_commands> ) ( <processing_commands> ) ... ( <processing_commands> )
15+
```
16+
17+
**Description**
18+
19+
The `FORK` processing command creates multiple execution branches to operate
20+
on the same input data and combines the results in a single output table. A discriminator column (`_fork`) is added to identify which branch each row came from.
21+
22+
**Branch identification:**
23+
- The `_fork` column identifies each branch with values like `fork1`, `fork2`, `fork3`
24+
- Values correspond to the order branches are defined
25+
- `fork1` always indicates the first branch
26+
27+
**Column handling:**
28+
- `FORK` branches can output different columns
29+
- Columns with the same name must have the same data type across all branches
30+
- Missing columns are filled with `null` values
31+
32+
**Row ordering:**
33+
- `FORK` preserves row order within each branch
34+
- Rows from different branches may be interleaved
35+
- Use `SORT _fork` to group results by branch
36+
37+
::::{note}
38+
`FORK` branches default to `LIMIT 1000` if no `LIMIT` is provided.
39+
::::
40+
41+
**Limitations**
42+
43+
- `FORK` supports at most 8 execution branches.
44+
- Using remote cluster references and `FORK` is not supported.
45+
- Using more than one `FORK` command in a query is not supported.
46+
47+
**Examples**
48+
49+
In the following example, each `FORK` branch returns one row.
50+
Notice how `FORK` adds a `_fork` column that indicates which row the branch originates from:
51+
52+
:::{include} ../examples/fork.csv-spec/simpleFork.md
53+
54+
The next example, returns total number of rows that match the query along with
55+
the top five rows sorted by score.
56+
57+
:::{include} ../examples/fork.csv-spec/simpleForkWithStats.md

‎docs/reference/query-languages/esql/_snippets/lists/aggregation-functions.md‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,10 @@
11
* [`AVG`](../../functions-operators/aggregation-functions.md#esql-avg)
2-
* [unavailable] [`AVG_OVER_TIME`](../../functions-operators/aggregation-functions.md#esql-avg_over_time)
32
* [`COUNT`](../../functions-operators/aggregation-functions.md#esql-count)
43
* [`COUNT_DISTINCT`](../../functions-operators/aggregation-functions.md#esql-count_distinct)
54
* [`MAX`](../../functions-operators/aggregation-functions.md#esql-max)
6-
* [unavailable] [`MAX_OVER_TIME`](../../functions-operators/aggregation-functions.md#esql-max_over_time)
75
* [`MEDIAN`](../../functions-operators/aggregation-functions.md#esql-median)
86
* [`MEDIAN_ABSOLUTE_DEVIATION`](../../functions-operators/aggregation-functions.md#esql-median_absolute_deviation)
97
* [`MIN`](../../functions-operators/aggregation-functions.md#esql-min)
10-
* [unavailable] [`MIN_OVER_TIME`](../../functions-operators/aggregation-functions.md#esql-min_over_time)
118
* [`PERCENTILE`](../../functions-operators/aggregation-functions.md#esql-percentile)
129
* [preview] [`ST_CENTROID_AGG`](../../functions-operators/aggregation-functions.md#esql-st_centroid_agg)
1310
* [preview] [`ST_EXTENT_AGG`](../../functions-operators/aggregation-functions.md#esql-st_extent_agg)
@@ -16,6 +13,3 @@
1613
* [`TOP`](../../functions-operators/aggregation-functions.md#esql-top)
1714
* [preview] [`VALUES`](../../functions-operators/aggregation-functions.md#esql-values)
1815
* [`WEIGHTED_AVG`](../../functions-operators/aggregation-functions.md#esql-weighted_avg)
19-
* [unavailable] [`FIRST_OVER_TIME`](../../functions-operators/aggregation-functions.md#esql-first_over_time)
20-
* [unavailable] [`LAST_OVER_TIME`](../../functions-operators/aggregation-functions.md#esql-last_over_time)
21-
* [unavailable] [`RATE`](../../functions-operators/aggregation-functions.md#esql-rate)

0 commit comments

Comments
 (0)