Skip to content

Commit d2d6c08

Browse files
ioanatialeemthompo
ioanatia
and
leemthompo
authored
ES|QL: Docs for FUSE (elastic#135693) (elastic#135890)
* Docs for FUSE * Add missing md file * restructure, copyedit, add links and code annotations * **copyedits** * **added tabbed sections** for rrf vs linear parameters instead of plain text * **inserted links** to related docs * **restructured content** with proper markdown headers * **gave examples descriptive titles** and added inline code annotations * wording * Add tip linking to ESQL for search page * backticks back alriiight --------- Co-authored-by: Liam Thompson <[email protected]>
1 parent 9617724 commit d2d6c08

File tree

4 files changed

+149
-0
lines changed

4 files changed

+149
-0
lines changed

‎docs/reference/query-languages/esql/_snippets/commands/layout/fuse.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
```yaml {applies_to}
2+
serverless: preview
3+
stack: preview 9.2.0
4+
```
5+
6+
The `FUSE` [processing command](/reference/query-languages/esql/commands/processing-commands.md) merges rows from multiple result sets and assigns
7+
new relevance scores.
8+
9+
`FUSE` enables [hybrid search](/reference/query-languages/esql/esql-search-tutorial.md#perform-hybrid-search) to combine and score results from multiple queries, together with the [`FORK`](/reference/query-languages/esql/commands/fork.md) command.
10+
11+
`FUSE` works by:
12+
13+
1. Merging rows with matching `<key_columns>` values
14+
2. Assigning new relevance scores using the specified `<fuse_method>` algorithm
15+
and the values from the `<group_column>` and `<score_column>`
16+
17+
:::::{tip}
18+
`FUSE` is for search use cases: it merges ranked result sets and computes relevance.
19+
Learn more about [how search works in ES|QL](docs-content://solutions/search/esql-for-search.md#how-search-works-in-esql).
20+
:::::
21+
22+
## Syntax
23+
24+
Use default parameters:
25+
26+
```esql
27+
FUSE
28+
```
29+
30+
Specify custom parameters:
31+
32+
```esql
33+
FUSE <fuse_method> SCORE BY <score_column> GROUP BY <group_column> KEY BY <key_columns> WITH <options>
34+
```
35+
36+
## Parameters
37+
38+
`fuse_method`
39+
: Defaults to `RRF`. Can be one of `RRF` (for [Reciprocal Rank Fusion](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf)) or `LINEAR` (for linear combination of scores).
40+
Designates which method to use to assign new relevance scores.
41+
42+
`options`
43+
: Options for the `fuse_method`.
44+
45+
::::{tab-set}
46+
:::{tab-item} RRF
47+
When `fuse_method` is `RRF`, `options` supports the following parameters:
48+
49+
`rank_constant`
50+
: Defaults to `60`. Represents the `rank_constant` used in the RRF formula.
51+
52+
`weights`
53+
: Defaults to `{}`. Allows you to set different weights for RRF scores based on `group_column` values. Refer to the [Set custom weights](#set-custom-weights) example.
54+
:::
55+
56+
:::{tab-item} LINEAR
57+
When `fuse_method` is `LINEAR`, `options` supports the following parameters:
58+
59+
`normalizer`
60+
: Defaults to `none`. Can be one of `none` or `minmax`. Specifies which score normalization method to apply.
61+
62+
`weights`
63+
: Defaults to `{}`. Allows you to different weights for scores based on `group_column` values. Refer to the [Set custom weights](#set-custom-weights) example.
64+
:::
65+
::::
66+
67+
`score_column`
68+
: Defaults to `_score`. Designates which column to use to retrieve the relevance scores of the input row
69+
and where to output the new relevance scores of the merged rows.
70+
71+
`group_column`
72+
: Defaults to `_fork`. Designates which column represents the result set.
73+
74+
`key_columns`
75+
: Defaults to `_id, _index`. Rows with matching `key_columns` values are merged.
76+
77+
## Examples
78+
79+
### Use RRF
80+
81+
In the following example, we use the [`FORK`](/reference/query-languages/esql/commands/fork.md) command to run two different queries: a lexical and a semantic query.
82+
We then use `FUSE` to merge the results (applies `RRF` by default):
83+
84+
```esql
85+
FROM books METADATA _id, _index, _score # Include document ID, index name, and relevance score
86+
| FORK (WHERE title:"Shakespeare" | SORT _score DESC) # Fork 1: Lexical search on title field, sorted by relevance score
87+
(WHERE semantic_title:"Shakespeare" | SORT _score DESC) # Fork 2: Semantic search on semantic_title field, sorted by relevance score
88+
| FUSE # Merge results using RRF algorithm by default
89+
```
90+
91+
### Use linear combination
92+
93+
`FUSE` can also use linear score combination:
94+
95+
```esql
96+
FROM books METADATA _id, _index, _score
97+
| FORK (WHERE title:"Shakespeare" | SORT _score DESC) # Fork 1: Lexical search on title
98+
(WHERE semantic_title:"Shakespeare" | SORT _score DESC) # Fork 2: Semantic search on semantic_title
99+
| FUSE LINEAR # Merge results using linear combination of scores (equal weights by default)
100+
```
101+
102+
### Normalize scores
103+
104+
When combining results from semantic and lexical queries through linear combination, we recommend first normalizing the scores from each result set.
105+
106+
The following example uses `minmax` score normalization.
107+
This means the scores normalize and assign values between 0 and 1, before combining the rows:
108+
109+
```esql
110+
FROM books METADATA _id, _index, _score
111+
| FORK (WHERE title:"Shakespeare" | SORT _score DESC) # Fork 1: Lexical search
112+
(WHERE semantic_title:"Shakespeare" | SORT _score DESC) # Fork 2: Semantic search
113+
| FUSE LINEAR WITH { "normalizer": "minmax" } # Linear combination with min-max normalization (scales scores to 0-1 range)
114+
```
115+
116+
### Set custom weights
117+
118+
`FUSE` allows you to specify different weights to scores, based on the `_fork` column values, enabling you to control the relative importance of each query branch in the final results.
119+
120+
```esql
121+
FROM books METADATA _id, _index, _score
122+
| FORK (WHERE title:"Shakespeare" | SORT _score DESC) # Fork 1: Lexical search
123+
(WHERE semantic_title:"Shakespeare" | SORT _score DESC) # Fork 2: Semantic search
124+
| FUSE LINEAR WITH { "weights": { "fork1": 0.7, "fork2": 0.3 }, "normalizer": "minmax" } # Weighted linear combination: 70% lexical, 30% semantic, with min-max normalization
125+
```
126+
127+
## Limitations
128+
129+
These limitations can be present either when:
130+
131+
- `FUSE` is not combined with [`FORK`](/reference/query-languages/esql/commands/fork.md)
132+
- `FUSE` doesn't use the default [metadata](/reference/query-languages/esql/esql-metadata-fields.md) columns `_id`, `_index`, `_score` and `_fork`
133+
134+
1. `FUSE` assumes that `key_columns` are single valued. When `key_columns` are multivalued, `FUSE` can produce unreliable relevance scores.
135+
1. `FUSE` automatically assigns a score value of `NULL` if the `<score_column>` or `<group_column>` are multivalued.
136+
1. `FUSE` assumes that the combination of `key_columns` and `group_column` is unique. If not, `FUSE` can produce unreliable relevance scores.
137+

‎docs/reference/query-languages/esql/_snippets/lists/processing-commands.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
* [`EVAL`](/reference/query-languages/esql/commands/eval.md)
77
* [`GROK`](/reference/query-languages/esql/commands/grok.md)
88
* {applies_to}`stack: preview` {applies_to}`serverless: preview` [`FORK`](/reference/query-languages/esql/commands/fork.md)
9+
* {applies_to}`stack: preview` {applies_to}`serverless: preview` [`FUSE`](/reference/query-languages/esql/commands/fuse.md)
910
* [`KEEP`](/reference/query-languages/esql/commands/keep.md)
1011
* [`LIMIT`](/reference/query-languages/esql/commands/limit.md)
1112
* [`LOOKUP JOIN`](/reference/query-languages/esql/commands/lookup-join.md)

‎docs/reference/query-languages/esql/commands/fuse.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
navigation_title: "FUSE"
3+
mapped_pages:
4+
- https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-fuse
5+
---
6+
7+
# `FUSE` [esql-fuse]
8+
9+
:::{include} ../_snippets/commands/layout/fuse.md
10+
:::

‎docs/reference/query-languages/toc.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@ toc:
107107
- file: esql/commands/enrich.md
108108
- file: esql/commands/eval.md
109109
- file: esql/commands/fork.md
110+
- file: esql/commands/fuse.md
110111
- file: esql/commands/grok.md
111112
- file: esql/commands/inlinestats-by.md
112113
- file: esql/commands/keep.md

0 commit comments

Comments
 (0)