Skip to content

Commit 71adaa8

Browse files
julian-elasticjulian-elastic
committed
Change documentation
1 parent 6bd2937 commit 71adaa8

File tree

1 file changed

+88
-32
lines changed

1 file changed

+88
-32
lines changed

‎docs/reference/query-languages/esql/esql-lookup-join.md‎

Lines changed: 88 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,41 @@
11
---
22
navigation_title: "Join data with LOOKUP JOIN"
33
mapped_pages:
4-
- https://www.elastic.co/guide/en/elasticsearch/reference/8.18/_lookup_join.html
4+
- https://www.elastic.co/guide/en/elasticsearch/reference/8.18/_lookup_join.html
55
---
66

7-
# Join data from multiple indices with `LOOKUP JOIN` [esql-lookup-join-reference]
7+
# Join data from multiple indices with
88

9-
The {{esql}} [`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) processing command combines data from your {{esql}} query results table with matching records from a specified lookup index. It adds fields from the lookup index as new columns to your results table based on matching values in the join field.
9+
`LOOKUP JOIN` [esql-lookup-join-reference]
1010

11-
Teams often have data scattered across multiple indices – like logs, IPs, user IDs, hosts, employees etc. Without a direct way to enrich or correlate each event with reference data, root-cause analysis, security checks, and operational insights become time-consuming.
11+
The {{esql}} [
12+
`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join)
13+
processing command combines data from your {{esql}} query results table with
14+
matching records from a specified lookup index. It adds fields from the lookup
15+
index as new columns to your results table based on matching values in the join
16+
field.
17+
18+
Teams often have data scattered across multiple indices – like logs, IPs, user
19+
IDs, hosts, employees etc. Without a direct way to enrich or correlate each
20+
event with reference data, root-cause analysis, security checks, and operational
21+
insights become time-consuming.
1222

1323
For example, you can use `LOOKUP JOIN` to:
1424

15-
* Retrieve environment or ownership details for each host to correlate your metrics data.
25+
* Retrieve environment or ownership details for each host to correlate your
26+
metrics data.
1627
* Quickly see if any source IPs match known malicious addresses.
17-
* Tag logs with the owning team or escalation info for faster triage and incident response.
28+
* Tag logs with the owning team or escalation info for faster triage and
29+
incident response.
1830

1931
## Compare with `ENRICH`
2032

21-
[`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) is similar to [`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich) in the fact that they both help you join data together. You should use `LOOKUP JOIN` when:
33+
[
34+
`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join)
35+
is similar to [
36+
`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich)
37+
in the fact that they both help you join data together. You should use
38+
`LOOKUP JOIN` when:
2239

2340
* Your enrichment data changes frequently
2441
* You want to avoid index-time processing
@@ -30,27 +47,36 @@ For example, you can use `LOOKUP JOIN` to:
3047

3148
## How the command works [esql-how-lookup-join-works]
3249

33-
The `LOOKUP JOIN` command adds fields from the lookup index as new columns to your results table based on matching values in the join field.
50+
The `LOOKUP JOIN` command adds fields from the lookup index as new columns to
51+
your results table based on matching values in the join field.
3452

3553
The command requires two parameters:
36-
- The name of the lookup index (which must have the `lookup` [`index.mode setting`](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting))
54+
55+
- The name of the lookup index (which must have the `lookup` [
56+
`index.mode setting`](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting))
3757
- The name of the field to join on
3858

3959
```esql
4060
LOOKUP JOIN <lookup_index> ON <field_name>
4161
```
4262

4363
:::{image} ../images/esql-lookup-join.png
44-
:alt: Illustration of the `LOOKUP JOIN` command, where the input table is joined with a lookup index to create an enriched output table.
64+
:alt: Illustration of the `LOOKUP JOIN` command, where the input table is joined
65+
with a lookup index to create an enriched output table.
4566
:::
4667

47-
If you're familiar with SQL, `LOOKUP JOIN` has left-join behavior. This means that if no rows match in the lookup index, the incoming row is retained and `null`s are added. If many rows in the lookup index match, `LOOKUP JOIN` adds one row per match.
68+
If you're familiar with SQL, `LOOKUP JOIN` has left-join behavior. This means
69+
that if no rows match in the lookup index, the incoming row is retained and
70+
`null`s are added. If many rows in the lookup index match, `LOOKUP JOIN` adds
71+
one row per match.
4872

4973
## Example
5074

51-
You can run this example for yourself if you'd like to see how it works, by setting up the indices and adding sample data.
75+
You can run this example for yourself if you'd like to see how it works, by
76+
setting up the indices and adding sample data.
5277

5378
### Sample data
79+
5480
:::{dropdown} Expand for setup instructions
5581

5682
**Set up indices**
@@ -73,6 +99,7 @@ PUT threat_list
7399
}
74100
}
75101
```
102+
76103
```console
77104
PUT firewall_logs
78105
{
@@ -90,7 +117,9 @@ PUT firewall_logs
90117

91118
**Add sample data**
92119

93-
Next, let's add some sample data to both indices. The `threat_list` index contains known malicious IPs, while the `firewall_logs` index contains logs of network traffic.
120+
Next, let's add some sample data to both indices. The `threat_list` index
121+
contains known malicious IPs, while the `firewall_logs` index contains logs of
122+
network traffic.
94123

95124
```console
96125
POST threat_list/_bulk
@@ -113,6 +142,7 @@ POST firewall_logs/_bulk
113142
{"index":{}}
114143
{"timestamp":"2025-04-23T10:00:30Z","source.ip":"192.0.2.1","destination.ip":"10.0.0.100","action":"allow","bytes_transferred":512}
115144
```
145+
116146
:::
117147

118148
### Query the data
@@ -128,31 +158,38 @@ FROM firewall_logs # The source index
128158

129159
### Response
130160

131-
A successful query will output a table. In this example, you can see that the `source.ip` field from the `firewall_logs` index is matched with the `source.ip` field in the `threat_list` index, and the corresponding `threat_level` and `threat_type` fields are added to the output.
161+
A successful query will output a table. In this example, you can see that the
162+
`source.ip` field from the `firewall_logs` index is matched with the `source.ip`
163+
field in the `threat_list` index, and the corresponding `threat_level` and
164+
`threat_type` fields are added to the output.
132165

133-
|source.ip|action|threat_type|threat_level|
134-
|---|---|---|---|
135-
|203.0.113.5|allow|C2_SERVER|high|
136-
|198.51.100.2|block|SCANNER|medium|
137-
|203.0.113.5|allow|C2_SERVER|high|
166+
| source.ip | action | threat_type | threat_level |
167+
|--------------|--------|-------------|--------------|
168+
| 203.0.113.5 | allow | C2_SERVER | high |
169+
| 198.51.100.2 | block | SCANNER | medium |
170+
| 203.0.113.5 | allow | C2_SERVER | high |
138171

139172
### Additional examples
140173

141-
Refer to the examples section of the [`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) command reference for more examples.
174+
Refer to the examples section of the [
175+
`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join)
176+
command reference for more examples.
142177

143178
## Prerequisites [esql-lookup-join-prereqs]
144179

145180
### Index configuration
146181

147-
Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
182+
Indices used for lookups must be configured with the [
183+
`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
148184

149185
### Data type compatibility
150186

151-
Join keys must have compatible data types between the source and lookup indices. Types within the same compatibility group can be joined together:
187+
Join keys must have compatible data types between the source and lookup indices.
188+
Types within the same compatibility group can be joined together:
152189

153190
| Compatibility group | Types | Notes |
154191
|------------------------|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|
155-
| **Numeric family** | `byte`, `short`, `integer`, `long`, `half_float`, `float`, `scaled_float`, `double` | All compatible |
192+
| **Numeric family** | `byte`, `short`, `integer`, `long`, `half_float`, `float`, `scaled_float`, `double` | All compatible |
156193
| **Keyword family** | `keyword`, `text.keyword` | Text fields only as join key on left-hand side and must have `.keyword` subfield |
157194
| **Date (Exact)** | `date` | Must match exactly |
158195
| **Date Nanos (Exact)** | `date_nanos` | Must match exactly |
@@ -164,7 +201,9 @@ To obtain a join key with a compatible type, use a [conversion function](/refere
164201

165202
### Unsupported Types
166203

167-
In addition to the [{{esql}} unsupported field types](/reference/query-languages/esql/limitations.md#_unsupported_types), `LOOKUP JOIN` does not support:
204+
In addition to
205+
the [{{esql}} unsupported field types](/reference/query-languages/esql/limitations.md#_unsupported_types),
206+
`LOOKUP JOIN` does not support:
168207

169208
* `VERSION`
170209
* `UNSIGNED_LONG`
@@ -177,11 +216,14 @@ For a complete list of all types supported in `LOOKUP JOIN`, refer to the [`LOOK
177216

178217
## Usage notes
179218

180-
This section covers important details about `LOOKUP JOIN` that impact query behavior and results. Review these details to ensure your queries work as expected and to troubleshoot unexpected results.
219+
This section covers important details about `LOOKUP JOIN` that impact query
220+
behavior and results. Review these details to ensure your queries work as
221+
expected and to troubleshoot unexpected results.
181222

182223
### Handling name collisions
183224

184-
When fields from the lookup index match existing column names, the new columns override the existing ones.
225+
When fields from the lookup index match existing column names, the new columns
226+
override the existing ones.
185227
Before the `LOOKUP JOIN` command, preserve columns by either:
186228

187229
* Using `RENAME` to assign non-conflicting names
@@ -197,10 +239,24 @@ any `LOOKUP JOIN`s.
197239

198240
The following are the current limitations with `LOOKUP JOIN`:
199241

200-
* Indices in [`lookup` mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting) are always single-sharded.
201-
* Cross cluster search is unsupported initially. Both source and lookup indices must be local.
242+
* Indices in [
243+
`lookup` mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting)
244+
are always single-sharded.
245+
* Cross cluster search is unsupported initially. Both source and lookup indices
246+
must be local.
202247
* Currently, only matching on equality is supported.
203-
* `LOOKUP JOIN` can only use a single match field and a single index. Wildcards are not supported.
204-
* Aliases, datemath, and datastreams are supported, as long as the index pattern matches a single concrete index {applies_to}`stack: ga 9.1.0`.
205-
* The name of the match field in `LOOKUP JOIN lu_idx ON match_field` must match an existing field in the query. This may require `RENAME`s or `EVAL`s to achieve.
206-
* The query will circuit break if there are too many matching documents in the lookup index, or if the documents are too large. More precisely, `LOOKUP JOIN` works in batches of, normally, about 10,000 rows; a large amount of heap space is needed if the matching documents from the lookup index for a batch are multiple megabytes or larger. This is roughly the same as for `ENRICH`.
248+
* `LOOKUP JOIN` can only use a single match field and a single index. Wildcards
249+
are not supported.
250+
* Aliases, datemath, and datastreams are supported, as long as the index
251+
pattern matches a single concrete index {applies_to}`stack: ga 9.1.0`.
252+
* Limitation on matching on a single field is removed. You can use a
253+
comma separated list of fields in the `ON` clause
254+
{applies_to}`stack: ga 9.2.0`.
255+
* The name of the match field in `LOOKUP JOIN lu_idx ON match_field` must match
256+
an existing field in the query. This may require `RENAME`s or `EVAL`s to
257+
achieve.
258+
* The query will circuit break if there are too many matching documents in the
259+
lookup index, or if the documents are too large. More precisely, `LOOKUP JOIN`
260+
works in batches of, normally, about 10,000 rows; a large amount of heap space
261+
is needed if the matching documents from the lookup index for a batch are
262+
multiple megabytes or larger. This is roughly the same as for `ENRICH`.

0 commit comments

Comments
 (0)