Skip to content

Commit 3e3a6b0

Browse files
leemthompoleemthompo
authored
[DOCS][ESQL] Cleanup and cross-reference LOOKUP JOIN reference and landing pages (#127215) (#127232)
* [DOCS][ESQL] Cleanup and cross-reference LOOKUP JOIN reference and landing pages **lookup-join.md (syntax reference)**: - removed tip formatting for simpler direct link to landing page - improved parameter formatting and descriptions - fixed template variable from `{esql}` to `{{esql}}` **esql-lookup-join.md (landing page)**: - added "compare with enrich" section header - simplified "how the command works" with clearer parameter explanation - added code example in how it works section - improved image alt text for accessibility - organized example section with better context and SQL comparison - added dropdown for sample tables to reduce visual clutter - added "query" subheading for clearer organization - included reference to additional examples in command reference - removed excessive whitespace * Improve example, add setup code replaced abstract employee/language example with security monitoring use case added setup instructions for creating test indices included sample data loading via bulk api new practical query example joining firewall logs with threat data simplified results output showing threat detection scenario added note about left-join behavior improved code comments and structure added required index.mode: lookup setting info
1 parent 4868af6 commit 3e3a6b0

File tree

2 files changed

+104
-67
lines changed

2 files changed

+104
-67
lines changed

‎docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ SLA of official GA features.
1111
index, to your {{esql}} query results, simplifying data enrichment
1212
and analysis workflows.
1313

14+
Refer to [the high-level landing page](../../../../esql/esql-lookup-join.md) for an overview of the `LOOKUP JOIN` command, including use cases, prerequisites, and current limitations.
15+
1416
**Syntax**
1517

1618
```esql
@@ -21,18 +23,14 @@ FROM <source_index>
2123
**Parameters**
2224

2325
`<lookup_index>`
24-
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster
25-
references are not supported.
26+
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
2627

2728
`<field_name>`
28-
: The field to join on. This field must exist
29-
in both your current query results and in the lookup index. If the field
30-
contains multi-valued entries, those entries will not match anything
31-
(the added fields will contain `null` for those rows).
29+
: The field to join on. This field must exist in both your current query results and in the lookup index. If the field contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
3230

3331
**Description**
3432

35-
The `LOOKUP JOIN` command adds new columns to your {esql} query
33+
The `LOOKUP JOIN` command adds new columns to your {{esql}} query
3634
results table by finding documents in a lookup index that share the same
3735
join field value as your result rows.
3836

‎docs/reference/query-languages/esql/esql-lookup-join.md‎

Lines changed: 99 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,9 @@ For example, you can use `LOOKUP JOIN` to:
1616
* Quickly see if any source IPs match known malicious addresses.
1717
* Tag logs with the owning team or escalation info for faster triage and incident response.
1818

19-
[`LOOKUP join`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) is similar to [`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich) in the fact that they both help you join data together. You should use `LOOKUP JOIN` when:
19+
## Compare with `ENRICH`
20+
21+
[`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) is similar to [`ENRICH`](/reference/query-languages/esql/commands/processing-commands.md#esql-enrich) in the fact that they both help you join data together. You should use `LOOKUP JOIN` when:
2022

2123
* Your enrichment data changes frequently
2224
* You want to avoid index-time processing
@@ -26,82 +28,119 @@ For example, you can use `LOOKUP JOIN` to:
2628
* You want to restrict users to use only specific lookup indices
2729
* You do not need to match using ranges or spatial relations
2830

29-
## How the `LOOKUP JOIN` command works [esql-how-lookup-join-works]
31+
## How the command works [esql-how-lookup-join-works]
32+
33+
The `LOOKUP JOIN` command adds fields from the lookup index as new columns to your results table based on matching values in the join field.
34+
35+
The command requires two parameters:
36+
- The name of the lookup index (which must have the `lookup` [`index.mode setting`](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting))
37+
- The name of the field to join on
3038

31-
The `LOOKUP JOIN` command adds new columns to a table, with data from {{es}} indices.
39+
```esql
40+
LOOKUP JOIN <lookup_index> ON <field_name>
41+
```
3242

3343
:::{image} ../images/esql-lookup-join.png
34-
:alt: esql lookup join
44+
:alt: Illustration of the `LOOKUP JOIN` command, where the input table is joined with a lookup index to create an enriched output table.
3545
:::
3646

37-
`<lookup_index>`
38-
: The name of the lookup index. This must be a specific index name - wildcards, aliases, and remote cluster references are not supported. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).
39-
40-
`<field_name>`
41-
: The field to join on. This field must exist in both your current query results and in the lookup index. If the field contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows).
47+
If you're familiar with SQL, `LOOKUP JOIN` has left-join behavior. This means that if no rows match in the lookup index, the incoming row is retained and `null`s are added. If many rows in the lookup index match, `LOOKUP JOIN` adds one row per match.
4248

4349
## Example
4450

45-
`LOOKUP JOIN` has left-join behavior. If no rows match in the lookup index, `LOOKUP JOIN` retains the incoming row and adds `null`s. If many rows in the lookup index match, `LOOKUP JOIN` adds one row per match.
46-
47-
In this example, we have two sample tables:
51+
You can run this example for yourself if you'd like to see how it works, by setting up the indices and adding sample data.
52+
53+
### Sample data
54+
:::{dropdown} Expand for setup instructions
55+
56+
**Set up indices**
57+
58+
First let's create two indices with mappings: `threat_list` and `firewall_logs`.
59+
60+
```console
61+
PUT threat_list
62+
{
63+
"settings": {
64+
"index.mode": "lookup" # The lookup index must use this mode
65+
},
66+
"mappings": {
67+
"properties": {
68+
"source.ip": { "type": "ip" },
69+
"threat_level": { "type": "keyword" },
70+
"threat_type": { "type": "keyword" },
71+
"last_updated": { "type": "date" }
72+
}
73+
}
74+
}
75+
```
76+
```console
77+
PUT firewall_logs
78+
{
79+
"mappings": {
80+
"properties": {
81+
"timestamp": { "type": "date" },
82+
"source.ip": { "type": "ip" },
83+
"destination.ip": { "type": "ip" },
84+
"action": { "type": "keyword" },
85+
"bytes_transferred": { "type": "long" }
86+
}
87+
}
88+
}
89+
```
4890

49-
**employees**
91+
**Add sample data**
5092

51-
| birth_date|emp_no|first_name|gender|hire_date|language|
52-
|---|---|---|---|---|---|
53-
|1955-10-04T00:00:00Z|10091|Amabile |M|1992-11-18T00:00:00Z|3|
54-
|1964-10-18T00:00:00Z|10092|Valdiodio |F|1989-09-22T00:00:00Z|1|
55-
|1964-06-11T00:00:00Z|10093|Sailaja |M|1996-11-05T00:00:00Z|3|
56-
|1957-05-25T00:00:00Z|10094|Arumugam |F|1987-04-18T00:00:00Z|5|
57-
|1965-01-03T00:00:00Z|10095|Hilari |M|1986-07-15T00:00:00Z|4|
93+
Next, let's add some sample data to both indices. The `threat_list` index contains known malicious IPs, while the `firewall_logs` index contains logs of network traffic.
5894

59-
**languages_non_unique_key**
95+
```console
96+
POST threat_list/_bulk
97+
{"index":{}}
98+
{"source.ip":"203.0.113.5","threat_level":"high","threat_type":"C2_SERVER","last_updated":"2025-04-22"}
99+
{"index":{}}
100+
{"source.ip":"198.51.100.2","threat_level":"medium","threat_type":"SCANNER","last_updated":"2025-04-23"}
101+
```
60102

61-
|language_code|language_name|country|
62-
|---|---|---|
63-
|1|English|Canada|
64-
|1|English|
65-
|1||United Kingdom|
66-
|1|English|United States of America|
67-
|2|German|[Germany\|Austria]|
68-
|2|German|Switzerland|
69-
|2|German|
70-
|4|Spanish|
71-
|5||France|
72-
|[6\|7]|Mv-Lang|Mv-Land|
73-
|[7\|8]|Mv-Lang2|Mv-Land2|
74-
||Null-Lang|Null-Land|
75-
||Null-Lang2|Null-Land2|
103+
```console
104+
POST firewall_logs/_bulk
105+
{"index":{}}
106+
{"timestamp":"2025-04-23T10:00:01Z","source.ip":"192.0.2.1","destination.ip":"10.0.0.100","action":"allow","bytes_transferred":1024}
107+
{"index":{}}
108+
{"timestamp":"2025-04-23T10:00:05Z","source.ip":"203.0.113.5","destination.ip":"10.0.0.55","action":"allow","bytes_transferred":2048}
109+
{"index":{}}
110+
{"timestamp":"2025-04-23T10:00:08Z","source.ip":"198.51.100.2","destination.ip":"10.0.0.200","action":"block","bytes_transferred":0}
111+
{"index":{}}
112+
{"timestamp":"2025-04-23T10:00:15Z","source.ip":"203.0.113.5","destination.ip":"10.0.0.44","action":"allow","bytes_transferred":4096}
113+
{"index":{}}
114+
{"timestamp":"2025-04-23T10:00:30Z","source.ip":"192.0.2.1","destination.ip":"10.0.0.100","action":"allow","bytes_transferred":512}
115+
```
116+
:::
76117

77-
Running the following query would provide the results shown below.
118+
### Query the data
78119

79120
```esql
80-
FROM employees
81-
| EVAL language_code = emp_no % 10
82-
| LOOKUP JOIN languages_lookup_non_unique_key ON language_code
83-
| WHERE emp_no > 10090 AND emp_no < 10096
84-
| SORT emp_no, country
85-
| KEEP emp_no, language_code, language_name, country;
121+
FROM firewall_logs # The source index
122+
| LOOKUP JOIN threat_list ON source.ip # The lookup index and join field
123+
| WHERE threat_level IS NOT NULL # Filter for rows non-null threat levels
124+
| SORT timestamp # LOOKUP JOIN does not guarantee output order, so you must explicitly sort the results if needed
125+
| KEEP timestamp, source.ip, destination.ip, action, threat_level, threat_type # Keep only relevant fields
126+
| LIMIT 10 # Limit the output to 10 rows
86127
```
87128

88-
|emp_no|language_code|language_name|country|
89-
|---|---|---|---|
90-
| 10091 | 1 | English | Canada|
91-
| 10091 | 1 | null | United Kingdom|
92-
| 10091 | 1 | English | United States of America|
93-
| 10091 | 1 | English | null|
94-
| 10092 | 2 | German | [Germany, Austria]|
95-
| 10092 | 2 | German | Switzerland|
96-
| 10092 | 2 | German | null|
97-
| 10093 | 3 | null | null|
98-
| 10094 | 4 | Spanish | null|
99-
| 10095 | 5 | null | France|
100-
101-
::::{important}
102-
`LOOKUP JOIN` does not guarantee the output to be in any particular order. If a certain order is required, users should use a [`SORT`](/reference/query-languages/esql/commands/processing-commands.md#esql-sort) somewhere after the `LOOKUP JOIN`.
103-
104-
::::
129+
### Response
130+
131+
A successful query will output a table. In this example, you can see that the `source.ip` field from the `firewall_logs` index is matched with the `source.ip` field in the `threat_list` index, and the corresponding `threat_level` and `threat_type` fields are added to the output.
132+
133+
```
134+
source.ip | action | threat_type | threat_level
135+
---------------+---------------+---------------+---------------
136+
203.0.113.5 |allow |C2_SERVER |high
137+
198.51.100.2 |block |SCANNER |medium
138+
203.0.113.5 |allow |C2_SERVER |high
139+
```
140+
141+
### Additional examples
142+
143+
Refer to the examples section of the [`LOOKUP JOIN`](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join) command reference for more examples.
105144

106145
## Prerequisites [esql-lookup-join-prereqs]
107146

0 commit comments

Comments
 (0)