Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ results, the output will contain one row for each matching combination.
For important information about using `LOOKUP JOIN`, refer to [Usage notes](../../../../esql/esql-lookup-join.md#usage-notes).
::::

:::{include} ../types/lookup-join.md
Copy link
Contributor

@alex-spies alex-spies Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would move that below the examples, this is a bulky little table :)

That looks inconsistent with like/rlike - but actually, that one has a duplicate Supported Types table and if we remove the upper one, we'd again have the supported types neatly below the examples!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I've made the types table concise, does it look better? Having it above the examples is consistent with all the functions and operators docs, so I had hoped to keep it that way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you're right. I also re-checked like/rlike's type tables in the WHERE documentation, and they are actually consistent as well. Let's keep it.

:::

**Examples**

**IP Threat correlation**: This query would allow you to see if any source
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this is generated from a different class, actually

Copy link
Contributor Author

@craigtaverner craigtaverner Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of think the increase in entropy from the fix is not worth it. Perhaps if we have more tests that do this...


**Supported types**

| field from the left index | field from the lookup index |
| --- | --- |
| boolean | boolean |
| byte | half_float, float, double, scaled_float, byte, short, integer, long |
| date | date |
| date_nanos | date_nanos |
| double | half_float, float, double, scaled_float, byte, short, integer, long |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, much better now in this format!

| float | half_float, float, double, scaled_float, byte, short, integer, long |
| half_float | half_float, float, double, scaled_float, byte, short, integer, long |
| integer | half_float, float, double, scaled_float, byte, short, integer, long |
| ip | ip |
| keyword | keyword |
| long | half_float, float, double, scaled_float, byte, short, integer, long |
| scaled_float | half_float, float, double, scaled_float, byte, short, integer, long |
| short | half_float, float, double, scaled_float, byte, short, integer, long |
| text | keyword |

37 changes: 28 additions & 9 deletions docs/reference/query-languages/esql/esql-lookup-join.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,19 +142,38 @@ Refer to the examples section of the [`LOOKUP JOIN`](/reference/query-languages/

## Prerequisites [esql-lookup-join-prereqs]

To use `LOOKUP JOIN`, the following requirements must be met:
### Index configuration

* Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting)
* **Compatible data types**: The join key and join field in the lookup index must have compatible data types. This means:
* The data types must either be identical or be internally represented as the same type in {{esql}}
* Numeric types follow these compatibility rules:
* `short` and `byte` are compatible with `integer` (all represented as `int`)
* `float`, `half_float`, and `scaled_float` are compatible with `double` (all represented as `double`)
* For text fields: You can only use text fields as the join key on the left-hand side of the join and only if they have a `.keyword` subfield
Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting).

### Data type compatibility

Join keys must have compatible data types between the source and lookup indices. Types within the same compatibility group can be joined together:

| Compatibility group | Types | Notes |
|------------------------|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|
| **Numeric family** | `byte`, `short`, `integer`, `long`, `half_float`, `float`, `scaled_float`, `double` | All compatible |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bikeshed: this is nice, but may get stale as it's duplicating the more explicit table.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely, but it's much nicer than list or prose, so let's keep an eye and make sure it doesn't become stale, otherwise we could just render the full table here too and be 100% in sync :)

| **Keyword family** | `keyword`, `text.keyword` | Text fields only as join key on left-hand side and must have `.keyword` subfield |
| **Date (Exact)** | `date` | Must match exactly |
| **Date Nanos (Exact)** | `date_nanos` | Must match exactly |
| **Boolean** | `boolean` | Must match exactly |

```{tip}
To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed.
```

For a complete list of supported data types and their internal representations, see the [Supported Field Types documentation](/reference/query-languages/esql/limitations.md#_supported_types).
### Unsupported Types

In addition to the [{{esql}} unsupported field types](/reference/query-languages/esql/limitations.md#_unsupported_types), `LOOKUP JOIN` does not support:

* `VERSION`
* `UNSIGNED_LONG`
* Spatial types like `GEO_POINT`, `GEO_SHAPE`
* Temporal intervals like `DURATION`, `PERIOD`

```{note}
For a complete list of all types supported in `LOOKUP JOIN`, refer to the [`LOOKUP JOIN` supported types table](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join).
```

## Usage notes

Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

41 changes: 41 additions & 0 deletions x-pack/plugin/esql/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,47 @@ tasks.named("test").configure {
}
}

// This is similar to the test task above, but needed for the LookupJoinTypesIT which runs in the internalClusterTest task
// and generates a types table for the LOOKUP JOIN command. It is possible in future we might have move tests that do this.
tasks.named("internalClusterTest").configure {
if (buildParams.ci == false) {
systemProperty 'generateDocs', true
def injected = project.objects.newInstance(Injected)
// Define the folder to delete and recreate
def tempDir = file("build/testrun/internalClusterTest/temp/esql")
doFirst {
injected.fs.delete {
it.delete(tempDir)
}
// Re-create this folder so we can save a table of generated examples to extract from csv-spec tests
tempDir.mkdirs() // Recreate the folder
}
File snippetsFolder = file("build/testrun/internalClusterTest/temp/esql/_snippets")
def snippetsDocFolder = file("${rootDir}/docs/reference/query-languages/esql/_snippets")
def snippetsTree = fileTree(snippetsFolder).matching {
include "**/types/*.md" // Recursively include all types/*.md files (effectively counting functions and operators)
}

doLast {
def snippets = snippetsTree.files.collect { it.name }
int countSnippets = snippets.size()
if (countSnippets == 0) {
logger.quiet("ESQL Docs: No function/operator snippets created. Skipping sync.")
} else {
logger.quiet("ESQL Docs: Found $countSnippets generated function/operator snippets to patch into docs")
injected.fs.sync {
from snippetsFolder
into snippetsDocFolder
include '**/*.md'
preserve {
include '**/*.md'
}
}
}
}
}
}

/****************************************************************
* Enable QA/rest integration tests for snapshot builds only *
* TODO: Enable for all builds upon this feature release *
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
import org.elasticsearch.xpack.core.esql.action.ColumnInfo;
import org.elasticsearch.xpack.esql.VerificationException;
import org.elasticsearch.xpack.esql.core.type.DataType;
import org.elasticsearch.xpack.esql.expression.function.DocsV3Support;
import org.elasticsearch.xpack.esql.expression.function.EsqlFunctionRegistry;
import org.elasticsearch.xpack.esql.plan.logical.join.Join;
import org.elasticsearch.xpack.esql.plugin.EsqlPlugin;
import org.elasticsearch.xpack.spatial.SpatialPlugin;
Expand All @@ -36,6 +38,7 @@
import java.util.Map;
import java.util.Set;
import java.util.function.Consumer;
import java.util.function.Supplier;
import java.util.stream.Collectors;

import static org.elasticsearch.test.ESIntegTestCase.Scope.SUITE;
Expand Down Expand Up @@ -265,6 +268,22 @@ private static boolean existingIndex(Collection<TestConfigs> existing, DataType
return existing.stream().anyMatch(c -> c.exists(indexName));
}

/** This test generates documentation for the supported output types of the lookup join. */
public void testOutputSupportedTypes() throws Exception {
Map<List<DataType>, DataType> signatures = new LinkedHashMap<>();
for (TestConfigs configs : testConfigurations.values()) {
if (configs.group.equals("unsupported") || configs.group.equals("union-types")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should use static string constants rather than typing out unsupported and union-types verbatim.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, but perhaps out of scope, since this test uses string literals like this in many places and I think they could all be fixed together. Perhaps moved into an enum, and use a switch somewhere to assert none are left out?

continue;
}
for (TestConfig config : configs.configs.values()) {
if (config instanceof TestConfigPasses) {
signatures.put(List.of(config.mainType(), config.lookupType()), null);
}
}
}
saveJoinTypes(() -> signatures);
}

public void testLookupJoinStrings() {
testLookupJoinTypes("strings");
}
Expand Down Expand Up @@ -747,4 +766,18 @@ public void doTest() {
private boolean isValidDataType(DataType dataType) {
return UNDER_CONSTRUCTION.get(dataType) == null || UNDER_CONSTRUCTION.get(dataType).isEnabled();
}

private static void saveJoinTypes(Supplier<Map<List<DataType>, DataType>> signatures) throws Exception {
ArrayList<EsqlFunctionRegistry.ArgSignature> args = new ArrayList<>();
args.add(new EsqlFunctionRegistry.ArgSignature("field from the left index", null, null, false, false));
args.add(new EsqlFunctionRegistry.ArgSignature("field from the lookup index", null, null, false, false));
DocsV3Support.CommandsDocsSupport docs = new DocsV3Support.CommandsDocsSupport(
"lookup-join",
LookupJoinTypesIT.class,
null,
args,
signatures
);
docs.renderDocs();
}
}
Loading
Loading