Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ results, the output will contain one row for each matching combination.
For important information about using `LOOKUP JOIN`, refer to [Usage notes](../../../../esql/esql-lookup-join.md#usage-notes).
::::

:::{include} ../types/lookup-join.md
Copy link
Contributor

@alex-spies alex-spies Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would move that below the examples, this is a bulky little table :)

That looks inconsistent with like/rlike - but actually, that one has a duplicate Supported Types table and if we remove the upper one, we'd again have the supported types neatly below the examples!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I've made the types table concise, does it look better? Having it above the examples is consistent with all the functions and operators docs, so I had hoped to keep it that way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you're right. I also re-checked like/rlike's type tables in the WHERE documentation, and they are actually consistent as well. Let's keep it.

:::

**Examples**

**IP Threat correlation**: This query would allow you to see if any source
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this is generated from a different class, actually

Copy link
Contributor Author

@craigtaverner craigtaverner Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of think the increase in entropy from the fix is not worth it. Perhaps if we have more tests that do this...


**Supported types**

| main | join |
| --- | --- |
| boolean | boolean |
| byte | byte |
| byte | double |
| byte | float |
| byte | half_float |
| byte | integer |
| byte | long |
| byte | scaled_float |
| byte | short |
| date | date |
| date_nanos | date_nanos |
| double | byte |
| double | double |
| double | float |
| double | half_float |
| double | integer |
| double | long |
| double | scaled_float |
| double | short |
| float | byte |
| float | double |
| float | float |
| float | half_float |
| float | integer |
| float | long |
| float | scaled_float |
| float | short |
| half_float | byte |
| half_float | double |
| half_float | float |
| half_float | half_float |
| half_float | integer |
| half_float | long |
| half_float | scaled_float |
| half_float | short |
| integer | byte |
| integer | double |
| integer | float |
| integer | half_float |
| integer | integer |
| integer | long |
| integer | scaled_float |
| integer | short |
| ip | ip |
| keyword | keyword |
| long | byte |
| long | double |
| long | float |
| long | half_float |
| long | integer |
| long | long |
| long | scaled_float |
| long | short |
| scaled_float | byte |
| scaled_float | double |
| scaled_float | float |
| scaled_float | half_float |
| scaled_float | integer |
| scaled_float | long |
| scaled_float | scaled_float |
| scaled_float | short |
| short | byte |
| short | double |
| short | float |
| short | half_float |
| short | integer |
| short | long |
| short | scaled_float |
| short | short |
| text | keyword |

7 changes: 6 additions & 1 deletion docs/reference/query-languages/esql/esql-lookup-join.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,10 +151,15 @@
* `short` and `byte` are compatible with `integer` (all represented as `int`)
* `float`, `half_float`, and `scaled_float` are compatible with `double` (all represented as `double`)
* For text fields: You can only use text fields as the join key on the left-hand side of the join and only if they have a `.keyword` subfield
* `DATE` and `DATE_NANOS` can only be joined against the exact same type.

To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed.

For a complete list of supported data types and their internal representations, see the [Supported Field Types documentation](/reference/query-languages/esql/limitations.md#_supported_types).
The list of unsupported fields includes all types not supported by {{esql}} as described in the [Unsupported Field Types documentation](/reference/query-languages/esql/limitations.md#_unsupported_types).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@craigtaverner I think this section is a little messy now, and technically this should be all part of the bulleted list

I suggest we revamp the Prerequisites section and use subheadings, and instead of prose/list items for the supported/unsupported types, perhaps a table would be a good alternative?

Copy link
Contributor

@leemthompo leemthompo Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an approach, in docsv3 syntax:

Prerequisites

Index configuration

Indices used for lookups must be configured with the lookup index mode.

Data type compatibility

Join keys must have compatible data types between the source and lookup indices. Types within the same compatibility group can be joined together:

Compatibility group Types Notes
Integer family byte, short, integer All interchangeable
Float family half_float, float, scaled_float, double All interchangeable
Keyword family keyword, text.keyword Text fields only as join key on left-hand side and must have .keyword subfield
Date (Exact) DATE Must match exactly
Date Nanos (Exact) DATE_NANOS Must match exactly
Boolean boolean Must match exactly
For a complete list of all types supported in `LOOKUP JOIN`, refer to the [`LOOKUP JOIN` supported types table](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join).
To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed.

Unsupported Types

In addition to the {{esql}} unsupported field types, LOOKUP JOIN does not support:

  • VERSION
  • UNSIGNED_LONG
  • Spatial types like GEO_POINT, GEO_SHAPE
  • Temporal intervals like DURATION, PERIOD

Copy link
Contributor

@leemthompo leemthompo Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or perhaps we can lose the overview table/list entirely and delegate to the new autogenerated table?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anyways just food for thought, I think the data type compatibility info should be a standalone section in any case, whether it's under prerequisites or not :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I incorporated your changes. I agree it makes things look more similar with two tables. But I think users might find both places independently, so it is not too bad to have two ways. The generated one is more up-to-date. I did make one fix to your table, because we can join between integers and float now too.

as well as the following: `VERSION`, `UNSIGNED_LONG`, all spatial types like `GEO_POINT`, `GEO_SHAPE`, and all
temporal periods like `DURATION` and `PERIOD`.

For a complete list of all types supported in `LOOKUP JOIN`, refer to the [`LOOKUP JOIN` supported types table](/reference/query-languages/esql/commands/processing-commands#esql-lookup-join).

Check failure on line 162 in docs/reference/query-languages/esql/esql-lookup-join.md

View workflow job for this annotation

GitHub Actions / docs-preview / build

`/reference/query-languages/esql/commands/processing-commands` does not exist. resolved to `/github/workspace/docs/reference/query-languages/esql/commands/processing-commands

## Usage notes

Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

39 changes: 39 additions & 0 deletions x-pack/plugin/esql/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,45 @@ tasks.named("test").configure {
}
}

tasks.named("internalClusterTest").configure {
if (buildParams.ci == false) {
systemProperty 'generateDocs', true
def injected = project.objects.newInstance(Injected)
// Define the folder to delete and recreate
def tempDir = file("build/testrun/internalClusterTest/temp/esql")
doFirst {
injected.fs.delete {
it.delete(tempDir)
}
// Re-create this folder so we can save a table of generated examples to extract from csv-spec tests
tempDir.mkdirs() // Recreate the folder
}
File snippetsFolder = file("build/testrun/internalClusterTest/temp/esql/_snippets")
def snippetsDocFolder = file("${rootDir}/docs/reference/query-languages/esql/_snippets")
def snippetsTree = fileTree(snippetsFolder).matching {
include "**/types/*.md" // Recursively include all types/*.md files (effectively counting functions and operators)
}

doLast {
def snippets = snippetsTree.files.collect { it.name }
int countSnippets = snippets.size()
if (countSnippets == 0) {
logger.quiet("ESQL Docs: No function/operator snippets created. Skipping sync.")
} else {
logger.quiet("ESQL Docs: Found $countSnippets generated function/operator snippets to patch into docs")
injected.fs.sync {
from snippetsFolder
into snippetsDocFolder
include '**/*.md'
preserve {
include '**/*.md'
}
}
}
}
}
}

/****************************************************************
* Enable QA/rest integration tests for snapshot builds only *
* TODO: Enable for all builds upon this feature release *
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
import org.elasticsearch.xpack.core.esql.action.ColumnInfo;
import org.elasticsearch.xpack.esql.VerificationException;
import org.elasticsearch.xpack.esql.core.type.DataType;
import org.elasticsearch.xpack.esql.expression.function.DocsV3Support;
import org.elasticsearch.xpack.esql.expression.function.EsqlFunctionRegistry;
import org.elasticsearch.xpack.esql.plan.logical.join.Join;
import org.elasticsearch.xpack.esql.plugin.EsqlPlugin;
import org.elasticsearch.xpack.spatial.SpatialPlugin;
Expand All @@ -36,6 +38,7 @@
import java.util.Map;
import java.util.Set;
import java.util.function.Consumer;
import java.util.function.Supplier;
import java.util.stream.Collectors;

import static org.elasticsearch.test.ESIntegTestCase.Scope.SUITE;
Expand Down Expand Up @@ -265,6 +268,21 @@ private static boolean existingIndex(Collection<TestConfigs> existing, DataType
return existing.stream().anyMatch(c -> c.exists(indexName));
}

public void testOutputSupportedTypes() throws Exception {
Map<List<DataType>, DataType> signatures = new LinkedHashMap<>();
for (TestConfigs configs : testConfigurations.values()) {
if (configs.group.equals("unsupported") || configs.group.equals("union-types")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should use static string constants rather than typing out unsupported and union-types verbatim.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, but perhaps out of scope, since this test uses string literals like this in many places and I think they could all be fixed together. Perhaps moved into an enum, and use a switch somewhere to assert none are left out?

continue;
}
for (TestConfig config : configs.configs.values()) {
if (config instanceof TestConfigPasses) {
signatures.put(List.of(config.mainType(), config.lookupType()), null);
}
}
}
saveJoinTypes(() -> signatures);
}

public void testLookupJoinStrings() {
testLookupJoinTypes("strings");
}
Expand Down Expand Up @@ -747,4 +765,18 @@ public void doTest() {
private boolean isValidDataType(DataType dataType) {
return UNDER_CONSTRUCTION.get(dataType) == null || UNDER_CONSTRUCTION.get(dataType).isEnabled();
}

private static void saveJoinTypes(Supplier<Map<List<DataType>, DataType>> signatures) throws Exception {
ArrayList<EsqlFunctionRegistry.ArgSignature> args = new ArrayList<>();
args.add(new EsqlFunctionRegistry.ArgSignature("main", null, "Field from the main index", false, false));
args.add(new EsqlFunctionRegistry.ArgSignature("join", null, "Field from the join index", false, false));
DocsV3Support.CommandsDocsSupport docs = new DocsV3Support.CommandsDocsSupport(
"lookup-join",
LookupJoinTypesIT.class,
null,
args,
signatures
);
docs.renderDocs();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.function.Function;
import java.util.function.Supplier;
Expand Down Expand Up @@ -859,9 +860,11 @@ private ObservabilityTier getObservabilityTier() {
/** Command specific docs generating, currently very empty since we only render kibana definition files */
public static class CommandsDocsSupport extends DocsV3Support {
private final LogicalPlan command;
private List<EsqlFunctionRegistry.ArgSignature> args;
private final XPackLicenseState licenseState;
private final ObservabilityTier observabilityTier;

/** Used in CommandLicenseTests to generate Kibana docs with licensing information for commands */
public CommandsDocsSupport(
String name,
Class<?> testClass,
Expand All @@ -875,15 +878,36 @@ public CommandsDocsSupport(
this.observabilityTier = observabilityTier;
}

/** Used in LookupJoinTypesIT to generate table of supported types for join field */
public CommandsDocsSupport(
String name,
Class<?> testClass,
LogicalPlan command,
List<EsqlFunctionRegistry.ArgSignature> args,
Supplier<Map<List<DataType>, DataType>> signatures
) {
super("commands", name, testClass, signatures);
this.command = command;
this.args = args;
this.licenseState = null;
this.observabilityTier = null;
}

@Override
public void renderSignature() throws IOException {
// Unimplemented until we make command docs dynamically generated
}

@Override
public void renderDocs() throws Exception {
// Currently we only render kibana definition files, but we could expand to rendering much more if we decide to
renderKibanaCommandDefinition();
// Currently we only render either signatures or kibana definition files,
// but we could expand to rendering much more if we decide to
if (args != null) {
renderTypes(name, args);
}
if (licenseState != null) {
renderKibanaCommandDefinition();
}
}

void renderKibanaCommandDefinition() throws Exception {
Expand Down Expand Up @@ -927,15 +951,18 @@ void renderParametersList(List<String> argNames, List<String> argDescriptions) t
}

void renderTypes(String name, List<EsqlFunctionRegistry.ArgSignature> args) throws IOException {
boolean showResultColumn = signatures.get().values().stream().anyMatch(Objects::nonNull);
StringBuilder header = new StringBuilder("| ");
StringBuilder separator = new StringBuilder("| ");
List<String> argNames = args.stream().map(EsqlFunctionRegistry.ArgSignature::name).toList();
for (String arg : argNames) {
header.append(arg).append(" | ");
separator.append("---").append(" | ");
}
header.append("result |");
separator.append("--- |");
if (showResultColumn) {
header.append("result |");
separator.append("--- |");
}

List<String> table = new ArrayList<>();
for (Map.Entry<List<DataType>, DataType> sig : this.signatures.get().entrySet()) { // TODO flip to using sortedSignatures
Expand All @@ -945,7 +972,7 @@ void renderTypes(String name, List<EsqlFunctionRegistry.ArgSignature> args) thro
if (sig.getKey().size() > argNames.size()) { // skip variadic [test] cases (but not those with optional parameters)
continue;
}
table.add(getTypeRow(args, sig, argNames));
table.add(getTypeRow(args, sig, argNames, showResultColumn));
}
Collections.sort(table);
if (table.isEmpty()) {
Expand All @@ -964,7 +991,8 @@ void renderTypes(String name, List<EsqlFunctionRegistry.ArgSignature> args) thro
private static String getTypeRow(
List<EsqlFunctionRegistry.ArgSignature> args,
Map.Entry<List<DataType>, DataType> sig,
List<String> argNames
List<String> argNames,
boolean showResultColumn
) {
StringBuilder b = new StringBuilder("| ");
for (int i = 0; i < sig.getKey().size(); i++) {
Expand All @@ -978,8 +1006,10 @@ private static String getTypeRow(
b.append(" | ");
}
b.append("| ".repeat(argNames.size() - sig.getKey().size()));
b.append(sig.getValue().esNameIfPossible());
b.append(" |");
if (showResultColumn) {
b.append(sig.getValue().esNameIfPossible());
b.append(" |");
}
return b.toString();
}

Expand Down
Loading