Support types table in lookup join docs #130410

craigtaverner · 2025-07-01T20:36:38Z

The LOOKUP JOIN docs are very brief in their description of supported types, saying only that the fields must be compatible. This PR tries to fix that by:

In the overview of LOOKUP JOIN describing at a high level the supported and unsupported types
In the command reference including a table of types generated from the LookupJoinTypeIT test, and so automatically updated

github-actions · 2025-07-01T20:36:48Z

🔍 Preview links for changed docs:

docs/reference/query-languages/esql/esql-lookup-join.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

elasticsearchmachine · 2025-07-01T20:37:03Z

Pinging @elastic/es-docs (Team:Docs)

elasticsearchmachine · 2025-07-01T20:37:03Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

…s_table

docs/reference/query-languages/esql/esql-lookup-join.md

Co-authored-by: Liam Thompson <[email protected]>

github-actions · 2025-07-02T09:33:39Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/esql-lookup-join.md

leemthompo · 2025-07-02T15:42:40Z

docs/reference/query-languages/esql/esql-lookup-join.md

 To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed.

-For a complete list of supported data types and their internal representations, see the [Supported Field Types documentation](/reference/query-languages/esql/limitations.md#_supported_types).
+The list of unsupported fields includes all types not supported by {{esql}} as described in the [Unsupported Field Types documentation](/reference/query-languages/esql/limitations.md#_unsupported_types).


@craigtaverner I think this section is a little messy now, and technically this should be all part of the bulleted list

I suggest we revamp the Prerequisites section and use subheadings, and instead of prose/list items for the supported/unsupported types, perhaps a table would be a good alternative?

Here's an approach, in docsv3 syntax:

Prerequisites

Index configuration

Indices used for lookups must be configured with the lookup index mode.

Data type compatibility

Join keys must have compatible data types between the source and lookup indices. Types within the same compatibility group can be joined together:

Compatibility group Types Notes

Integer family byte, short, integer All interchangeable

Float family half_float, float, scaled_float, double All interchangeable

Keyword family keyword, text.keyword Text fields only as join key on left-hand side and must have .keyword subfield

Date (Exact) DATE Must match exactly

Date Nanos (Exact) DATE_NANOS Must match exactly

Boolean boolean Must match exactly

For a complete list of all types supported in `LOOKUP JOIN`, refer to the [`LOOKUP JOIN` supported types table](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join).

To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed.

Unsupported Types

In addition to the {{esql}} unsupported field types, LOOKUP JOIN does not support:

VERSION

UNSIGNED_LONG

Spatial types like GEO_POINT, GEO_SHAPE

Temporal intervals like DURATION, PERIOD

Or perhaps we can lose the overview table/list entirely and delegate to the new autogenerated table?

anyways just food for thought, I think the data type compatibility info should be a standalone section in any case, whether it's under prerequisites or not :)

I incorporated your changes. I agree it makes things look more similar with two tables. But I think users might find both places independently, so it is not too bad to have two ways. The generated one is more up-to-date. I did make one fix to your table, because we can join between integers and float now too.

alex-spies

This is neat, thanks @craigtaverner !

Only minor comments from me :)

alex-spies · 2025-07-02T15:24:47Z

docs/reference/query-languages/esql/_snippets/commands/types/lookup-join.md

@@ -0,0 +1,77 @@
+% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.


nit: this is generated from a different class, actually

I kind of think the increase in entropy from the fix is not worth it. Perhaps if we have more tests that do this...

docs/reference/query-languages/esql/esql-lookup-join.md

docs/reference/query-languages/esql/_snippets/commands/types/lookup-join.md

alex-spies · 2025-07-02T15:33:36Z

docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md

 For important information about using `LOOKUP JOIN`, refer to [Usage notes](../../../../esql/esql-lookup-join.md#usage-notes).
 ::::

+:::{include} ../types/lookup-join.md


nit: would move that below the examples, this is a bulky little table :)

That looks inconsistent with like/rlike - ~~but actually, that one has a duplicate Supported Types table and if we remove the upper one, we'd again have the supported types neatly below the examples!~~

Now that I've made the types table concise, does it look better? Having it above the examples is consistent with all the functions and operators docs, so I had hoped to keep it that way.

Yeah, you're right. I also re-checked like/rlike's type tables in the WHERE documentation, and they are actually consistent as well. Let's keep it.

x-pack/plugin/esql/build.gradle

alex-spies · 2025-07-02T15:40:51Z

...esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/LookupJoinTypesIT.java

+    public void testOutputSupportedTypes() throws Exception {
+        Map<List<DataType>, DataType> signatures = new LinkedHashMap<>();
+        for (TestConfigs configs : testConfigurations.values()) {
+            if (configs.group.equals("unsupported") || configs.group.equals("union-types")) {


maybe we should use static string constants rather than typing out unsupported and union-types verbatim.

Good idea, but perhaps out of scope, since this test uses string literals like this in many places and I think they could all be fixed together. Perhaps moved into an enum, and use a switch somewhere to assert none are left out?

...esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/LookupJoinTypesIT.java

Co-authored-by: Alexander Spies <[email protected]>

…er/elasticsearch into esql_lookup_join_types_table

alex-spies

Thanks @craigtaverner !

alex-spies · 2025-07-02T16:52:11Z

docs/reference/query-languages/esql/_snippets/commands/types/lookup-join.md

+| byte | half_float, float, double, scaled_float, byte, short, integer, long |
+| date | date |
+| date_nanos | date_nanos |
+| double | half_float, float, double, scaled_float, byte, short, integer, long |


Oh, much better now in this format!

alex-spies · 2025-07-02T16:55:38Z

docs/reference/query-languages/esql/esql-lookup-join.md

+
+| Compatibility group    | Types                                                                               | Notes                                                                            |
+|------------------------|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|
+| **Numeric family**     | `byte`, `short`, `integer`, `long`, `half_float`, `float`, `scaled_float`, `double` | All compatible                    |


bikeshed: this is nice, but may get stale as it's duplicating the more explicit table.

Definitely, but it's much nicer than list or prose, so let's keep an eye and make sure it doesn't become stale, otherwise we could just render the full table here too and be 100% in sync :)

* Support types table in lookup join docs * Don't show a results column in the join types * Make LOOKUP JOIN types table more compact * Update docs/reference/query-languages/esql/esql-lookup-join.md Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]>

craigtaverner · 2025-07-03T09:45:32Z

Backport to 9.1 in #130537

* Support types table in lookup join docs * Don't show a results column in the join types * Make LOOKUP JOIN types table more compact * Update docs/reference/query-languages/esql/esql-lookup-join.md Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]>

…#130537) * Support types table in lookup join docs (#130410) * Support types table in lookup join docs * Don't show a results column in the join types * Make LOOKUP JOIN types table more compact * Update docs/reference/query-languages/esql/esql-lookup-join.md Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]> * Fix the ESQL docs for the null predicates (#130440) Fix null predicates docs * This also includes moving the note above the examples in the Kibana inline docs. * Add missing knn docs * Refine newlines in kibana docs --------- Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]>

…lastic#130410) (elastic#130537) * Support types table in lookup join docs (elastic#130410) * Support types table in lookup join docs * Don't show a results column in the join types * Make LOOKUP JOIN types table more compact * Update docs/reference/query-languages/esql/esql-lookup-join.md Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]> * Fix the ESQL docs for the null predicates (elastic#130440) Fix null predicates docs * This also includes moving the note above the examples in the Kibana inline docs. * Add missing knn docs * Refine newlines in kibana docs --------- Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]>

craigtaverner · 2025-07-04T08:42:48Z

Backport to 9.0 in #130613

…#130613) * [9.1] Fixes to null-predicates and lookup-join docs (#130440 #130410) (#130537) * Support types table in lookup join docs (#130410) * Support types table in lookup join docs * Don't show a results column in the join types * Make LOOKUP JOIN types table more compact * Update docs/reference/query-languages/esql/esql-lookup-join.md Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]> * Fix the ESQL docs for the null predicates (#130440) Fix null predicates docs * This also includes moving the note above the examples in the Kibana inline docs. * Add missing knn docs * Refine newlines in kibana docs --------- Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]> * Fix Null predicates after cherry-pick * [DOCS] Update ES|QL generated docs to consistently use the `applies_to` metadata (#128576) - Add PREVIEW annotations to all preview functions - Update docs generation logic to use annotations instead of preview boolean * Changed stack: ga 9.1 to stack: coming in multiple places - Match.java: Updated the options parameter description - MultiMatch.java: Updated the options parameter description - ToLower.java: Reformatted parameter description and updated version annotation - ToUpper.java: Removed the appliesTo annotation entirely and reformatted parameter description - updated applies_to annotations to specify both preview 9.0.0 and ga 9.1.0 lifecycle stages - added version-specific documentation examples with applies_to markers for ga 9.1.0 features - enhanced to_lower and to_upper functions to indicate support for multi-valued expressions in ga 9.1.0 - added version guards around function parameters and descriptions using applies_to syntax - updated function parameter descriptions to indicate ga 9.1.0 availability for named parameters - use detailedDescription + and fix match_phrase applies_to syntax - strip inline applies_to from kibana docs - update roundto, matchphrase lifecycles - fix match named params info - various spatial functions are preview - unsigned long is preview - update qstr - Update TO_LOWER/TO_UPPER parameter descriptions for clarity - hide spatial functions per #129839 - added `stack: ga 9.1.0` blocks to match_phrase.md and qstr.md examples - simplified term.md version from `preview 9.0.0` to just `preview` - added `applies_to = "stack: ga 9.1.0"` to matchphrase and querystring java annotations - removed version `9.0.0` from term function annotation - deleted unused `makeCallout()` and `appendLifeCycleAndVersion()` methods Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Nik Everett <[email protected]> * First round cleanup of docs after cherry-picking Includes partial support for the grammer fix to the header comment. * Second round, just replacing `no` with `not` * Fix lookupjointypes test after backport --------- Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Alexander Spies <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Nik Everett <[email protected]>

Support types table in lookup join docs

854d292

craigtaverner requested a review from alex-spies July 1, 2025 20:36

craigtaverner added >docs General docs changes Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL labels Jul 1, 2025

elasticsearchmachine added Team:Docs Meta label for docs team v9.2.0 labels Jul 1, 2025

craigtaverner added 4 commits July 1, 2025 22:52

Don't show a results column in the join types

bad33c1

Merge remote-tracking branch 'origin/main' into esql_lookup_join_type…

a4af15e

…s_table

Newline changes after running tests

edf5047

Fix compile error after merging main

8e470d7

leemthompo reviewed Jul 2, 2025

View reviewed changes

docs/reference/query-languages/esql/esql-lookup-join.md Outdated Show resolved Hide resolved

craigtaverner and others added 2 commits July 2, 2025 11:32

Update docs/reference/query-languages/esql/esql-lookup-join.md

f928275

Co-authored-by: Liam Thompson <[email protected]>

Merge branch 'main' into esql_lookup_join_types_table

4620139

Merge branch 'main' into esql_lookup_join_types_table

138e489

leemthompo reviewed Jul 2, 2025

View reviewed changes

alex-spies approved these changes Jul 2, 2025

View reviewed changes

craigtaverner and others added 6 commits July 2, 2025 17:45

Make LOOKUP JOIN types table more compact

78e10d8

Merge branch 'main' into esql_lookup_join_types_table

8e554e9

Update docs/reference/query-languages/esql/esql-lookup-join.md

2092a23

Co-authored-by: Alexander Spies <[email protected]>

Code review suggestions

724d3d4

Merge branch 'esql_lookup_join_types_table' of github.com:craigtavern…

6949a31

…er/elasticsearch into esql_lookup_join_types_table

Merge branch 'main' into esql_lookup_join_types_table

33401a7

alex-spies approved these changes Jul 2, 2025

View reviewed changes

craigtaverner added auto-backport Automatically create backport pull requests when merged v9.0.4 labels Jul 3, 2025

craigtaverner added the v9.1.1 label Jul 3, 2025

craigtaverner merged commit efd1aaf into elastic:main Jul 3, 2025
32 checks passed

elasticsearchmachine added the backport pending label Jul 3, 2025

craigtaverner mentioned this pull request Jul 3, 2025

[9.1] Fixes to null-predicates and lookup-join docs (#130440 #130410) #130537

Merged

craigtaverner mentioned this pull request Jul 4, 2025

[9.0] Fixes to null-predicates and lookup-join docs (#130440 #130410) #130613

Merged

craigtaverner removed the backport pending label Jul 4, 2025

Compatibility group	Types	Notes
Integer family	`byte`, `short`, `integer`	All interchangeable
Float family	`half_float`, `float`, `scaled_float`, `double`	All interchangeable
Keyword family	`keyword`, `text.keyword`	Text fields only as join key on left-hand side and must have `.keyword` subfield
Date (Exact)	`DATE`	Must match exactly
Date Nanos (Exact)	`DATE_NANOS`	Must match exactly
Boolean	`boolean`	Must match exactly

		@@ -0,0 +1,77 @@
		% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.

Support types table in lookup join docs #130410

Support types table in lookup join docs #130410

Uh oh!

Conversation

craigtaverner commented Jul 1, 2025

Uh oh!

github-actions bot commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 1, 2025

Uh oh!

elasticsearchmachine commented Jul 1, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leemthompo Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Prerequisites

Index configuration

Data type compatibility

Unsupported Types

Uh oh!

leemthompo Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

craigtaverner Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alex-spies Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

craigtaverner commented Jul 3, 2025

Uh oh!

craigtaverner commented Jul 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

github-actions bot commented Jul 1, 2025 •

edited

Loading

github-actions bot commented Jul 2, 2025 •

edited

Loading

leemthompo Jul 2, 2025 •

edited

Loading

leemthompo Jul 2, 2025 •

edited

Loading

craigtaverner Jul 2, 2025 •

edited

Loading

alex-spies Jul 2, 2025 •

edited

Loading