-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Support types table in lookup join docs #130410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
854d292
bad33c1
a4af15e
edf5047
8e470d7
f928275
4620139
138e489
78e10d8
8e554e9
2092a23
724d3d4
6949a31
33401a7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| % This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: this is generated from a different class, actually There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I kind of think the increase in entropy from the fix is not worth it. Perhaps if we have more tests that do this... |
||
|
|
||
| **Supported types** | ||
|
|
||
| | field from the left index | field from the lookup index | | ||
| | --- | --- | | ||
| | boolean | boolean | | ||
| | byte | half_float, float, double, scaled_float, byte, short, integer, long | | ||
| | date | date | | ||
| | date_nanos | date_nanos | | ||
| | double | half_float, float, double, scaled_float, byte, short, integer, long | | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, much better now in this format! |
||
| | float | half_float, float, double, scaled_float, byte, short, integer, long | | ||
| | half_float | half_float, float, double, scaled_float, byte, short, integer, long | | ||
| | integer | half_float, float, double, scaled_float, byte, short, integer, long | | ||
| | ip | ip | | ||
| | keyword | keyword | | ||
| | long | half_float, float, double, scaled_float, byte, short, integer, long | | ||
| | scaled_float | half_float, float, double, scaled_float, byte, short, integer, long | | ||
| | short | half_float, float, double, scaled_float, byte, short, integer, long | | ||
| | text | keyword | | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -142,19 +142,38 @@ Refer to the examples section of the [`LOOKUP JOIN`](/reference/query-languages/ | |
|
|
||
| ## Prerequisites [esql-lookup-join-prereqs] | ||
|
|
||
| To use `LOOKUP JOIN`, the following requirements must be met: | ||
| ### Index configuration | ||
|
|
||
| * Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting) | ||
| * **Compatible data types**: The join key and join field in the lookup index must have compatible data types. This means: | ||
| * The data types must either be identical or be internally represented as the same type in {{esql}} | ||
| * Numeric types follow these compatibility rules: | ||
| * `short` and `byte` are compatible with `integer` (all represented as `int`) | ||
| * `float`, `half_float`, and `scaled_float` are compatible with `double` (all represented as `double`) | ||
| * For text fields: You can only use text fields as the join key on the left-hand side of the join and only if they have a `.keyword` subfield | ||
| Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting). | ||
|
|
||
| ### Data type compatibility | ||
|
|
||
| Join keys must have compatible data types between the source and lookup indices. Types within the same compatibility group can be joined together: | ||
|
|
||
| | Compatibility group | Types | Notes | | ||
| |------------------------|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| | ||
| | **Numeric family** | `byte`, `short`, `integer`, `long`, `half_float`, `float`, `scaled_float`, `double` | All compatible | | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. bikeshed: this is nice, but may get stale as it's duplicating the more explicit table. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Definitely, but it's much nicer than list or prose, so let's keep an eye and make sure it doesn't become stale, otherwise we could just render the full table here too and be 100% in sync :) |
||
| | **Keyword family** | `keyword`, `text.keyword` | Text fields only as join key on left-hand side and must have `.keyword` subfield | | ||
| | **Date (Exact)** | `date` | Must match exactly | | ||
| | **Date Nanos (Exact)** | `date_nanos` | Must match exactly | | ||
| | **Boolean** | `boolean` | Must match exactly | | ||
|
|
||
| ```{tip} | ||
| To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed. | ||
| ``` | ||
|
|
||
| For a complete list of supported data types and their internal representations, see the [Supported Field Types documentation](/reference/query-languages/esql/limitations.md#_supported_types). | ||
| ### Unsupported Types | ||
|
|
||
| In addition to the [{{esql}} unsupported field types](/reference/query-languages/esql/limitations.md#_unsupported_types), `LOOKUP JOIN` does not support: | ||
|
|
||
| * `VERSION` | ||
| * `UNSIGNED_LONG` | ||
| * Spatial types like `GEO_POINT`, `GEO_SHAPE` | ||
| * Temporal intervals like `DURATION`, `PERIOD` | ||
|
|
||
| ```{note} | ||
| For a complete list of all types supported in `LOOKUP JOIN`, refer to the [`LOOKUP JOIN` supported types table](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join). | ||
| ``` | ||
|
|
||
| ## Usage notes | ||
|
|
||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,6 +19,8 @@ | |
| import org.elasticsearch.xpack.core.esql.action.ColumnInfo; | ||
| import org.elasticsearch.xpack.esql.VerificationException; | ||
| import org.elasticsearch.xpack.esql.core.type.DataType; | ||
| import org.elasticsearch.xpack.esql.expression.function.DocsV3Support; | ||
| import org.elasticsearch.xpack.esql.expression.function.EsqlFunctionRegistry; | ||
| import org.elasticsearch.xpack.esql.plan.logical.join.Join; | ||
| import org.elasticsearch.xpack.esql.plugin.EsqlPlugin; | ||
| import org.elasticsearch.xpack.spatial.SpatialPlugin; | ||
|
|
@@ -36,6 +38,7 @@ | |
| import java.util.Map; | ||
| import java.util.Set; | ||
| import java.util.function.Consumer; | ||
| import java.util.function.Supplier; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| import static org.elasticsearch.test.ESIntegTestCase.Scope.SUITE; | ||
|
|
@@ -265,6 +268,22 @@ private static boolean existingIndex(Collection<TestConfigs> existing, DataType | |
| return existing.stream().anyMatch(c -> c.exists(indexName)); | ||
| } | ||
|
|
||
| /** This test generates documentation for the supported output types of the lookup join. */ | ||
| public void testOutputSupportedTypes() throws Exception { | ||
alex-spies marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Map<List<DataType>, DataType> signatures = new LinkedHashMap<>(); | ||
| for (TestConfigs configs : testConfigurations.values()) { | ||
| if (configs.group.equals("unsupported") || configs.group.equals("union-types")) { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe we should use static string constants rather than typing out There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good idea, but perhaps out of scope, since this test uses string literals like this in many places and I think they could all be fixed together. Perhaps moved into an enum, and use a switch somewhere to assert none are left out? |
||
| continue; | ||
| } | ||
| for (TestConfig config : configs.configs.values()) { | ||
| if (config instanceof TestConfigPasses) { | ||
| signatures.put(List.of(config.mainType(), config.lookupType()), null); | ||
| } | ||
| } | ||
| } | ||
| saveJoinTypes(() -> signatures); | ||
| } | ||
|
|
||
| public void testLookupJoinStrings() { | ||
| testLookupJoinTypes("strings"); | ||
| } | ||
|
|
@@ -747,4 +766,18 @@ public void doTest() { | |
| private boolean isValidDataType(DataType dataType) { | ||
| return UNDER_CONSTRUCTION.get(dataType) == null || UNDER_CONSTRUCTION.get(dataType).isEnabled(); | ||
| } | ||
|
|
||
| private static void saveJoinTypes(Supplier<Map<List<DataType>, DataType>> signatures) throws Exception { | ||
| ArrayList<EsqlFunctionRegistry.ArgSignature> args = new ArrayList<>(); | ||
| args.add(new EsqlFunctionRegistry.ArgSignature("field from the left index", null, null, false, false)); | ||
| args.add(new EsqlFunctionRegistry.ArgSignature("field from the lookup index", null, null, false, false)); | ||
| DocsV3Support.CommandsDocsSupport docs = new DocsV3Support.CommandsDocsSupport( | ||
| "lookup-join", | ||
| LookupJoinTypesIT.class, | ||
| null, | ||
| args, | ||
| signatures | ||
| ); | ||
| docs.renderDocs(); | ||
| } | ||
| } | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: would move that below the examples, this is a bulky little table :)
That looks inconsistent with like/rlike -
but actually, that one has a duplicateSupported Typestable and if we remove the upper one, we'd again have the supported types neatly below the examples!There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that I've made the types table concise, does it look better? Having it above the examples is consistent with all the functions and operators docs, so I had hoped to keep it that way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you're right. I also re-checked like/rlike's type tables in the
WHEREdocumentation, and they are actually consistent as well. Let's keep it.