-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Support types table in lookup join docs #130410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
854d292
bad33c1
a4af15e
edf5047
8e470d7
f928275
4620139
138e489
78e10d8
8e554e9
2092a23
724d3d4
6949a31
33401a7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| % This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: this is generated from a different class, actually There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I kind of think the increase in entropy from the fix is not worth it. Perhaps if we have more tests that do this... |
||
|
|
||
| **Supported types** | ||
|
|
||
| | main | join | | ||
alex-spies marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| | --- | --- | | ||
| | boolean | boolean | | ||
| | byte | byte | | ||
| | byte | double | | ||
| | byte | float | | ||
| | byte | half_float | | ||
| | byte | integer | | ||
| | byte | long | | ||
| | byte | scaled_float | | ||
| | byte | short | | ||
| | date | date | | ||
| | date_nanos | date_nanos | | ||
| | double | byte | | ||
| | double | double | | ||
| | double | float | | ||
| | double | half_float | | ||
| | double | integer | | ||
| | double | long | | ||
| | double | scaled_float | | ||
| | double | short | | ||
| | float | byte | | ||
| | float | double | | ||
| | float | float | | ||
| | float | half_float | | ||
| | float | integer | | ||
| | float | long | | ||
| | float | scaled_float | | ||
| | float | short | | ||
| | half_float | byte | | ||
| | half_float | double | | ||
| | half_float | float | | ||
| | half_float | half_float | | ||
| | half_float | integer | | ||
| | half_float | long | | ||
| | half_float | scaled_float | | ||
| | half_float | short | | ||
| | integer | byte | | ||
| | integer | double | | ||
| | integer | float | | ||
| | integer | half_float | | ||
| | integer | integer | | ||
| | integer | long | | ||
| | integer | scaled_float | | ||
| | integer | short | | ||
| | ip | ip | | ||
| | keyword | keyword | | ||
| | long | byte | | ||
| | long | double | | ||
| | long | float | | ||
| | long | half_float | | ||
| | long | integer | | ||
| | long | long | | ||
| | long | scaled_float | | ||
| | long | short | | ||
| | scaled_float | byte | | ||
| | scaled_float | double | | ||
| | scaled_float | float | | ||
| | scaled_float | half_float | | ||
| | scaled_float | integer | | ||
| | scaled_float | long | | ||
| | scaled_float | scaled_float | | ||
| | scaled_float | short | | ||
| | short | byte | | ||
| | short | double | | ||
| | short | float | | ||
| | short | half_float | | ||
| | short | integer | | ||
| | short | long | | ||
| | short | scaled_float | | ||
| | short | short | | ||
| | text | keyword | | ||
|
|
||
| Original file line number | Diff line number | Diff line change | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -151,10 +151,15 @@ | ||||||||||||||||||||||
| * `short` and `byte` are compatible with `integer` (all represented as `int`) | |||||||||||||||||||||||
| * `float`, `half_float`, and `scaled_float` are compatible with `double` (all represented as `double`) | |||||||||||||||||||||||
| * For text fields: You can only use text fields as the join key on the left-hand side of the join and only if they have a `.keyword` subfield | |||||||||||||||||||||||
| * `DATE` and `DATE_NANOS` can only be joined against the exact same type. | |||||||||||||||||||||||
|
|
|||||||||||||||||||||||
| To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed. | |||||||||||||||||||||||
|
|
|||||||||||||||||||||||
| For a complete list of supported data types and their internal representations, see the [Supported Field Types documentation](/reference/query-languages/esql/limitations.md#_supported_types). | |||||||||||||||||||||||
| The list of unsupported fields includes all types not supported by {{esql}} as described in the [Unsupported Field Types documentation](/reference/query-languages/esql/limitations.md#_unsupported_types). | |||||||||||||||||||||||
craigtaverner marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
|||||||||||||||||||||||
| Compatibility group | Types | Notes |
|---|---|---|
| Integer family | byte, short, integer |
All interchangeable |
| Float family | half_float, float, scaled_float, double |
All interchangeable |
| Keyword family | keyword, text.keyword |
Text fields only as join key on left-hand side and must have .keyword subfield |
| Date (Exact) | DATE |
Must match exactly |
| Date Nanos (Exact) | DATE_NANOS |
Must match exactly |
| Boolean | boolean |
Must match exactly |
For a complete list of all types supported in `LOOKUP JOIN`, refer to the [`LOOKUP JOIN` supported types table](/reference/query-languages/esql/commands/processing-commands.md#esql-lookup-join).
To obtain a join key with a compatible type, use a [conversion function](/reference/query-languages/esql/functions-operators/type-conversion-functions.md) if needed.
Unsupported Types
In addition to the {{esql}} unsupported field types, LOOKUP JOIN does not support:
VERSIONUNSIGNED_LONG- Spatial types like
GEO_POINT,GEO_SHAPE - Temporal intervals like
DURATION,PERIOD
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or perhaps we can lose the overview table/list entirely and delegate to the new autogenerated table?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
anyways just food for thought, I think the data type compatibility info should be a standalone section in any case, whether it's under prerequisites or not :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I incorporated your changes. I agree it makes things look more similar with two tables. But I think users might find both places independently, so it is not too bad to have two ways. The generated one is more up-to-date. I did make one fix to your table, because we can join between integers and float now too.
Check failure on line 162 in docs/reference/query-languages/esql/esql-lookup-join.md
GitHub Actions / docs-preview / build
`/reference/query-languages/esql/commands/processing-commands` does not exist. resolved to `/github/workspace/docs/reference/query-languages/esql/commands/processing-commands
craigtaverner marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,6 +19,8 @@ | |
| import org.elasticsearch.xpack.core.esql.action.ColumnInfo; | ||
| import org.elasticsearch.xpack.esql.VerificationException; | ||
| import org.elasticsearch.xpack.esql.core.type.DataType; | ||
| import org.elasticsearch.xpack.esql.expression.function.DocsV3Support; | ||
| import org.elasticsearch.xpack.esql.expression.function.EsqlFunctionRegistry; | ||
| import org.elasticsearch.xpack.esql.plan.logical.join.Join; | ||
| import org.elasticsearch.xpack.esql.plugin.EsqlPlugin; | ||
| import org.elasticsearch.xpack.spatial.SpatialPlugin; | ||
|
|
@@ -36,6 +38,7 @@ | |
| import java.util.Map; | ||
| import java.util.Set; | ||
| import java.util.function.Consumer; | ||
| import java.util.function.Supplier; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| import static org.elasticsearch.test.ESIntegTestCase.Scope.SUITE; | ||
|
|
@@ -265,6 +268,21 @@ private static boolean existingIndex(Collection<TestConfigs> existing, DataType | |
| return existing.stream().anyMatch(c -> c.exists(indexName)); | ||
| } | ||
|
|
||
| public void testOutputSupportedTypes() throws Exception { | ||
alex-spies marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Map<List<DataType>, DataType> signatures = new LinkedHashMap<>(); | ||
| for (TestConfigs configs : testConfigurations.values()) { | ||
| if (configs.group.equals("unsupported") || configs.group.equals("union-types")) { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe we should use static string constants rather than typing out There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good idea, but perhaps out of scope, since this test uses string literals like this in many places and I think they could all be fixed together. Perhaps moved into an enum, and use a switch somewhere to assert none are left out? |
||
| continue; | ||
| } | ||
| for (TestConfig config : configs.configs.values()) { | ||
| if (config instanceof TestConfigPasses) { | ||
| signatures.put(List.of(config.mainType(), config.lookupType()), null); | ||
| } | ||
| } | ||
| } | ||
| saveJoinTypes(() -> signatures); | ||
| } | ||
|
|
||
| public void testLookupJoinStrings() { | ||
| testLookupJoinTypes("strings"); | ||
| } | ||
|
|
@@ -747,4 +765,18 @@ public void doTest() { | |
| private boolean isValidDataType(DataType dataType) { | ||
| return UNDER_CONSTRUCTION.get(dataType) == null || UNDER_CONSTRUCTION.get(dataType).isEnabled(); | ||
| } | ||
|
|
||
| private static void saveJoinTypes(Supplier<Map<List<DataType>, DataType>> signatures) throws Exception { | ||
| ArrayList<EsqlFunctionRegistry.ArgSignature> args = new ArrayList<>(); | ||
| args.add(new EsqlFunctionRegistry.ArgSignature("main", null, "Field from the main index", false, false)); | ||
| args.add(new EsqlFunctionRegistry.ArgSignature("join", null, "Field from the join index", false, false)); | ||
| DocsV3Support.CommandsDocsSupport docs = new DocsV3Support.CommandsDocsSupport( | ||
| "lookup-join", | ||
| LookupJoinTypesIT.class, | ||
| null, | ||
| args, | ||
| signatures | ||
| ); | ||
| docs.renderDocs(); | ||
| } | ||
| } | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: would move that below the examples, this is a bulky little table :)
That looks inconsistent with like/rlike -
but actually, that one has a duplicateSupported Typestable and if we remove the upper one, we'd again have the supported types neatly below the examples!There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that I've made the types table concise, does it look better? Having it above the examples is consistent with all the functions and operators docs, so I had hoped to keep it that way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you're right. I also re-checked like/rlike's type tables in the
WHEREdocumentation, and they are actually consistent as well. Let's keep it.