Skip to content

ESQL: Document LU JOIN/MV_EXPAND not respecting SORT #127718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 7, 2025
Merged

ESQL: Document LU JOIN/MV_EXPAND not respecting SORT #127718

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1d670f4
Document LU JOIN/MV_EXPAND not respecting SORT
alex-spies May 5, 2025
c1cc777
Create implementation details section, move collision, sorting tips t…
leemthompo May 6, 2025
e5643da
fix typo
leemthompo May 6, 2025
87ef558
rename heading, tweak opening sentence, add collision info
leemthompo May 7, 2025
4a30c20
fix link, tweak wording
leemthompo May 7, 2025
cd68742
Update docs/reference/query-languages/esql/esql-lookup-join.md
leemthompo May 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ If multiple documents in the lookup index match a single row in your
results, the output will contain one row for each matching combination.

::::{tip}
In case of name collisions, the newly created columns will override existing columns.
For important information about using `LOOKUP JOIN`, refer to [Implementation details](../../../../esql/esql-lookup-join.md#implementation-details).
::::

**Examples**
Expand Down
6 changes: 6 additions & 0 deletions docs/reference/query-languages/esql/_snippets/commands/layout/mv_expand.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ MV_EXPAND column
`column`
: The multivalued column to expand.

::::{warning}
The output rows produced by `MV_EXPAND` can be in any order and may not respect
preceding `SORT`s. To guarantee a certain ordering, place a `SORT` after any
`MV_EXPAND`s.
::::

**Example**

:::{include} ../examples/mv_expand.csv-spec/simple.md
Expand Down
16 changes: 15 additions & 1 deletion docs/reference/query-languages/esql/esql-lookup-join.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,9 +156,23 @@ To obtain a join key with a compatible type, use a [conversion function](/refere

For a complete list of supported data types and their internal representations, see the [Supported Field Types documentation](/reference/query-languages/esql/limitations.md#_supported_types).

## Implementation details
Copy link
Contributor Author

@alex-spies alex-spies May 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, that's not really an implementation detail, that's more the semantics that the command provides.

Maybe "behavior details" or "semantic details" or "results" or something along these lines?

Copy link
Contributor

@leemthompo leemthompo May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup agreed that was rough placeholder— we'll go with Usage notes for future-proofery :)


This section covers key implementation aspects of `LOOKUP JOIN` that affect query behavior and results. Review these details to ensure your queries work as expected and to troubleshoot unexpected results.

### Handling name collisions

In case of name collisions, the newly created columns will override existing columns.
Copy link
Contributor

@leemthompo leemthompo May 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alex-spies @costin this is pre-existing info unchanged in this PR, but was wondering if there was additional guidance about workarounds we could give for dealing with name collisions?

Copy link
Contributor Author

@alex-spies alex-spies May 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, good idea!

Like this?

Such columns can be preserved by using `RENAME` to assign non-conflicting names before the `LOOKUP JOIN`.

And yes, we realize this is clunky and we want to have a more ergonomic approach to avoid this renaming dance.


### Sorting behavior

The output rows produced by `LOOKUP JOIN` can be in any order and may not
respect preceding `SORT`s. To guarantee a certain ordering, place a `SORT` after
any `LOOKUP JOIN`s.

## Limitations

The following are the current limitations with `LOOKUP JOIN`
The following are the current limitations with `LOOKUP JOIN`:

* Indices in [`lookup` mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting) are always single-sharded.
* Cross cluster search is unsupported initially. Both source and lookup indices must be local.
Expand Down
Loading