ESQL: Make LOOKUP more left-joiny (#118889) (#119475) #120232
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics. Right now if LOOKUP JOIN matches many rows in the `lookup` index we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN. This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH.
This makes `LOOKUP` return multiple rows if there are multiple matches. This is the way SQL works so it's *probably* what folks will expect. Even if it isn't, it allows for more optimizations. Like, this change doesn't optimize anything - it just changes the behavior. But there are optimizations you can do *later* that are transparent when we have *this* behavior, but not with the old behavior. Example: ``` - 2 | [German, German, German] | [Austria, Germany, Switzerland] + 2 | German | [Austria, Germany] + 2 | German | Switzerland + 2 | German | null ``` Relates: elastic#118781
This was referenced Jan 15, 2025
| ROW language_code = [4, 5, 6, 7] | ||
| | LOOKUP JOIN languages_lookup_non_unique_key ON language_code | ||
| | EVAL language_name = MV_SORT(language_name), country = MV_SORT(country) | ||
| | KEEP language_code, language_name, country |
There was a problem hiding this comment.
I just realized that this backport is still pending.
Do you mind adding | SORT language_code, language_name, country here to avoid flakiness with multi-node?
See #120259
idegtiarenko
approved these changes
Jan 16, 2025
|
I've stolen @idegtiarenko's fix in #120259 and am checking it locally against this code. Should be fine. I'll push it in a bit and auto-merge this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is combined backport for two commits, #118889 and #119475. I just accidentally didn't bring back the first one until the second one was ready.
First
This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics.
Right now if LOOKUP JOIN matches many rows in the lookup index we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN.
This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH.
Second
This makes
LOOKUPreturn multiple rows if there are multiple matches. This is the way SQL works so it's probably what folks will expect. Even if it isn't, it allows for more optimizations. Like, this change doesn't optimize anything - it just changes the behavior. But there are optimizations you can do later that are transparent when we have this behavior, but not with the old behavior.Example:
Relates: #118781