ESQL: Pushdown Lookup Join past Project #129503

alex-spies · 2025-06-16T18:09:32Z

Closes #119082.

Alternative approach to #127776 that doesn't rely on renaming the lookup fields that LOOKUP JOIN adds.

In general, we're doing this simple transformation:

Join[LEFT][leftKeyField][rightKeyField]
|_Project[leftKeyField, leftOtherField]
| \_...
\_EsRelation[LOOKUP][rightKeyField, rightOtherField]

->

Project[leftKeyField, leftOtherField, rightOtherField]
\_Join[LEFT][leftKeyField][rightKeyField]
  |_...
  \_EsRelation[LOOKUP][rightKeyField, rightOtherField]

The tricky part is dealing with the case where the lookup fields added by LOOKUP JOIN would shadow some attributes if performed after the Project; in this case, we leave Evals in place that assign temporary names to any would-be shadowed attributes. This is the same approach that other pushdown rules take when they push down past an Order (SORT) - see PushDownUtils#pushGeneratingPlanPastProjectAndOrderBy for reference.

Example with shadowing:

Join[LEFT][leftKeyField][rightKeyField]
|_Project[leftKeyField, nameConflictField AS renamedField]
| \_...
\_EsRelation[LOOKUP][rightKeyField, nameConflictField]

->

Project[leftKeyField, $$nameConflictField$temp_name$ AS renamedField, nameConflictField]
\_Join[LEFT][leftKeyField][rightKeyField]
  |_Eval[nameConflictField AS $$nameConflictField$temp_name$]
  |  \_...
  \_EsRelation[LOOKUP][rightKeyField, nameConflictField]

…st-project-take-2

We only add an optimization, it must still work with older nodes.

…st-project-take-2

alex-spies · 2025-06-17T15:58:41Z

This still needs greater test coverage. But I think the productive code is ready to review, so I'll undraft this.

elasticsearchmachine · 2025-06-17T15:58:50Z

Hi @alex-spies, I've created a changelog YAML for you.

elasticsearchmachine · 2025-06-17T15:59:19Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

…st-project-take-2

...n/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownUtils.java

alex-spies · 2025-06-17T16:03:41Z

.../src/test/java/org/elasticsearch/xpack/esql/optimizer/AbstractLogicalPlanOptimizerTests.java

I split out the setup part of the LogicalPlanOptimizerTests so that we can stuff related tests into a dedicated class rather than having one big pool.

bpintea

Nice, LGTM.

.../java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownJoinPastProjectTests.java

.../src/test/java/org/elasticsearch/xpack/esql/optimizer/AbstractLogicalPlanOptimizerTests.java

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/local/EsqlProject.java

...n/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownUtils.java

.../main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownJoinPastProject.java

bpintea · 2025-06-18T17:37:52Z

.../main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownJoinPastProject.java

+
+            for (NamedExpression proj : newProjections) {
+                // TODO: add assert to Project that ensures Alias to attr or pure attr.
+                Attribute coreAttr = (Attribute) (proj instanceof Alias as ? as.child() : proj);


Suggested change

Attribute coreAttr = (Attribute) (proj instanceof Alias as ? as.child() : proj);

Attribute coreAttr = (Attribute) Alias.unwrap(proj);

Actually, even better, maybe skip iterating twice over join's output attributes -- do all in one single loop? You then won't need to test for unwrapped object (here and below).
Edit: and the cast won't be necessary anymore (now it feels like it would deserve a comment as to why it's safe).

The single loop would get rid of the unwrapping and casting, but that makes the handling of shadowing less isolated - which I find harder to explain nicely.

.../main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownJoinPastProject.java

astefan

LGTM

.../main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownJoinPastProject.java

…st-project-take-2

alex-spies · 2025-06-20T16:38:38Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Project.java

    public Project(Source source, LogicalPlan child, List<? extends NamedExpression> projections) {
        super(source, child);
        this.projections = projections;
+        assert validateProjections(projections);


Additional safety measure: there is an unwritten invariant of Projects, let's write it out.

…st-project-take-2

elasticsearchmachine · 2025-06-23T10:07:06Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 129503

alex-spies · 2025-06-23T10:18:01Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

Add a new logical plan optimization: When there is a Project (KEEP/DROP/RENAME/renaming EVALs) in a LOOKUP JOIN's left child (the "main" side), perform the Project after the LOOKUP JOIN. This prevents premature field extractions when the lookup join happens on data nodes. (cherry picked from commit 809dab1) # Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/plan/logical/CommandLicenseTests.java

Add a new logical plan optimization: When there is a Project (KEEP/DROP/RENAME/renaming EVALs) in a LOOKUP JOIN's left child (the "main" side), perform the Project after the LOOKUP JOIN. This prevents premature field extractions when the lookup join happens on data nodes.

alex-spies added 5 commits June 16, 2025 11:59

Add first test + some skaffolding

ab3f000

Improve naming in pushGeneratingPlanPastProjectAndOrderBy

6efc938

Sketch out algorithm

c11e139

Implement algo

c2c401b

Fix first bug, lol

1d830ac

elasticsearchmachine added the v9.1.0 label Jun 16, 2025

alex-spies added 11 commits June 16, 2025 20:24

Fix another bug :)

9f30384

Merge remote-tracking branch 'upstream/main' into pushdown-lu-join-pa…

1ce823a

…st-project-take-2

Remove redundant capability

83978b0

We only add an optimization, it must still work with older nodes.

Add comment to EsqlProject

4610340

Merge remote-tracking branch 'upstream/main' into pushdown-lu-join-pa…

2d6bfd8

…st-project-take-2

Update test expectations

896d406

Do not apply to INLINESTATS

1056e37

Start splitting LogicalPlanOptimizerTests

1dd89d1

Move new test to its own class

ce38db5

Add first optimizer test

ac13587

Add first edge case optimizer tests

21b7bde

alex-spies added >enhancement auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v8.19.0 labels Jun 17, 2025

Update docs/changelog/129503.yaml

a9f3ac3

alex-spies marked this pull request as ready for review June 17, 2025 15:58

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 17, 2025

Merge remote-tracking branch 'upstream/main' into pushdown-lu-join-pa…

4de2fc2

…st-project-take-2

alex-spies changed the title ~~ESQL: Pushdown Lookup Join past Project take 2~~ ESQL: Pushdown Lookup Join past Project Jun 17, 2025

Update 129503.yaml

c04ca94

alex-spies commented Jun 17, 2025

View reviewed changes

...n/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownUtils.java Show resolved Hide resolved

alex-spies commented Jun 17, 2025

View reviewed changes

alex-spies requested review from astefan and bpintea June 17, 2025 16:04

bpintea approved these changes Jun 18, 2025

View reviewed changes

astefan approved these changes Jun 19, 2025

View reviewed changes

.../main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownJoinPastProject.java Show resolved Hide resolved

alex-spies added 9 commits June 20, 2025 15:34

Merge remote-tracking branch 'upstream/main' into pushdown-lu-join-pa…

b42e30a

…st-project-take-2

Implement feedback

ffa8b99

Validation for Project

0512094

Fix visibility

50f864d

Fix validation

3ece38e

Improve readability

d5ba763

Spotless

c8edad4

Add more complex unit test

dc5c91e

More spec tests

1f82f93

alex-spies commented Jun 20, 2025

View reviewed changes

alex-spies added 2 commits June 23, 2025 10:07

Make tests respect Project's invariant

fb13584

Merge remote-tracking branch 'upstream/main' into pushdown-lu-join-pa…

2b02b6f

…st-project-take-2

alex-spies merged commit 809dab1 into elastic:main Jun 23, 2025
32 checks passed

elasticsearchmachine added the backport pending label Jun 23, 2025

alex-spies mentioned this pull request Jun 23, 2025

[8.19] ESQL: Pushdown Lookup Join past Project (#129503) #129842

Merged

alex-spies mentioned this pull request Sep 11, 2025

ESQL: LOOKUP JOIN with expressions #134098

Merged

	Attribute coreAttr = (Attribute) (proj instanceof Alias as ? as.child() : proj);
	Attribute coreAttr = (Attribute) Alias.unwrap(proj);

ESQL: Pushdown Lookup Join past Project #129503

ESQL: Pushdown Lookup Join past Project #129503

Uh oh!

Conversation

alex-spies commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alex-spies commented Jun 17, 2025

Uh oh!

elasticsearchmachine commented Jun 17, 2025

Uh oh!

elasticsearchmachine commented Jun 17, 2025

Uh oh!

Uh oh!

alex-spies Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bpintea Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

bpintea Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-spies Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-spies Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 23, 2025

💔 Backport failed

Uh oh!

alex-spies commented Jun 23, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alex-spies commented Jun 16, 2025 •

edited

Loading

bpintea Jun 18, 2025 •

edited

Loading