[SPARK-47672][SQL] Avoid double eval from filter pushDown w/ projection pushdown#46143

Open

holdenk wants to merge 70 commits intoapache:masterfrom

holdenk:SPARK-47672-avoid-double-eval-from-filter-pushdown-split-projection

Commits on Nov 3, 2025

Start adding a test for relationship pushdown where we don't push down an 'expensive' projected operation (rlike)

holdenk
authored and
sfc-gh-hkarau
committed
Break up the filter pushdown test into two parts with || and && since we can't do pawrtial on the ||

holdenk
authored and
sfc-gh-hkarau
committed
Add an expectedCost to keep track of if elements are expensive or not (idk seems better than the hack of determining if something is eligible for pushdown, but we could also use that maybe? idk) & …

holdenk
authored and
sfc-gh-hkarau
committed
Get the filter push down/not-push down working.

holdenk
authored and
sfc-gh-hkarau
committed
A few more "expensive" type operations we would want to use the project to 'save'

holdenk
authored and
sfc-gh-hkarau
committed
Fix push to do alias replace with a non-empty stay up and non-empty pushDown

holdenk
authored and
sfc-gh-hkarau
committed
Fill out how we do out filter push down to also push the parts of the projection that we are using

holdenk
authored and
sfc-gh-hkarau
committed
Fix up tests, add a bit more coverage, fix the case where there are no aliases

holdenk
authored and
sfc-gh-hkarau
committed
Make projection column ordering stable w/ filter pushdown.

holdenk
authored and
sfc-gh-hkarau
committed
Simplify alias use tracking, simplify non-alias filter pushdown, simplify leftBehindAliases tracking

holdenk
authored and
sfc-gh-hkarau
committed
Remove some printlns & use the heuristic for pushing.

holdenk
authored and
sfc-gh-hkarau
committed
Another attempt at using Advisory filters, but looking at the code we'd have to change PhysicalOperation's type to take Filter[s] instead of Exprs and that's a no-go

holdenk
authored and
sfc-gh-hkarau
committed
Revert "Another attempt at using Advisory filters, but looking at the code we'd have to change PhysicalOperation's type to take Filter[s] instead of Exprs and that's a no-go"

holdenk
authored and
sfc-gh-hkarau
committed
Avoid introducing new filters for \'cheap\' aliases that we're going to push all the way anyways.

holdenk
authored and
sfc-gh-hkarau
committed
Handle expensive only filter

holdenk
authored and
sfc-gh-hkarau
committed
Add an extra test to catch the no cheap refs only expensive refs

holdenk
authored and
sfc-gh-hkarau
committed

Commits on Nov 4, 2025

Make the no double filter eval logic configurable (default to true to restore pre-bug behaviour).

sfc-gh-hkarau
and
holdenk
committed

Commits on Nov 7, 2025

Switch from numeric cost to boolean expensive, add a quick heuristic for regexes.

sfc-gh-hkarau
and
holdenk
committed

Commits on Nov 14, 2025

Commits on Nov 27, 2025

Commits on Dec 10, 2025

Commits on Dec 12, 2025

Commits on Dec 17, 2025

A bit more hacking on cleaning up the refs (still needs more TLC)

sfc-gh-hkarau
and
holdenk
committed

Commits on Dec 18, 2025

Commits on Dec 22, 2025

Commits on Dec 23, 2025

Commits on Dec 29, 2025

Commits on Jan 12, 2026

Commits on Feb 18, 2026

Commits on Feb 20, 2026

Commits on Feb 21, 2026