Support "pre-image" for pruning predicate evaluation #18789

sdf-jkl · 2025-11-17T21:43:48Z

Which issue does this PR close?

Closes Support "pre-image" for pruning predicate evaluation #18320.

Rationale for this change

What changes are included in this PR?

Adding a new rule to expr_simplifier library - udf_preimage

This rule performs the optimization for the following operators:

The rule currently supports optimization for date_part function given 'year' literal is passed as the interval parameter.

Are these changes tested?

Tested for all comparison operators above and all possible datatypes in unit tests and sqllogictests.

Are there any user-facing changes?

sdf-jkl · 2025-11-17T21:51:27Z

@alamb This works as is right now, but I like the idea of adding a new method ScalarUDFImpl::simplify_predicate() as a unified api for all scalar functions. We then can move this code in the DatePartFunc implementation of ScalarUDFImpl and create a simplification rule for scalar functions.

The rule will check if there is a scalar function is present in the predicate expression and match the corresponding ScalarUDFImpl::simplify_predicate() to simplify the expression.

alamb

Thanks @sdf-jkl -- this is looking quite cool

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

2010YOUY01

Thank you, this is great 👍🏼

The implementation idea looks good to me. This should be good to go after the end-to-end tests are added, and also have the test coverage double checked.

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

datafusion/optimizer/src/simplify_expressions/unwrap_date_part.rs

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

Co-authored-by: Yongting You <[email protected]>

alamb · 2025-12-09T22:02:51Z

BTW if you want my active notes (aka I tried a few things to refine the API) you can see what I was trying in this commit: alamb@49057e9

…sion into pre-image-support

sdf-jkl

@alamb thanks for your review.

I've addressed your comments and ready for another go. We should be very close to closing this.

sdf-jkl · 2025-12-12T20:37:11Z

datafusion/expr/src/udf.rs

+    pub fn column_expr(&self, args: &[Expr]) -> Option<Expr> {
+        self.inner.column_expr(args)


Added a little helper method to ScalarUDFImpl to extract the inner columnar Expr

sdf-jkl · 2025-12-12T20:39:46Z

datafusion/functions/src/datetime/date_part.rs

    }
 }

+fn date_to_scalar(date: NaiveDate, target_type: &DataType) -> Option<ScalarValue> {


Cleaned this up a little

datafusion/functions/src/datetime/date_part.rs

datafusion/functions/Cargo.toml

sdf-jkl · 2025-12-12T20:43:29Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+                if get_preimage(&left, &right, info)?.0.is_some()
+                    && get_preimage(&left, &right, info)?.1.is_some() =>


Looking forward to if let to be stabilized in match guards

sdf-jkl · 2025-12-12T20:44:37Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+    };
+    Ok((
+        func.preimage(args, right_expr, info)?,
+        func.column_expr(args),


Had to add it here, because we already extract the func, args here.

sdf-jkl · 2025-12-12T20:45:33Z

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

+/// [ClickHouse Paper]:  https://www.vldb.org/pvldb/vol17/p3731-schulze.pdf
+/// [preimage]: https://en.wikipedia.org/wiki/Image_(mathematics)#Inverse_image
+///
+pub(super) fn rewrite_with_preimage(


Added a little more comments

…image-support

sdf-jkl · 2025-12-15T18:00:52Z

Hey @alamb please check when you're available!

alamb · 2025-12-16T12:41:58Z

Thanks -- I'll try and find some time over the next day or two

…image-support

sdf-jkl · 2025-12-19T17:08:26Z

@alamb 👀

…image-support

…r it

sdf-jkl

Updated IsDistinctFrom and IsNotDistinctFrom null handling logic and updated tests.

datafusion/core/tests/optimizer/mod.rs

sdf-jkl · 2025-12-23T16:58:09Z

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

+        //
+        // <expr> is not distinct from x ==> (<expr> is NULL and x is NULL) or ((<expr> >= lower) and (<expr> < upper))
+        // but since x is always not NULL => (<expr> >= lower) and (<expr> < upper)
+        Operator::Eq | Operator::IsNotDistinctFrom => and(


IsNotDistinctFrom acts like Eq, but handles Null differently.
For IsNotDistinctFrom:
Null = Null => True,
While Eq:
Null = Null => False.

Preimage optimization requires an interval to perform this optimization, therefore, rhs can't be Null. We can safely remove the Null handling part.
This makes IsNotDistinctFrom behavior same as Eq

sdf-jkl · 2025-12-23T17:00:25Z

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

+        ),
+        // <expr> is distinct from x ==> (<expr> < lower) or (<expr> >= upper) or (<expr> is NULL and x is not NULL) or (<expr> is not NULL and x is NULL)
+        // but given that x is always not NULL => (<expr> < lower) or (<expr> >= upper) or (<expr> is NULL)
+        Operator::IsDistinctFrom => Expr::BinaryExpr(BinaryExpr {


Same as above, for IsDistinctFrom rhs can't be Null.
We can simplify the Null handling part: (<expr> is NULL and x is not NULL) or (<expr> is not NULL and x is NULL) to (<expr> is NULL).

sdf-jkl · 2025-12-29T15:38:24Z

Hey @alamb, I've made some changes and ready for another go. Please take a look when you have time, thanks!

alamb · 2026-01-09T19:00:09Z

Sorry for the dealy here @sdf-jkl -- I am trying to find time to review this but I have been out and other things came up that were higher priority. I hope to return to this next week

alamb · 2026-01-09T19:00:33Z

(it is hard for me to find time to review 1000 line PRs, unfortunately)

sdf-jkl · 2026-01-09T19:47:47Z

@alamb no worries. Maybe I restructure this into some smaller PRs, so it will be easier to review?

alamb · 2026-01-09T19:59:47Z

@alamb no worries. Maybe I restructure this into some smaller PRs, so it will be easier to review?

That would certainly help me 🙏

sdf-jkl · 2026-01-10T17:54:59Z

@alamb I split it in two:

Support "pre-image" for pruning predicate evaluation #1 #19722
preimage support logic here
Support "pre-image" for pruning predicate evaluation #2 #19733
preimage impl for data_part udf

Hope this will be helpful

sdf-jkl and others added 5 commits November 6, 2025 16:19

create first test

de30cb0

year_literal_to_type_with_op function

052ab95

kinda works

d9a4253

getting_there

83bae02

Merge branch 'apache:main' into pre-image-support

eec27c1

github-actions bot added the optimizer Optimizer rules label Nov 17, 2025

2010YOUY01 self-requested a review November 18, 2025 04:49

sdf-jkl added 3 commits November 19, 2025 11:08

taplo format

1e794c3

add op_swap test

2ec0b9a

Fix comment

630f5c5

alamb reviewed Nov 19, 2025

View reviewed changes

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs Show resolved Hide resolved

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs Outdated Show resolved Hide resolved

commented out attempt for inlist support

fbcfb97

alamb mentioned this pull request Nov 19, 2025

[Epic] Better support for DateTime XiangpengHao/liquid-cache#393

Open

4 tasks

2010YOUY01 reviewed Nov 20, 2025

View reviewed changes

restructure the changes

2b54409

github-actions bot added logical-expr Logical plan and expressions functions Changes to functions implementation labels Nov 20, 2025

sdf-jkl and others added 5 commits November 20, 2025 11:50

Update datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

3260ee9

Co-authored-by: Yongting You <[email protected]>

Merge branch 'main' into pre-image-support

cdfc2f5

Fix merge error

d011b51

cargo fmt plus some fixes

cd390df

Add sqllogictest

36bb529

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Nov 21, 2025

sdf-jkl and others added 5 commits November 21, 2025 13:46

Add other than year date_part tests

7c4dd9c

Fix docs

cdb9cb6

cargo fmt

57ee667

Cargo fmt + doc changes

bb50792

Merge branch 'main' into pre-image-support

fa3b31c

This comment was marked as outdated.

Sign in to view

sdf-jkl added 2 commits December 12, 2025 15:30

Implementing suggestions

e1c358f

Merge branch 'pre-image-support' of https://github.com/sdf-jkl/datafu…

63761a1

…sion into pre-image-support

sdf-jkl commented Dec 12, 2025

View reviewed changes

sdf-jkl added 4 commits December 12, 2025 16:36

fix ci

c79e937

Merge branch 'main' of https://github.com/apache/datafusion into pre-…

0318c62

…image-support

cargo fmt

352b3ff

Move tests to core/optimizer/tests

7dbdc00

github-actions bot added the core Core DataFusion crate label Dec 15, 2025

sdf-jkl requested a review from alamb December 15, 2025 18:05

Merge branch 'main' of https://github.com/apache/datafusion into pre-…

19ba392

…image-support

sdf-jkl added 4 commits December 22, 2025 14:29

Merge branch 'main' of https://github.com/apache/datafusion into pre-…

5456976

…image-support

Merge branch 'main' of https://github.com/apache/datafusion into pre-…

114eb50

…image-support

Improve IsDistinctFrom, IsNotDistinctFrom logic and add unit tests fo…

130413d

…r it

Return sqllogictests + add unit tests

ae05293

sdf-jkl commented Dec 23, 2025

View reviewed changes

alamb mentioned this pull request Jan 5, 2026

Andrew Lamb Weekly-ish Open Source plan - 2026-01-05 #19652

Open

40 tasks

Merge branch 'main' into pre-image-support

bc91039

sdf-jkl marked this pull request as draft January 9, 2026 21:08

This was referenced Jan 9, 2026

Support "pre-image" for pruning predicate evaluation #1 #19722

Open

Support "pre-image" for pruning predicate evaluation #2 #19733

Open

		pub fn column_expr(&self, args: &[Expr]) -> Option<Expr> {
		self.inner.column_expr(args)

		if get_preimage(&left, &right, info)?.0.is_some()
		&& get_preimage(&left, &right, info)?.1.is_some() =>

Support "pre-image" for pruning predicate evaluation #18789

Are you sure you want to change the base?

Support "pre-image" for pruning predicate evaluation #18789

Conversation

sdf-jkl commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

sdf-jkl commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

2010YOUY01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

alamb commented Dec 9, 2025

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sdf-jkl Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Dec 15, 2025

Uh oh!

alamb commented Dec 16, 2025

Uh oh!

sdf-jkl commented Dec 19, 2025

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sdf-jkl Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Dec 29, 2025

Uh oh!

alamb commented Jan 9, 2026

Uh oh!

alamb commented Jan 9, 2026

Uh oh!

sdf-jkl commented Jan 9, 2026

Uh oh!

alamb commented Jan 9, 2026

Uh oh!

sdf-jkl commented Jan 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

sdf-jkl commented Nov 17, 2025 •

edited

Loading

sdf-jkl commented Nov 17, 2025 •

edited

Loading

sdf-jkl commented Jan 10, 2026 •

edited

Loading