[CALCITE-7122] Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP/CEIL/FLOOR #4488

xuzifu666 · 2025-08-03T02:48:18Z

jira: https://issues.apache.org/jira/browse/CALCITE-7122

xiedeyantu · 2025-08-03T13:28:01Z

core/src/main/java/org/apache/calcite/rex/RexSimplify.java

+  /**
+   * Runs simplification unary function by eliminating idempotent.
+   */
+  private RexCall simplifUnaryFunction(RexCall rexCall) {


Maybe we can simplify lower(upper(lower(x))) to upper(lower(x)) or lower(upper(x))?

This is a bit like a special case, because the two functions have opposite effects. Maybe can hand this case, and it will not affect the default behavior.

Apologies, my example was too specific. What I meant is that there are two idempotent functions, f and g, which may have multiple repetitions and can be combined arbitrarily, but will ultimately simplify to either f(g(x)) or g(f(x)). Of course, special cases like the one I mentioned above are exceptions.

Yes，currently this goal can be achieved based on this PR. For example, lower(lower(upper(uper('a')))) will be simplified to lower(upper('a')).

As @julianhyde provided in the case of Jira, these two functions do not have opposite effects, so this case is not need to be simplified.

I think simplifyIdempotentUnaryFunction is more appropriate.

xiedeyantu · 2025-08-03T13:35:08Z

core/src/main/java/org/apache/calcite/rex/RexSimplify.java

      if (e.getClass() == RexCall.class) {
-        return simplifyGenericNode((RexCall) e);
+        RexCall rexCall = (RexCall) e;
+        rexCall = simplifUnaryFunction(rexCall);


Maybe you should keep the same format as before, using case upper/lower/abs.

I was going to do this at first, but it is handled according to SqlKind. The SqlKind of upper/lower/abs is OTHER_FUNCTION. If I need to add a new SqlKind and modify the SqlKind of these functions because of this, it doesn’t feel necessary.
For example, if a new XXX function is added in future, the corresponding SqlKind must also be added. The coupling feels a bit strong, but it is very convenient to put it directly in the List.

julianhyde · 2025-08-03T18:56:46Z

Can we take this discussion to Jira? We need to have a conversation about specification. Those conversations belong in Jira.

Also please STOP logging a jira case just the moment before you push a PR. We know that you have been working on the PR for a few days. Give the rest of us chance to weigh in on the specification before you write the implementation.

Please fix the reference to the jira case. Is it 7122 or 7112?

xuzifu666 · 2025-08-04T01:37:47Z

Can we take this discussion to Jira? We need to have a conversation about specification. Those conversations belong in Jira.

Also please STOP logging a jira case just the moment before you push a PR. We know that you have been working on the PR for a few days. Give the rest of us chance to weigh in on the specification before you write the implementation.

Please fix the reference to the jira case. Is it 7122 or 7112?

OK, thanks for the reminder. I will list the JIRA in advance so that everyone can evaluate it together. @julianhyde

silundong · 2025-08-04T03:21:40Z

core/src/main/java/org/apache/calcite/rex/RexSimplify.java

+  /**
+   * Runs simplification unary function by eliminating idempotent.
+   */
+  private RexCall simplifUnaryFunction(RexCall rexCall) {


I think simplifyIdempotentUnaryFunction is more appropriate.

silundong · 2025-08-04T04:12:27Z

core/src/main/java/org/apache/calcite/rex/RexSimplify.java

  private static final Strong STRONG = new Strong();

+  private static final List<SqlOperator> IDEMOTENT_UNARY_FUNCTIONS =
+      Arrays.asList(SqlStdOperatorTable.UPPER, SqlStdOperatorTable.LOWER, SqlStdOperatorTable.ABS);


ImmutableList is better

I agree with your points and had changed them, thanks @silundong

xiedeyantu · 2025-08-06T03:04:11Z

core/src/test/java/org/apache/calcite/test/RexImplicationCheckerTest.java

+  /** Test case for
+   * <a href="https://issues.apache.org/jira/browse/CALCITE-7122">[CALCITE-7122]
+   * Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP</a>. */
+  @Test void testSimplifyIdempotentUnaryFunctions() {


Can you demonstrate this simplification capability using toSql? For example, if you write select upper(upper(lower(lower('x')))) it can be simplified to upper(lower('x')). This kind of test is not very easy to understand.

In order to facilitate intuitive understanding of the case, I added SQL testing.

No. RexImplicationCheckerTest is a unit test. If you drag in the functionality that goes to and from SQL, it is not longer a unit test.

To make the tests easier to understand, add comments. E.g. "Test that abs(abs(x)) is simplified to abs(x)".

OK, I added more detail comments for these tests and remove sql tests. @julianhyde

core/src/main/java/org/apache/calcite/rex/RexSimplify.java

ILuffZhe

LGTM! Let's see if there are new comments come in the next 2 days.

…UPPER/LOWER/ABS/INITCAP

xiedeyantu · 2025-08-09T06:32:50Z

Can idempotent functions such as ceil and floor also be considered together?

xuzifu666 · 2025-08-09T07:44:28Z

Can idempotent functions such as ceil and floor also be considered together?

The ceil and floor functions have already been separately processed before, and the logic is a little complex, so they should be no need to handle them here.

xiedeyantu · 2025-08-09T08:49:07Z

Can idempotent functions such as ceil and floor also be considered together?

The ceil and floor functions have already been separately processed before, and the logic is a little complex, so they should be no need to handle them here.

I see, these two functions are binary functions. I don't have any other questions for now.

julianhyde · 2025-08-09T19:34:36Z

I see, these two functions are binary functions.

Numeric CEIL and FLOOR are unary. E.g. CEIL(CEIL(3.14)) = CEIL(3.14) = 4.

Datetime CEIL and FLOOR are binary. E.g. FLOOR(FLOOR(DATE '2025-02-08' TO MONTH)) = FLOOR(DATE '2025-02-08' TO MONTH) = DATE '2025-02-01'. The second parameter is a time unit, basically a literal over an enum type, so we could regard the functions as a family of unary functions, e.g. FLOOR_MONTH, CEIL_MONTH, FLOOR_YEAR, etc.

xiedeyantu · 2025-08-10T00:11:06Z

I see, these two functions are binary functions.

Numeric CEIL and FLOOR are unary. E.g. CEIL(CEIL(3.14)) = CEIL(3.14) = 4.

Datetime CEIL and FLOOR are binary. E.g. FLOOR(FLOOR(DATE '2025-02-08' TO MONTH)) = FLOOR(DATE '2025-02-08' TO MONTH) = DATE '2025-02-01'. The second parameter is a time unit, basically a literal over an enum type, so we could regard the functions as a family of unary functions, e.g. FLOOR_MONTH, CEIL_MONTH, FLOOR_YEAR, etc.

I agree with @julianhyde 's idea. I also have a suggestion—though I’m not sure how feasible it is—to consolidate all the currently identified unary idempotent functions into a unified encapsulation while ensuring compatibility with the existing ones in simplify . The goal would be to extract a shared logic layer to avoid scattering similar implementations across multiple places, which could deduce future maintenance costs. Currently, only four functions are handled uniformly, and without careful review, others might mistakenly treat CEIL and FLOOR as non-idempotent or assume they were overlooked. If modifying this proves too difficult, we could at least document all the relevant functions in the comments to prevent future developers from thinking the feature was implemented incompletely.
I’m not entirely sure if this suggestion is valid. If Julian has time, I’d appreciate it if you could take a look as well.

xuzifu666 · 2025-08-10T00:46:22Z

Floor and ceil are not purely unary functions also support binary, and they were previously handled separately, so I did not include them in the logic of the unary function list. They were not accidentally omitted, and I think it is not a problem to handle them uniformly in unary way.

But regarding the scalability mentioned，personally I believe that putting all idempotent functions on a unified list and identifying other types is consistent with the current approach. If there is no clear transformation plan, there is no need to abstract it, which may increase the understanding cost for subsequent developers or let someone with a deep understanding of the plan to come up with a clear plan to follow up separately.

sonarqubecloud · 2025-08-11T19:00:23Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mihaibudiu · 2025-09-24T17:34:28Z

what is the status of this PR? It is marked as "changes requested". Have all change request been addressed?

xuzifu666 · 2025-09-25T09:34:09Z

what is the status of this PR? It is marked as "changes requested". Have all change request been addressed?

IMO it had been addressed all comments now. @mihaibudiu

mihaibudiu · 2025-09-25T16:17:52Z

@xiedeyantu can you please confirm that you are satisfied with these changes?

xiedeyantu · 2025-09-25T22:53:00Z

@xiedeyantu can you please confirm that you are satisfied with these changes?

@mihaibudiu I'm just concerned that the current implementation might be too loose and not conducive to long-term maintenance. What I was trying to express is whether each function itself could have a common property to indicate whether it is an idempotent function. If there were a way to achieve this point, it would also benefit functional dependency analysis later, such as determining whether a function is an injective function. Or perhaps I'm being too critical - logically I don't think there should be any errors in the current approach. If you think the current implementation is fine, I have no objection.

mihaibudiu · 2025-09-25T23:01:25Z

So your suggestion is to add a property to SqlFunction which says whether the function is idempotent?
Frankly, I think there would be too many such properties to create for various interesting semantic properties of functions that could be used by the optimizer.
Moreover, I think that the kinds of programs optimized by this PR never occur in practice, because this is just not how people write programs. So I think this is a lot of work for very little benefit.

xiedeyantu · 2025-09-25T23:12:11Z

So your suggestion is to add a property to SqlFunction which says whether the function is idempotent? Frankly, I think there would be too many such properties to create for various interesting semantic properties of functions that could be used by the optimizer. Moreover, I think that the kinds of programs optimized by this PR never occur in practice, because this is just not how people write programs. So I think this is a lot of work for very little benefit.

If I understand correctly, you also hope that all functions can have some necessary common properties, although not necessarily the idempotent property. If this PR can achieve minimal intrusive changes, I’m also fine with it. However, I also think that the optimization in the current PR is not very cost-effective.

@mihaibudiu I'm considering exploring whether it's feasible to add some useful attributes to functions. At a certain stage, achieving function dependency capabilities requires handling specific functions, but I'm not sure if adding attributes is a worthwhile effort for Calcite.

mihaibudiu · 2025-09-25T23:30:04Z

There are some properties like determinism, behavior with nulls, etc. which are very useful for optimizations.
Idempotence - perhaps less.
I think this should be judged on a case by case basis.

mihaibudiu · 2025-09-25T23:31:09Z

So let's review this once more and merge it if it works; we are spending a lot of time on an optimization which will probably be never applied in practice.

rubenada · 2025-10-08T17:41:49Z

What's the status of this PR? Does it fulfill the scope that was agreed upon?

mihaibudiu · 2025-10-08T17:43:51Z

There is some concern that this is relatively intrusive, making code more complex and hard to maintain, and it does not really solve a practical problem. That's why it wasn't merged yet.

mihaibudiu · 2025-10-08T17:44:38Z

I would be more confident if someone could supply a real example where this would be beneficial. The only case I can imagine is for machine-generated SQL.

xuzifu666 · 2025-10-09T02:26:18Z

I would be more confident if someone could supply a real example where this would be beneficial. The only case I can imagine is for machine-generated SQL.

This should be an optimization for functions, similar to generating more efficient SQL. This idempotency elimination is a relatively obvious SQL optimization, but it is hard to say how much efficiency it can improve. For example, some enterprise SQL platforms will first optimize SQL, requiring the most concise and efficient SQL output, and then send it to the compute engine for execution. I think this optimization is profitable, although not much. @mihaibudiu

mihaibudiu · 2025-10-09T17:06:45Z

Can you show a (non-artificial) program written by some person or machine which would benefit from this optimization?

mihaibudiu · 2025-10-10T22:31:58Z

core/src/main/java/org/apache/calcite/rex/RexInterpreter.java

+    if (values.size() == 1) {
+      if (values.get(0) instanceof BigDecimal) {
+        BigDecimal bigDecimal = (BigDecimal) values.get(0);
+        return bigDecimal.longValue();


I think this is wrong for example for UNSIGNED BIGINT.
You have to keep the value BigDecimal.
But I don't understand what this change has to do with the issue being solved.

mihaibudiu · 2025-10-10T22:32:23Z

core/src/main/java/org/apache/calcite/rex/RexSimplify.java


  private static final Strong STRONG = new Strong();

+  private static final ImmutableList<SqlOperator> IDEMOTENT_UNARY_FUNCTIONS =


typo in "idempotent"

mihaibudiu · 2025-10-10T22:35:33Z

core/src/main/java/org/apache/calcite/rex/RexSimplify.java

+        break;
+      }
+    }
+    return simplifyGenericNode(rexCall);


This action was not invoked in the original code.

mihaibudiu · 2025-10-10T22:36:15Z

core/src/main/java/org/apache/calcite/rex/RexSimplify.java

    if (e.getOperands().size() != 2) {
      // Bail out since we only simplify floor <date>
-      return e;
+      return simplifyIdempotentUnaryFunction(e);


If you call from this point isn't the operator always in the IDEMPOTENT_UNARY_FUNCTIONS?
Is that check still necessary for other paths?

mihaibudiu · 2025-10-10T22:37:19Z

core/src/test/java/org/apache/calcite/test/RexImplicationCheckerTest.java

+   * <a href="https://issues.apache.org/jira/browse/CALCITE-7122">[CALCITE-7122]
+   * Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP/CEIL/FLOOR</a>.
+   * */
+  @Test void testSimplifyIdempotentUnaryFunctions() {


these tests are fine, but a test that goes through RelToSql may be easier to read and write.

xuzifu666 · 2025-11-05T02:19:27Z

I hadn't noticed this information before, but I see that it mentions other architectural design suggestions, so we might need to discuss it further.

github-actions · 2025-12-05T03:48:43Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 90 days if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

xiedeyantu requested changes Aug 3, 2025

View reviewed changes

xuzifu666 changed the title ~~[CALCITE-7112] Eliminate nested calls for idempotent unary functions upper/lower/abs~~ [CALCITE-7122] Eliminate nested calls for idempotent unary functions upper/lower/abs Aug 4, 2025

xuzifu666 force-pushed the idempotents branch from ea67822 to 1a03a23 Compare August 4, 2025 02:06

silundong reviewed Aug 4, 2025

View reviewed changes

xuzifu666 changed the title ~~[CALCITE-7122] Eliminate nested calls for idempotent unary functions upper/lower/abs~~ [CALCITE-7122] Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP Aug 6, 2025

xiedeyantu reviewed Aug 6, 2025

View reviewed changes

ILuffZhe reviewed Aug 6, 2025

View reviewed changes

core/src/main/java/org/apache/calcite/rex/RexSimplify.java Show resolved Hide resolved

core/src/main/java/org/apache/calcite/rex/RexSimplify.java Show resolved Hide resolved

ILuffZhe approved these changes Aug 7, 2025

View reviewed changes

[CALCITE-7122] Eliminate nested calls for idempotent unary functions …

974f5f1

…UPPER/LOWER/ABS/INITCAP

xuzifu666 force-pushed the idempotents branch from d9f753b to 974f5f1 Compare August 8, 2025 02:14

Addressed

32f6847

xuzifu666 changed the title ~~[CALCITE-7122] Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP~~ [CALCITE-7122] Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP/CEIL/FLOOR Aug 11, 2025

mihaibudiu reviewed Oct 10, 2025

View reviewed changes

xiedeyantu mentioned this pull request Oct 12, 2025

[CALCITE-7122] Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP #4577

Closed

github-actions bot added the stale label Dec 5, 2025

github-actions bot removed the stale label Jan 12, 2026


		private static final Strong STRONG = new Strong();

		private static final ImmutableList<SqlOperator> IDEMOTENT_UNARY_FUNCTIONS =

[CALCITE-7122] Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP/CEIL/FLOOR #4488

Are you sure you want to change the base?

[CALCITE-7122] Eliminate nested calls for idempotent unary functions UPPER/LOWER/ABS/INITCAP/CEIL/FLOOR #4488

Uh oh!

Conversation

xuzifu666 commented Aug 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Aug 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Aug 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julianhyde commented Aug 3, 2025

Uh oh!

xuzifu666 commented Aug 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ILuffZhe left a comment

Choose a reason for hiding this comment

Uh oh!

xiedeyantu commented Aug 9, 2025

Uh oh!

xuzifu666 commented Aug 9, 2025

Uh oh!

xiedeyantu commented Aug 9, 2025

Uh oh!

julianhyde commented Aug 9, 2025

Uh oh!

xiedeyantu commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xuzifu666 commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 11, 2025

Quality Gate passed

Uh oh!

mihaibudiu commented Sep 24, 2025

Uh oh!

xuzifu666 commented Sep 25, 2025

Uh oh!

mihaibudiu commented Sep 25, 2025

Uh oh!

xiedeyantu commented Sep 25, 2025

Uh oh!

mihaibudiu commented Sep 25, 2025

xuzifu666 Aug 3, 2025 •

edited

Loading

xuzifu666 Aug 3, 2025 •

edited

Loading

xuzifu666 Aug 4, 2025 •

edited

Loading

xuzifu666 Aug 8, 2025 •

edited

Loading

xiedeyantu commented Aug 10, 2025 •

edited

Loading

xuzifu666 commented Aug 10, 2025 •

edited

Loading

xiedeyantu commented Sep 25, 2025 •

edited

Loading