Skip to content

Commit d14c440

Browse files
mbutrovichmbutrovich
committed
- Changed transformUpWithSubqueries -> transformUp to preserve ReusedSubqueryExec.
- New plans.
1 parent b8d46c3 commit d14c440

File tree

656 files changed

+3595
-3526
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

656 files changed

+3595
-3526
lines changed

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -319,7 +319,10 @@ case class CometExecRule(session: SparkSession) extends Rule[SparkPlan] {
319319
* CometBroadcastExchange, enabling broadcast reuse.
320320
*/
321321
private def transformSubqueryBroadcasts(plan: SparkPlan): SparkPlan = {
322-
plan.transformUpWithSubqueries { case p =>
322+
// Use transformUp instead of transformUpWithSubqueries to avoid breaking ReusedSubqueryExec
323+
// references. InSubqueryExec (for DPP) only appears at the top level of filter expressions
324+
// on scans, not nested inside scalar subqueries, so we don't need to recurse into subqueries.
325+
plan.transformUp { case p =>
323326
p.transformExpressions { case sub: InSubqueryExec =>
324327
sub.plan match {
325328
case s: SubqueryBroadcastExec =>

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1.native_datafusion/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@ CometColumnarToRow
1515
: : : :- CometFilter
1616
: : : : +- CometNativeScan parquet spark_catalog.default.store_returns
1717
: : : : +- SubqueryBroadcast
18-
: : : : +- BroadcastExchange
19-
: : : : +- CometColumnarToRow
18+
: : : : +- CometColumnarToRow
19+
: : : : +- CometBroadcastExchange
2020
: : : : +- CometProject
2121
: : : : +- CometFilter
2222
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
@@ -50,4 +50,4 @@ CometColumnarToRow
5050
+- CometFilter
5151
+- CometNativeScan parquet spark_catalog.default.customer
5252

53-
Comet accelerated 46 out of 49 eligible operators (93%). Final plan contains 2 transitions between Spark and Comet.
53+
Comet accelerated 47 out of 49 eligible operators (95%). Final plan contains 2 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1.native_iceberg_compat/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ TakeOrderedAndProject
1616
: : : : +- ColumnarToRow
1717
: : : : +- Scan parquet spark_catalog.default.store_returns [COMET: Dynamic Partition Pruning is not supported with spark.comet.scan.impl=native_iceberg_compat.]
1818
: : : : +- SubqueryBroadcast
19-
: : : : +- BroadcastExchange
20-
: : : : +- CometColumnarToRow
19+
: : : : +- CometColumnarToRow
20+
: : : : +- CometBroadcastExchange
2121
: : : : +- CometProject
2222
: : : : +- CometFilter
2323
: : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
@@ -58,4 +58,4 @@ TakeOrderedAndProject
5858
+- CometFilter
5959
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer
6060

61-
Comet accelerated 18 out of 49 eligible operators (36%). Final plan contains 10 transitions between Spark and Comet.
61+
Comet accelerated 19 out of 49 eligible operators (38%). Final plan contains 10 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q10.native_datafusion/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@ TakeOrderedAndProject
2020
: : : : +- CometBroadcastHashJoin
2121
: : : : :- CometNativeScan parquet spark_catalog.default.store_sales
2222
: : : : : +- SubqueryBroadcast
23-
: : : : : +- BroadcastExchange
24-
: : : : : +- CometColumnarToRow
23+
: : : : : +- CometColumnarToRow
24+
: : : : : +- CometBroadcastExchange
2525
: : : : : +- CometProject
2626
: : : : : +- CometFilter
2727
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
@@ -60,4 +60,4 @@ TakeOrderedAndProject
6060
+- CometFilter
6161
+- CometNativeScan parquet spark_catalog.default.customer_demographics
6262

63-
Comet accelerated 35 out of 54 eligible operators (64%). Final plan contains 7 transitions between Spark and Comet.
63+
Comet accelerated 36 out of 54 eligible operators (66%). Final plan contains 7 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q10.native_iceberg_compat/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@ TakeOrderedAndProject
2121
: : : : :- ColumnarToRow
2222
: : : : : +- Scan parquet spark_catalog.default.store_sales [COMET: Dynamic Partition Pruning is not supported with spark.comet.scan.impl=native_iceberg_compat.]
2323
: : : : : +- SubqueryBroadcast
24-
: : : : : +- BroadcastExchange
25-
: : : : : +- CometColumnarToRow
24+
: : : : : +- CometColumnarToRow
25+
: : : : : +- CometBroadcastExchange
2626
: : : : : +- CometProject
2727
: : : : : +- CometFilter
2828
: : : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
@@ -64,4 +64,4 @@ TakeOrderedAndProject
6464
+- CometFilter
6565
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer_demographics
6666

67-
Comet accelerated 21 out of 54 eligible operators (38%). Final plan contains 11 transitions between Spark and Comet.
67+
Comet accelerated 22 out of 54 eligible operators (40%). Final plan contains 11 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q11.native_datafusion/extended.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@ CometColumnarToRow
2121
: : : : +- CometFilter
2222
: : : : +- CometNativeScan parquet spark_catalog.default.store_sales
2323
: : : : +- SubqueryBroadcast
24-
: : : : +- BroadcastExchange
25-
: : : : +- CometColumnarToRow
24+
: : : : +- CometColumnarToRow
25+
: : : : +- CometBroadcastExchange
2626
: : : : +- CometFilter
2727
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
2828
: : : +- CometBroadcastExchange
@@ -43,8 +43,8 @@ CometColumnarToRow
4343
: : : +- CometFilter
4444
: : : +- CometNativeScan parquet spark_catalog.default.store_sales
4545
: : : +- SubqueryBroadcast
46-
: : : +- BroadcastExchange
47-
: : : +- CometColumnarToRow
46+
: : : +- CometColumnarToRow
47+
: : : +- CometBroadcastExchange
4848
: : : +- CometFilter
4949
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
5050
: : +- CometBroadcastExchange
@@ -88,4 +88,4 @@ CometColumnarToRow
8888
+- CometFilter
8989
+- CometNativeScan parquet spark_catalog.default.date_dim
9090

91-
Comet accelerated 80 out of 86 eligible operators (93%). Final plan contains 3 transitions between Spark and Comet.
91+
Comet accelerated 82 out of 86 eligible operators (95%). Final plan contains 3 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q11.native_iceberg_compat/extended.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,8 @@ TakeOrderedAndProject
2323
: : : : +- ColumnarToRow
2424
: : : : +- Scan parquet spark_catalog.default.store_sales [COMET: Dynamic Partition Pruning is not supported with spark.comet.scan.impl=native_iceberg_compat.]
2525
: : : : +- SubqueryBroadcast
26-
: : : : +- BroadcastExchange
27-
: : : : +- CometColumnarToRow
26+
: : : : +- CometColumnarToRow
27+
: : : : +- CometBroadcastExchange
2828
: : : : +- CometFilter
2929
: : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
3030
: : : +- BroadcastExchange
@@ -49,8 +49,8 @@ TakeOrderedAndProject
4949
: : : +- ColumnarToRow
5050
: : : +- Scan parquet spark_catalog.default.store_sales [COMET: Dynamic Partition Pruning is not supported with spark.comet.scan.impl=native_iceberg_compat.]
5151
: : : +- SubqueryBroadcast
52-
: : : +- BroadcastExchange
53-
: : : +- CometColumnarToRow
52+
: : : +- CometColumnarToRow
53+
: : : +- CometBroadcastExchange
5454
: : : +- CometFilter
5555
: : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
5656
: : +- BroadcastExchange
@@ -103,4 +103,4 @@ TakeOrderedAndProject
103103
+- CometFilter
104104
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
105105

106-
Comet accelerated 28 out of 86 eligible operators (32%). Final plan contains 18 transitions between Spark and Comet.
106+
Comet accelerated 30 out of 86 eligible operators (34%). Final plan contains 18 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q12.native_datafusion/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ TakeOrderedAndProject
1414
: :- CometFilter
1515
: : +- CometNativeScan parquet spark_catalog.default.web_sales
1616
: : +- SubqueryBroadcast
17-
: : +- BroadcastExchange
18-
: : +- CometColumnarToRow
17+
: : +- CometColumnarToRow
18+
: : +- CometBroadcastExchange
1919
: : +- CometProject
2020
: : +- CometFilter
2121
: : +- CometNativeScan parquet spark_catalog.default.date_dim
@@ -28,4 +28,4 @@ TakeOrderedAndProject
2828
+- CometFilter
2929
+- CometNativeScan parquet spark_catalog.default.date_dim
3030

31-
Comet accelerated 22 out of 27 eligible operators (81%). Final plan contains 2 transitions between Spark and Comet.
31+
Comet accelerated 23 out of 27 eligible operators (85%). Final plan contains 2 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q12.native_iceberg_compat/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ TakeOrderedAndProject
1616
: : +- ColumnarToRow
1717
: : +- Scan parquet spark_catalog.default.web_sales [COMET: Dynamic Partition Pruning is not supported with spark.comet.scan.impl=native_iceberg_compat.]
1818
: : +- SubqueryBroadcast
19-
: : +- BroadcastExchange
20-
: : +- CometColumnarToRow
19+
: : +- CometColumnarToRow
20+
: : +- CometBroadcastExchange
2121
: : +- CometProject
2222
: : +- CometFilter
2323
: : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
@@ -32,4 +32,4 @@ TakeOrderedAndProject
3232
+- CometFilter
3333
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
3434

35-
Comet accelerated 12 out of 27 eligible operators (44%). Final plan contains 6 transitions between Spark and Comet.
35+
Comet accelerated 13 out of 27 eligible operators (48%). Final plan contains 6 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q13.native_datafusion/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@ CometColumnarToRow
1515
: : : : :- CometFilter
1616
: : : : : +- CometNativeScan parquet spark_catalog.default.store_sales
1717
: : : : : +- SubqueryBroadcast
18-
: : : : : +- BroadcastExchange
19-
: : : : : +- CometColumnarToRow
18+
: : : : : +- CometColumnarToRow
19+
: : : : : +- CometBroadcastExchange
2020
: : : : : +- CometProject
2121
: : : : : +- CometFilter
2222
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
@@ -39,4 +39,4 @@ CometColumnarToRow
3939
+- CometFilter
4040
+- CometNativeScan parquet spark_catalog.default.household_demographics
4141

42-
Comet accelerated 36 out of 38 eligible operators (94%). Final plan contains 2 transitions between Spark and Comet.
42+
Comet accelerated 37 out of 38 eligible operators (97%). Final plan contains 2 transitions between Spark and Comet.

0 commit comments

Comments
 (0)