Skip to content

Commit 7dccd13

Browse files
compheadcomphead
committed
chore: Fallback to Spark for windows
1 parent 481b688 commit 7dccd13

File tree

294 files changed

+14924
-17028
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

294 files changed

+14924
-17028
lines changed

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q12.native_iceberg_compat/explain.txt‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
== Physical Plan ==
2-
TakeOrderedAndProject (24)
3-
+- * Project (23)
4-
+- Window (22)
5-
+- * CometColumnarToRow (21)
2+
* CometColumnarToRow (24)
3+
+- CometTakeOrderedAndProject (23)
4+
+- CometProject (22)
5+
+- CometWindowExec (21)
66
+- CometSort (20)
77
+- CometColumnarExchange (19)
88
+- * HashAggregate (18)
@@ -120,20 +120,20 @@ Arguments: hashpartitioning(i_class#12, 5), ENSURE_REQUIREMENTS, CometColumnarSh
120120
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11]
121121
Arguments: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11], [i_class#12 ASC NULLS FIRST]
122122

123-
(21) CometColumnarToRow [codegen id : 2]
123+
(21) CometWindowExec
124124
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11]
125+
Arguments: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11, _we0#20], [sum(_w0#19) windowspecdefinition(i_class#12, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS _we0#20], [i_class#12]
125126

126-
(22) Window
127-
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11]
128-
Arguments: [sum(_w0#19) windowspecdefinition(i_class#12, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS _we0#20], [i_class#12]
129-
130-
(23) Project [codegen id : 3]
131-
Output [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, ((_w0#19 * 100) / _we0#20) AS revenueratio#21, i_item_id#11]
127+
(22) CometProject
132128
Input [8]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11, _we0#20]
129+
Arguments: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21, i_item_id#11], [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, ((_w0#19 * 100) / _we0#20) AS revenueratio#21, i_item_id#11]
133130

134-
(24) TakeOrderedAndProject
131+
(23) CometTakeOrderedAndProject
135132
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21, i_item_id#11]
136-
Arguments: 100, [i_category#13 ASC NULLS FIRST, i_class#12 ASC NULLS FIRST, i_item_id#11 ASC NULLS FIRST, i_item_desc#7 ASC NULLS FIRST, revenueratio#21 ASC NULLS FIRST], [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21]
133+
Arguments: TakeOrderedAndProject(limit=100, orderBy=[i_category#13 ASC NULLS FIRST,i_class#12 ASC NULLS FIRST,i_item_id#11 ASC NULLS FIRST,i_item_desc#7 ASC NULLS FIRST,revenueratio#21 ASC NULLS FIRST], output=[i_item_desc#7,i_category#13,i_class#12,i_current_price#8,itemrevenue#18,revenueratio#21]), [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21], 100, 0, [i_category#13 ASC NULLS FIRST, i_class#12 ASC NULLS FIRST, i_item_id#11 ASC NULLS FIRST, i_item_desc#7 ASC NULLS FIRST, revenueratio#21 ASC NULLS FIRST], [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21]
134+
135+
(24) CometColumnarToRow [codegen id : 2]
136+
Input [6]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21]
137137

138138
===== Subqueries =====
139139

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q12.native_iceberg_compat/extended.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
TakeOrderedAndProject [COMET: ]
2-
+- Project
3-
+- Window
4-
+- CometColumnarToRow
1+
CometColumnarToRow
2+
+- CometTakeOrderedAndProject
3+
+- CometProject
4+
+- CometWindowExec
55
+- CometSort
66
+- CometColumnarExchange
77
+- HashAggregate [COMET: makedecimal is not supported, Unsupported result expressions found in: List(i_item_desc#1, i_category#2, i_class#3, i_current_price#4, MakeDecimal(sum(UnscaledValue(ws_ext_sales_price#5))#6,17,2) AS itemrevenue#7, MakeDecimal(sum(UnscaledValue(ws_ext_sales_price#5))#6,17,2) AS _w0#8, i_item_id#9)]
@@ -29,4 +29,4 @@
2929
+- CometFilter
3030
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
3131

32-
Comet accelerated 21 out of 27 eligible operators (77%). Final plan contains 3 transitions between Spark and Comet.
32+
Comet accelerated 24 out of 27 eligible operators (88%). Final plan contains 3 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q12.native_iceberg_compat/simplified.txt‎

Lines changed: 36 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,36 @@
1-
TakeOrderedAndProject [i_category,i_class,i_item_id,i_item_desc,revenueratio,i_current_price,itemrevenue]
2-
WholeStageCodegen (3)
3-
Project [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,_we0,i_item_id]
4-
InputAdapter
5-
Window [_w0,i_class]
6-
WholeStageCodegen (2)
7-
CometColumnarToRow
8-
InputAdapter
9-
CometSort [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,i_item_id]
10-
CometColumnarExchange [i_class] #1
11-
WholeStageCodegen (1)
12-
HashAggregate [i_item_id,i_item_desc,i_category,i_class,i_current_price,sum] [sum(UnscaledValue(ws_ext_sales_price)),itemrevenue,_w0,sum]
13-
CometColumnarToRow
14-
InputAdapter
15-
CometExchange [i_item_id,i_item_desc,i_category,i_class,i_current_price] #2
16-
CometHashAggregate [ws_ext_sales_price] [i_item_id,i_item_desc,i_category,i_class,i_current_price,sum]
17-
CometProject [ws_ext_sales_price,i_item_id,i_item_desc,i_current_price,i_class,i_category]
18-
CometBroadcastHashJoin [ws_ext_sales_price,ws_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category,d_date_sk]
19-
CometProject [ws_ext_sales_price,ws_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
20-
CometBroadcastHashJoin [ws_item_sk,ws_ext_sales_price,ws_sold_date_sk,i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
21-
CometFilter [ws_item_sk,ws_ext_sales_price,ws_sold_date_sk]
22-
CometScan [native_iceberg_compat] parquet spark_catalog.default.web_sales [ws_item_sk,ws_ext_sales_price,ws_sold_date_sk]
23-
SubqueryBroadcast [d_date_sk] #1
24-
BroadcastExchange #3
25-
WholeStageCodegen (1)
26-
CometColumnarToRow
27-
InputAdapter
28-
CometProject [d_date_sk]
29-
CometFilter [d_date_sk,d_date]
30-
CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim [d_date_sk,d_date]
31-
CometBroadcastExchange [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category] #4
32-
CometProject [i_item_id,i_class,i_category] [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
33-
CometFilter [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
34-
CometScan [native_iceberg_compat] parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
35-
CometBroadcastExchange [d_date_sk] #5
36-
CometProject [d_date_sk]
37-
CometFilter [d_date_sk,d_date]
38-
CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim [d_date_sk,d_date]
1+
WholeStageCodegen (2)
2+
CometColumnarToRow
3+
InputAdapter
4+
CometTakeOrderedAndProject [i_item_desc,i_category,i_class,i_current_price,itemrevenue,revenueratio,i_item_id]
5+
CometProject [_w0,_we0] [i_item_desc,i_category,i_class,i_current_price,itemrevenue,revenueratio,i_item_id]
6+
CometWindowExec [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,i_item_id,_we0]
7+
CometSort [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,i_item_id]
8+
CometColumnarExchange [i_class] #1
9+
WholeStageCodegen (1)
10+
HashAggregate [i_item_id,i_item_desc,i_category,i_class,i_current_price,sum] [sum(UnscaledValue(ws_ext_sales_price)),itemrevenue,_w0,sum]
11+
CometColumnarToRow
12+
InputAdapter
13+
CometExchange [i_item_id,i_item_desc,i_category,i_class,i_current_price] #2
14+
CometHashAggregate [ws_ext_sales_price] [i_item_id,i_item_desc,i_category,i_class,i_current_price,sum]
15+
CometProject [ws_ext_sales_price,i_item_id,i_item_desc,i_current_price,i_class,i_category]
16+
CometBroadcastHashJoin [ws_ext_sales_price,ws_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category,d_date_sk]
17+
CometProject [ws_ext_sales_price,ws_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
18+
CometBroadcastHashJoin [ws_item_sk,ws_ext_sales_price,ws_sold_date_sk,i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
19+
CometFilter [ws_item_sk,ws_ext_sales_price,ws_sold_date_sk]
20+
CometScan [native_iceberg_compat] parquet spark_catalog.default.web_sales [ws_item_sk,ws_ext_sales_price,ws_sold_date_sk]
21+
SubqueryBroadcast [d_date_sk] #1
22+
BroadcastExchange #3
23+
WholeStageCodegen (1)
24+
CometColumnarToRow
25+
InputAdapter
26+
CometProject [d_date_sk]
27+
CometFilter [d_date_sk,d_date]
28+
CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim [d_date_sk,d_date]
29+
CometBroadcastExchange [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category] #4
30+
CometProject [i_item_id,i_class,i_category] [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
31+
CometFilter [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
32+
CometScan [native_iceberg_compat] parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
33+
CometBroadcastExchange [d_date_sk] #5
34+
CometProject [d_date_sk]
35+
CometFilter [d_date_sk,d_date]
36+
CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim [d_date_sk,d_date]

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q12/explain.txt‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
== Physical Plan ==
2-
TakeOrderedAndProject (24)
3-
+- * Project (23)
4-
+- Window (22)
5-
+- * CometColumnarToRow (21)
2+
* CometColumnarToRow (24)
3+
+- CometTakeOrderedAndProject (23)
4+
+- CometProject (22)
5+
+- CometWindowExec (21)
66
+- CometSort (20)
77
+- CometColumnarExchange (19)
88
+- * HashAggregate (18)
@@ -120,20 +120,20 @@ Arguments: hashpartitioning(i_class#12, 5), ENSURE_REQUIREMENTS, CometColumnarSh
120120
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11]
121121
Arguments: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11], [i_class#12 ASC NULLS FIRST]
122122

123-
(21) CometColumnarToRow [codegen id : 2]
123+
(21) CometWindowExec
124124
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11]
125+
Arguments: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11, _we0#20], [sum(_w0#19) windowspecdefinition(i_class#12, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS _we0#20], [i_class#12]
125126

126-
(22) Window
127-
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11]
128-
Arguments: [sum(_w0#19) windowspecdefinition(i_class#12, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS _we0#20], [i_class#12]
129-
130-
(23) Project [codegen id : 3]
131-
Output [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, ((_w0#19 * 100) / _we0#20) AS revenueratio#21, i_item_id#11]
127+
(22) CometProject
132128
Input [8]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, _w0#19, i_item_id#11, _we0#20]
129+
Arguments: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21, i_item_id#11], [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, ((_w0#19 * 100) / _we0#20) AS revenueratio#21, i_item_id#11]
133130

134-
(24) TakeOrderedAndProject
131+
(23) CometTakeOrderedAndProject
135132
Input [7]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21, i_item_id#11]
136-
Arguments: 100, [i_category#13 ASC NULLS FIRST, i_class#12 ASC NULLS FIRST, i_item_id#11 ASC NULLS FIRST, i_item_desc#7 ASC NULLS FIRST, revenueratio#21 ASC NULLS FIRST], [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21]
133+
Arguments: TakeOrderedAndProject(limit=100, orderBy=[i_category#13 ASC NULLS FIRST,i_class#12 ASC NULLS FIRST,i_item_id#11 ASC NULLS FIRST,i_item_desc#7 ASC NULLS FIRST,revenueratio#21 ASC NULLS FIRST], output=[i_item_desc#7,i_category#13,i_class#12,i_current_price#8,itemrevenue#18,revenueratio#21]), [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21], 100, 0, [i_category#13 ASC NULLS FIRST, i_class#12 ASC NULLS FIRST, i_item_id#11 ASC NULLS FIRST, i_item_desc#7 ASC NULLS FIRST, revenueratio#21 ASC NULLS FIRST], [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21]
134+
135+
(24) CometColumnarToRow [codegen id : 2]
136+
Input [6]: [i_item_desc#7, i_category#13, i_class#12, i_current_price#8, itemrevenue#18, revenueratio#21]
137137

138138
===== Subqueries =====
139139

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q12/extended.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
TakeOrderedAndProject [COMET: ]
2-
+- Project
3-
+- Window
4-
+- CometColumnarToRow
1+
CometColumnarToRow
2+
+- CometTakeOrderedAndProject
3+
+- CometProject
4+
+- CometWindowExec
55
+- CometSort
66
+- CometColumnarExchange
77
+- HashAggregate [COMET: makedecimal is not supported, Unsupported result expressions found in: List(i_item_desc#1, i_category#2, i_class#3, i_current_price#4, MakeDecimal(sum(UnscaledValue(ws_ext_sales_price#5))#6,17,2) AS itemrevenue#7, MakeDecimal(sum(UnscaledValue(ws_ext_sales_price#5))#6,17,2) AS _w0#8, i_item_id#9)]
@@ -29,4 +29,4 @@
2929
+- CometFilter
3030
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
3131

32-
Comet accelerated 21 out of 27 eligible operators (77%). Final plan contains 3 transitions between Spark and Comet.
32+
Comet accelerated 24 out of 27 eligible operators (88%). Final plan contains 3 transitions between Spark and Comet.

0 commit comments

Comments
 (0)