Skip to content

Commit 0cac7d9

Browse files
andygroveandygrove
authored
chore: Update stability suite to use auto scan instead of native_comet (#2178)
1 parent 528822a commit 0cac7d9

File tree

812 files changed

+9355
-9360
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

812 files changed

+9355
-9360
lines changed

‎docs/source/contributor-guide/development.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -108,15 +108,15 @@ Note that the output files get written to `$SPARK_HOME`.
108108
The tests can be run with:
109109

110110
```sh
111-
export SPARK_HOME=`pwd` COMET_PARQUET_SCAN_IMPL=native_comet
111+
export SPARK_HOME=`pwd`
112112
./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.4 -nsu test
113113
./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.5 -nsu test
114114
./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-4.0 -nsu test
115115
```
116116

117117
and
118118
```sh
119-
export SPARK_HOME=`pwd` COMET_PARQUET_SCAN_IMPL=native_comet
119+
export SPARK_HOME=`pwd`
120120
./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.4 -nsu test
121121
./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.5 -nsu test
122122
./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-4.0 -nsu test
@@ -126,16 +126,16 @@ If your pull request changes the query plans generated by Comet, you should rege
126126
To regenerate the golden files, you can run the following commands.
127127

128128
```sh
129-
export SPARK_HOME=`pwd` COMET_PARQUET_SCAN_IMPL=native_comet
129+
export SPARK_HOME=`pwd`
130130
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.4 -nsu test
131131
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.5 -nsu test
132132
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-4.0 -nsu test
133133
```
134134

135135
and
136136
```sh
137-
export SPARK_HOME=`pwd` COMET_PARQUET_SCAN_IMPL=native_comet
138-
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -nsu test
137+
export SPARK_HOME=`pwd`
138+
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.4 -nsu test
139139
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.5 -nsu test
140140
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-4.0 -nsu test
141141
```

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1/explain.txt‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,11 @@
1414
: : : +- CometProject (8)
1515
: : : +- CometBroadcastHashJoin (7)
1616
: : : :- CometFilter (2)
17-
: : : : +- CometScan [native_comet] parquet spark_catalog.default.store_returns (1)
17+
: : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns (1)
1818
: : : +- CometBroadcastExchange (6)
1919
: : : +- CometProject (5)
2020
: : : +- CometFilter (4)
21-
: : : +- CometScan [native_comet] parquet spark_catalog.default.date_dim (3)
21+
: : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim (3)
2222
: : +- CometBroadcastExchange (25)
2323
: : +- CometFilter (24)
2424
: : +- CometHashAggregate (23)
@@ -30,19 +30,19 @@
3030
: : +- CometProject (17)
3131
: : +- CometBroadcastHashJoin (16)
3232
: : :- CometFilter (14)
33-
: : : +- CometScan [native_comet] parquet spark_catalog.default.store_returns (13)
33+
: : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns (13)
3434
: : +- ReusedExchange (15)
3535
: +- CometBroadcastExchange (31)
3636
: +- CometProject (30)
3737
: +- CometFilter (29)
38-
: +- CometScan [native_comet] parquet spark_catalog.default.store (28)
38+
: +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store (28)
3939
+- CometBroadcastExchange (37)
4040
+- CometProject (36)
4141
+- CometFilter (35)
42-
+- CometScan [native_comet] parquet spark_catalog.default.customer (34)
42+
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer (34)
4343

4444

45-
(1) CometScan [native_comet] parquet spark_catalog.default.store_returns
45+
(1) CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns
4646
Output [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
4747
Batched: true
4848
Location: InMemoryFileIndex []
@@ -54,7 +54,7 @@ ReadSchema: struct<sr_customer_sk:int,sr_store_sk:int,sr_return_amt:decimal(7,2)
5454
Input [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
5555
Condition : (isnotnull(sr_store_sk#2) AND isnotnull(sr_customer_sk#1))
5656

57-
(3) CometScan [native_comet] parquet spark_catalog.default.date_dim
57+
(3) CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
5858
Output [2]: [d_date_sk#6, d_year#7]
5959
Batched: true
6060
Location [not included in comparison]/{warehouse_dir}/date_dim]
@@ -100,7 +100,7 @@ Functions [1]: [sum(UnscaledValue(sr_return_amt#3))]
100100
Input [3]: [ctr_customer_sk#9, ctr_store_sk#10, ctr_total_return#11]
101101
Condition : isnotnull(ctr_total_return#11)
102102

103-
(13) CometScan [native_comet] parquet spark_catalog.default.store_returns
103+
(13) CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns
104104
Output [4]: [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14, sr_returned_date_sk#15]
105105
Batched: true
106106
Location: InMemoryFileIndex []
@@ -169,7 +169,7 @@ Arguments: [ctr_store_sk#10], [ctr_store_sk#19], Inner, (cast(ctr_total_return#1
169169
Input [5]: [ctr_customer_sk#9, ctr_store_sk#10, ctr_total_return#11, (avg(ctr_total_return) * 1.2)#23, ctr_store_sk#19]
170170
Arguments: [ctr_customer_sk#9, ctr_store_sk#10], [ctr_customer_sk#9, ctr_store_sk#10]
171171

172-
(28) CometScan [native_comet] parquet spark_catalog.default.store
172+
(28) CometScan [native_iceberg_compat] parquet spark_catalog.default.store
173173
Output [2]: [s_store_sk#24, s_state#25]
174174
Batched: true
175175
Location [not included in comparison]/{warehouse_dir}/store]
@@ -197,7 +197,7 @@ Arguments: [ctr_store_sk#10], [s_store_sk#24], Inner, BuildRight
197197
Input [3]: [ctr_customer_sk#9, ctr_store_sk#10, s_store_sk#24]
198198
Arguments: [ctr_customer_sk#9], [ctr_customer_sk#9]
199199

200-
(34) CometScan [native_comet] parquet spark_catalog.default.customer
200+
(34) CometScan [native_iceberg_compat] parquet spark_catalog.default.customer
201201
Output [2]: [c_customer_sk#26, c_customer_id#27]
202202
Batched: true
203203
Location [not included in comparison]/{warehouse_dir}/customer]
@@ -239,10 +239,10 @@ BroadcastExchange (46)
239239
+- * CometColumnarToRow (45)
240240
+- CometProject (44)
241241
+- CometFilter (43)
242-
+- CometScan [native_comet] parquet spark_catalog.default.date_dim (42)
242+
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim (42)
243243

244244

245-
(42) CometScan [native_comet] parquet spark_catalog.default.date_dim
245+
(42) CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
246246
Output [2]: [d_date_sk#6, d_year#7]
247247
Batched: true
248248
Location [not included in comparison]/{warehouse_dir}/date_dim]

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1/simplified.txt‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,19 +15,19 @@ WholeStageCodegen (1)
1515
CometProject [sr_customer_sk,sr_store_sk,sr_return_amt]
1616
CometBroadcastHashJoin [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk,d_date_sk]
1717
CometFilter [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
18-
CometScan [native_comet] parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
18+
CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
1919
SubqueryBroadcast [d_date_sk] #1
2020
BroadcastExchange #2
2121
WholeStageCodegen (1)
2222
CometColumnarToRow
2323
InputAdapter
2424
CometProject [d_date_sk]
2525
CometFilter [d_date_sk,d_year]
26-
CometScan [native_comet] parquet spark_catalog.default.date_dim [d_date_sk,d_year]
26+
CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim [d_date_sk,d_year]
2727
CometBroadcastExchange [d_date_sk] #3
2828
CometProject [d_date_sk]
2929
CometFilter [d_date_sk,d_year]
30-
CometScan [native_comet] parquet spark_catalog.default.date_dim [d_date_sk,d_year]
30+
CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim [d_date_sk,d_year]
3131
CometBroadcastExchange [(avg(ctr_total_return) * 1.2),ctr_store_sk] #4
3232
CometFilter [(avg(ctr_total_return) * 1.2),ctr_store_sk]
3333
CometHashAggregate [(avg(ctr_total_return) * 1.2),ctr_store_sk,sum,count,avg(ctr_total_return)]
@@ -39,14 +39,14 @@ WholeStageCodegen (1)
3939
CometProject [sr_customer_sk,sr_store_sk,sr_return_amt]
4040
CometBroadcastHashJoin [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk,d_date_sk]
4141
CometFilter [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
42-
CometScan [native_comet] parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
42+
CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
4343
ReusedSubquery [d_date_sk] #1
4444
ReusedExchange [d_date_sk] #3
4545
CometBroadcastExchange [s_store_sk] #7
4646
CometProject [s_store_sk]
4747
CometFilter [s_store_sk,s_state]
48-
CometScan [native_comet] parquet spark_catalog.default.store [s_store_sk,s_state]
48+
CometScan [native_iceberg_compat] parquet spark_catalog.default.store [s_store_sk,s_state]
4949
CometBroadcastExchange [c_customer_sk,c_customer_id] #8
5050
CometProject [c_customer_id] [c_customer_sk,c_customer_id]
5151
CometFilter [c_customer_sk,c_customer_id]
52-
CometScan [native_comet] parquet spark_catalog.default.customer [c_customer_sk,c_customer_id]
52+
CometScan [native_iceberg_compat] parquet spark_catalog.default.customer [c_customer_sk,c_customer_id]

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q10/explain.txt‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -15,40 +15,40 @@ TakeOrderedAndProject (47)
1515
: : : :- * CometColumnarToRow (12)
1616
: : : : +- CometBroadcastHashJoin (11)
1717
: : : : :- CometFilter (2)
18-
: : : : : +- CometScan [native_comet] parquet spark_catalog.default.customer (1)
18+
: : : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer (1)
1919
: : : : +- CometBroadcastExchange (10)
2020
: : : : +- CometProject (9)
2121
: : : : +- CometBroadcastHashJoin (8)
22-
: : : : :- CometScan [native_comet] parquet spark_catalog.default.store_sales (3)
22+
: : : : :- CometScan [native_iceberg_compat] parquet spark_catalog.default.store_sales (3)
2323
: : : : +- CometBroadcastExchange (7)
2424
: : : : +- CometProject (6)
2525
: : : : +- CometFilter (5)
26-
: : : : +- CometScan [native_comet] parquet spark_catalog.default.date_dim (4)
26+
: : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim (4)
2727
: : : +- BroadcastExchange (18)
2828
: : : +- * CometColumnarToRow (17)
2929
: : : +- CometProject (16)
3030
: : : +- CometBroadcastHashJoin (15)
31-
: : : :- CometScan [native_comet] parquet spark_catalog.default.web_sales (13)
31+
: : : :- CometScan [native_iceberg_compat] parquet spark_catalog.default.web_sales (13)
3232
: : : +- ReusedExchange (14)
3333
: : +- BroadcastExchange (25)
3434
: : +- * CometColumnarToRow (24)
3535
: : +- CometProject (23)
3636
: : +- CometBroadcastHashJoin (22)
37-
: : :- CometScan [native_comet] parquet spark_catalog.default.catalog_sales (20)
37+
: : :- CometScan [native_iceberg_compat] parquet spark_catalog.default.catalog_sales (20)
3838
: : +- ReusedExchange (21)
3939
: +- BroadcastExchange (33)
4040
: +- * CometColumnarToRow (32)
4141
: +- CometProject (31)
4242
: +- CometFilter (30)
43-
: +- CometScan [native_comet] parquet spark_catalog.default.customer_address (29)
43+
: +- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer_address (29)
4444
+- BroadcastExchange (40)
4545
+- * CometColumnarToRow (39)
4646
+- CometProject (38)
4747
+- CometFilter (37)
48-
+- CometScan [native_comet] parquet spark_catalog.default.customer_demographics (36)
48+
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer_demographics (36)
4949

5050

51-
(1) CometScan [native_comet] parquet spark_catalog.default.customer
51+
(1) CometScan [native_iceberg_compat] parquet spark_catalog.default.customer
5252
Output [3]: [c_customer_sk#3, c_current_cdemo_sk#4, c_current_addr_sk#5]
5353
Batched: true
5454
Location [not included in comparison]/{warehouse_dir}/customer]
@@ -59,14 +59,14 @@ ReadSchema: struct<c_customer_sk:int,c_current_cdemo_sk:int,c_current_addr_sk:in
5959
Input [3]: [c_customer_sk#3, c_current_cdemo_sk#4, c_current_addr_sk#5]
6060
Condition : (isnotnull(c_current_addr_sk#5) AND isnotnull(c_current_cdemo_sk#4))
6161

62-
(3) CometScan [native_comet] parquet spark_catalog.default.store_sales
62+
(3) CometScan [native_iceberg_compat] parquet spark_catalog.default.store_sales
6363
Output [2]: [ss_customer_sk#6, ss_sold_date_sk#7]
6464
Batched: true
6565
Location: InMemoryFileIndex []
6666
PartitionFilters: [isnotnull(ss_sold_date_sk#7), dynamicpruningexpression(ss_sold_date_sk#7 IN dynamicpruning#8)]
6767
ReadSchema: struct<ss_customer_sk:int>
6868

69-
(4) CometScan [native_comet] parquet spark_catalog.default.date_dim
69+
(4) CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
7070
Output [3]: [d_date_sk#9, d_year#10, d_moy#11]
7171
Batched: true
7272
Location [not included in comparison]/{warehouse_dir}/date_dim]
@@ -106,7 +106,7 @@ Arguments: [c_customer_sk#3], [ss_customer_sk#6], LeftSemi, BuildRight
106106
(12) CometColumnarToRow [codegen id : 5]
107107
Input [3]: [c_customer_sk#3, c_current_cdemo_sk#4, c_current_addr_sk#5]
108108

109-
(13) CometScan [native_comet] parquet spark_catalog.default.web_sales
109+
(13) CometScan [native_iceberg_compat] parquet spark_catalog.default.web_sales
110110
Output [2]: [ws_bill_customer_sk#12, ws_sold_date_sk#13]
111111
Batched: true
112112
Location: InMemoryFileIndex []
@@ -138,7 +138,7 @@ Right keys [1]: [ws_bill_customer_sk#12]
138138
Join type: ExistenceJoin(exists#2)
139139
Join condition: None
140140

141-
(20) CometScan [native_comet] parquet spark_catalog.default.catalog_sales
141+
(20) CometScan [native_iceberg_compat] parquet spark_catalog.default.catalog_sales
142142
Output [2]: [cs_ship_customer_sk#16, cs_sold_date_sk#17]
143143
Batched: true
144144
Location: InMemoryFileIndex []
@@ -178,7 +178,7 @@ Condition : (exists#2 OR exists#1)
178178
Output [2]: [c_current_cdemo_sk#4, c_current_addr_sk#5]
179179
Input [5]: [c_customer_sk#3, c_current_cdemo_sk#4, c_current_addr_sk#5, exists#2, exists#1]
180180

181-
(29) CometScan [native_comet] parquet spark_catalog.default.customer_address
181+
(29) CometScan [native_iceberg_compat] parquet spark_catalog.default.customer_address
182182
Output [2]: [ca_address_sk#20, ca_county#21]
183183
Batched: true
184184
Location [not included in comparison]/{warehouse_dir}/customer_address]
@@ -210,7 +210,7 @@ Join condition: None
210210
Output [1]: [c_current_cdemo_sk#4]
211211
Input [3]: [c_current_cdemo_sk#4, c_current_addr_sk#5, ca_address_sk#20]
212212

213-
(36) CometScan [native_comet] parquet spark_catalog.default.customer_demographics
213+
(36) CometScan [native_iceberg_compat] parquet spark_catalog.default.customer_demographics
214214
Output [9]: [cd_demo_sk#22, cd_gender#23, cd_marital_status#24, cd_education_status#25, cd_purchase_estimate#26, cd_credit_rating#27, cd_dep_count#28, cd_dep_employed_count#29, cd_dep_college_count#30]
215215
Batched: true
216216
Location [not included in comparison]/{warehouse_dir}/customer_demographics]
@@ -274,10 +274,10 @@ BroadcastExchange (52)
274274
+- * CometColumnarToRow (51)
275275
+- CometProject (50)
276276
+- CometFilter (49)
277-
+- CometScan [native_comet] parquet spark_catalog.default.date_dim (48)
277+
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim (48)
278278

279279

280-
(48) CometScan [native_comet] parquet spark_catalog.default.date_dim
280+
(48) CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
281281
Output [3]: [d_date_sk#9, d_year#10, d_moy#11]
282282
Batched: true
283283
Location [not included in comparison]/{warehouse_dir}/date_dim]

0 commit comments

Comments
 (0)