Skip to content

Commit 6166b4e

Browse files
andygroveandygrove
committed
4.0 golden files
1 parent 10af6d7 commit 6166b4e

File tree

270 files changed

+51118
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

270 files changed

+51118
-0
lines changed

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q1.native_iceberg_compat/explain.txt‎

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
== Physical Plan ==
2+
TakeOrderedAndProject (45)
3+
+- * Project (44)
4+
+- * BroadcastHashJoin Inner BuildRight (43)
5+
:- * Project (37)
6+
: +- * BroadcastHashJoin Inner BuildRight (36)
7+
: :- * Project (30)
8+
: : +- * BroadcastHashJoin Inner BuildRight (29)
9+
: : :- * Filter (13)
10+
: : : +- * HashAggregate (12)
11+
: : : +- * CometColumnarToRow (11)
12+
: : : +- CometExchange (10)
13+
: : : +- CometHashAggregate (9)
14+
: : : +- CometProject (8)
15+
: : : +- CometBroadcastHashJoin (7)
16+
: : : :- CometFilter (2)
17+
: : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns (1)
18+
: : : +- CometBroadcastExchange (6)
19+
: : : +- CometProject (5)
20+
: : : +- CometFilter (4)
21+
: : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim (3)
22+
: : +- BroadcastExchange (28)
23+
: : +- * Filter (27)
24+
: : +- * HashAggregate (26)
25+
: : +- * CometColumnarToRow (25)
26+
: : +- CometColumnarExchange (24)
27+
: : +- * HashAggregate (23)
28+
: : +- * HashAggregate (22)
29+
: : +- * CometColumnarToRow (21)
30+
: : +- CometExchange (20)
31+
: : +- CometHashAggregate (19)
32+
: : +- CometProject (18)
33+
: : +- CometBroadcastHashJoin (17)
34+
: : :- CometFilter (15)
35+
: : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns (14)
36+
: : +- ReusedExchange (16)
37+
: +- BroadcastExchange (35)
38+
: +- * CometColumnarToRow (34)
39+
: +- CometProject (33)
40+
: +- CometFilter (32)
41+
: +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store (31)
42+
+- BroadcastExchange (42)
43+
+- * CometColumnarToRow (41)
44+
+- CometProject (40)
45+
+- CometFilter (39)
46+
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer (38)
47+
48+
49+
(1) CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns
50+
Output [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
51+
Batched: true
52+
Location: InMemoryFileIndex []
53+
PartitionFilters: [isnotnull(sr_returned_date_sk#4)]
54+
PushedFilters: [IsNotNull(sr_store_sk), IsNotNull(sr_customer_sk)]
55+
ReadSchema: struct<sr_customer_sk:int,sr_store_sk:int,sr_return_amt:decimal(7,2)>
56+
57+
(2) CometFilter
58+
Input [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
59+
Condition : (isnotnull(sr_store_sk#2) AND isnotnull(sr_customer_sk#1))
60+
61+
(3) CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
62+
Output [2]: [d_date_sk#5, d_year#6]
63+
Batched: true
64+
Location [not included in comparison]/{warehouse_dir}/date_dim]
65+
PushedFilters: [IsNotNull(d_year), EqualTo(d_year,2000), IsNotNull(d_date_sk)]
66+
ReadSchema: struct<d_date_sk:int,d_year:int>
67+
68+
(4) CometFilter
69+
Input [2]: [d_date_sk#5, d_year#6]
70+
Condition : ((isnotnull(d_year#6) AND (d_year#6 = 2000)) AND isnotnull(d_date_sk#5))
71+
72+
(5) CometProject
73+
Input [2]: [d_date_sk#5, d_year#6]
74+
Arguments: [d_date_sk#5], [d_date_sk#5]
75+
76+
(6) CometBroadcastExchange
77+
Input [1]: [d_date_sk#5]
78+
Arguments: [d_date_sk#5]
79+
80+
(7) CometBroadcastHashJoin
81+
Left output [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
82+
Right output [1]: [d_date_sk#5]
83+
Arguments: [sr_returned_date_sk#4], [d_date_sk#5], Inner, BuildRight
84+
85+
(8) CometProject
86+
Input [5]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4, d_date_sk#5]
87+
Arguments: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3], [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3]
88+
89+
(9) CometHashAggregate
90+
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3]
91+
Keys [2]: [sr_customer_sk#1, sr_store_sk#2]
92+
Functions [1]: [partial_sum(UnscaledValue(sr_return_amt#3))]
93+
94+
(10) CometExchange
95+
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#7]
96+
Arguments: hashpartitioning(sr_customer_sk#1, sr_store_sk#2, 5), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=1]
97+
98+
(11) CometColumnarToRow [codegen id : 5]
99+
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#7]
100+
101+
(12) HashAggregate [codegen id : 5]
102+
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#7]
103+
Keys [2]: [sr_customer_sk#1, sr_store_sk#2]
104+
Functions [1]: [sum(UnscaledValue(sr_return_amt#3))]
105+
Aggregate Attributes [1]: [sum(UnscaledValue(sr_return_amt#3))#8]
106+
Results [3]: [sr_customer_sk#1 AS ctr_customer_sk#9, sr_store_sk#2 AS ctr_store_sk#10, MakeDecimal(sum(UnscaledValue(sr_return_amt#3))#8,17,2) AS ctr_total_return#11]
107+
108+
(13) Filter [codegen id : 5]
109+
Input [3]: [ctr_customer_sk#9, ctr_store_sk#10, ctr_total_return#11]
110+
Condition : isnotnull(ctr_total_return#11)
111+
112+
(14) CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns
113+
Output [4]: [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14, sr_returned_date_sk#15]
114+
Batched: true
115+
Location: InMemoryFileIndex []
116+
PartitionFilters: [isnotnull(sr_returned_date_sk#15)]
117+
PushedFilters: [IsNotNull(sr_store_sk)]
118+
ReadSchema: struct<sr_customer_sk:int,sr_store_sk:int,sr_return_amt:decimal(7,2)>
119+
120+
(15) CometFilter
121+
Input [4]: [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14, sr_returned_date_sk#15]
122+
Condition : isnotnull(sr_store_sk#13)
123+
124+
(16) ReusedExchange [Reuses operator id: 6]
125+
Output [1]: [d_date_sk#16]
126+
127+
(17) CometBroadcastHashJoin
128+
Left output [4]: [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14, sr_returned_date_sk#15]
129+
Right output [1]: [d_date_sk#16]
130+
Arguments: [sr_returned_date_sk#15], [d_date_sk#16], Inner, BuildRight
131+
132+
(18) CometProject
133+
Input [5]: [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14, sr_returned_date_sk#15, d_date_sk#16]
134+
Arguments: [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14], [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14]
135+
136+
(19) CometHashAggregate
137+
Input [3]: [sr_customer_sk#12, sr_store_sk#13, sr_return_amt#14]
138+
Keys [2]: [sr_customer_sk#12, sr_store_sk#13]
139+
Functions [1]: [partial_sum(UnscaledValue(sr_return_amt#14))]
140+
141+
(20) CometExchange
142+
Input [3]: [sr_customer_sk#12, sr_store_sk#13, sum#17]
143+
Arguments: hashpartitioning(sr_customer_sk#12, sr_store_sk#13, 5), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=2]
144+
145+
(21) CometColumnarToRow [codegen id : 1]
146+
Input [3]: [sr_customer_sk#12, sr_store_sk#13, sum#17]
147+
148+
(22) HashAggregate [codegen id : 1]
149+
Input [3]: [sr_customer_sk#12, sr_store_sk#13, sum#17]
150+
Keys [2]: [sr_customer_sk#12, sr_store_sk#13]
151+
Functions [1]: [sum(UnscaledValue(sr_return_amt#14))]
152+
Aggregate Attributes [1]: [sum(UnscaledValue(sr_return_amt#14))#8]
153+
Results [2]: [sr_store_sk#13 AS ctr_store_sk#18, MakeDecimal(sum(UnscaledValue(sr_return_amt#14))#8,17,2) AS ctr_total_return#19]
154+
155+
(23) HashAggregate [codegen id : 1]
156+
Input [2]: [ctr_store_sk#18, ctr_total_return#19]
157+
Keys [1]: [ctr_store_sk#18]
158+
Functions [1]: [partial_avg(ctr_total_return#19)]
159+
Aggregate Attributes [2]: [sum#20, count#21]
160+
Results [3]: [ctr_store_sk#18, sum#22, count#23]
161+
162+
(24) CometColumnarExchange
163+
Input [3]: [ctr_store_sk#18, sum#22, count#23]
164+
Arguments: hashpartitioning(ctr_store_sk#18, 5), ENSURE_REQUIREMENTS, CometColumnarShuffle, [plan_id=3]
165+
166+
(25) CometColumnarToRow [codegen id : 2]
167+
Input [3]: [ctr_store_sk#18, sum#22, count#23]
168+
169+
(26) HashAggregate [codegen id : 2]
170+
Input [3]: [ctr_store_sk#18, sum#22, count#23]
171+
Keys [1]: [ctr_store_sk#18]
172+
Functions [1]: [avg(ctr_total_return#19)]
173+
Aggregate Attributes [1]: [avg(ctr_total_return#19)#24]
174+
Results [2]: [(avg(ctr_total_return#19)#24 * 1.2) AS (avg(ctr_total_return) * 1.2)#25, ctr_store_sk#18]
175+
176+
(27) Filter [codegen id : 2]
177+
Input [2]: [(avg(ctr_total_return) * 1.2)#25, ctr_store_sk#18]
178+
Condition : isnotnull((avg(ctr_total_return) * 1.2)#25)
179+
180+
(28) BroadcastExchange
181+
Input [2]: [(avg(ctr_total_return) * 1.2)#25, ctr_store_sk#18]
182+
Arguments: HashedRelationBroadcastMode(List(cast(input[1, int, true] as bigint)),false), [plan_id=4]
183+
184+
(29) BroadcastHashJoin [codegen id : 5]
185+
Left keys [1]: [ctr_store_sk#10]
186+
Right keys [1]: [ctr_store_sk#18]
187+
Join type: Inner
188+
Join condition: (cast(ctr_total_return#11 as decimal(24,7)) > (avg(ctr_total_return) * 1.2)#25)
189+
190+
(30) Project [codegen id : 5]
191+
Output [2]: [ctr_customer_sk#9, ctr_store_sk#10]
192+
Input [5]: [ctr_customer_sk#9, ctr_store_sk#10, ctr_total_return#11, (avg(ctr_total_return) * 1.2)#25, ctr_store_sk#18]
193+
194+
(31) CometScan [native_iceberg_compat] parquet spark_catalog.default.store
195+
Output [2]: [s_store_sk#26, s_state#27]
196+
Batched: true
197+
Location [not included in comparison]/{warehouse_dir}/store]
198+
PushedFilters: [IsNotNull(s_state), IsNotNull(s_store_sk)]
199+
ReadSchema: struct<s_store_sk:int,s_state:string>
200+
201+
(32) CometFilter
202+
Input [2]: [s_store_sk#26, s_state#27]
203+
Condition : ((isnotnull(s_state#27) AND (static_invoke(CharVarcharCodegenUtils.readSidePadding(s_state#27, 2)) = TN)) AND isnotnull(s_store_sk#26))
204+
205+
(33) CometProject
206+
Input [2]: [s_store_sk#26, s_state#27]
207+
Arguments: [s_store_sk#26], [s_store_sk#26]
208+
209+
(34) CometColumnarToRow [codegen id : 3]
210+
Input [1]: [s_store_sk#26]
211+
212+
(35) BroadcastExchange
213+
Input [1]: [s_store_sk#26]
214+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=5]
215+
216+
(36) BroadcastHashJoin [codegen id : 5]
217+
Left keys [1]: [ctr_store_sk#10]
218+
Right keys [1]: [s_store_sk#26]
219+
Join type: Inner
220+
Join condition: None
221+
222+
(37) Project [codegen id : 5]
223+
Output [1]: [ctr_customer_sk#9]
224+
Input [3]: [ctr_customer_sk#9, ctr_store_sk#10, s_store_sk#26]
225+
226+
(38) CometScan [native_iceberg_compat] parquet spark_catalog.default.customer
227+
Output [2]: [c_customer_sk#28, c_customer_id#29]
228+
Batched: true
229+
Location [not included in comparison]/{warehouse_dir}/customer]
230+
PushedFilters: [IsNotNull(c_customer_sk)]
231+
ReadSchema: struct<c_customer_sk:int,c_customer_id:string>
232+
233+
(39) CometFilter
234+
Input [2]: [c_customer_sk#28, c_customer_id#29]
235+
Condition : isnotnull(c_customer_sk#28)
236+
237+
(40) CometProject
238+
Input [2]: [c_customer_sk#28, c_customer_id#29]
239+
Arguments: [c_customer_sk#28, c_customer_id#30], [c_customer_sk#28, static_invoke(CharVarcharCodegenUtils.readSidePadding(c_customer_id#29, 16)) AS c_customer_id#30]
240+
241+
(41) CometColumnarToRow [codegen id : 4]
242+
Input [2]: [c_customer_sk#28, c_customer_id#30]
243+
244+
(42) BroadcastExchange
245+
Input [2]: [c_customer_sk#28, c_customer_id#30]
246+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=6]
247+
248+
(43) BroadcastHashJoin [codegen id : 5]
249+
Left keys [1]: [ctr_customer_sk#9]
250+
Right keys [1]: [c_customer_sk#28]
251+
Join type: Inner
252+
Join condition: None
253+
254+
(44) Project [codegen id : 5]
255+
Output [1]: [c_customer_id#30]
256+
Input [3]: [ctr_customer_sk#9, c_customer_sk#28, c_customer_id#30]
257+
258+
(45) TakeOrderedAndProject
259+
Input [1]: [c_customer_id#30]
260+
Arguments: 100, [c_customer_id#30 ASC NULLS FIRST], [c_customer_id#30]
261+

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q1.native_iceberg_compat/simplified.txt‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
TakeOrderedAndProject [c_customer_id]
2+
WholeStageCodegen (5)
3+
Project [c_customer_id]
4+
BroadcastHashJoin [ctr_customer_sk,c_customer_sk]
5+
Project [ctr_customer_sk]
6+
BroadcastHashJoin [ctr_store_sk,s_store_sk]
7+
Project [ctr_customer_sk,ctr_store_sk]
8+
BroadcastHashJoin [ctr_store_sk,ctr_store_sk,ctr_total_return,(avg(ctr_total_return) * 1.2)]
9+
Filter [ctr_total_return]
10+
HashAggregate [sr_customer_sk,sr_store_sk,sum] [sum(UnscaledValue(sr_return_amt)),ctr_customer_sk,ctr_store_sk,ctr_total_return,sum]
11+
CometColumnarToRow
12+
InputAdapter
13+
CometExchange [sr_customer_sk,sr_store_sk] #1
14+
CometHashAggregate [sr_return_amt] [sr_customer_sk,sr_store_sk,sum]
15+
CometProject [sr_customer_sk,sr_store_sk,sr_return_amt]
16+
CometBroadcastHashJoin [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk,d_date_sk]
17+
CometFilter [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
18+
CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
19+
CometBroadcastExchange [d_date_sk] #2
20+
CometProject [d_date_sk]
21+
CometFilter [d_date_sk,d_year]
22+
CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim [d_date_sk,d_year]
23+
InputAdapter
24+
BroadcastExchange #3
25+
WholeStageCodegen (2)
26+
Filter [(avg(ctr_total_return) * 1.2)]
27+
HashAggregate [ctr_store_sk,sum,count] [avg(ctr_total_return),(avg(ctr_total_return) * 1.2),sum,count]
28+
CometColumnarToRow
29+
InputAdapter
30+
CometColumnarExchange [ctr_store_sk] #4
31+
WholeStageCodegen (1)
32+
HashAggregate [ctr_store_sk,ctr_total_return] [sum,count,sum,count]
33+
HashAggregate [sr_customer_sk,sr_store_sk,sum] [sum(UnscaledValue(sr_return_amt)),ctr_store_sk,ctr_total_return,sum]
34+
CometColumnarToRow
35+
InputAdapter
36+
CometExchange [sr_customer_sk,sr_store_sk] #5
37+
CometHashAggregate [sr_return_amt] [sr_customer_sk,sr_store_sk,sum]
38+
CometProject [sr_customer_sk,sr_store_sk,sr_return_amt]
39+
CometBroadcastHashJoin [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk,d_date_sk]
40+
CometFilter [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
41+
CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
42+
ReusedExchange [d_date_sk] #2
43+
InputAdapter
44+
BroadcastExchange #6
45+
WholeStageCodegen (3)
46+
CometColumnarToRow
47+
InputAdapter
48+
CometProject [s_store_sk]
49+
CometFilter [s_store_sk,s_state]
50+
CometScan [native_iceberg_compat] parquet spark_catalog.default.store [s_store_sk,s_state]
51+
InputAdapter
52+
BroadcastExchange #7
53+
WholeStageCodegen (4)
54+
CometColumnarToRow
55+
InputAdapter
56+
CometProject [c_customer_id] [c_customer_sk,c_customer_id]
57+
CometFilter [c_customer_sk,c_customer_id]
58+
CometScan [native_iceberg_compat] parquet spark_catalog.default.customer [c_customer_sk,c_customer_id]

0 commit comments

Comments
 (0)