Skip to content

Commit d9dc391

Browse files
compheadcomphead
committed
chore: Fallback to Spark for windows
1 parent 7dccd13 commit d9dc391

File tree

343 files changed

+17142
-15035
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

343 files changed

+17142
-15035
lines changed

‎common/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -251,7 +251,7 @@ object CometConf extends ShimCometConf {
251251
val COMET_EXEC_EXPAND_ENABLED: ConfigEntry[Boolean] =
252252
createExecEnabledConfig("expand", defaultValue = true)
253253
val COMET_EXEC_WINDOW_ENABLED: ConfigEntry[Boolean] =
254-
createExecEnabledConfig("window", defaultValue = false)
254+
createExecEnabledConfig("window", defaultValue = true)
255255
val COMET_EXEC_TAKE_ORDERED_AND_PROJECT_ENABLED: ConfigEntry[Boolean] =
256256
createExecEnabledConfig("takeOrderedAndProject", defaultValue = true)
257257

‎docs/source/user-guide/latest/configs.md‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -139,23 +139,23 @@ These settings can be used to determine which parts of the plan are accelerated
139139
<!--BEGIN:CONFIG_TABLE[enable_exec]-->
140140
| Config | Description | Default Value |
141141
|--------|-------------|---------------|
142-
| `spark.comet.exec.aggregate.enabled` | Whether to enable aggregate by default. | true |
143-
| `spark.comet.exec.broadcastExchange.enabled` | Whether to enable broadcastExchange by default. | true |
144-
| `spark.comet.exec.broadcastHashJoin.enabled` | Whether to enable broadcastHashJoin by default. | true |
145-
| `spark.comet.exec.coalesce.enabled` | Whether to enable coalesce by default. | true |
146-
| `spark.comet.exec.collectLimit.enabled` | Whether to enable collectLimit by default. | true |
147-
| `spark.comet.exec.expand.enabled` | Whether to enable expand by default. | true |
148-
| `spark.comet.exec.filter.enabled` | Whether to enable filter by default. | true |
149-
| `spark.comet.exec.globalLimit.enabled` | Whether to enable globalLimit by default. | true |
150-
| `spark.comet.exec.hashJoin.enabled` | Whether to enable hashJoin by default. | true |
151-
| `spark.comet.exec.localLimit.enabled` | Whether to enable localLimit by default. | true |
152-
| `spark.comet.exec.project.enabled` | Whether to enable project by default. | true |
153-
| `spark.comet.exec.sort.enabled` | Whether to enable sort by default. | true |
154-
| `spark.comet.exec.sortMergeJoin.enabled` | Whether to enable sortMergeJoin by default. | true |
155-
| `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled` | Experimental support for Sort Merge Join with filter | false |
156-
| `spark.comet.exec.takeOrderedAndProject.enabled` | Whether to enable takeOrderedAndProject by default. | true |
157-
| `spark.comet.exec.union.enabled` | Whether to enable union by default. | true |
158-
| `spark.comet.exec.window.enabled` | Whether to enable window by default. | false |
142+
| `spark.comet.exec.aggregate.enabled` | Whether to enable aggregate by default. | true |
143+
| `spark.comet.exec.broadcastExchange.enabled` | Whether to enable broadcastExchange by default. | true |
144+
| `spark.comet.exec.broadcastHashJoin.enabled` | Whether to enable broadcastHashJoin by default. | true |
145+
| `spark.comet.exec.coalesce.enabled` | Whether to enable coalesce by default. | true |
146+
| `spark.comet.exec.collectLimit.enabled` | Whether to enable collectLimit by default. | true |
147+
| `spark.comet.exec.expand.enabled` | Whether to enable expand by default. | true |
148+
| `spark.comet.exec.filter.enabled` | Whether to enable filter by default. | true |
149+
| `spark.comet.exec.globalLimit.enabled` | Whether to enable globalLimit by default. | true |
150+
| `spark.comet.exec.hashJoin.enabled` | Whether to enable hashJoin by default. | true |
151+
| `spark.comet.exec.localLimit.enabled` | Whether to enable localLimit by default. | true |
152+
| `spark.comet.exec.project.enabled` | Whether to enable project by default. | true |
153+
| `spark.comet.exec.sort.enabled` | Whether to enable sort by default. | true |
154+
| `spark.comet.exec.sortMergeJoin.enabled` | Whether to enable sortMergeJoin by default. | true |
155+
| `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled` | Experimental support for Sort Merge Join with filter | false |
156+
| `spark.comet.exec.takeOrderedAndProject.enabled` | Whether to enable takeOrderedAndProject by default. | true |
157+
| `spark.comet.exec.union.enabled` | Whether to enable union by default. | true |
158+
| `spark.comet.exec.window.enabled` | Whether to enable window by default. | true |
159159
<!--END:CONFIG_TABLE-->
160160

161161
## Enabling or Disabling Individual Scalar Expressions

‎spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1125,6 +1125,9 @@ object QueryPlanSerde extends Logging with CometExprShim {
11251125

11261126
case WindowExec(windowExpression, partitionSpec, orderSpec, child)
11271127
if CometConf.COMET_EXEC_WINDOW_ENABLED.get(conf) =>
1128+
withInfo(op, "Window expressions are not supported")
1129+
None
1130+
/*
11281131
val output = child.output
11291132
11301133
val winExprs: Array[WindowExpression] = windowExpression.flatMap { expr =>
@@ -1165,7 +1168,7 @@ object QueryPlanSerde extends Logging with CometExprShim {
11651168
Some(builder.setWindow(windowBuilder).build())
11661169
} else {
11671170
None
1168-
}
1171+
} */
11691172

11701173
case aggregate: BaseAggregateExec
11711174
if (aggregate.isInstanceOf[HashAggregateExec] ||

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q12.native_iceberg_compat/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
TakeOrderedAndProject [COMET: ]
22
+- Project
3-
+- Window
3+
+- Window [COMET: Window expressions are not supported]
44
+- CometColumnarToRow
55
+- CometSort
66
+- CometExchange

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q12/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
TakeOrderedAndProject [COMET: ]
22
+- Project
3-
+- Window
3+
+- Window [COMET: Window expressions are not supported]
44
+- CometColumnarToRow
55
+- CometSort
66
+- CometExchange

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q20.native_iceberg_compat/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
TakeOrderedAndProject [COMET: ]
22
+- Project
3-
+- Window
3+
+- Window [COMET: Window expressions are not supported]
44
+- CometColumnarToRow
55
+- CometSort
66
+- CometExchange

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q20/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
TakeOrderedAndProject [COMET: ]
22
+- Project
3-
+- Window
3+
+- Window [COMET: Window expressions are not supported]
44
+- CometColumnarToRow
55
+- CometSort
66
+- CometExchange

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q36.native_iceberg_compat/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
TakeOrderedAndProject [COMET: ]
22
+- Project
3-
+- Window
3+
+- Window [COMET: Window expressions are not supported]
44
+- CometColumnarToRow
55
+- CometSort
66
+- CometExchange

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q36/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
TakeOrderedAndProject [COMET: ]
22
+- Project
3-
+- Window
3+
+- Window [COMET: Window expressions are not supported]
44
+- CometColumnarToRow
55
+- CometSort
66
+- CometExchange

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q47.native_iceberg_compat/extended.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
: : +- Filter
88
: : +- Window
99
: : +- Filter
10-
: : +- Window
10+
: : +- Window [COMET: Window expressions are not supported]
1111
: : +- CometColumnarToRow
1212
: : +- CometSort
1313
: : +- CometExchange
@@ -39,7 +39,7 @@
3939
: : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store
4040
: +- BroadcastExchange
4141
: +- Project
42-
: +- Window
42+
: +- Window [COMET: Window expressions are not supported]
4343
: +- CometColumnarToRow
4444
: +- CometSort
4545
: +- CometExchange
@@ -71,7 +71,7 @@
7171
: +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store
7272
+- BroadcastExchange
7373
+- Project
74-
+- Window
74+
+- Window [COMET: Window expressions are not supported]
7575
+- CometColumnarToRow
7676
+- CometSort
7777
+- CometExchange

0 commit comments

Comments
 (0)