Skip to content

fix: Mark WindowsExec as incompatible #2748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
+308 −306
Open

fix: Mark WindowsExec as incompatible #2748

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
bd4323c
use map lookup and clean up CometExecRule
andygrove Nov 9, 2025
e9bb02e
move operators into package
andygrove Nov 9, 2025
9bd17e2
more cleanup
andygrove Nov 9, 2025
7c64c4b
simplify
andygrove Nov 9, 2025
de1a058
revert CometExecRule changes
andygrove Nov 9, 2025
d754f57
rebase
andygrove Nov 9, 2025
5c52971
revert
andygrove Nov 9, 2025
34a99ff
revert a change
andygrove Nov 9, 2025
af0fccf
remove unused import
andygrove Nov 9, 2025
32beb22
golden files for 3.5
andygrove Nov 9, 2025
2690d87
golden files for 3.4
andygrove Nov 9, 2025
a9a1579
golden files for 4.0
andygrove Nov 9, 2025
bc66fe2
fix
andygrove Nov 10, 2025
597f588
fix
andygrove Nov 10, 2025
49747d1
fix
andygrove Nov 10, 2025
50c5972
fix
andygrove Nov 10, 2025
3f27bef
upmerge [skip ci]
andygrove Nov 10, 2025
64035ff
golden files
andygrove Nov 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source/user-guide/latest/compatibility.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.

## Window Functions

Comet's support for window functions is incomplete and known to be incorrect. It is disabled by default and
Comet's support for window functions is incomplete and known to be incorrect. It is disabled by default and
should not be used in production. The feature will be enabled in a future release. Tracking issue: [#2721](https://github.com/apache/datafusion-comet/issues/2721).

## Cast
Expand Down
9 changes: 2 additions & 7 deletions spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,8 @@ object QueryPlanSerde extends Logging with CometExprShim {
classOf[ShuffledHashJoinExec] -> CometShuffleHashJoin,
classOf[SortMergeJoinExec] -> CometSortMergeJoin,
classOf[SortExec] -> CometSort,
classOf[LocalTableScanExec] -> CometLocalTableScan)
classOf[LocalTableScanExec] -> CometLocalTableScan,
classOf[WindowExec] -> CometWindow)

private val arrayExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(
classOf[ArrayAppend] -> CometArrayAppend,
Expand Down Expand Up @@ -929,7 +930,6 @@ object QueryPlanSerde extends Logging with CometExprShim {
* converted to a native operator.
*/
def operator2Proto(op: SparkPlan, childOp: Operator*): Option[Operator] = {
val conf = op.conf
val builder = OperatorOuterClass.Operator.newBuilder().setPlanId(op.id)
childOp.foreach(builder.addChildren)

Expand All @@ -953,10 +953,6 @@ object QueryPlanSerde extends Logging with CometExprShim {
case scan: CometScanExec if scan.scanImpl == CometConf.SCAN_NATIVE_DATAFUSION =>
CometNativeScan.convert(scan, builder, childOp: _*)

case _: WindowExec if CometConf.COMET_EXEC_WINDOW_ENABLED.get(conf) =>
withInfo(op, "Window expressions are not supported")
None

case op if isCometSink(op) =>
val supportedTypes =
op.output.forall(a => supportedDataType(a.dataType, allowComplex = true))
Expand Down Expand Up @@ -1084,7 +1080,6 @@ object QueryPlanSerde extends Logging with CometExprShim {
case _: CollectLimitExec => true
case _: UnionExec => true
case _: TakeOrderedAndProjectExec => true
case _: WindowExec => true
Copy link
Member Author

@andygrove andygrove Nov 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this line fixes #2737. I wish I could explain why, but I cannot.

Copy link
Contributor

@comphead comphead Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WindowExec was added here because it was related to TakeOrderedProjectExec which was added here previuosly, but you are right, it is most likely the reason, why the operator doesn't fallback to Spark in clean way

case _ => false
}
}
Expand Down
6 changes: 5 additions & 1 deletion spark/src/main/scala/org/apache/comet/serde/operator/CometWindow.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import org.apache.spark.sql.execution.window.WindowExec

import org.apache.comet.{CometConf, ConfigEntry}
import org.apache.comet.CometSparkSessionExtensions.withInfo
import org.apache.comet.serde.{CometOperatorSerde, OperatorOuterClass}
import org.apache.comet.serde.{CometOperatorSerde, Incompatible, OperatorOuterClass, SupportLevel}
import org.apache.comet.serde.OperatorOuterClass.Operator
import org.apache.comet.serde.QueryPlanSerde.{exprToProto, windowExprToProto}

Expand All @@ -36,6 +36,10 @@ object CometWindow extends CometOperatorSerde[WindowExec] {
override def enabledConfig: Option[ConfigEntry[Boolean]] = Some(
CometConf.COMET_EXEC_WINDOW_ENABLED)

override def getSupportLevel(op: WindowExec): SupportLevel = {
Incompatible(Some("Native WindowExec has known correctness issues"))
}

override def convert(
op: WindowExec,
builder: Operator.Builder,
Expand Down
2 changes: 1 addition & 1 deletion .../tpcds-plan-stability/approved-plans-v1_4-spark3_5/q12.native_iceberg_compat/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
2 changes: 1 addition & 1 deletion spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q12/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
2 changes: 1 addition & 1 deletion .../tpcds-plan-stability/approved-plans-v1_4-spark3_5/q20.native_iceberg_compat/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
2 changes: 1 addition & 1 deletion spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q20/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
2 changes: 1 addition & 1 deletion .../tpcds-plan-stability/approved-plans-v1_4-spark3_5/q36.native_iceberg_compat/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
2 changes: 1 addition & 1 deletion spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q36/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
6 changes: 3 additions & 3 deletions .../tpcds-plan-stability/approved-plans-v1_4-spark3_5/q47.native_iceberg_compat/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
: : +- Filter
: : +- Window
: : +- Filter
: : +- Window [COMET: Window expressions are not supported]
: : +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: : +- CometColumnarToRow
: : +- CometSort
: : +- CometExchange
Expand Down Expand Up @@ -39,7 +39,7 @@
: : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store
: +- BroadcastExchange
: +- Project
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand Down Expand Up @@ -71,7 +71,7 @@
: +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store
+- BroadcastExchange
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
6 changes: 3 additions & 3 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q47/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
: : +- Filter
: : +- Window
: : +- Filter
: : +- Window [COMET: Window expressions are not supported]
: : +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: : +- CometColumnarToRow
: : +- CometSort
: : +- CometExchange
Expand Down Expand Up @@ -39,7 +39,7 @@
: : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store
: +- BroadcastExchange
: +- Project
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand Down Expand Up @@ -71,7 +71,7 @@
: +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store
+- BroadcastExchange
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
6 changes: 3 additions & 3 deletions .../tpcds-plan-stability/approved-plans-v1_4-spark3_5/q49.native_iceberg_compat/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ CometColumnarToRow
: +- Filter
: +- Window
: +- Sort
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand Down Expand Up @@ -40,7 +40,7 @@ CometColumnarToRow
: +- Filter
: +- Window
: +- Sort
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand All @@ -67,7 +67,7 @@ CometColumnarToRow
+- Filter
+- Window
+- Sort
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
6 changes: 3 additions & 3 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q49/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ CometColumnarToRow
: +- Filter
: +- Window
: +- Sort
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand Down Expand Up @@ -40,7 +40,7 @@ CometColumnarToRow
: +- Filter
: +- Window
: +- Sort
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand All @@ -67,7 +67,7 @@ CometColumnarToRow
+- Filter
+- Window
+- Sort
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
6 changes: 3 additions & 3 deletions .../tpcds-plan-stability/approved-plans-v1_4-spark3_5/q51.native_iceberg_compat/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Filter
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand All @@ -9,7 +9,7 @@
:- CometSort
: +- CometColumnarExchange
: +- Project
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand All @@ -33,7 +33,7 @@
+- CometSort
+- CometColumnarExchange
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
6 changes: 3 additions & 3 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q51/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
TakeOrderedAndProject [COMET: ]
+- Filter
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand All @@ -9,7 +9,7 @@
:- CometSort
: +- CometColumnarExchange
: +- Project
: +- Window [COMET: Window expressions are not supported]
: +- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
: +- CometColumnarToRow
: +- CometSort
: +- CometExchange
Expand All @@ -33,7 +33,7 @@
+- CometSort
+- CometColumnarExchange
+- Project
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
2 changes: 1 addition & 1 deletion .../tpcds-plan-stability/approved-plans-v1_4-spark3_5/q53.native_iceberg_compat/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Filter
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
2 changes: 1 addition & 1 deletion spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q53/extended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
TakeOrderedAndProject [COMET: ]
+- Project
+- Filter
+- Window [COMET: Window expressions are not supported]
+- Window [COMET: WindowExec is not fully compatible with Spark (Native WindowExec has known correctness issues). To enable it anyway, set spark.comet.operator.WindowExec.allowIncompatible=true. For more information, refer to the Comet Compatibility Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html).]
+- CometColumnarToRow
+- CometSort
+- CometExchange
Expand Down
Loading
Loading