Skip to content

Commit 7bc853d

Browse files
committed
[SPARK-24033][SQL] Fix Mismatched of Window Frame specifiedwindowframe(RowFrame, -1, -1)
## What changes were proposed in this pull request? When the OffsetWindowFunction's frame is `UnaryMinus(Literal(1))` but the specified window frame has been simplified to `Literal(-1)` by some optimizer rules e.g., `ConstantFolding`. Thus, they do not match and cause the following error: ``` org.apache.spark.sql.AnalysisException: Window Frame specifiedwindowframe(RowFrame, -1, -1) must match the required frame specifiedwindowframe(RowFrame, -1, -1); at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:41) at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:91) at ``` ## How was this patch tested? Added a test Author: gatorsmile <[email protected]> Closes apache#21115 from gatorsmile/fixLag.
1 parent 32b4bcd commit 7bc853d

File tree

2 files changed

+18
-1
lines changed

2 files changed

+18
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -342,7 +342,10 @@ abstract class OffsetWindowFunction
342342
override lazy val frame: WindowFrame = {
343343
val boundary = direction match {
344344
case Ascending => offset
345-
case Descending => UnaryMinus(offset)
345+
case Descending => UnaryMinus(offset) match {
346+
case e: Expression if e.foldable => Literal.create(e.eval(EmptyRow), e.dataType)
347+
case o => o
348+
}
346349
}
347350
SpecifiedWindowFrame(RowFrame, boundary, boundary)
348351
}

sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFramesSuite.scala

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -402,4 +402,18 @@ class DataFrameWindowFramesSuite extends QueryTest with SharedSQLContext {
402402
Row(7, 3000) :: Row(8, 3000) :: Row(9, 5500) ::
403403
Row(10, 6000) :: Nil)
404404
}
405+
406+
test("SPARK-24033: Analysis Failure of OffsetWindowFunction") {
407+
val ds = Seq((1, 1), (1, 2), (1, 3), (2, 1), (2, 2)).toDF("n", "i")
408+
val res =
409+
Row(1, 1, null) :: Row (1, 2, 1) :: Row(1, 3, 2) :: Row(2, 1, null) :: Row(2, 2, 1) :: Nil
410+
checkAnswer(
411+
ds.withColumn("m",
412+
lead("i", -1).over(Window.partitionBy("n").orderBy("i").rowsBetween(-1, -1))),
413+
res)
414+
checkAnswer(
415+
ds.withColumn("m",
416+
lag("i", 1).over(Window.partitionBy("n").orderBy("i").rowsBetween(-1, -1))),
417+
res)
418+
}
405419
}

0 commit comments

Comments
 (0)