Skip to content

Commit fb8f4c0

Browse files
mgaido91gatorsmile
authored andcommitted
[SPARK-25505][SQL][FOLLOWUP] Fix for attributes cosmetically different in Pivot clause
## What changes were proposed in this pull request? apache#22519 introduced a bug when the attributes in the pivot clause are cosmetically different from the output ones (eg. different case). In particular, the problem is that the PR used a `Set[Attribute]` instead of an `AttributeSet`. ## How was this patch tested? added UT Closes apache#22582 from mgaido91/SPARK-25505_followup. Authored-by: Marco Gaido <[email protected]> Signed-off-by: gatorsmile <[email protected]>
1 parent 4da541a commit fb8f4c0

File tree

3 files changed

+6
-6
lines changed

3 files changed

+6
-6
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -555,8 +555,7 @@ class Analyzer(
555555
}
556556
// Group-by expressions coming from SQL are implicit and need to be deduced.
557557
val groupByExprs = groupByExprsOpt.getOrElse {
558-
val pivotColAndAggRefs =
559-
(pivotColumn.references ++ aggregates.flatMap(_.references)).toSet
558+
val pivotColAndAggRefs = pivotColumn.references ++ AttributeSet(aggregates)
560559
child.output.filterNot(pivotColAndAggRefs.contains)
561560
}
562561
val singleAgg = aggregates.size == 1

sql/core/src/test/resources/sql-tests/inputs/pivot.sql

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -289,11 +289,12 @@ PIVOT (
289289
);
290290

291291
-- grouping columns output in the same order as input
292+
-- correctly handle pivot columns with different cases
292293
SELECT * FROM (
293294
SELECT course, earnings, "a" as a, "z" as z, "b" as b, "y" as y, "c" as c, "x" as x, "d" as d, "w" as w
294295
FROM courseSales
295296
)
296297
PIVOT (
297-
sum(earnings)
298-
FOR course IN ('dotNET', 'Java')
298+
sum(Earnings)
299+
FOR Course IN ('dotNET', 'Java')
299300
);

sql/core/src/test/resources/sql-tests/results/pivot.sql.out

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -484,8 +484,8 @@ SELECT * FROM (
484484
FROM courseSales
485485
)
486486
PIVOT (
487-
sum(earnings)
488-
FOR course IN ('dotNET', 'Java')
487+
sum(Earnings)
488+
FOR Course IN ('dotNET', 'Java')
489489
)
490490
-- !query 31 schema
491491
struct<a:string,z:string,b:string,y:string,c:string,x:string,d:string,w:string,dotNET:bigint,Java:bigint>

0 commit comments

Comments
 (0)