Skip to content

Commit f29c2b5

Browse files
忍冬srowen
authored andcommitted
[SPARK-25256][SQL][TEST] Plan mismatch errors in Hive tests in Scala 2.12
## What changes were proposed in this pull request? ### For `SPARK-5775 read array from partitioned_parquet_with_key_and_complextypes`: scala2.12 ``` scala> (1 to 10).toString res4: String = Range 1 to 10 ``` scala2.11 ``` scala> (1 to 10).toString res2: String = Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) ``` And ``` def prepareAnswer(answer: Seq[Row], isSorted: Boolean): Seq[Row] = { val converted: Seq[Row] = answer.map(prepareRow) if (!isSorted) converted.sortBy(_.toString()) else converted } ``` sortBy `_.toString` is not a good idea. ### Other failures are caused by ``` Array(Int.box(1)).toSeq == Array(Double.box(1.0)).toSeq ``` It is false in 2.12.2 + and is true in 2.11.x , 2.12.0, 2.12.1 ## How was this patch tested? This is a patch on a specific unit test. Closes apache#22264 from sadhen/SPARK25256. Authored-by: 忍冬 <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent bb3e6ed commit f29c2b5

File tree

2 files changed

+11
-1
lines changed

2 files changed

+11
-1
lines changed

sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,16 @@ object QueryTest {
290290
Row.fromSeq(row.toSeq.map {
291291
case null => null
292292
case d: java.math.BigDecimal => BigDecimal(d)
293+
// Equality of WrappedArray differs for AnyVal and AnyRef in Scala 2.12.2+
294+
case seq: Seq[_] => seq.map {
295+
case b: java.lang.Byte => b.byteValue
296+
case s: java.lang.Short => s.shortValue
297+
case i: java.lang.Integer => i.intValue
298+
case l: java.lang.Long => l.longValue
299+
case f: java.lang.Float => f.floatValue
300+
case d: java.lang.Double => d.doubleValue
301+
case x => x
302+
}
293303
// Convert array to Seq for easy equality check.
294304
case b: Array[_] => b.toSeq
295305
case r: Row => prepareRow(r)

sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1068,7 +1068,7 @@ abstract class ParquetPartitioningTest extends QueryTest with SQLTestUtils with
10681068
test(s"SPARK-5775 read array from $table") {
10691069
checkAnswer(
10701070
sql(s"SELECT arrayField, p FROM $table WHERE p = 1"),
1071-
(1 to 10).map(i => Row(1 to i, 1)))
1071+
(1 to 10).map(i => Row((1 to i).toArray, 1)))
10721072
}
10731073
}
10741074

0 commit comments

Comments
 (0)