Skip to content

Commit e9fce2a

Browse files
wangyumcloud-fan
authored andcommitted
[SPARK-24716][TESTS][FOLLOW-UP] Test Hive metastore schema and parquet schema are in different letter cases
## What changes were proposed in this pull request? Since apache#21696. Spark uses Parquet schema instead of Hive metastore schema to do pushdown. That change can avoid wrong records returned when Hive metastore schema and parquet schema are in different letter cases. This pr add a test case for it. More details: https://issues.apache.org/jira/browse/SPARK-25206 ## How was this patch tested? unit tests Closes apache#22267 from wangyum/SPARK-24716-TESTS. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent 56bc700 commit e9fce2a

File tree

1 file changed

+16
-0
lines changed

1 file changed

+16
-0
lines changed

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSuite.scala

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ package org.apache.spark.sql.hive
2020
import org.apache.spark.sql.{QueryTest, Row}
2121
import org.apache.spark.sql.execution.datasources.parquet.ParquetTest
2222
import org.apache.spark.sql.hive.test.TestHiveSingleton
23+
import org.apache.spark.sql.internal.SQLConf
2324

2425
case class Cases(lower: String, UPPER: String)
2526

@@ -76,4 +77,19 @@ class HiveParquetSuite extends QueryTest with ParquetTest with TestHiveSingleton
7677
}
7778
}
7879
}
80+
81+
test("SPARK-25206: wrong records are returned by filter pushdown " +
82+
"when Hive metastore schema and parquet schema are in different letter cases") {
83+
withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> true.toString) {
84+
withTempPath { path =>
85+
val data = spark.range(1, 10).toDF("id")
86+
data.write.parquet(path.getCanonicalPath)
87+
withTable("SPARK_25206") {
88+
sql("CREATE TABLE SPARK_25206 (ID LONG) USING parquet LOCATION " +
89+
s"'${path.getCanonicalPath}'")
90+
checkAnswer(sql("select id from SPARK_25206 where id > 0"), data)
91+
}
92+
}
93+
}
94+
}
7995
}

0 commit comments

Comments
 (0)