Skip to content

Commit 332e593

Browse files
cxzl25HyukjinKwon
authored andcommitted
[SPARK-29943][SQL] Improve error messages for unsupported data type
### What changes were proposed in this pull request? Improve error messages for unsupported data type. ### Why are the changes needed? When the spark reads the hive table and encounters an unsupported field type, the exception message has only one unsupported type, and the user cannot know which field of which table. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? ```create view t AS SELECT STRUCT('a' AS `$a`, 1 AS b) as q;``` current: org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int> change: org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>, column: q ```select * from t,t_normal_1,t_normal_2``` current: org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int> change: org.apache.spark.SparkException: Cannot recognize hive type string: struct<$a:string,b:int>, column: q, db: default, table: t Closes apache#26577 from cxzl25/unsupport_data_type_msg. Authored-by: sychen <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent 68034a8 commit 332e593

File tree

1 file changed

+9
-3
lines changed

1 file changed

+9
-3
lines changed

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -440,8 +440,13 @@ private[hive] class HiveClientImpl(
440440
private def convertHiveTableToCatalogTable(h: HiveTable): CatalogTable = {
441441
// Note: Hive separates partition columns and the schema, but for us the
442442
// partition columns are part of the schema
443-
val cols = h.getCols.asScala.map(fromHiveColumn)
444-
val partCols = h.getPartCols.asScala.map(fromHiveColumn)
443+
val (cols, partCols) = try {
444+
(h.getCols.asScala.map(fromHiveColumn), h.getPartCols.asScala.map(fromHiveColumn))
445+
} catch {
446+
case ex: SparkException =>
447+
throw new SparkException(
448+
s"${ex.getMessage}, db: ${h.getDbName}, table: ${h.getTableName}", ex)
449+
}
445450
val schema = StructType(cols ++ partCols)
446451

447452
val bucketSpec = if (h.getNumBuckets > 0) {
@@ -982,7 +987,8 @@ private[hive] object HiveClientImpl {
982987
CatalystSqlParser.parseDataType(hc.getType)
983988
} catch {
984989
case e: ParseException =>
985-
throw new SparkException("Cannot recognize hive type string: " + hc.getType, e)
990+
throw new SparkException(
991+
s"Cannot recognize hive type string: ${hc.getType}, column: ${hc.getName}", e)
986992
}
987993
}
988994

0 commit comments

Comments
 (0)