Skip to content

Commit 27d625d

Browse files
gengliangwangcloud-fan
authored andcommitted
[SPARK-27459][SQL] Revise the exception message of schema inference failure in file source V2
## What changes were proposed in this pull request? Since https://github.com/apache/spark/pull/23383/files#diff-db4a140579c1ac4b1dbec7fe5057eecaR36, the exception message of schema inference failure in file source V2 is `tableName`, which is equivalent to `shortName + path`. While in file source V1, the message is `Unable to infer schema from ORC/CSV/JSON...`. We should make the message in V2 consistent with V1, so that in the future migration the related test cases don't need to be modified. apache#24058 (review) ## How was this patch tested? Revert the modified unit test cases in https://github.com/apache/spark/pull/24005/files#diff-b9ddfbc9be8d83ecf100b3b8ff9610b9R431 and https://github.com/apache/spark/pull/23383/files#diff-9ab56940ee5a53f2bb81e3c008653362R577, and test with them. Closes apache#24369 from gengliangwang/reviseInferSchemaMessage. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent 0bb716b commit 27d625d

File tree

3 files changed

+3
-3
lines changed

3 files changed

+3
-3
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileTable.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ abstract class FileTable(
5454
inferSchema(fileIndex.allFiles())
5555
}.getOrElse {
5656
throw new AnalysisException(
57-
s"Unable to infer schema for $name. It must be specified manually.")
57+
s"Unable to infer schema for $formatName. It must be specified manually.")
5858
}.asNullable
5959

6060
override lazy val schema: StructType = {

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -580,7 +580,7 @@ abstract class OrcQueryTest extends OrcTest {
580580
val m1 = intercept[AnalysisException] {
581581
testAllCorruptFiles()
582582
}.getMessage
583-
assert(m1.contains("Unable to infer schema"))
583+
assert(m1.contains("Unable to infer schema for ORC"))
584584
testAllCorruptFilesWithoutSchemaInfer()
585585
}
586586

sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -428,7 +428,7 @@ class DataFrameReaderWriterSuite extends QueryTest with SharedSQLContext with Be
428428
val message = intercept[AnalysisException] {
429429
testRead(spark.read.csv(), Seq.empty, schema)
430430
}.getMessage
431-
assert(message.toLowerCase(Locale.ROOT).contains("unable to infer schema for csv"))
431+
assert(message.contains("Unable to infer schema for CSV. It must be specified manually."))
432432

433433
testRead(spark.read.csv(dir), data, schema)
434434
testRead(spark.read.csv(dir, dir), data ++ data, schema)

0 commit comments

Comments
 (0)