Skip to content

Commit 5d8aee5

Browse files
dilipbiswaldongjoon-hyun
authored andcommitted
[SPARK-27445][SQL][TEST] Update SQLQueryTestSuite to process files ending with .sql
## What changes were proposed in this pull request? While using vi or vim to edit the test files the .swp or .swo files are created and attempt to run the test suite in the presence of these files causes errors like below : ``` nfo] - subquery/exists-subquery/.exists-basic.sql.swp *** FAILED *** (117 milliseconds) [info] java.io.FileNotFoundException: /Users/dbiswal/mygit/apache/spark/sql/core/target/scala-2.12/test-classes/sql-tests/results/subquery/exists-subquery/.exists-basic.sql.swp.out (No such file or directory) [info] at java.io.FileInputStream.open0(Native Method) [info] at java.io.FileInputStream.open(FileInputStream.java:195) [info] at java.io.FileInputStream.<init>(FileInputStream.java:138) [info] at org.apache.spark.sql.catalyst.util.package$.fileToString(package.scala:49) [info] at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:247) [info] at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runTest$11(SQLQueryTestSuite.scala:192) ``` ~~This minor pr adds these temp files in the ignore list.~~ While computing the list of test files to process, only consider files with `.sql` extension. This makes sure the unwanted temp files created from various editors are ignored from processing. ## How was this patch tested? Verified manually. Closes apache#24333 from dilipbiswal/dkb_sqlquerytest. Authored-by: Dilip Biswal <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 4ec7f63 commit 5d8aee5

File tree

1 file changed

+7
-4
lines changed

1 file changed

+7
-4
lines changed

sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -105,11 +105,11 @@ class SQLQueryTestSuite extends QueryTest with SharedSQLContext {
105105
private val inputFilePath = new File(baseResourcePath, "inputs").getAbsolutePath
106106
private val goldenFilePath = new File(baseResourcePath, "results").getAbsolutePath
107107

108+
private val validFileExtensions = ".sql"
109+
108110
/** List of test cases to ignore, in lower cases. */
109111
private val blackList = Set(
110-
"blacklist.sql", // Do NOT remove this one. It is here to test the blacklist functionality.
111-
".DS_Store" // A meta-file that may be created on Mac by Finder App.
112-
// We should ignore this file from processing.
112+
"blacklist.sql" // Do NOT remove this one. It is here to test the blacklist functionality.
113113
)
114114

115115
// Create all the test cases.
@@ -329,7 +329,10 @@ class SQLQueryTestSuite extends QueryTest with SharedSQLContext {
329329
/** Returns all the files (not directories) in a directory, recursively. */
330330
private def listFilesRecursively(path: File): Seq[File] = {
331331
val (dirs, files) = path.listFiles().partition(_.isDirectory)
332-
files ++ dirs.flatMap(listFilesRecursively)
332+
// Filter out test files with invalid extensions such as temp files created
333+
// by vi (.swp), Mac (.DS_Store) etc.
334+
val filteredFiles = files.filter(_.getName.endsWith(validFileExtensions))
335+
filteredFiles ++ dirs.flatMap(listFilesRecursively)
333336
}
334337

335338
/** Load built-in test tables into the SparkSession. */

0 commit comments

Comments
 (0)