Skip to content

Commit e47f48c

Browse files
steveloughranHyukjinKwon
authored andcommitted
[SPARK-20886][CORE] HadoopMapReduceCommitProtocol to handle FileOutputCommitter.getWorkPath==null
## What changes were proposed in this pull request? Handles the situation where a `FileOutputCommitter.getWorkPath()` returns `null` by downgrading to the supplied `path` argument. The existing code does an `Option(workPath.toString).getOrElse(path)`, which triggers an NPE in the `toString()` operation if the workPath == null. The code apparently was meant to handle this (hence the getOrElse() clause, but as the NPE has already occurred at that point the else-clause never gets invoked. ## How was this patch tested? Manually, with some later code review. Author: Steve Loughran <[email protected]> Closes apache#18111 from steveloughran/cloud/SPARK-20886-committer-NPE.
1 parent 3d0e174 commit e47f48c

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,8 @@ class HadoopMapReduceCommitProtocol(jobId: String, path: String)
7373

7474
val stagingDir: String = committer match {
7575
// For FileOutputCommitter it has its own staging path called "work path".
76-
case f: FileOutputCommitter => Option(f.getWorkPath.toString).getOrElse(path)
76+
case f: FileOutputCommitter =>
77+
Option(f.getWorkPath).map(_.toString).getOrElse(path)
7778
case _ => path
7879
}
7980

0 commit comments

Comments
 (0)