Skip to content

Commit 3628242

Browse files
gaborgsomogyisrowen
authored andcommitted
[MINOR][DSTREAMS] Add DStreamCheckpointData.cleanup warning if delete returns false
## What changes were proposed in this pull request? While I was reviewing apache#24235 I've found a minor addition possibility. Namely `FileSystem.delete` returns a boolean which is not yet checked. In this PR I've added a warning message when it returns false. I've added this as MINOR because no control flow change introduced. ## How was this patch tested? Existing unit tests. Closes apache#24263 from gaborgsomogyi/SPARK-27301-minor. Authored-by: Gabor Somogyi <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent d7dd59a commit 3628242

File tree

1 file changed

+5
-2
lines changed

1 file changed

+5
-2
lines changed

streaming/src/main/scala/org/apache/spark/streaming/dstream/DStreamCheckpointData.scala

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,9 +87,12 @@ class DStreamCheckpointData[T: ClassTag](dstream: DStream[T])
8787
if (fileSystem == null) {
8888
fileSystem = path.getFileSystem(dstream.ssc.sparkContext.hadoopConfiguration)
8989
}
90-
fileSystem.delete(path, true)
90+
if (fileSystem.delete(path, true)) {
91+
logInfo("Deleted checkpoint file '" + file + "' for time " + time)
92+
} else {
93+
logWarning(s"Error deleting old checkpoint file '$file' for time $time")
94+
}
9195
timeToCheckpointFile -= time
92-
logInfo("Deleted checkpoint file '" + file + "' for time " + time)
9396
} catch {
9497
case e: Exception =>
9598
logWarning("Error deleting old checkpoint file '" + file + "' for time " + time, e)

0 commit comments

Comments
 (0)