Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Commit 22b111e

Browse files
caneGuyjerryshao
authored andcommitted
[SPARK-21902][CORE] Print root cause for BlockManager#doPut
## What changes were proposed in this pull request? As logging below, actually exception will be hidden when removeBlockInternal throw an exception. `2017-08-31,10:26:57,733 WARN org.apache.spark.storage.BlockManager: Putting block broadcast_110 failed due to an exception 2017-08-31,10:26:57,734 WARN org.apache.spark.broadcast.BroadcastManager: Failed to create a new broadcast in 1 attempts java.io.IOException: Failed to create local dir in /tmp/blockmgr-5bb5ac1e-c494-434a-ab89-bd1808c6b9ed/2e. at org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:70) at org.apache.spark.storage.DiskStore.remove(DiskStore.scala:115) at org.apache.spark.storage.BlockManager.removeBlockInternal(BlockManager.scala:1339) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:910) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:726) at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1233) at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:122) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.BroadcastManager$$anonfun$newBroadcast$1.apply$mcVI$sp(BroadcastManager.scala:60) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:58) at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1415) at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1002) at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:924) at org.apache.spark.scheduler.DAGScheduler$$anonfun$submitWaitingChildStages$6.apply(DAGScheduler.scala:771) at org.apache.spark.scheduler.DAGScheduler$$anonfun$submitWaitingChildStages$6.apply(DAGScheduler.scala:770) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at org.apache.spark.scheduler.DAGScheduler.submitWaitingChildStages(DAGScheduler.scala:770) at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1235) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1662) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)` In this pr i will print exception first make troubleshooting more conveniently. PS: This one split from [PR-19133](apache#19133) ## How was this patch tested? Exsist unit test Author: zhoukang <[email protected]> Closes apache#19171 from caneGuy/zhoukang/print-rootcause.
1 parent 8866174 commit 22b111e

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

core/src/main/scala/org/apache/spark/storage/BlockManager.scala

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -988,11 +988,16 @@ private[spark] class BlockManager(
988988
logWarning(s"Putting block $blockId failed")
989989
}
990990
res
991+
} catch {
992+
// Since removeBlockInternal may throw exception,
993+
// we should print exception first to show root cause.
994+
case NonFatal(e) =>
995+
logWarning(s"Putting block $blockId failed due to exception $e.")
996+
throw e
991997
} finally {
992998
// This cleanup is performed in a finally block rather than a `catch` to avoid having to
993999
// catch and properly re-throw InterruptedException.
9941000
if (exceptionWasThrown) {
995-
logWarning(s"Putting block $blockId failed due to an exception")
9961001
// If an exception was thrown then it's possible that the code in `putBody` has already
9971002
// notified the master about the availability of this block, so we need to send an update
9981003
// to remove this block location.

0 commit comments

Comments
 (0)