Skip to content

Commit 529f847

Browse files
jiangxb1987cloud-fan
authored andcommitted
[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.
## What changes were proposed in this pull request? Address apache#20449 (comment), If `resultIter` is already a `InterruptibleIterator`, don't double wrap it. ## How was this patch tested? Existing tests. Author: Xingbo Jiang <[email protected]> Closes apache#20920 from jiangxb1987/SPARK-23040.
1 parent 15298b9 commit 529f847

File tree

1 file changed

+8
-3
lines changed

1 file changed

+8
-3
lines changed

core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -111,8 +111,13 @@ private[spark] class BlockStoreShuffleReader[K, C](
111111
case None =>
112112
aggregatedIter
113113
}
114-
// Use another interruptible iterator here to support task cancellation as aggregator or(and)
115-
// sorter may have consumed previous interruptible iterator.
116-
new InterruptibleIterator[Product2[K, C]](context, resultIter)
114+
115+
resultIter match {
116+
case _: InterruptibleIterator[Product2[K, C]] => resultIter
117+
case _ =>
118+
// Use another interruptible iterator here to support task cancellation as aggregator
119+
// or(and) sorter may have consumed previous interruptible iterator.
120+
new InterruptibleIterator[Product2[K, C]](context, resultIter)
121+
}
117122
}
118123
}

0 commit comments

Comments
 (0)