Skip to content

Commit f28b1a4

Browse files
juliuszsompolskihvanhovell
authored andcommitted
[SPARK-22721] BytesToBytesMap peak memory not updated.
## What changes were proposed in this pull request? Follow-up to earlier commit. The peak memory of BytesToBytesMap is not updated in more places - spill() and destructiveIterator(). ## How was this patch tested? Manually. Author: Juliusz Sompolski <[email protected]> Closes #19923 from juliuszsompolski/SPARK-22721cd.
1 parent f88a67b commit f28b1a4

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -347,6 +347,8 @@ public long spill(long numBytes) throws IOException {
347347
return 0L;
348348
}
349349

350+
updatePeakMemoryUsed();
351+
350352
// TODO: use existing ShuffleWriteMetrics
351353
ShuffleWriteMetrics writeMetrics = new ShuffleWriteMetrics();
352354

@@ -424,6 +426,7 @@ public MapIterator iterator() {
424426
* `lookup()`, the behavior of the returned iterator is undefined.
425427
*/
426428
public MapIterator destructiveIterator() {
429+
updatePeakMemoryUsed();
427430
return new MapIterator(numValues, loc, true);
428431
}
429432

0 commit comments

Comments
 (0)