Skip to content

Commit a5ccbce

Browse files
madianjundongjoon-hyun
authored andcommitted
[SPARK-30067][CORE] Fix fragment offset comparison in getBlockHosts
### What changes were proposed in this pull request? A bug fixed about the code in getBlockHosts() function. In the case "The fragment ends at a position within this block", the end of fragment should be before the end of block,where the "end of block" means `b.getOffset + b.getLength`,not `b.getLength`. ### Why are the changes needed? When comparing the fragment end and the block end,we should use fragment's `offset + length`,and then compare to the block's `b.getOffset + b.getLength`, not the block's length. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? No test. Closes apache#26650 from mdianjun/fix-getBlockHosts. Authored-by: madianjun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent da27f91 commit a5ccbce

File tree

1 file changed

+4
-3
lines changed

1 file changed

+4
-3
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/PartitionedFileUtil.scala

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,13 +64,14 @@ object PartitionedFileUtil {
6464
offset: Long,
6565
length: Long): Array[String] = {
6666
val candidates = blockLocations.map {
67-
// The fragment starts from a position within this block
67+
// The fragment starts from a position within this block. It handles the case where the
68+
// fragment is fully contained in the block.
6869
case b if b.getOffset <= offset && offset < b.getOffset + b.getLength =>
6970
b.getHosts -> (b.getOffset + b.getLength - offset).min(length)
7071

7172
// The fragment ends at a position within this block
72-
case b if offset <= b.getOffset && offset + length < b.getLength =>
73-
b.getHosts -> (offset + length - b.getOffset).min(length)
73+
case b if b.getOffset < offset + length && offset + length < b.getOffset + b.getLength =>
74+
b.getHosts -> (offset + length - b.getOffset)
7475

7576
// The fragment fully contains this block
7677
case b if offset <= b.getOffset && b.getOffset + b.getLength <= offset + length =>

0 commit comments

Comments
 (0)