MAPREDUCE-7508.FileInputFormat can throw ArrayIndexOutofBoundsException because of some concurrent execution. #7859
+35
−37
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…on because of some concurrent execution.
|
LGTM! |
|
💔 -1 overall
This message was automatically generated. |
Hexiaoqiao
reviewed
Aug 8, 2025
| @@ -369,6 +369,9 @@ public InputSplit[] getSplits(JobConf job, int numSplits) | |||
| } else { | |||
| blkLocations = fs.getFileBlockLocations(file, 0, length); | |||
| } | |||
| if (blkLocations.length == 0){ | |||
There was a problem hiding this comment.
Thanks for your contributions. Just wonder why it will meet length != 0 but blkLocations.length == 0, some corner case that lead inconsistent metadata of NameNode?
There was a problem hiding this comment.
@Hexiaoqiao thanks for review.
It is a rare corner case, only happens when using spark streaming monitors a path where the upstream system sometimes starts two identical tasks that attempt to create and write to the same HDFS file simultaneously. This can lead to conflicts where a file is created and written to twice in quick succession.
When Spark scans for files, it uses the FileInputFormat.getSplits() method to split the file. The first step in getSplits is to retrieve the file's length. If the file length is not zero, the next step is to get the block locations array for that file. However, if the two upstream programs rapidly create and write to the same file (i.e., the file is overwritten or appended to almost simultaneously), a race condition may occur:
The file's length is already non-zero, but calling getFileBlockLocations() returns an empty array because the file is being overwritten or is not yet fully written.
There was a problem hiding this comment.
Thanks. Got it. Make sense to me. +1 to check in.
BTW, the root cause here is invoke listStatus to get file status and invoke another interface getFileBlockLocations to get block location, but file has changed between this steps, right? If it is true, is it proper to use blkLocations only as condition at L364 rather than file.length which could be not the correct result here? Thanks again.
There was a problem hiding this comment.
Thanks. Got it. Make sense to me. +1 to check in. BTW, the root cause here is invoke
listStatusto get file status and invoke another interfacegetFileBlockLocationsto get block location, but file has changed between this steps, right? If it is true, is it proper to useblkLocationsonly as condition at L364 rather thanfile.lengthwhich could be not the correct result here? Thanks again.
Thanks for your review. It is a good solution to use blkLocations only as condition at L364, if blkLocations array is empty, file.length is also empty, this will not effect the later opretions.
|
💔 -1 overall
This message was automatically generated. |
|
LGTM. Please try to run CI again and better to get happy result. Thanks @liangyu-1 |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
@Hexiaoqiao |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…on because of some concurrent execution.
Description of PR
As Described in MAPREDUCE-7508
When Spark scans for files, it uses the FileInputFormat.getSplits() method to split the file. The first step in getSplits is to retrieve the file's length. If the file length is not zero, the next step is to get the block locations array for that file. However, if the two upstream programs rapidly create and write to the same file (i.e., the file is overwritten or appended to almost simultaneously), a race condition may occur:
The file's length is already non-zero,
but calling getFileBlockLocations() returns an empty array because the file is being overwritten or is not yet fully written.
When this happens, subsequent logic in getSplits (such as accessing the last element of the block locations array) will throw an ArrayIndexOutOfBoundsException because the block locations array is unexpectedly empty.
How was this patch tested?
I rebuild the project and ran on our cluster, spark did not throw Execptions.
For code changes:
If Array
blkLocationsis empty, it will continue to next iteration, so that it will now find the the last blockLocation of this file.LICENSE,LICENSE-binary,NOTICE-binaryfiles?