[SPARK-54928] Add shuffle migration statistics to MigrationInfo #53706
+133
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit enhances the decommissioning process by adding detailed shuffle migration tracking to the `MigrationInfo` case class. The new `MigrationStat` case class tracks: - `numBlocksLeft`: Number of shuffle blocks remaining to be migrated - `totalMigratedSize`: Total size in bytes of migrated shuffle blocks - `numMigratedBlock`: Number of shuffle blocks successfully migrated - `totalBlocks`: Total number of shuffle blocks discovered - `totalSize`: Total size of all shuffle blocks (migrated + remaining) - `deletedBlocks`: Number of shuffle blocks deleted during migration This provides comprehensive visibility into shuffle block migration progress and helps with monitoring decommissioning efficiency. A unit test with enhanced size verification is included to ensure all statistics are calculated and reported correctly. Change-Id: I21a8913343ca92e353243c5a34b144a2e43f828c
JIRA Issue Information=== Improvement SPARK-54928 === This comment was automatically generated by GitHub Actions |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This adds detailed shuffle migration statistics to block manager decommissioning. A new
MigrationInfo/MigrationStatexposes counts and sizes for discovered, migrated, remaining,and deleted shuffle blocks, and the executor shutdown path uses the new struct. A unit test
verifies the stats and size accounting.
Why are the changes needed?
Decommissioning currently exposes only a boolean for “all blocks migrated.” The additional
stats provide visibility into shuffle migration progress and help diagnose slow/failed
migrations.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
./build/sbt "testOnly org.apache.spark.storage.BlockManagerDecommissionUnitSuite"(Note: the suite passed, but the sbt run ended with
sql-api / Antlr4 / antlr4Generateexit code 1.)Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)