[SPARK-54947][CORE] Refactor block mapping with BlockInfoGroup for better block management

[Ngone51]

What changes were proposed in this pull request?

This PR refactors the block mapping optimization by #52646. The refator contains these main changes:

• Instead of using the HashMap for maintaining the block mappings, this PR introduces a new class BlockInfoGroup.Collection to manage the block mapping, providing rich APIs for group level block operations, e.g., size, get. Blocks that do not naturally having the mapping will use BlockInfoGroup.Singleton instead.

• Integrates the existing blockInfoWrappers with BlockInfoGroup for centralized block management rather than maintaing multiple separate HashMaps. And this changes us to operate blocks on the group level first and then the individual blocks.

• Replace the block mapping APIs (e.g., getBlockIdsFromMapping , removeFromMapping) that are added by [SPARK-53446][CORE] Optimize BlockManager remove operations with cached block mappings #52646 with the new block group APIs, e.g., getBlockIdsForGroup, removeBlockGroup.

Why are the changes needed?

This refactor centralizes the block management with the new data structure and intorduces better block management semantics. It should be good for codebase maintainence for the long term.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Updated tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[github-actions]

JIRA Issue Information

=== Improvement SPARK-54947 ===
Summary: Refactor block mapping by reusing blockInfoWrappers for better block management
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions