Adding metadata to support shard split and to resolve shard ids based on child shard hash#20859
Conversation
PR Reviewer Guide 🔍(Review updated until commit 8d02fc7)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 8d02fc7 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 49efa6a
Suggestions up to commit 9a4e365
Suggestions up to commit 4618229
Suggestions up to commit ec0bf77
Suggestions up to commit 2db9022
|
|
❌ Gradle check result for a79f554: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
a79f554 to
276231f
Compare
|
Persistent review updated to latest commit 276231f |
276231f to
6f0508b
Compare
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 6f0508b.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
server/src/main/java/org/opensearch/cluster/metadata/SplitShardsMetadata.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/cluster/metadata/ShardRange.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/cluster/metadata/ShardRange.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/cluster/metadata/ShardRange.java
Outdated
Show resolved
Hide resolved
6f0508b to
cc1269d
Compare
|
Persistent review updated to latest commit cc1269d |
cc1269d to
f642557
Compare
|
Persistent review updated to latest commit f642557 |
|
Thanks @andrross for the review! Addressed your comments. Can you please take a look again? |
|
❌ Gradle check result for f642557: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
f642557 to
dd496ee
Compare
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit dd496ee.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
|
Persistent review updated to latest commit dd496ee |
dd496ee to
33769e5
Compare
|
Persistent review updated to latest commit 33769e5 |
|
❌ Gradle check result for 33769e5: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
33769e5 to
2db9022
Compare
|
Persistent review updated to latest commit 2db9022 |
|
❌ Gradle check result for 2db9022: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
2db9022 to
ec0bf77
Compare
|
Persistent review updated to latest commit ec0bf77 |
ec0bf77 to
4618229
Compare
|
Persistent review updated to latest commit 4618229 |
|
❌ Gradle check result for 4618229: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
4618229 to
9a4e365
Compare
|
Persistent review updated to latest commit 9a4e365 |
|
❌ Gradle check result for 9a4e365: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
9a4e365 to
49efa6a
Compare
|
Persistent review updated to latest commit 49efa6a |
|
❌ Gradle check result for 49efa6a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
… on child shard hash Signed-off-by: vikasvb90 <vikasvb@amazon.com>
49efa6a to
8d02fc7
Compare
|
Persistent review updated to latest commit 8d02fc7 |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #20859 +/- ##
============================================
- Coverage 73.32% 73.31% -0.02%
- Complexity 72272 72410 +138
============================================
Files 5797 5804 +7
Lines 330323 330885 +562
Branches 47676 47822 +146
============================================
+ Hits 242215 242576 +361
- Misses 68663 68828 +165
- Partials 19445 19481 +36 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This reverts commit ed2e100. The change to IndexMetadata has resulted in bwc failures with the remote cluster state feature.
This reverts commit ed2e100. The change to IndexMetadata has resulted in bwc failures with the remote cluster state feature. Signed-off-by: Andrew Ross <andrross@amazon.com>
Description
Introduces metadata infrastructure for online (live) shard splitting in OpenSearch.
ShardRange - Represents a hash range owned by a child shard, mapping a shard ID to a contiguous range of hash values. Supports range containment checks.
SplitShardsMetadata - Tracks the shard split tree within IndexMetadata. Maintains root-to-child shard mappings, in-progress split state, and active shard IDs. Given a document's routing hash, it resolves the correct child shard by walking the split tree. Ensures shard range consistency.
primaryTermsMap in IndexMetadata - Adds a Map<Integer, Long> alongside the existing long[] array to support non-contiguous shard IDs that arise after splits. The map becomes the source of truth for primary term look-ups.
OperationRouting - Updated to resolve document routing to the correct child shard by delegating to SplitShardsMetadata.getShardIdOfHash().
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
This is based on the in-place shard split design.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.