Add support for delegating write to split target #136241

Tim-Brooks · 2025-10-09T03:45:24Z

This commit adds the logic to delegate bulk shard requests to the split
target when a primary receives a request from a stale coordinator.

…itRequestOnSourceTwoPass Refresh

…com:ankikuma/elasticsearch into 09162025/ReshardSplitRequestOnSourceTwoPass merged

…itRequestOnSourceTwoPass Refresh

…com:ankikuma/elasticsearch into 09162025/ReshardSplitRequestOnSourceTwoPass pull

…itRequestOnSourceTwoPass Refresh

…com:ankikuma/elasticsearch into 09162025/ReshardSplitRequestOnSourceTwoPass Pull

…itRequestOnSourceTwoPass Refresh

…com:ankikuma/elasticsearch into ankikuma-09162025/ReshardSplitRequestOnSourceTwoPass

elasticsearchmachine · 2025-10-09T03:45:48Z

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

…hardSplitRequestOnSourceTwoPass

...r/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

server/src/main/java/org/elasticsearch/action/bulk/ShardBulkSplitHelper.java

lkts · 2025-10-10T16:59:06Z

server/src/main/java/org/elasticsearch/action/bulk/ShardBulkSplitHelper.java

+    public static Tuple<BulkShardResponse, Exception> combineResponses(
+        BulkShardRequest originalRequest,
+        Map<ShardId, BulkShardRequest> splitRequests,
+        Map<ShardId, Tuple<BulkShardResponse, Exception>> responses


We need Either type :)

server/src/main/java/org/elasticsearch/cluster/routing/IndexRouting.java

ankikuma · 2025-10-10T17:25:27Z

LGTM

…hardSplitRequestOnSourceTwoPass

bcully

Looks great. I had a few questions/suggestions but nothing major. We probably want some more end-to-end ITs when this lands.

bcully · 2025-10-14T18:50:57Z

server/src/main/java/org/elasticsearch/action/bulk/ShardBulkSplitHelper.java

+
+    private ShardBulkSplitHelper() {}
+
+    public static Map<ShardId, BulkShardRequest> splitRequests(BulkShardRequest request, ProjectMetadata project) {


can we document this a bit? Like I'm assuming that the caller has blocked handoff here by taking permits. It looks like there's also an expectation that the caller won't call this function unless it has determined that the coordinator's shard summary doesn't match the shard's, so it doesn't need to fast-path for that.
And there looks like there's an expectation on the caller that if the request has no items, an entry for the source shard with no items will be returned, but that otherwise this function should not generate empty sub-requests (e.g., if it created a map eagerly for both source and target that would be a bug).

bcully · 2025-10-14T18:59:37Z

server/src/main/java/org/elasticsearch/action/bulk/ShardBulkSplitHelper.java

+            }
+        }
+        BulkShardResponse bulkShardResponse = new BulkShardResponse(originalRequest.shardId(), bulkItemResponses);
+        // TODO: Decide how to handle


I think you looked at consumers of this? If so could you leave a breadcrumb to your investigation? If not, we should ticket that and link the ticket here.

bcully · 2025-10-14T22:40:12Z

server/src/main/java/org/elasticsearch/action/support/replication/ReplicationSplitHelper.java

+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Supplier;
+
+public class ReplicationSplitHelper<


it would be nice to have a little javadoc explaining what this class is for if you get a chance

bcully · 2025-10-14T23:07:14Z

server/src/main/java/org/elasticsearch/action/bulk/ShardBulkSplitHelper.java

+                new ShardId(index, newShardId),
+                shardNum -> new ArrayList<>()
+            );
+            shardRequests.add(new BulkItemRequest(bulkItemRequest.id(), bulkItemRequest.request()));


should we assert anything about the shard id in bulkItemRequest.request()? From skimming code I think it's probably null (thin serialization) but I don't know if it always is, or how it would be used if it's not null.

bcully · 2025-10-14T23:14:37Z

server/src/main/java/org/elasticsearch/action/DocWriteRequest.java

     */
    int route(IndexRouting indexRouting);

+    int rerouteAtSourceDuringResharding(IndexRouting indexRouting);


doc comment? I think they're nice for interface/abstract methods.

bcully · 2025-10-14T23:14:52Z

server/src/main/java/org/elasticsearch/cluster/routing/IndexRouting.java

     */
    public abstract int indexShard(IndexRequest indexRequest);

+    public abstract int rerouteToTarget(IndexRequest indexRequest);


doc comment? :)

bcully · 2025-10-14T23:17:23Z

server/src/main/java/org/elasticsearch/cluster/routing/IndexRouting.java

+                }
+                return indexShard(indexRequest);
+            } else if (addIdWithRoutingHash) {
+                // TODO: is this correct?


we're probably going to have to generate test cases for tsdb/logsdb

bcully · 2025-10-14T23:19:31Z

.../src/test/java/org/elasticsearch/action/support/replication/ReplicationSplitHelperTests.java

+        IndexMetadata indexMetadata = IndexMetadata.builder(indexName).settings(settings).build();
+        indexMetadata = IndexMetadata.builder(indexMetadata).reshardAddShards(2).build();
+
+        SplitShardCountSummary staleSummary = SplitShardCountSummary.fromInt(1);


personally I'd prefer to generate this from a 1 shard metadata instead of assuming the serialization meaning, but I suppose if we changed serialization we'd notice.

ankikuma and others added 28 commits September 17, 2025 17:13

Split requestwq

53a338d

commit

064f3ec

Merge remote-tracking branch 'upstream/main' into 09162025/ReshardSpl…

984063e

…itRequestOnSourceTwoPass Refresh

es

b93104a

fix reroute at source

9b79168

[CI] Auto commit changes from spotless

12e46af

fix reroute at source bugs

770e0f1

Merge branch '09162025/ReshardSplitRequestOnSourceTwoPass' of github.…

3e72e7a

…com:ankikuma/elasticsearch into 09162025/ReshardSplitRequestOnSourceTwoPass merged

[CI] Update transport version definitions

0cf593c

Merge remote-tracking branch 'upstream/main' into 09162025/ReshardSpl…

826d11a

…itRequestOnSourceTwoPass Refresh

Merge branch '09162025/ReshardSplitRequestOnSourceTwoPass' of github.…

a793d5c

…com:ankikuma/elasticsearch into 09162025/ReshardSplitRequestOnSourceTwoPass pull

refresh

b153030

commit

0dc91f1

Merge remote-tracking branch 'upstream/main' into 09162025/ReshardSpl…

9dc5079

…itRequestOnSourceTwoPass Refresh

fix reroute logic

a9c9885

spotless

c2c5490

Merge remote-tracking branch 'upstream/main' into 09162025/ReshardSpl…

df26a61

…itRequestOnSourceTwoPass Refresh

Merge remote-tracking branch 'upstream/main' into 09162025/ReshardSpl…

636225d

…itRequestOnSourceTwoPass Refresh

commit

4584a23

[CI] Auto commit changes from spotless

c9c32d8

commit

fe1d3c3

Merge remote-tracking branch 'upstream/main' into 09162025/ReshardSpl…

fd43a75

…itRequestOnSourceTwoPass Refresh

Merge branch '09162025/ReshardSplitRequestOnSourceTwoPass' of github.…

7aca078

…com:ankikuma/elasticsearch into 09162025/ReshardSplitRequestOnSourceTwoPass Pull

Merge remote-tracking branch 'upstream/main' into 09162025/ReshardSpl…

f1c4b2d

…itRequestOnSourceTwoPass Refresh

Merge branch '09162025/ReshardSplitRequestOnSourceTwoPass' of github.…

0c16c45

…com:ankikuma/elasticsearch into ankikuma-09162025/ReshardSplitRequestOnSourceTwoPass

Changes

5fcc140

Change

c14623a

Change

9946884

Tim-Brooks added >non-issue :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. labels Oct 9, 2025

Tim-Brooks added the v9.3.0 label Oct 9, 2025

elasticsearchmachine added the Team:Distributed Indexing Meta label for Distributed Indexing team label Oct 9, 2025

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Oct 9, 2025

Tim-Brooks and others added 5 commits October 9, 2025 16:43

Changes

646bdbe

Merge remote-tracking branch 'origin/main' into ankikuma-09162025/Res…

7c5235a

…hardSplitRequestOnSourceTwoPass

Change

701d19c

Merge remote-tracking branch 'origin/main' into ankikuma-09162025/Res…

ed015ee

…hardSplitRequestOnSourceTwoPass

[CI] Auto commit changes from spotless

d3648e9

ankikuma reviewed Oct 10, 2025

View reviewed changes

...r/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java Outdated Show resolved Hide resolved

lkts approved these changes Oct 10, 2025

View reviewed changes

ankikuma approved these changes Oct 10, 2025

View reviewed changes

Tim-Brooks added 5 commits October 13, 2025 12:26

Change

5ba4e73

Merge remote-tracking branch 'origin/main' into ankikuma-09162025/Res…

aa8505e

…hardSplitRequestOnSourceTwoPass

Change

dc0fb02

Merge remote-tracking branch 'origin/main' into ankikuma-09162025/Res…

6cc838c

…hardSplitRequestOnSourceTwoPass

Fix

292f8cc

bcully approved these changes Oct 14, 2025

View reviewed changes


		private ShardBulkSplitHelper() {}

		public static Map<ShardId, BulkShardRequest> splitRequests(BulkShardRequest request, ProjectMetadata project) {

Add support for delegating write to split target #136241

Are you sure you want to change the base?

Add support for delegating write to split target #136241

Conversation

Tim-Brooks commented Oct 9, 2025

Uh oh!

elasticsearchmachine commented Oct 9, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ankikuma commented Oct 10, 2025

Uh oh!

bcully left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants