Resharding - Adding shards to an existing index #121082

ankikuma · 2025-01-28T18:22:24Z

This POC attempts to double the number of shards of an existing index.

ankikuma · 2025-01-28T18:25:29Z

I have not tested this out yet. The main approach is in MetadataAutoshardIndexService.java

…OC_addShards Refresh to latest

…/elasticsearch into 01172025/AutoShardPOC_addShards git pull

bcully

I know this operation isn't an actual safe reshard yet - no data is relocated or anything. But it might be nice to include a starter test that sets up, say, a single shard index, then reshards it and then submits some indexing and search operations and confirms that they are handled correctly (all data indexed and returned on search) and that the data spans shards?

...rg/elasticsearch/action/admin/indices/autoshard/AutoshardIndexClusterStateUpdateRequest.java

bcully · 2025-01-28T22:58:17Z

...rg/elasticsearch/action/admin/indices/autoshard/AutoshardIndexClusterStateUpdateRequest.java

+    }
+
+    public String index() {
+        return index;


Do we generally refer to indices by name internally? I had imagined that they would come in that way at the client side because that's the client-friendly way to refer to one, but that we might translate them to something less ambiguous like the uuid internally so we wouldn't have to think about things like ABA races.

From what I have seen, we use the name and the Index data structure quite interchangeably in the code. Not sure if there are some guidelines around which of the two to pass around internally.

bcully · 2025-01-28T22:58:55Z

...rg/elasticsearch/action/admin/indices/autoshard/AutoshardIndexClusterStateUpdateRequest.java

+    }
+
+    public String cause() {
+        return cause;


I guess this is some standard request component?

Not really, I think a request can contain whatever we think we need to put in the request. I was modeling this after CreateIndexClusterStateUpdateRequest and a cause field seemed useful for a re-sharding request. But really this is as bare bones a request as possible for re-sharding, we might need to add more stuff to this.

bcully · 2025-01-28T23:16:38Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataAutoshardIndexService.java

+                    waitForActiveShardsTimeout,
+                    delegate.map(shardsAcknowledged -> {
+                        if (shardsAcknowledged == false) {
+                            logger.debug(


Might be good for this to be higher than debug, it's I guess not going to be spammy.

bcully · 2025-01-28T23:17:12Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataAutoshardIndexService.java

+                    })
+                );
+            } else {
+                logger.trace("index creation not acknowledged for [{}]", request);


When does this happen? Is it an error condition?

bcully · 2025-01-28T23:18:28Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataAutoshardIndexService.java

+
+        Settings.Builder settingsBuilder = Settings.builder().put(sourceMetadata.getSettings());
+        settingsBuilder.remove(IndexMetadata.SETTING_NUMBER_OF_SHARDS);
+        settingsBuilder.put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, sourceNumShards * 2);


I suppose it might be for later, but seems like it might be nice for the new number of shards to be a request parameter.

…OC_addShards Move TransportStatelessAutoshardAction to stateless.java

…OC_addShards

…OC_addShards Refresh

…OC_addShards Update to latest

…OC_addShards Update branch

…/elasticsearch into 01172025/AutoShardPOC_addShards pull

server/src/main/java/org/elasticsearch/cluster/routing/UnassignedInfo.java

…OC_addShards Refresh to latest

…/elasticsearch into 01172025/AutoShardPOC_addShards git pull

…OC_addShards Latest

server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java

…OC_addShards Refresh branch

…OC_addShards refresh to latest

henningandersen

Looks good though I am in doubt about the routing-num-shards part.

henningandersen · 2025-03-05T11:05:44Z

server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java

+            settings = Settings.builder().put(settings).put(SETTING_NUMBER_OF_SHARDS, shardCount).build();
+            var newPrimaryTerms = new long[shardCount];
+            System.arraycopy(primaryTerms, 0, newPrimaryTerms, 0, this.primaryTerms.length);
+            primaryTerms = newPrimaryTerms;


I think we should fill the remaining part of the primary terms array with UNASSIGNED_PRIMARY_TERM, just like in initializePrimaryTerms and leave it to the client to fill those in.

henningandersen · 2025-03-05T11:10:15Z

server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java

+            var newPrimaryTerms = new long[shardCount];
+            System.arraycopy(primaryTerms, 0, newPrimaryTerms, 0, this.primaryTerms.length);
+            primaryTerms = newPrimaryTerms;
+            routingNumShards = shardCount;


I am not sure about this bit. I think we discussed it previously but do not recall the conclusion. When I look at MetadataCreateIndexService.calculateNumRoutingShards, it looks somewhat different to this.

henningandersen · 2025-03-05T11:11:43Z

server/src/main/java/org/elasticsearch/cluster/routing/UnassignedInfo.java

-        }
+        } else if (reason.equals(Reason.RESHARD_ADDED)
+            && out.getTransportVersion().before(TransportVersions.UNASSIGENEDINFO_RESHARD_ADDED)) {
+                out.writeByte((byte) Reason.FORCED_EMPTY_PRIMARY.ordinal());


Can we assert false here? I think we want this to never happen, i.e., do resharding in mixed clusters. I'd probably also prefer throwing over sending a more or less random reason here.

ok sure. But then should we also assert in the Unassigned constructor ?

…_addShards

Tim-Brooks

LGTM

elasticsearchmachine · 2025-03-05T23:22:24Z

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

This POC attempts to double the number of shards of an existing index.

Autoshard add shards

b622631

elasticsearchmachine added serverless-linked Added by automation, don't add manually v9.0.0 labels Jan 28, 2025

ankikuma requested review from Tim-Brooks, bcully and henningandersen January 28, 2025 18:25

elasticsearchmachine and others added 4 commits January 28, 2025 18:32

[CI] Auto commit changes from spotless

1644094

Add ActionModule and NodeConstruction

72a6aec

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

e92f449

…OC_addShards Refresh to latest

Merge branch '01172025/AutoShardPOC_addShards' of github.com:ankikuma…

a6fac03

…/elasticsearch into 01172025/AutoShardPOC_addShards git pull

ankikuma added the WIP label Jan 28, 2025

bcully reviewed Jan 28, 2025

View reviewed changes

ankikuma added 5 commits January 29, 2025 09:44

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

5f20573

…OC_addShards Move TransportStatelessAutoshardAction to stateless.java

Remove Stateless action from ActionModule

ce8c01f

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

6ac0171

…OC_addShards

NodeConstruction bind Autoshard service

c1d32f1

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

23a41ba

…OC_addShards Refresh

elasticsearchmachine added v9.1.0 and removed v9.0.0 labels Jan 30, 2025

ankikuma removed the serverless-linked Added by automation, don't add manually label Feb 7, 2025

ankikuma and others added 7 commits February 12, 2025 10:55

save work

8e2efee

Update Routing table builder

91d747d

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

c45e7f3

…OC_addShards Update to latest

[CI] Auto commit changes from spotless

1d7a1f3

UnassignedInfoTests

5ac2601

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

31e787a

…OC_addShards Update branch

Merge branch '01172025/AutoShardPOC_addShards' of github.com:ankikuma…

b38a5ef

…/elasticsearch into 01172025/AutoShardPOC_addShards pull

Tim-Brooks reviewed Feb 20, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/cluster/routing/UnassignedInfo.java Outdated Show resolved Hide resolved

Add testUpdateNumberOfShards

12ee0c6

ankikuma and others added 10 commits February 24, 2025 19:12

refactor RoutingTable

74b7508

[CI] Auto commit changes from spotless

b5c7d23

format

31aa598

move out reshardUpdateNumberOfShards

457a172

[CI] Auto commit changes from spotless

56ad696

commit

233be31

[CI] Auto commit changes from spotless

5dbfdba

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

76a18ed

…OC_addShards Refresh to latest

Merge branch '01172025/AutoShardPOC_addShards' of github.com:ankikuma…

475f9ab

…/elasticsearch into 01172025/AutoShardPOC_addShards git pull

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

e356913

…OC_addShards Latest

ankikuma marked this pull request as ready for review February 25, 2025 20:33

Tim-Brooks reviewed Feb 25, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java Show resolved Hide resolved

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

785939e

…OC_addShards Refresh branch

ankikuma changed the title ~~POC [Autoshard] Add shards to existing Index~~ Resharding - Adding shards to an existing index Mar 4, 2025

ankikuma added 2 commits March 4, 2025 15:04

transportVersions

b6a372b

Merge remote-tracking branch 'upstream/main' into 01172025/AutoShardP…

6457842

…OC_addShards refresh to latest

henningandersen reviewed Mar 5, 2025

View reviewed changes

Tim-Brooks added 4 commits March 5, 2025 12:07

Merge remote-tracking branch 'origin/main' into 01172025/AutoShardPOC…

1504271

…_addShards

Changes

3e6025b

Changes

d89616b

Change

ecdfe4d

Tim-Brooks approved these changes Mar 5, 2025

View reviewed changes

Tim-Brooks added :Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. >non-issue and removed WIP labels Mar 5, 2025

elasticsearchmachine added the Team:Distributed Indexing Meta label for Distributed Indexing team label Mar 5, 2025

Tim-Brooks merged commit 473c4da into elastic:main Mar 6, 2025
17 checks passed

georgewallace pushed a commit to georgewallace/elasticsearch that referenced this pull request Mar 11, 2025

Resharding - Adding shards to an existing index (elastic#121082)

87e0ef6

This POC attempts to double the number of shards of an existing index.

costin pushed a commit to costin/elasticsearch that referenced this pull request Mar 15, 2025

Resharding - Adding shards to an existing index (elastic#121082)

69afdcb

This POC attempts to double the number of shards of an existing index.

Resharding - Adding shards to an existing index #121082

Resharding - Adding shards to an existing index #121082

Uh oh!

Conversation

ankikuma commented Jan 28, 2025

Uh oh!

ankikuma commented Jan 28, 2025

Uh oh!

bcully left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

henningandersen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim-Brooks left a comment

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Mar 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ankikuma Feb 20, 2025 •

edited

Loading