Suspend Index throttling when relocating #128797

ankikuma · 2025-06-03T03:56:11Z

Addresses ES-11770.

If index throttling is enabled such that it pauses all indexing threads that try to index into a shard, this can starve other tasks such as relocation that try to acquire all indexing permits. This PR addresses this by suspending throttling to allow the indexing threads that are holding the permits to pass.

…exingForPermits Refresh

elasticsearchmachine · 2025-06-03T03:57:02Z

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

ankikuma · 2025-06-03T04:02:14Z

I noticed during testing that throttling gets disabled once a shard is moved. I guess this is because of the way the engine is created for the relocated shard. But I haven't had a chance to dig into the relocation code to verify that this is expected behaviour.

henningandersen

Thanks for working on this. Left a number of comments.

henningandersen · 2025-06-03T04:31:30Z

server/src/main/java/org/elasticsearch/index/shard/IndexShardOperationPermits.java

+        indexShard.suspendThrottling();
        waitUntilBlocked(ActionListener.assertOnce(onAcquired), timeout, timeUnit, executor);
+        // TODO: Does this do anything ? Looks like the relocated shard does not have throttling enabled
+        indexShard.resumeThrottling();


I think I would prefer to handle this outside this class, we can make a method in IndexShard that wraps blockOperations and does this, avoiding sending an object to this method and the effect on testing etc.

Also, notice that this is sort of incorrect as is in that we sometimes call this with the executor set to the generic thread pool. We should instead resume throttling when the listener is called, that will handle all cases.

henningandersen · 2025-06-03T04:35:34Z

server/src/main/java/org/elasticsearch/index/shard/IndexShard.java

        }
    }

+    public boolean isIndexingPaused() {


This does not seem necessary to expose outside IndexShard?

I was calling it from RelocationIT. I can remove it. Just for my understanding, why is it risky to expose this ?

Removed in the latest upload

It exposes internal state from the engine. As such it is not "risky", but exposing more than necessary breaks encapsulation. In particular this one is only there for testing and can be fetched just as easily without this. The IndexShard interface is huge and I'd like to keep the surface it has down.

Thanks for explaining Henning

server/src/main/java/org/elasticsearch/index/shard/IndexShard.java

henningandersen · 2025-06-03T09:26:02Z

server/src/internalClusterTest/java/org/elasticsearch/recovery/RelocationIT.java

+        logger.info("--> index more docs so we have something in the translog");
+        for (int i = 10; i < 20; i++) {
+            prepareIndex("test").setId(Integer.toString(i)).setSource("field", "value" + i).get();
+        }


I do not folllow why this is important to the test?

It is not. I wrote this test by modifying testRelocationWhileIndexingRandom() so it's just a carry over from there. Removed it.

henningandersen · 2025-06-03T09:33:01Z

server/src/internalClusterTest/java/org/elasticsearch/recovery/RelocationIT.java

+        assertHitCount(prepareSearch("test").setSize(0), 20);
+
+        logger.info("--> relocate the shard from node1 to node2");
+        ClusterRerouteUtils.reroute(client(), new MoveAllocationCommand("test", 0, node_1, node_2));


I prefer to set an allocation rule through index settings. Someting like index.routing.allocation.include._id = node_2.

I am not sure I follow this comment.

Or maybe I do. Like this ?
updateIndexSettings(Settings.builder().put("index.routing.allocation.include._id", node_2), "test");

Do you want me to change this everywhere in this file ?

I am not sure how to make that work. I tried this:
updateIndexSettings(Settings.builder().put("index.routing.allocation.include._id", nodes[toNode]), "test");
ensureGreen(ACCEPTABLE_RELOCATION_TIME, "test");

But it looks like this is not enough to ensure that the shard has moved to the target node.

You need to use ._name if you use node_2, like done here (though that one excludes, you can do that too - or use include, both should work).

henningandersen · 2025-06-03T09:36:26Z

server/src/internalClusterTest/java/org/elasticsearch/recovery/RelocationIT.java

+        assertThat(clusterHealthResponse.isTimedOut(), equalTo(false));
+
+        // Relocated shard is not throttled
+        assertThat(shard.isIndexingPaused(), equalTo(false));


This seems surprising, why is it not throttled?

I initially thought it might be because the node that we relocate the shard to does not have PAUSE_THROTTLING enabled. But that doesn't help either. So I am guessing it has to do with how we do the relocation, wouldn't we have to recreate the engine on the new node and it probably will not transfer throttling ?

But wait, this is the original source shard we are talking about, not the relocated target shard, so it should have throttling enabled after we resume throttling. I will need to look into this a bit more.

Oh I figured it out, it's because the engine is null for the source shard. I will just get rid of this check, I don't think it is useful.

server/src/internalClusterTest/java/org/elasticsearch/recovery/RelocationIT.java

…exingForPermits Refresh

…exingForPermits Refresh branch

…uma/elasticsearch into 05192025/UnpauseIndexingForPermits pull

…exingForPermits refresh branch

…uma/elasticsearch into 05192025/UnpauseIndexingForPermits Refresh branch

…exingForPermits refresh branch

…exingForPermits Refresh branch

…uma/elasticsearch into 05192025/UnpauseIndexingForPermits pull

ankikuma · 2025-07-13T18:33:28Z

There was a problem with RelocationIT#testRelocationWhileIndexingRandom() where we were relocating the replica and not the primary. I changed it so we are relocating the primary, and it works fine now.

henningandersen

LGTM.

henningandersen · 2025-07-30T10:00:09Z

server/src/main/java/org/elasticsearch/index/shard/IndexShard.java

+     * @param timeUnit   the time unit of the {@code timeout} argument
+     * @param executor   executor on which to wait for in-flight operations to finish and acquire all permits
+     */
+    public void blockOperations(


Can this be private?

Suggested change

public void blockOperations(

private void blockOperations(

…exingForPermits Refresh branch

…exingForPermits Refresh

If index throttling is enabled such that it pauses all indexing threads that try to index into a shard, this can starve other tasks such as relocation that try to acquire all indexing permits. This PR addresses this by suspending throttling to allow the indexing threads that are holding the permits to pass. Addresses ES-11770.

ankikuma added 9 commits May 23, 2025 13:59

pause indexing and race condition diags

dfe639f

commit

2601960

commit

ec91a19

refresh branch

3ddb78b

commit

f12949e

commit

90670f3

commit

45e3799

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

cd43ab3

…exingForPermits Refresh

commit

bf91cab

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.1.0 labels Jun 3, 2025

ankikuma added :Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. and removed needs:triage Requires assignment of a team area label v9.1.0 labels Jun 3, 2025

elasticsearchmachine added the Team:Distributed Indexing Meta label for Distributed Indexing team label Jun 3, 2025

ankikuma added >non-issue and removed Team:Distributed Indexing Meta label for Distributed Indexing team labels Jun 3, 2025

elasticsearchmachine added the Team:Distributed Indexing Meta label for Distributed Indexing team label Jun 3, 2025

ankikuma added v9.1.0 and removed Team:Distributed Indexing Meta label for Distributed Indexing team labels Jun 3, 2025

elasticsearchmachine added the Team:Distributed Indexing Meta label for Distributed Indexing team label Jun 3, 2025

ankikuma requested a review from henningandersen June 3, 2025 04:03

henningandersen reviewed Jun 3, 2025

View reviewed changes

ankikuma added 5 commits June 3, 2025 20:14

address review comments

e642fea

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

a249357

…exingForPermits Refresh

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

a11d2fd

…exingForPermits Refresh branch

test failure

82a37f5

remove commented code

560a035

ankikuma and others added 13 commits June 5, 2025 10:51

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

6315189

…exingForPermits Refresh branch

address comments

5e81c31

test

661fa12

[CI] Auto commit changes from spotless

77058ad

test

1d872c1

Merge branch '05192025/UnpauseIndexingForPermits' of github.com:ankik…

129622a

…uma/elasticsearch into 05192025/UnpauseIndexingForPermits pull

old changes

610bc0f

pull changes

e50c200

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

343d9f2

…exingForPermits refresh branch

Merge branch '05192025/UnpauseIndexingForPermits' of github.com:ankik…

94f212a

…uma/elasticsearch into 05192025/UnpauseIndexingForPermits Refresh branch

fix test

6a2bf2b

[CI] Auto commit changes from spotless

ee22887

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

8e9d456

…exingForPermits refresh branch

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

ankikuma and others added 5 commits July 9, 2025 17:27

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

371dfbe

…exingForPermits Refresh branch

fix test + throttle only for primary

c9438b4

[CI] Auto commit changes from spotless

31fe364

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

9729ebf

…exingForPermits Refresh branch

Merge branch '05192025/UnpauseIndexingForPermits' of github.com:ankik…

1134a5e

…uma/elasticsearch into 05192025/UnpauseIndexingForPermits pull

henningandersen approved these changes Jul 30, 2025

View reviewed changes

ankikuma added 5 commits July 30, 2025 09:18

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

8d6fc17

…exingForPermits Refresh branch

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

a1bf5a3

…exingForPermits Refresh branch

Merge remote-tracking branch 'upstream/main' into 05192025/UnpauseInd…

3571c46

…exingForPermits Refresh

relax assert that throttled shard is primary

2ee8dfb

add comment

f492df1

ankikuma merged commit 074f070 into elastic:main Jul 30, 2025
33 checks passed

Suspend Index throttling when relocating #128797

Suspend Index throttling when relocating #128797

Uh oh!

Conversation

ankikuma commented Jun 3, 2025

Uh oh!

elasticsearchmachine commented Jun 3, 2025

Uh oh!

ankikuma commented Jun 3, 2025

Uh oh!

henningandersen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ankikuma commented Jul 13, 2025

Uh oh!

henningandersen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ankikuma Jun 3, 2025 •

edited

Loading

ankikuma Jun 3, 2025 •

edited

Loading