[CI] FieldCapabilitiesIT testRelocation failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #10791 / ubuntu-2404_platform-support-unix
• elasticsearch-periodic-platform-support #10755 / sles-15_platform-support-unix

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.fieldcaps.FieldCapabilitiesIT.testCancel" -Dtests.seed=45D866EAAF1D6C45 -Dtests.locale=szl-Latn-PL -Dtests.timezone=Antarctica/Syowa -Druntime.java=25

Applicable branches:
9.0

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: null

Issue Reasons:

• [9.0] 2 failures in test testRelocation (1.3% fail rate in 155 executions)
• [9.0] 2 failures in pipeline elasticsearch-periodic-platform-support (28.6% fail rate in 7 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

There are several tests in the FieldCapabilitiesIT failing here (both scans I checked), the first one to fail always seems to be "testRelocation" with the following Uncaught Exception in a search_coordination thread:

[2025-09-22T13:49:07,301][INFO ][o.e.s.f.FieldCapabilitiesIT] [testRelocation] Index [7] docs async: [true] bulk: [false]
вер 22, 2025 1:49:07 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[#9377,elasticsearch[node_s2][search_coordination][T#1],5,TGRP-FieldCapabilitiesIT]
java.lang.AssertionError: safeGet: listener was completed exceptionally
	at __randomizedtesting.SeedInfo.seed([35954D228E05ECD6]:0)
	at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2510)
	at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2494)
	at org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteUtils.doReroute(ClusterRerouteUtils.java:47)
	at org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteUtils.reroute(ClusterRerouteUtils.java:35)
	at org.elasticsearch.search.fieldcaps.FieldCapabilitiesIT.moveOrCloseShardsOnNodes(FieldCapabilitiesIT.java:631)
	at org.elasticsearch.search.fieldcaps.FieldCapabilitiesIT.lambda$testRelocation$2(FieldCapabilitiesIT.java:653)
	at org.elasticsearch.test.transport.StubbableTransport.lambda$addRequestHandlingBehavior$0(StubbableTransport.java:89)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:86)
	at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:289)
	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:302)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: java.util.concurrent.ExecutionException: org.elasticsearch.transport.RemoteTransportException: [node_s0][127.0.0.1:18463][cluster:admin/reroute]
	at org.elasticsearch.action.support.PlainActionFuture$Sync.getValue(PlainActionFuture.java:279)
	at org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:253)
	at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:74)
	at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2505)
	... 17 more
Caused by: org.elasticsearch.transport.RemoteTransportException: [node_s0][127.0.0.1:18463][cluster:admin/reroute]
Caused by: java.lang.IllegalArgumentException: [move_allocation] can't move [new_index][8], shard is not started (state = INITIALIZING]
	at org.elasticsearch.cluster.routing.allocation.command.MoveAllocationCommand.execute(MoveAllocationCommand.java:157)
	at org.elasticsearch.cluster.routing.allocation.command.AllocationCommands.execute(AllocationCommands.java:74)
	at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocator.execute(ShardsAllocator.java:67)
	at org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceShardsAllocator.execute(DesiredBalanceShardsAllocator.java:296)
	at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:372)
	at org.elasticsearch.action.admin.cluster.reroute.TransportClusterRerouteAction$ClusterRerouteResponseAckedClusterStateUpdateTask.execute(TransportClusterRerouteAction.java:245)
	at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:573)
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1069)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1032)
	at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:244)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1685)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:454)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1682)
	at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1277)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:454)
	at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1256)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.lang.Thread.run(Thread.java:1447)

So the root cause is:

java.lang.IllegalArgumentException: [move_allocation] can't move [new_index][8], shard is not started (state = INITIALIZING]
	at org.elasticsearch.cluster.routing.allocation.command.MoveAllocationCommand.execute(MoveAllocationCommand.java:157)

I'll take a look if there is anything specific in this test setup that might cause shards not starting, also will check if there are any other similar looking tickets in that timeframe (Sep 24th).

[cbuescher]

Found #124227 which looks like this issue should have been fixed already. The two failures here are from 9.0 branch though. Will check if that fix was successfully backported

[cbuescher]

It looks like some of the backports of that previous fix failed. Will try to get those back to at least 9.0, maybe also 8x branches.

[cbuescher]

Closed by #136366