[CI] ReindexDatastreamIndexTransportActionIT testTsdbStartEndSet failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #28971 / part2
• elasticsearch-pull-request #94579 / part-2
• elasticsearch-intake #28969 / part2

Reproduction Line:

./gradlew ":x-pack:plugin:migrate:internalClusterTest" --tests "org.elasticsearch.xpack.migrate.action.ReindexDatastreamIndexTransportActionIT.testTsdbStartEndSet" -Dtests.seed=898E4FBF1A1263EB -Dtests.locale=kgp-BR -Dtests.timezone=Etc/GMT-6 -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: safeGet: listener was completed exceptionally

Issue Reasons:

• [main] 3 failures in test testTsdbStartEndSet (1.1% fail rate in 272 executions)
• [main] 2 failures in step part2 (4.5% fail rate in 44 executions)
• [main] 2 failures in pipeline elasticsearch-intake (4.5% fail rate in 44 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testTsdbStartEndSet (0.7% fail rate in 270 executions)

Build Scans:

• elasticsearch-pull-request #94579 / part-2
• elasticsearch-intake #28969 / part2

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

Here's the root cause:

                java.lang.IllegalArgumentException: [index.mode=time_series] requires a non-empty [index.routing_path]
                    at org.elasticsearch.index.IndexMode$2.validateWithOtherSettings(IndexMode.java:154)
                    at org.elasticsearch.index.IndexSettings$2.validate(IndexSettings.java:691)
                    at org.elasticsearch.index.IndexSettings$2.validate(IndexSettings.java:685)
                    at org.elasticsearch.common.settings.Setting.get(Setting.java:589)
                    at org.elasticsearch.common.settings.Setting.get(Setting.java:561)
                    at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:634)
                    at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:531)
                    at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:501)
                    at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:471)
                    at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.normalizeRequestSetting(MetadataCreateIndexService.java:377)
                    at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.onlyCreateIndex(MetadataCreateIndexService.java:338)
                    at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.createIndex(MetadataCreateIndexService.java:298)
                    at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.masterOperation(TransportCreateIndexAction.java:156)
                    at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.masterOperation(TransportCreateIndexAction.java:49)
                    at org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:113)
                    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:229)
                    at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
                    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
                    at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:315)
                    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:229)
                    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:166)
                    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)
                    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)
                    at org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)
                    at org.elasticsearch.action.support.TransportAction.executeDirect(TransportAction.java:55)
                    at org.elasticsearch.action.support.HandledTransportAction.lambda$new$0(HandledTransportAction.java:51)
                    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:86)
                    at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:319)
                    at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:297)
                    at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:127)
                    at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102)
                    at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:830)
                    at org.elasticsearch.transport.InboundPipeline.forwardFragment(InboundPipeline.java:116)
                    at org.elasticsearch.transport.InboundPipeline.lambda$doHandleBytes$1(InboundPipeline.java:78)
                    at org.elasticsearch.transport.InboundDecoder.finishMessage(InboundDecoder.java:144)
                    at org.elasticsearch.transport.InboundDecoder.internalDecode(InboundDecoder.java:129)
                    at org.elasticsearch.transport.InboundDecoder.decode(InboundDecoder.java:58)
                    at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:79)
                    at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:69)
                    at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:54)
                    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
                    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                    at org.elasticsearch.transport.netty4.NettyByteBufSizer.channelRead(NettyByteBufSizer.java:33)
                    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
                    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
                    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
                    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
                    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
                    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
                    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697)
                    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660)
                    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
                    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
                    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                    at java.lang.Thread.run(Thread.java:1474)

Looks like it might have been a result of #132566.