[CI] SyntheticSourceRollingUpgradeIT testIndexing {upgradedNodes=2} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #10472 / openjdk21_9.0.7_java-fips-matrix-bwc
• elasticsearch-intake #28065 / bc-upgrade-tests-part2
• elasticsearch-intake #28063 / bc-upgrade-tests-part2
• elasticsearch-intake #28060 / bc-upgrade-tests-part2
• elasticsearch-intake #28059 / bc-upgrade-tests-part2
• elasticsearch-intake #28057 / bc-upgrade-tests-part2
• elasticsearch-intake #28056 / bc-upgrade-tests-part2
• elasticsearch-intake #28055 / bc-upgrade-tests-part2
• elasticsearch-intake #28054 / bc-upgrade-tests-part2
• elasticsearch-intake #28053 / bc-upgrade-tests-part2

Reproduction Line:

./gradlew ":x-pack:plugin:logsdb:qa:rolling-upgrade:v9.0.7#bwcTest" -Dtests.class="org.elasticsearch.upgrades.SyntheticSourceRollingUpgradeIT" -Dtests.method="testIndexing {upgradedNodes=2}" -Dtests.seed=42FBBE63F4A4E55B -Dtests.bwc=true -Dtests.locale=es-DO -Dtests.timezone=Chile/EasterIsland -Druntime.java=21 -Dtests.fips.enabled=true

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.ClassCastException: class java.lang.String cannot be cast to class java.util.List (java.lang.String and java.util.List are in module java.base of loader 'bootstrap')

Issue Reasons:

• [main] 59 consecutive failures in step bc-upgrade-tests-part2
• [main] 4 consecutive failures in pipeline elasticsearch-intake
• [main] 65 failures in test testIndexing {upgradedNodes=2} (6.6% fail rate in 981 executions)
• [main] 2 failures in step openjdk21_9.0.7_java-fips-matrix-bwc (33.3% fail rate in 6 executions)
• [main] 2 failures in step 9.1.4_part_2_bwc-snapshots (0.9% fail rate in 225 executions)
• [main] 59 failures in step bc-upgrade-tests-part2 (100.0% fail rate in 59 executions)
• [main] 2 failures in pipeline elasticsearch-periodic (33.3% fail rate in 6 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (1.3% fail rate in 229 executions)
• [main] 59 failures in pipeline elasticsearch-intake (98.3% fail rate in 60 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 57 consecutive failures in step bc-upgrade-tests-part2
• [main] 58 failures in test testIndexing {upgradedNodes=2} (6.1% fail rate in 951 executions)
• [main] 57 failures in step bc-upgrade-tests-part2 (100.0% fail rate in 57 executions)
• [main] 57 failures in pipeline elasticsearch-intake (96.6% fail rate in 59 executions)

Build Scans:

• elasticsearch-periodic #10460 / openjdk21_9.0.7_java-fips-matrix-bwc
• elasticsearch-intake #28029 / bc-upgrade-tests-part2
• elasticsearch-intake #28027 / bc-upgrade-tests-part2
• elasticsearch-intake #28026 / bc-upgrade-tests-part2
• elasticsearch-intake #28025 / bc-upgrade-tests-part2
• elasticsearch-intake #28024 / bc-upgrade-tests-part2
• elasticsearch-intake #28015 / bc-upgrade-tests-part2
• elasticsearch-intake #28013 / bc-upgrade-tests-part2
• elasticsearch-intake #28012 / bc-upgrade-tests-part2
• elasticsearch-intake #28011 / bc-upgrade-tests-part2

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[kkrik-es]

The logs show that the upgraded node can't join because of versioning:

Caused by: java.lang.IllegalStateException: node with version [9.1.4-9.2.0] may not join a cluster with minimum version [9.2.0] ```  

https://gradle-enterprise.elastic.co/s/3rdi4g4ujyhwe/tests/overview?outcome=FAILED

[martijnvg]

I don't think this is a synthetic source test issue. Looking at this ci run. Looks like node version issue. I've looked at the es logging of the test cluster and see things like this:

[2025-09-05T05:46:43,473][WARN ][o.e.t.TransportService   ] [test-cluster-1] remote node [{test-cluster-0}{INl1-pchQyWqeTUBreJSMQ}{Chg9ivQmQcyWANj2kNoxxg}{test-cluster-0}{127.0.0.1}{127.0.0.1:36323}{cdfhilmrstw}{9.2.0}{8000099-9036000}{transform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=2, ml.max_jvm_size=805306368, ml.config_version=12.0.0, ml.machine_memory=126604197888, ml.allocated_processors_double=2.0, testattr=test}] is build [b72b3a87fac77abc2ce32f8c89b03ae23ad8c24e] of version [9.2.0] but this node is build [b229c65a50fb286f48ad28832f74f9e181090a5f] of version [9.2.0] which has an incompatible wire format

org.elasticsearch.transport.RemoteTransportException: [test-cluster-1][127.0.0.1:42631][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: node with version [9.1.4-9.2.0] may not join a cluster with minimum version [9.2.0]
	at org.elasticsearch.cluster.version.CompatibilityVersions.ensureVersionsCompatibility(CompatibilityVersions.java:83) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.coordination.NodeJoinExecutor.execute(NodeJoinExecutor.java:176) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1076) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1039) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:246) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1692) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:465) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1689) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1284) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:465) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1263) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1067) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.2.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) ~[?:?]
	at java.lang.Thread.run(Thread.java:1447) ~[?:?]

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)