[CI] MultiClusterSpecIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #31899 / 9.2.3_bwc-snapshots
• elasticsearch-intake #31898 / 9.2.3_bwc-snapshots
• elasticsearch-pull-request #107439 / 9.2.3_part_3_bwc-snapshots
• elasticsearch-pull-request #107438 / 9.2.3_part_3_bwc-snapshots
• elasticsearch-pull-request #107435 / 9.2.3_part_3_bwc-snapshots
• elasticsearch-intake #31897 / 9.2.3_bwc-snapshots
• elasticsearch-pull-request #107433 / 9.2.3_part_3_bwc-snapshots
• elasticsearch-pull-request #107431 / 9.2.3_part_3_bwc-snapshots
• elasticsearch-pull-request #107432 / 9.2.3_part_3_bwc-snapshots
• elasticsearch-pull-request #107430 / 9.2.3_part_3_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:esql:qa:server:multi-clusters:v9.2.3#newToOld" -Dtests.class="org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT" -Dtests.method="test {csv-spec:union_types.MultiIndexIndirectUseOfUnionTypesInEnrich}" -Dtests.seed=D63C9C5529D179DD -Dtests.bwc=true -Dtests.locale=shi -Dtests.timezone=Europe/Uzhgorod -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:35577], URI [/_query/async/FjhaQ2FvN2hZUmhlYTUtVDZJVFpRU2ccYUp1YXoxRFVSNUNsZ05RMGdJOVgwdzoxODQ3OA==?wait_for_completion_timeout=6000s&pretty=true&error_trace=true], status line [HTTP/1.1 500 Internal Server Error]
¿eerror¿jroot_cause�¿dtypewillegal_state_exceptionfreasonx�Message not fully read (request) for requestId [10286], action [cluster:monitor/xpack/enrich/esql/resolve_policy], available [3]; resettingkstack_trace�y��org.elasticsearch.ElasticsearchException$1: Message not fully read (request) for requestId [10286], action [cluster:monitor/xpack/enrich/esql/resolve_policy], available [3]; resetting
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:797)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:711)
	at org.elasticsearch.server@9.3.0-
[truncated]

Issue Reasons:

• [main] 6 consecutive failures in step 9.2.3_bwc-snapshots
• [main] 29 consecutive failures in step 9.2.3_part_3_bwc-snapshots
• [main] 46 failures in class org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT (4.6% fail rate in 1000 executions)
• [main] 6 failures in step 9.2.3_bwc-snapshots (100.0% fail rate in 6 executions)
• [main] 3 failures in step 8.19.8_part_3_bwc-snapshots (4.2% fail rate in 71 executions)
• [main] 2 failures in step 9.3.0_part_3_bwc-snapshots (2.9% fail rate in 70 executions)
• [main] 29 failures in step 9.2.3_part_3_bwc-snapshots (100.0% fail rate in 29 executions)
• [main] 2 failures in step 9.1.8_part_3_bwc-snapshots (9.1% fail rate in 22 executions)
• [main] 3 failures in step 9.2.2_part_3_bwc-snapshots (6.3% fail rate in 48 executions)
• [main] 6 failures in pipeline elasticsearch-intake (31.6% fail rate in 19 executions)
• [main] 27 failures in pipeline elasticsearch-pull-request (36.0% fail rate in 75 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[alex-spies]

9.2.3 bwc tests are surely failing because of this pending backport #138893 to 9.2. The transport version for 9.2.3 was automatically bumped to a newer version than what #138893 introduces, so nodes on 9.3/main think that 9.2.3 nodes can do things that they can't until the backport gets merged.

\tat java.base/java.lang.Thread.run(Thread.java:1474)\nCaused by: java.lang.IllegalStateException:\	
      \ unexpected byte [0x92]\n\tat org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readBoolean(StreamInput.java:674)\n\	
      \tat org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readBoolean(StreamInput.java:664)\n\	
      \tat org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readOptionalTimeValue(StreamInput.java:1527)\n\	
      \tat org.elasticsearch.xpack.esql.action.EsqlExecutionInfo.<init>(EsqlExecutionInfo.java:139)\n\	
      \tat org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readOptional(StreamInput.java:1178)\n\	
      \tat org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:1168)\n\	
      \tat org.elasticsearch.xpack.esql.action.EsqlQueryResponse.deserialize(EsqlQueryResponse.java:157)\n\

[alex-spies]

Should be fixed now that #138893 is merged.