[CI] CrossClusterAsyncQueryIT testBadAsyncId failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #26442 / part3
• elasticsearch-pull-request #83863 / part-3
• elasticsearch-pull-request #83742 / part-3
• elasticsearch-pull-request #83629 / part-3
• elasticsearch-periodic-platform-support #9696 / windows-2025_checkpart3_platform-support-windows

Reproduction Line:

./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryIT.testBadAsyncId" -Dtests.seed=5E01BD734B783073 -Dtests.locale=kxv-Telu -Dtests.timezone=Asia/Ashgabat -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: java.lang.IllegalStateException: unexpected byte [0x08]

Issue Reasons:

• [main] 5 failures in test testBadAsyncId (0.9% fail rate in 574 executions)
• [main] 3 failures in step part-3 (1.1% fail rate in 276 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (1.1% fail rate in 274 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 5 failures in test testBadAsyncId (0.9% fail rate in 574 executions)
• [main] 3 failures in step part-3 (1.1% fail rate in 276 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (1.1% fail rate in 274 executions)

Build Scans:

• elasticsearch-intake #26442 / part3
• elasticsearch-pull-request #83863 / part-3
• elasticsearch-pull-request #83742 / part-3
• elasticsearch-pull-request #83629 / part-3
• elasticsearch-periodic-platform-support #9696 / windows-2025_checkpart3_platform-support-windows

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[fang-xing-esql]

@smalyshev Do you mind to take a look at this first? Thanks in advance!

[alex-spies]

@smalyshev , I hope you don't mind if I assign you as you added the failing test in http://github.com/elastic/elasticsearch/pull/118122.

[smalyshev]

This is weird, because where it fails is in deserializing EsqlExecutionInfo which doesn't seem to have anything to do with the test in question.

[smalyshev]

@idegtiarenko You refactored this part recently, could it be something was changed inavertently?

[idegtiarenko]

I managed to reproduce the issue locally using background load produced by stress-ng --cpu 16.
I also added following logging:

            log.info("Written {} cluster infos (1)", clusterInfo.size());
            out.writeCollection(clusterInfo.values());
            log.info("Written {} cluster infos (2)", clusterInfo.size());

and

        this.clusterInfo = in.readMapValues(EsqlExecutionInfo.Cluster::new, Cluster::getClusterAlias, ConcurrentHashMap::new);
        log.info("Read {} cluster infos", clusterInfo.size());

This resulted in the following failure: https://gradle-enterprise.elastic.co/s/57x2olqlrfxua/tests/task/:x-pack:plugin:esql:internalClusterTest/details/org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryIT/testBadAsyncId%20%7Bseed%3D%5BF45B9FACA14B081:BBD300AD74B0BDE6%5D%7D?top-execution=1

[2025-08-12T07:41:53,094][INFO ][o.e.x.e.a.EsqlExecutionInfo][main-cluster-2][search][T#5] Written 0 cluster infos (1)	
[2025-08-12T07:41:53,094][INFO ][o.e.x.e.a.EsqlExecutionInfo][main-cluster-2][search][T#5] Written 3 cluster infos (2)	
[2025-08-12T07:41:53,094][INFO ][o.e.x.e.a.EsqlExecutionInfo][main-cluster-0][transport_worker][T#3] Read 2 cluster infos

This indicates concurrent modification while writing.

elasticsearch/server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java

Lines 1133 to 1138 in de40ac4

	public <T> void writeCollection(final Collection<T> collection, final Writer<T> writer) throws IOException {
	writeVInt(collection.size());
	for (final T val : collection) {
	writer.write(this, val);
	}
	}

writeCollection implementation uses 2 distinct calls to write size and obtain iterator. I suspect it wrote a smaller size than later was supplied by iterator.

I will bring back the defensive copy to avoid this, however I think it is an issue that EsqlExecutionInfo is serialized while it is still being mutated. @smalyshev , do you mind taking a look why it might be happening?