[CI] MixedClusterDownsampleRestIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #24068 / 8.18.3_bwc-snapshots
• elasticsearch-periodic #8683 / openjdk17_8.17.8_java-fips-matrix-bwc
• elasticsearch-periodic #8659 / openjdk17_8.18.3_java-fips-matrix-bwc

Reproduction Line:

undefined

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [8.19] 3 failures in class org.elasticsearch.xpack.downsample.MixedClusterDownsampleRestIT (0.4% fail rate in 723 executions)
• [8.19] 2 failures in pipeline elasticsearch-periodic (9.5% fail rate in 21 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[gmarouli]

This is appears to be an environment failure:

[2025-06-16T01:19:38,920][WARN ][o.e.d.FileBasedSeedHostsProvider] [test-cluster-0] expected, but did not find, a dynamic hosts list at [/dev/shm/bk/bk-agent-prod-gcp-1750072106811557626/elastic/elasticsearch-intake/x-pack/plugin/downsample/qa/mixed-cluster/build/testrun/v8.18.3_yamlRestTest/temp/test-cluster13984399680053193577/test-cluster-0/config/unicast_hosts.txt]

This has affected also:

• [CI] HuggingFaceServiceUpgradeIT class failing #129475
• [CI] MixedClusterDownsampleRestIT class failing #128939
• [CI] MixedClusterDownsampleRestIT class failing #128680

[gmarouli]

RCA: In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally.

Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times.