[CI] DataNodeRequestSenderTests testRetryOnlyMovedShards failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #8290 / ubuntu-2004_platform-support-unix
• elasticsearch-periodic-platform-support #8224 / ubuntu-2004-aarch64_checkpart3_platform-support-arm
• elasticsearch-periodic-platform-support #8218 / windows-2019_checkpart3_platform-support-windows
• elasticsearch-periodic-platform-support #8176 / ubuntu-2004-aarch64_checkpart3_platform-support-arm

Reproduction Line:

./gradlew ":x-pack:plugin:esql:test" --tests "org.elasticsearch.xpack.esql.plugin.DataNodeRequestSenderTests.testRetryOnlyMovedShards" -Dtests.seed=B4179AA2A470F433 -Dtests.locale=pis-SB -Dtests.timezone=Eire -Druntime.java=24

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: <2>
     but: was <1>

Issue Reasons:

• [8.19] 4 failures in test testRetryOnlyMovedShards (0.6% fail rate in 637 executions)
• [8.19] 2 failures in step ubuntu-2004-aarch64_checkpart3_platform-support-arm (10.0% fail rate in 20 executions)
• [8.19] 4 failures in pipeline elasticsearch-periodic-platform-support (20.0% fail rate in 20 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[idegtiarenko]

It does not reproduces with a given seed and overall very rare.
I suspect this is fixed by #128062 as it addressed a concurrency issue when retrying more than a single shard at once.