[CI] SearchReplicaSelectionIT testNodeSelection failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #26136 / part1
• elasticsearch-intake #26023 / part1

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.routing.SearchReplicaSelectionIT.testNodeSelection" -Dtests.seed=73656387188EF471 -Dtests.locale=it-CH -Dtests.timezone=America/Phoenix -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: a value equal to or greater than <40.22649163>
     but: <9.010883549999999> was less than <40.22649163>

Issue Reasons:

• [main] 2 failures in test testNodeSelection (0.4% fail rate in 510 executions)
• [main] 2 failures in step part1 (4.0% fail rate in 50 executions)
• [main] 2 failures in pipeline elasticsearch-intake (4.0% fail rate in 50 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testNodeSelection (0.4% fail rate in 510 executions)
• [main] 2 failures in step part1 (4.0% fail rate in 50 executions)
• [main] 2 failures in pipeline elasticsearch-intake (4.0% fail rate in 50 executions)

Build Scans:

• elasticsearch-intake #26136 / part1
• elasticsearch-intake #26023 / part1

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[carlosdelest]

Unable to reproduce locally. Executed test suite 1000 times and didn't get a failure. Unmuting in #132092