[CI] IndexRecoveryIT testUsesFileBasedRecoveryIfOperationsBasedRecoveryWouldBeUnreasonable failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #10907 / debian-12_platform-support-unix
• elasticsearch-pull-request #95948 / part-1
• elasticsearch-pull-request #95870 / part-1
• elasticsearch-pull-request #95766 / part-1

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.indices.recovery.IndexRecoveryIT.testUsesFileBasedRecoveryIfOperationsBasedRecoveryWouldBeUnreasonable" -Dtests.seed=5A7C2DD8A952A4DA -Dtests.locale=en-DG -Dtests.timezone=Pacific/Palau -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: timed out waiting for green state

Issue Reasons:

• [main] 4 failures in test testUsesFileBasedRecoveryIfOperationsBasedRecoveryWouldBeUnreasonable (1.6% fail rate in 255 executions)
• [main] 3 failures in step part-1 (2.2% fail rate in 134 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (2.3% fail rate in 133 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 failures in test testUsesFileBasedRecoveryIfOperationsBasedRecoveryWouldBeUnreasonable (1.6% fail rate in 255 executions)
• [main] 3 failures in step part-1 (2.2% fail rate in 134 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (2.3% fail rate in 133 executions)

Build Scans:

• elasticsearch-periodic-platform-support #10907 / debian-12_platform-support-unix
• elasticsearch-pull-request #95948 / part-1
• elasticsearch-pull-request #95870 / part-1
• elasticsearch-pull-request #95766 / part-1

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[bcully]

One timeout in ensureGreen with peculiar explanation from allocation:

      "node_decision" : "yes",	
      "deciders" : [	
        {	
          "decider" : "write_load",	
          "decision" : "NOT_PREFERRED",	
          "explanation" : "The high utilization threshold of [0.900000] would be exceeded on node [_UaEPHURTMGQPULoBa7Gwg] with utilization [0.00] if shard [[test-index][0]] with estimated additional utilisation [1.00133] (write load [1.00133] / threads [1]) were assigned to it. Cannot allocate shard to node without risking increased write latencies."	
        }

[bcully]

They all look like this. Maybe related to the new write_load decider? I don't understand why it fails with node_decision of yes though.

I'll pass this to distributed coordination for a look.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)