[CI] WriteLoadForecasterIT testWriteLoadForecastGetsPopulatedDuringRollovers failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #28015 / part2
• elasticsearch-pull-request #90174 / part-2
• elasticsearch-pull-request #90133 / part-2
• elasticsearch-pull-request #90118 / part-2
• elasticsearch-pull-request #90094 / part-2
• elasticsearch-intake #27979 / part2
• elasticsearch-pull-request #90071 / part-2
• elasticsearch-intake #27977 / part2
• elasticsearch-pull-request #90060 / part-2
• elasticsearch-pull-request #90058 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:write-load-forecaster:internalClusterTest" --tests "org.elasticsearch.xpack.writeloadforecaster.WriteLoadForecasterIT.testWriteLoadForecastGetsPopulatedDuringRollovers" -Dtests.seed=E86E4A0F69058F93 -Dtests.locale=fr-MF -Dtests.timezone=Atlantic/Faeroe -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: timed out waiting for green state

Issue Reasons:

• [main] 29 failures in test testWriteLoadForecastGetsPopulatedDuringRollovers (10.0% fail rate in 290 executions)
• [main] 7 failures in step part2 (15.6% fail rate in 45 executions)
• [main] 22 failures in step part-2 (13.4% fail rate in 164 executions)
• [main] 7 failures in pipeline elasticsearch-intake (15.6% fail rate in 45 executions)
• [main] 20 failures in pipeline elasticsearch-pull-request (12.9% fail rate in 155 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 9.1

Mute Reasons:

• [9.1] 2 consecutive failures in test testWriteLoadForecastGetsPopulatedDuringRollovers
• [9.1] 2 consecutive failures in step part2
• [9.1] 2 consecutive failures in pipeline elasticsearch-intake
• [9.1] 2 failures in test testWriteLoadForecastGetsPopulatedDuringRollovers (2.3% fail rate in 87 executions)
• [9.1] 2 failures in step part2 (28.6% fail rate in 7 executions)
• [9.1] 2 failures in pipeline elasticsearch-intake (28.6% fail rate in 7 executions)

Build Scans:

• elasticsearch-intake #27949 / part2
• elasticsearch-intake #27945 / part2

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[ywangd]

Should be fixed by #134132