[CI] FileSettingsRoleMappingsRestartIT testReservedStatePersistsOnRestart failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #16610 / part4
• elasticsearch-periodic-platform-support #6045 / almalinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #6045 / ubuntu-2204_platform-support-unix
• elasticsearch-pull-request #53316 / part-4
• elasticsearch-pull-request #53266 / part-4
• elasticsearch-periodic-platform-support #6040 / rhel-8_platform-support-unix
• elasticsearch-periodic-platform-support #6040 / almalinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #6040 / ubuntu-2404_platform-support-unix
• elasticsearch-periodic #6051 / openjdk23_checkpart4_java-matrix
• elasticsearch-periodic #6046 / openjdk23_checkpart4_java-matrix

Reproduction Line:

./gradlew ":x-pack:plugin:security:internalClusterTest" --tests "org.elasticsearch.xpack.security.FileSettingsRoleMappingsRestartIT.testReservedStatePersistsOnRestart" -Dtests.seed=84B6A2BE7D7481F2 -Dtests.locale=sr -Dtests.timezone=America/Los_Angeles -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: iterable with items [] in any order
     but: no match for: "everyone_kibana_alone"

Issue Reasons:

• [8.x] 2 consecutive failures in step almalinux-9_platform-support-unix
• [8.x] 11 failures in test testReservedStatePersistsOnRestart (6.5% fail rate in 168 executions)
• [8.x] 3 failures in step part-4 (7.5% fail rate in 40 executions)
• [8.x] 2 failures in step almalinux-9_platform-support-unix (66.7% fail rate in 3 executions)
• [8.x] 2 failures in step openjdk23_checkpart4_java-matrix (66.7% fail rate in 3 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic (66.7% fail rate in 3 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (66.7% fail rate in 3 executions)
• [8.x] 3 failures in pipeline elasticsearch-pull-request (7.9% fail rate in 38 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in test testReservedStatePersistsOnRestart
• [main] 6 failures in test testReservedStatePersistsOnRestart (0.7% fail rate in 803 executions)
• [main] 3 failures in step part-4 (1.0% fail rate in 313 executions)
• [main] 2 failures in step part-4-fips (20.0% fail rate in 10 executions)
• [main] 5 failures in pipeline elasticsearch-pull-request (1.6% fail rate in 312 executions)

Build Scans:

• elasticsearch-intake #16445 / part4
• elasticsearch-pull-request #52934 / part-4
• elasticsearch-pull-request #52898 / part-4
• elasticsearch-pull-request #52148 / part-4-fips
• elasticsearch-pull-request #52001 / part-4-fips
• elasticsearch-pull-request #51955 / part-4

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ldematte]

Failures have spiked in the last 2 days, but this test has been randomly failing since the 17th of December. There hasn't been a change to ReservedClusterStateService or FileSettingsService since #118351, so I'm putting this down to medium-risk, as it might be a bug but it seems to happen rarely.
(it might also be a test bug; if it turns out to be that way, we can lower this to low-risk)
@prdoyle do you think #118351 may be related?

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 2 consecutive failures in step almalinux-9_platform-support-unix
• [8.x] 11 failures in test testReservedStatePersistsOnRestart (6.5% fail rate in 168 executions)
• [8.x] 3 failures in step part-4 (7.5% fail rate in 40 executions)
• [8.x] 2 failures in step almalinux-9_platform-support-unix (66.7% fail rate in 3 executions)
• [8.x] 2 failures in step openjdk23_checkpart4_java-matrix (66.7% fail rate in 3 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic (66.7% fail rate in 3 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (66.7% fail rate in 3 executions)
• [8.x] 3 failures in pipeline elasticsearch-pull-request (7.9% fail rate in 38 executions)

Build Scans:

• elasticsearch-intake #16610 / part4
• elasticsearch-periodic-platform-support #6045 / almalinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #6045 / ubuntu-2204_platform-support-unix
• elasticsearch-pull-request #53316 / part-4
• elasticsearch-pull-request #53266 / part-4
• elasticsearch-periodic-platform-support #6040 / rhel-8_platform-support-unix
• elasticsearch-periodic-platform-support #6040 / almalinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #6040 / ubuntu-2404_platform-support-unix
• elasticsearch-periodic #6051 / openjdk23_checkpart4_java-matrix
• elasticsearch-periodic #6046 / openjdk23_checkpart4_java-matrix

[prdoyle]

The failures are occurring in a call to assertRoleMappingsInClusterState, which these tests sometimes call instead of assertRoleMappingsInClusterStateWithAwait. I wonder why it's ok not to wait? Possibly because it's looking at the initial state, and it's doing so immediately after awaitFileSettingsWatcher.

Just wondering because I've just fixed a similar bug in #126720.