Fix security index creation race condition

[jfreden]

Resolves: #127694, #122863

The error occurs when trying to authenticate the activate profile request:

[2025-05-05T01:02:23,272][WARN ][o.e.x.s.a.RealmsAuthenticator][testActivateProfileForJWT] An error occurred while attempting to authenticate ['aud:es-01' 'groups:admin' 'iss:my-issuer-01' 'sub:me'] against realm [jwt0]
1> org.elasticsearch.action.UnavailableShardsException: at least one search shard for the index [.security-7] is unavailable

Before the test, the creation of the .security index is skipped:

[2025-05-05T01:02:23,174][INFO ][o.e.x.s.a.j.JwtRealmSingleNodeTests][testActivateProfileForJWT] Security index already exists, ignoring.

And before that we see:

[2025-05-05T01:02:23,127][INFO ][o.e.c.m.MetadataCreateIndexService][node_s_0][masterService#updateTask][T#1] creating index [.security-7] in project [default], cause [api], templates [], shards [1]/[1]
[2025-05-05T01:02:23,134][INFO ][o.e.c.r.a.AllocationService][node_s_0][masterService#updateTask][T#1] in project [default] updating number_of_replicas to [0] for indices [.security-7]

Even earlier we see:

[2025-05-05T01:02:22,854][INFO ][o.e.x.s.s.SecurityIndexManager][node_s_0][generic][T#6] security index does not exist, creating [.security-7] with alias [.security]

This is triggered by prepareIndexIfNeededThenExecute.

What I think is happening is that the index is created but not available and therefore it fails. If we check if exists and also wait for it to become available this might not happen.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[slobodanadamovic]

LGTM Thanks for fixing this 👍