Do not pass ProjectMetadata to lazy index permissions builder

[jfreden]

During a serverless incident (INC-4832) that was caused by frequent OOM exceptions it was discovered that ~30% of the heap was occupied by ProjectMetadata instances.

The ProjectMetadata instances were retained by a lambda in IndicesPermission, see this example of a path to gc root:

The reason the lambda exists is to make the index access control lazy. Because the lambda is lazy, it will hold on to the reference to ProjectMetadata for the full request life cycle (as opposed to building the index permissions and dropping the reference). This becomes a problem when there are many concurrent searches (index actions requiring us to check index permissions) coupled with frequent ProjectMetadata updates. Since the lambda holds a reference to ProjectMetadata it can't be garbage collected.

I've proven this by:

1. Adding a sleep to TransportSearchAction to simulate slow searches
2. Hook up visual vm to Elasticsearch
3. Launch "slow" searches with ProjectMetadata updates in between (triggered by creating new indices)
4. Trigger GC manually through visual vm
5. Observe memory usage by ProjectMetadata while the searches are hanging (to simulate request in flight)

Before any requests

While requests are in flight

Fix

To fix this issue I've moved the part that needed ProjectMetadata outside of the lambda.
ProjectMetadata was needed to resolved failure store indices. With this PR we will do some more work that #88708 tried to remove, but I think it's acceptable for the memory gain.

To validate that this fixed the issue I ran the same test as above and could see that ProjectMetadata could be garbaged collected as soon as authorization was finished.

[elasticsearchmachine]

Hi @jfreden, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[slobodanadamovic]

LGTM 👍

Nice job on tracking this down!