Do not pass ProjectMetadata to lazy index permissions builder (#135337)

During a serverless incident (INC-4832) that was caused by frequent OOM
exceptions it was discovered that ~30% of the heap was occupied by
`ProjectMetadata` instances. 

The `ProjectMetadata` instances were retained by a lambda in
`IndicesPermission`, see this example of a path to gc root:  <img
width="2760" height="915" alt="9a9f6dfd-bd11-41ac-a0e2-345a86ba0509"
src="https://github.com/user-attachments/assets/7de8b33e-6002-4330-87e0-28a6ab7aeeac"
/>

The reason the lambda exists is to make the [index access control
lazy](#88708). Because the
lambda is lazy, it will hold on to the reference to `ProjectMetadata`
for the full request life cycle (as opposed to building the index
permissions and dropping the reference). This becomes a problem when
there are many concurrent searches (index actions requiring us to check
index permissions) coupled with frequent `ProjectMetadata` updates.
Since the lambda holds a reference to `ProjectMetadata` it can't be
garbage collected. 

I've proven this by: 1. Adding a sleep to `TransportSearchAction` to
simulate slow searches 2. Hook up visual vm to Elasticsearch  3. Launch
"slow" searches with `ProjectMetadata` updates in between (triggered by
creating new indices) 4. Trigger GC manually through visual vm 5.
Observe memory usage by `ProjectMetadata` while the searches are hanging
(to simulate request in flight)

### Before any requests

<img width="814" height="619" alt="Screenshot 2025-09-24 at 13 52 34"
src="https://github.com/user-attachments/assets/5732a317-e298-4c90-bf8e-b5c211481c5d"
/>

### While requests are in flight <img width="798" height="668"
alt="Screenshot 2025-09-24 at 14 31 41"
src="https://github.com/user-attachments/assets/c5e5e9ab-e95c-4a78-b0fd-f4bcc5d3149e"
/>

### Fix To fix this issue I've moved the part that needed
`ProjectMetadata` outside of the lambda.  `ProjectMetadata` was needed
to resolved failure store indices. With this PR we will do some more
work that #88708 tried to
remove, but I think it's acceptable for the memory gain.

To validate that this fixed the issue I ran the same test as above and
could see that `ProjectMetadata` could be garbaged collected as soon as
authorization was finished. 

<img width="945" height="722" alt="Screenshot 2025-09-24 at 14 04 37"
src="https://github.com/user-attachments/assets/8f70b388-2150-4768-a5bf-ac10dd36b41c"
/>

Author: jfreden

docs/changelog/135337.yaml

@@ -0,0 +1,5 @@
+pr: 135337
+summary: Do not pass `ProjectMetadata` to lazy index permissions builder
+area: Security
+type: enhancement
+issues: []

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java

@@ -461,6 +461,12 @@ private IndexResource(String name, @Nullable IndexAbstraction abstraction, @Null
             this.selector = selector;
         }
 
+        public List<Index> getFailureIndices(ProjectMetadata metadata) {
+            return indexAbstraction != null && IndexComponentSelector.FAILURES.equals(selector)
+                ? indexAbstraction.getFailureIndices(metadata)
+                : List.of();
+        }
+
         /**
          * @return {@code true} if-and-only-if this object is related to a data-stream, either by having a
          * {@link IndexAbstraction#getType()} of {@link IndexAbstraction.Type#DATA_STREAM} or by being the backing index for a
@@ -535,13 +541,12 @@ public int size(Map<String, IndexAbstraction> lookup) {
             }
         }
 
-        public Collection<String> resolveConcreteIndices(ProjectMetadata metadata) {
+        public Collection<String> resolveConcreteIndices(List<Index> failureIndices) {
             if (indexAbstraction == null) {
                 return List.of();
             } else if (indexAbstraction.getType() == IndexAbstraction.Type.CONCRETE_INDEX) {
                 return List.of(indexAbstraction.getName());
             } else if (IndexComponentSelector.FAILURES.equals(selector)) {
-                final List<Index> failureIndices = indexAbstraction.getFailureIndices(metadata);
                 final List<String> concreteIndexNames = new ArrayList<>(failureIndices.size());
                 for (var idx : failureIndices) {
                     concreteIndexNames.add(idx.getName());
@@ -604,12 +609,16 @@ public IndicesAccessControl authorize(
 
         final boolean overallGranted = isActionGranted(action, resources.values());
         final int finalTotalResourceCount = totalResourceCount;
+        final var failureIndicesByResourceName = resources.entrySet()
+            .stream()
+            .collect(Collectors.toMap(Map.Entry::getKey, entry -> entry.getValue().getFailureIndices(metadata)));
+
         final Supplier<Map<String, IndicesAccessControl.IndexAccessControl>> indexPermissions = () -> buildIndicesAccessControl(
             action,
             resources,
             finalTotalResourceCount,
             fieldPermissionsCache,
-            metadata
+            failureIndicesByResourceName
         );
 
         return new IndicesAccessControl(overallGranted, indexPermissions);
@@ -620,7 +629,7 @@ private Map<String, IndicesAccessControl.IndexAccessControl> buildIndicesAccessC
         final Map<String, IndexResource> requestedResources,
         final int totalResourceCount,
         final FieldPermissionsCache fieldPermissionsCache,
-        final ProjectMetadata metadata
+        final Map<String, List<Index>> failureIndicesByIndexResource
     ) {
 
         // now... every index that is associated with the request, must be granted
@@ -636,7 +645,9 @@ private Map<String, IndicesAccessControl.IndexAccessControl> buildIndicesAccessC
             boolean granted = false;
             final String resourceName = resourceEntry.getKey();
             final IndexResource resource = resourceEntry.getValue();
-            final Collection<String> concreteIndices = resource.resolveConcreteIndices(metadata);
+            final Collection<String> concreteIndices = resource.resolveConcreteIndices(
+                failureIndicesByIndexResource.get(resourceEntry.getKey())
+            );
             for (Group group : groups) {
                 // the group covers the given index OR the given index is a backing index and the group covers the parent data stream
                 if (resource.checkIndex(group)) {