resolve indices for prefixed _all expressions #137330

richard-dennehy · 2025-10-29T11:23:47Z

Follow up to #135425

As identified, expressions such as *:_all currently don't resolve correctly (it appears that it currently tries to authorize the user against the literal index "_all").

This PR fixes the resolution of prefixed _all expressions.

Relates: ES-13213

richard-dennehy · 2025-10-29T13:33:51Z

...ity/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java

+        );
+    }
+
+    public void testResolveIndexWithRemotePrefix() {


I noticed that I could just set all cross project requests to go down the "all indices" path and all the tests would pass, even though this is obviously wrong; I've added this test which fails under that scenario

all the tests would pass

Do you mean all tests in this test class? I am surprised unless all existing tests are effectively resolving to all accessible indices. Is that the case? Also, I assume tests in other places, e.g. the serverless REST test, should fail?

I think it's because the majority of the other tests in this file have CPS mode disabled; I'll see if the REST test guards against this, but I don't want it to be the only test that would fail - it makes the feedback loop too long

Confirmed that the REST test fails when this functionality is implemented incorrectly

elasticsearchmachine · 2025-10-29T16:02:06Z

Pinging @elastic/es-security (Team:Security)

ywangd

This look mostly good. I have only minor comments.

Can you also have a serverless side PR to update the test scenarios?

ywangd · 2025-10-30T02:26:03Z

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

+                isAllIndices = crossProjectModeDecider.resolvesCrossProject(replaceable)
+                    ? IndexNameExpressionResolver.isAllIndices(
+                        indicesList(indicesRequest.indices()),
+                        (expr) -> CrossProjectIndexExpressionsRewriter.rewriteIndexExpression(
+                            expr,
+                            authorizedProjects.originProjectAlias(),
+                            authorizedProjects.allProjectAliases()
+                        ).localExpression()
+                    )
+                    : IndexNameExpressionResolver.isAllIndices(
+                        indicesList(indicesRequest.indices()),
+                        (expr) -> IndexNameExpressionResolver.splitSelectorExpression(expr).v1()
+                    );


This matches what I was thinking, i.e. let cases like *:_all go through the all indices handling. I suggest we use RemoteClusterAware#splitIndexName instead of rewriteIndexExpression. It is lighter and we don't care about resolving projects at this point. It also needs to handle syntax like selector. I'd prefer we don't error out at isAllIndices check. The unsupported syntax will still be caught later when we call rewriteIndexExpression. So overall I am suggeting something like

IndexNameExpressionResolver.isAllIndices( indicesList(indicesRequest.indices()), expr -> { IndexNameExpressionResolver.splitSelectorExpression(RemoteClusterAware.splitIndexName(expr)[1]).v1() } )

It appears this isn't enough, as this means any expression targeting _all will resolve all indices on the local project, even if that expression is e.g. some_remote_project:_all.

We need to check if the expression is _all, and that the prefix includes the local project. It looks to me like rewriteIndexExpression handles all the edge cases here, even if it does more than what's necessary here.

Yes we will resolve local indices unnecessarily in this case. But I think local indices should still be excluded because of this code. I was going to leave the unnecessary expansion for a future optimization.

Even if we use rewriteIndexExpression here, it is not sufficient due to project_routing, e.g. we will still expand local indices unnecessarily when the request has expression _all and project routing some_remote_project. If you are keen to avoid local expansion, I'd rather prefer we solve this more comprehensively to cover this TODO as well.

What I have in mind is something like the follows:

Still use the splitIndexName and splitSelectorExpression for checking isAllIndices which IMO should be interpreted as "is it an all indices pattern for any project".

Before expanding local indices, check crossProjectModeDecider.resolvesCrossProject and only expand locally if the final resolved projects include the origin project.

But I think local indices should still be excluded because of this code.

It appears that this logic only handles project routing filtering but not the qualified projects in index expression. So it does not work for linked_project:_all.

I am still a little suspicious on the latest change. IIUC, it means an expression like linked_project:_all goes through the else branch of the isAllIndices check? While expressions like *:_all or _all with project routing linked_project still go through the if branch. The later is functionally equivalent to linked_project:_all and it would great if they follow the same path. Maybe we just need to resolve the project routing before invoking rewriteIndexExpression? If we take this path, I think we should also rename the isAllIndices variable to something like containsLocalAllIndices?

I've switched back to using splitIndexName again, and extended the shouldExcludeLocalResolution check to check if localExpression is set, and this seems to work (including the REST test). Does this align with what you were thinking?

I could also take a look at moving the condition earlier to see if we can skip the redundant local expansion as well.

That works for me as well. We can have separate PR to address the TODO for skipping redundant local expansion.

ywangd · 2025-10-30T02:29:38Z

...ity/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java

+        );
+    }
+
+    public void testResolveIndexWithRemotePrefix() {


all the tests would pass

Do you mean all tests in this test class? I am surprised unless all existing tests are effectively resolving to all accessible indices. Is that the case? Also, I assume tests in other places, e.g. the serverless REST test, should fail?

ywangd

LGTM

…ixed-all-expressions

ywangd · 2025-11-06T01:48:55Z

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

+                    var unprefixed = crossProjectModeDecider.resolvesCrossProject(replaceable)
+                        ? RemoteClusterAware.splitIndexName(expr)[1]
+                        : expr;
+                    return unprefixed != null ? IndexNameExpressionResolver.splitSelectorExpression(unprefixed).v1() : null;


Nit: I think unprefixed cannot be null?

Follow up to elastic#135425 As identified, expressions such as `*:_all` currently don't resolve correctly (it appears that it currently tries to authorize the user against the literal index "_all"). This PR fixes the resolution of prefixed `_all` expressions. Relates: ES-13213

…-json * upstream/main: Mute org.elasticsearch.xpack.inference.action.filter.ShardBulkInferenceActionFilterBasicLicenseIT testLicenseInvalidForInference {p0=false} elastic#137691 Mute org.elasticsearch.xpack.inference.action.filter.ShardBulkInferenceActionFilterBasicLicenseIT testLicenseInvalidForInference {p0=true} elastic#137690 [LTR] Fix feature display order when using explain. (elastic#137671) Remove extra RemoteClusterService instances in unit test (elastic#137647) Fix `ComponentTemplatesFileSettingsIT.testSettingsApplied` (elastic#137669) Consolidates troubleshooting content into the "Returning semantic field embeddings in _source" section (elastic#137233) Update bundled JDK to 25.0.1 (elastic#137640) resolve indices for prefixed _all expressions (elastic#137330) ESQL: Add TopN support for exponential histograms (elastic#137313) allows field caps to be cross project (elastic#137530) ESQL: Add exponential histogram percentile function (elastic#137553) Wait for nodes to have downloaded databases in `GeoIpDownloaderIT` (elastic#137636) Tighten on when THROTTLE decision can be returned (elastic#136794) Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeMetricsIT test elastic#137655 Add a test for two little known conditional processor paths (elastic#137645) Extract a common ORIGIN constant (elastic#137612) Remove early phase failure in batched (elastic#136889) Returning correct index mode from get data streams api (elastic#137646) [ML] Manage AD results indices (elastic#136065)

Follow up to elastic#135425 As identified, expressions such as `*:_all` currently don't resolve correctly (it appears that it currently tries to authorize the user against the literal index "_all"). This PR fixes the resolution of prefixed `_all` expressions. Relates: ES-13213

resolve indices for prefixed _all expressions

9c7e04b

richard-dennehy added >non-issue :Security/Security Security issues without another label Team:Security Meta label for security team labels Oct 29, 2025

elasticsearchmachine added the v9.3.0 label Oct 29, 2025

richard-dennehy commented Oct 29, 2025

View reviewed changes

add test for origin prefix _all expression

2ecdc40

richard-dennehy marked this pull request as ready for review October 29, 2025 16:01

richard-dennehy requested a review from ywangd October 29, 2025 16:01

[CI] Auto commit changes from spotless

b19ec68

ywangd reviewed Oct 30, 2025

View reviewed changes

richard-dennehy added 2 commits October 30, 2025 10:48

use splitIndexName instead of rewriteIndexExpression

84ea19d

Merge branch 'main' into remote-project-prefixed-all-expressions

8bbfd8f

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Oct 30, 2025

richard-dennehy requested a review from ywangd October 30, 2025 16:54

ywangd approved these changes Oct 31, 2025

View reviewed changes

richard-dennehy added 4 commits October 31, 2025 14:36

use rewriteIndexExpression again

aaea6cf

Merge remote-tracking branch 'upstream/main' into remote-project-pref…

7812375

…ixed-all-expressions

extend shouldExcludeLocalResolution check

ef0417d

rename

13ceafc

ywangd approved these changes Nov 6, 2025

View reviewed changes

richard-dennehy added 2 commits November 6, 2025 09:19

Merge branch 'main' into remote-project-prefixed-all-expressions

d3470c9

review comment

9730e5e

richard-dennehy added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Nov 6, 2025

elasticsearchmachine merged commit 774cf6a into elastic:main Nov 6, 2025
40 checks passed

richard-dennehy deleted the remote-project-prefixed-all-expressions branch November 6, 2025 10:47

resolve indices for prefixed _all expressions #137330

resolve indices for prefixed _all expressions #137330

Uh oh!

Conversation

richard-dennehy commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 29, 2025

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

richard-dennehy commented Oct 29, 2025 •

edited

Loading