In a follow-on PR, I think we'll need to build some tests that simulate the case where one of the remote clusters is not available (not responding) and hits the timeout. We would do those tests in the serverless IT test and we can use some of the test plugin features we have in the core CCS IT tests to induce non-responsive linked projects.

You don't need a plugin for it. It is achievable by adding a "handling behaviour" to the underlying MockTransportService.

(optional) - I think you could make this method static (and non-private) so that you could write unit tests for it.

That may not be worth it since it does it 3 things and two of those (CrossProjectIndexResolutionValidator.validate and ResolvedIndices.resolveWithIndexExpressions) are already static methods with tests.

Is this the IndicesOptions we'll be passing in for CPS? If not, should we change it to match what the production code passes in (which I think would be from indicesOptionsForCrossProjectFanout)?

In the production code, it's resolutionIdxOpts, so that's the CPS options. So, yes, it'd be wiser to use them here.

it passes as-is - it can be any value

if it needs to be a specific value, should we add validation or an assertion (to both MRT = false and MRT = true paths)?

I feel like I'm missing something. This statement:

This method guarantees that the resulting remote project contains at least one index resolved for the mapped ResolvedIndexExpressions

doesn't seem right to me. The check that guarantees this is the call to CrossProjectIndexResolutionValidator.validate in mergeResolvedIndices, isn't it?

I think it's a guarantee on the map's (remoteIndices) structuring rather than a behaviour ascertaining. Either way, my opinion is that it's not immediately clear what the doc means, unless you're by default familiar with the CPS specifics. Could this be rewritten or clarified?

Lines 268 through 280 iterates over a map of ResolvedIndexExpressions and filters the internal list of ResolvedIndexExpression for entities that have .localIndexResolutionResult() == SUCCESS and at least one index.

For example, if project-1 had the original index expression logs* map to indices ["logs-1", "logs-2"], then the result has project-1 map to ["logs*"]. If project-1 had logs* map to [], then project-1 is not included in the resulting map.

So, each remote project that is keyed in the resulting ResolvedIndices has at least one index that is mapped from the original index expression.

The thing I would like to verify is the assumption that we made in the slack thread, where we set the remote project's index in the OriginalIndices to be the original index expression and not the resolved index (e.g. ["logs*"] and not ["logs-1", "logs-2"].

The thing I would like to verify is the assumption that we made in the slack thread, where we set the remote project's index in the OriginalIndices to be the original index expression and not the resolved index

That's right. We want to associate with the user input index expressions and not the resolved values. You're on the right track.

Is ResolvedIndices.resolveWithIndexExpressions the method that ensures the right subset of indices are assigned to each linked project?

In other words, if I have say GET logs,foo,bar/_search and two linked projects p1 and p2, suppose these are the only indices that exist across all 3 projects:

_origin: logs
p1:foo
p2:bar

is ResolvedIndices.resolveWithIndexExpressions the method that ensures that each project alias is mapped to only the indices/aliases/datastreams found on each project?

If yes, I'd like to see that scenario tested here. I think you are only testing on one linked project and I'm having trouble following both what ResolvedIndices.resolveWithIndexExpressions does (the code is very dense) and what is being tested here. I think laying it out with code comments would help.

++ on this.

is ResolvedIndices.resolveWithIndexExpressions the method that ensures that each project alias is mapped to only the indices/aliases/datastreams found on each project?

kinda, from our discussion in the thread, it maps to the original index expression that resolves to the indices/aliases/datastreams found on each project.

"once we validated the flat expression we can use the rewritten index expression, so p1:logs* is fine no need to change it to p1:logs-1"