Move some security APIs to using promises in place of callbacks

[original-brownbear]

We ahve some incredibly deep callstacks in security that seem to visibly raise context switch costs, make profiling more complicated and generally make the code rather hard to follow.
Also, deeply nested listener chains tend to make logic consume more heap because we almost inevitably continue holding on to things we no longer need as we go down the stack.
Since the methods adjusted here mostly return a result synchronously on the hot path, we can both save overhead (we can do a couple follow-ups to that effect) and make things a little easier to follow by using promises as returns in place of consuming callbacks.

A good example of the effects of this change would be this piece of search request REST handling that flattens out enormously.

Before:

After:

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[DaveCTurner]

Should we do something similar to flatten out RequestFilterChain#proceed rather than using recursion there too?

[DaveCTurner]

I'd rather add something like this to SubscribableListener over exposing rawResult():

    public boolean completeIfFailed(ActionListener<?> listener) {
        final Object currentState = state;
        if (currentState instanceof FailureResult result) {
            listener.onFailure(result.exception());
            return true;
        } else {
            return false;
        }
    }

[original-brownbear]

I went for just an isSuccess now like other solutions have. There isn't much of a reason to be tricky for the failure path and this way we have a fast solution for the hot success path. Let me know what you think :) Combining this approach with the get-results on listenable futures would allow neat simplifications + speedups in some places I believe (by avoiding allocating new listeners).

[original-brownbear]

Should we do something similar to flatten out RequestFilterChain#proceed rather than using recursion there too?

I think we should yea, gotta be a little careful maybe because of thread-context though. That kind of thing is made it impossible to go further with this kind of change in security for the time being when I tried because any thread-local context that only exists around the onResponse is lost if the listener hasn't been added before.

[DaveCTurner]

This bit LGTM, I'll leave the rest to the security team

[slobodanadamovic]

The changes LGTM 👍 Nice job!

(I left a non-blocking question and comment regarding adjusting the example plugin)

[slobodanadamovic]

The change to this interface will cause compilation errors of CustomAuthorizationEngine. Can you adjust the security example plugin as well?

[slobodanadamovic]

What would you think if we changed the request interceptors to be fully sync and not return a promise?

public interface RequestInterceptor {

    void intercept(RequestInfo requestInfo, AuthorizationEngine authorizationEngine, AuthorizationInfo authorizationInfo)
        throws ElasticsearchSecurityException;
}

then runRequestInterceptors would look like this

private void runRequestInterceptors(
    RequestInfo requestInfo,
    AuthorizationInfo authorizationInfo,
    AuthorizationEngine authorizationEngine,
    ActionListener<Void> listener
) {
    for (RequestInterceptor requestInterceptor : requestInterceptors) {
        try {
            requestInterceptor.intercept(requestInfo, authorizationEngine, authorizationInfo);
        } catch (Exception e) {
            listener.onFailure(e);
            return;
        }
    }
    listener.onResponse(null);
}

[original-brownbear]

Thanks you two!

I fixed up the plugin. + I'm all on board with a sync API here, but is that possible looking at the code it might fork today?

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 123812

[slobodanadamovic]

• I'm all on board with a sync API here, but is that possible looking at the code it might fork today?

You are right. 🤦