Skip to content

Fix SearchErrorTraceIT and friends to work with batched query execution #132227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Sep 4, 2025
Merged

Fix SearchErrorTraceIT and friends to work with batched query execution #132227

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
69bfa0a
Fix SearchErrorTraceIT and friends to work with batched query execution
original-brownbear Apr 22, 2025
ce6435c
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Jul 28, 2025
b185eb1
Convert BytesTransportResponse
benchaplin Jul 29, 2025
7684209
Merge branch 'main' into fixup-search-error-trace-it
drempapis Jul 30, 2025
925d730
Merge branch 'main' into fixup-search-error-trace-it
drempapis Jul 30, 2025
2ac4e98
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Jul 30, 2025
ebca2bb
Merge branch 'main' into fixup-search-error-trace-it
drempapis Jul 31, 2025
a36a3ca
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Aug 11, 2025
e647f43
Adjust approach for intercepting transport responses
benchaplin Sep 2, 2025
c9d625f
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Sep 2, 2025
9e5cfe8
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Sep 3, 2025
215b030
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Sep 3, 2025
512fce1
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Sep 3, 2025
3e68c7b
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Sep 4, 2025
b0791f1
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Sep 4, 2025
f19d164
Merge branch 'main' into fixup-search-error-trace-it
benchaplin Sep 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions ...oke-test-http/src/internalClusterTest/java/org/elasticsearch/http/SearchErrorTraceIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,6 @@ public static void setDebugLogLevel() {
@Before
public void setupMessageListener() {
hasStackTrace = ErrorTraceHelper.setupErrorTraceListener(internalCluster());
// TODO: make this test work with batched query execution by enhancing ErrorTraceHelper.setupErrorTraceListener
updateClusterSettings(Settings.builder().put(SearchService.BATCHED_QUERY_PHASE.getKey(), false));
}

@After
Expand Down
4 changes: 2 additions & 2 deletions server/src/main/java/org/elasticsearch/action/search/SearchQueryThenFetchAsyncAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ public static final class NodeQueryResponse extends TransportResponse {
private final SearchPhaseController.TopDocsStats topDocsStats;
private final QueryPhaseResultConsumer.MergeResult mergeResult;

NodeQueryResponse(StreamInput in) throws IOException {
public NodeQueryResponse(StreamInput in) throws IOException {
this.results = in.readArray(i -> i.readBoolean() ? new QuerySearchResult(i) : i.readException(), Object[]::new);
this.mergeResult = QueryPhaseResultConsumer.MergeResult.readFrom(in);
this.topDocsStats = SearchPhaseController.TopDocsStats.readFrom(in);
Expand Down Expand Up @@ -552,7 +552,7 @@ private void onNodeQueryFailure(Exception e, NodeQueryRequest request, CanMatchP
}
}

private static final String NODE_SEARCH_ACTION_NAME = "indices:data/read/search[query][n]";
public static final String NODE_SEARCH_ACTION_NAME = "indices:data/read/search[query][n]";

static void registerNodeSearchAction(
SearchTransportService searchTransportService,
Expand Down
2 changes: 2 additions & 0 deletions server/src/main/java/org/elasticsearch/transport/OutboundHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ void sendResponse(
) {
assert assertValidTransportVersion(transportVersion);
assert response.hasReferences();
var messageListener = this.messageListener;
messageListener.onBeforeResponseSent(requestId, action, response);
try {
sendMessage(
channel,
Expand Down
8 changes: 8 additions & 0 deletions server/src/main/java/org/elasticsearch/transport/TransportMessageListener.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ default void onRequestReceived(long requestId, String action) {}
*/
default void onResponseSent(long requestId, String action) {}

/**
* Called for every action response sent before the response has been passed to the underlying network implementation.
* @param requestId the request ID (unique per client)
* @param action the request action
* @param response response instance
*/
default void onBeforeResponseSent(long requestId, String action, TransportResponse response) {}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I follow the issue in the first place, as well as the proposed fix. The proposal is to add a new default method to TransportMessageListener, to then only ever use it in a test? Could you help me better understand what other options we have?

Copy link
Contributor Author

@benchaplin benchaplin Aug 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Async/SearchErrorTraceIT forces an error on all shards during a search. Before batched execution, this would trigger onResponseSent(long requestId, String action, Exception error) - called for every failed action response. We could then implement onResponseSent and inspect the exception to assert things (like whether the stack trace was present).

Batched execution sends a non-error response even if all searches fail. We still want to inspect this response to assert the same things that were once captured in the exception. But TransportMessageListener has no hook to inspect non-error payloads, so this PR introduces one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey Ben, sorry for the delay. I understand the problem better. I think this is a bit of an intrusive fix though. Isn't there another way to intercept responses in this specific test as opposed to using transport message listener? You may want to reach out to the distrib team and ask for advice on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@javanna I've reworked the tests using a nice suggestion from @DaveCTurner - thanks for pushing for a better solution!


/***
* Called for every failed action response after the response has been passed to the underlying network implementation.
* @param requestId the request ID (unique per client)
Expand Down
62 changes: 51 additions & 11 deletions test/framework/src/main/java/org/elasticsearch/search/ErrorTraceHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,22 @@

import org.apache.logging.log4j.Level;
import org.elasticsearch.ExceptionsHelper;
import org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction;
import org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.NodeQueryResponse;
import org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.index.query.QueryShardException;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.InternalTestCluster;
import org.elasticsearch.test.MockLog;
import org.elasticsearch.test.transport.MockTransportService;
import org.elasticsearch.transport.BytesTransportResponse;
import org.elasticsearch.transport.TransportMessageListener;
import org.elasticsearch.transport.TransportResponse;
import org.elasticsearch.transport.TransportService;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Arrays;
import java.util.Optional;
import java.util.concurrent.atomic.AtomicBoolean;
Expand All @@ -37,29 +45,61 @@ public enum ErrorTraceHelper {

public static BooleanSupplier setupErrorTraceListener(InternalTestCluster internalCluster) {
final AtomicBoolean transportMessageHasStackTrace = new AtomicBoolean(false);
internalCluster.getDataNodeInstances(TransportService.class)
.forEach(ts -> asInstanceOf(MockTransportService.class, ts).addMessageListener(new TransportMessageListener() {
internalCluster.getDataNodeInstances(TransportService.class).forEach(ts -> {
var mockTs = asInstanceOf(MockTransportService.class, ts);
mockTs.addMessageListener(new TransportMessageListener() {
// This is called when error_trace is false
@Override
public void onResponseSent(long requestId, String action, Exception error) {
TransportMessageListener.super.onResponseSent(requestId, action, error);
if (action.startsWith("indices:data/read/search")) {
Optional<Throwable> throwable = ExceptionsHelper.unwrapCausesAndSuppressed(
error,
t -> t.getStackTrace().length > 0
);
transportMessageHasStackTrace.set(throwable.isPresent());
checkStacktraceStateAndRemove(error, mockTs);
}
}
}));

// This is called when error_trace is true
@Override
public void onBeforeResponseSent(long requestId, String action, TransportResponse response) {
if (SearchQueryThenFetchAsyncAction.NODE_SEARCH_ACTION_NAME.equals(action)) {
var bytes = asInstanceOf(BytesTransportResponse.class, response);
NodeQueryResponse nodeQueryResponse = null;
try (StreamInput in = bytes.bytes().streamInput()) {
var namedWriteableAwareInput = new NamedWriteableAwareStreamInput(
in,
internalCluster.getNamedWriteableRegistry()
);
nodeQueryResponse = new NodeQueryResponse(namedWriteableAwareInput);
for (Object result : nodeQueryResponse.getResults()) {
if (result instanceof Exception error) {
checkStacktraceStateAndRemove(error, mockTs);
}
}
} catch (IOException e) {
throw new UncheckedIOException(e);
} finally {
if (nodeQueryResponse != null) {
nodeQueryResponse.decRef();
}
}
}
}

private void checkStacktraceStateAndRemove(Exception error, MockTransportService mockTs) {
Optional<Throwable> throwable = ExceptionsHelper.unwrapCausesAndSuppressed(error, t -> t.getStackTrace().length > 0);
transportMessageHasStackTrace.set(throwable.isPresent());
mockTs.removeMessageListener(this);
}
});
});
return transportMessageHasStackTrace::get;
}

/**
* Adds expectations for debug logging of a message and exception on each shard of the given index.
*
* @param numShards the number of shards in the index (an expectation will be added for each shard)
* @param mockLog the mock log
* @param errorTriggeringIndex the name of the index that will trigger the error
* @param numShards the number of shards in the index (an expectation will be added for each shard)
* @param mockLog the mock log
* @param errorTriggeringIndex the name of the index that will trigger the error
*/
public static void addSeenLoggingExpectations(int numShards, MockLog mockLog, String errorTriggeringIndex) {
String nodesDisjunction = format(
Expand Down
14 changes: 14 additions & 0 deletions test/framework/src/main/java/org/elasticsearch/test/transport/MockTransportService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
import org.elasticsearch.transport.TransportMessageListener;
import org.elasticsearch.transport.TransportRequest;
import org.elasticsearch.transport.TransportRequestOptions;
import org.elasticsearch.transport.TransportResponse;
import org.elasticsearch.transport.TransportService;
import org.elasticsearch.transport.TransportSettings;
import org.elasticsearch.transport.netty4.Netty4Transport;
Expand Down Expand Up @@ -846,6 +847,12 @@ public void onResponseSent(long requestId, String action) {
messageListener.onResponseSent(requestId, action);
}

@Override
public void onBeforeResponseSent(long requestId, String action, TransportResponse response) {
super.onBeforeResponseSent(requestId, action, response);
messageListener.onBeforeResponseSent(requestId, action, response);
}

@Override
public void onResponseSent(long requestId, String action, Exception e) {
super.onResponseSent(requestId, action, e);
Expand Down Expand Up @@ -898,6 +905,13 @@ public void onRequestSent(
}
}

@Override
public void onBeforeResponseSent(long requestId, String action, TransportResponse response) {
for (TransportMessageListener listener : listeners) {
listener.onBeforeResponseSent(requestId, action, response);
}
}

@Override
@SuppressWarnings("rawtypes")
public void onResponseReceived(long requestId, Transport.ResponseContext holder) {
Expand Down
2 changes: 0 additions & 2 deletions .../src/internalClusterTest/java/org/elasticsearch/xpack/search/AsyncSearchErrorTraceIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,6 @@ public static void setDebugLogLevel() {
@Before
public void setupMessageListener() {
transportMessageHasStackTrace = ErrorTraceHelper.setupErrorTraceListener(internalCluster());
// TODO: make this test work with batched query execution by enhancing ErrorTraceHelper.setupErrorTraceListener
updateClusterSettings(Settings.builder().put(SearchService.BATCHED_QUERY_PHASE.getKey(), false));
}

@After
Expand Down