fix: do not let _resolve/cluster hang if remote is unresponsive (elastic#119516) (elastic#119526)

* fix: do not let `_resolve/cluster` hang if remote is unresponsive

Previously, `_resolve/cluster` would wait for a response from a remote
as part of the connection strategy. If the remote were to be
unresponsive, this API would wait until `netty` would terminate the
connection with a handshake exception. The threshold for terminating the
connection is `10s`. This means that the API would wait for `10s` before
determining that the remote is unresponsive. This strategy is now
replaced with a fail fast where a response is sent back to the user
immediately rather than waiting for a connection termination.

* Update docs/changelog/119516.yaml

Author: pawankartik-elastic

docs/changelog/119516.yaml

@@ -0,0 +1,5 @@
+pr: 119516
+summary: "Fix: do not let `_resolve/cluster` hang if remote is unresponsive"
+area: Search
+type: bug
+issues: []

server/src/main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java

@@ -141,7 +141,7 @@ protected void doExecuteForked(Task task, ResolveClusterActionRequest request, A
                 RemoteClusterClient remoteClusterClient = remoteClusterService.getRemoteClusterClient(
                     clusterAlias,
                     searchCoordinationExecutor,
-                    RemoteClusterService.DisconnectedStrategy.RECONNECT_IF_DISCONNECTED
+                    RemoteClusterService.DisconnectedStrategy.FAIL_IF_DISCONNECTED
                 );
                 var remoteRequest = new ResolveClusterActionRequest(originalIndices.indices(), request.indicesOptions());
                 // allow cancellation requests to propagate to remote clusters