Skip to content

Clarify the behavior of remote/info and resolve/cluster for connected status of remotes #118993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jan 29, 2025
Merged

Clarify the behavior of remote/info and resolve/cluster for connected status of remotes #118993

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
0af647e
Clarify the behavior of remote/info and resolve/cluster for connected…
quux00 Dec 18, 2024
4e08de4
Revised resolve-cluster.asciidoc further
quux00 Dec 19, 2024
5b9e749
Merge remote-tracking branch 'elastic/main' into docs/remote-info
quux00 Dec 19, 2024
fd48388
Rewording of docs language before PR review
quux00 Dec 19, 2024
d3d5d6e
Merge remote-tracking branch 'elastic/main' into docs/remote-info
quux00 Jan 7, 2025
277d334
Adjusted wording around 'force a reconnect' to 'attempt to reconnect'
quux00 Jan 7, 2025
be35beb
Merge remote-tracking branch 'elastic/main' into docs/remote-info
quux00 Jan 9, 2025
53c0b54
Merge remote-tracking branch 'elastic/main' into docs/remote-info
quux00 Jan 23, 2025
496586e
Minor tweaks
quux00 Jan 23, 2025
b77477c
Merge remote-tracking branch 'elastic/main' into docs/remote-info
quux00 Jan 28, 2025
d5fd9bb
Added docs on the new timeout param for _resolve/cluster
quux00 Jan 28, 2025
405aeb8
Revert "Added docs on the new timeout param for _resolve/cluster"
quux00 Jan 28, 2025
45bb773
Fixed error in example of disconnected cluster
quux00 Jan 28, 2025
0d4958a
Merge remote-tracking branch 'elastic/main' into docs/remote-info
quux00 Jan 29, 2025
84ff8e9
Removed comments from resolve-cluster docs that don't belong in 8.17.…
quux00 Jan 29, 2025
d8dfa44
PR feedback change
quux00 Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion docs/reference/cluster/remote-info.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ The cluster remote info API allows you to retrieve all of the configured
remote cluster information. It returns connection and endpoint information keyed
by the configured remote cluster alias.

TIP: This endpoint returns the information that reflects state on the cluster
you are querying. The `connected` field does not necessarily reflect whether
the remote cluster is down or unavailable, only whether there is currently an open
connection to it. Elasticsearch does not spontaneously try to reconnect to a
disconnected remote cluster. To trigger a reconnection, attempt a
<<modules-cross-cluster-search,{ccs}>>, <<esql-cross-clusters,{esql} {ccs}>>,
or try the <<indices-resolve-cluster-api,resolve cluster>> endpoint.


[[cluster-remote-info-api-response-body]]
==== {api-response-body-title}
Expand All @@ -34,7 +42,10 @@ by the configured remote cluster alias.
`proxy`.

`connected`::
True if there is at least one connection to the remote cluster.
True if there is at least one open connection to the remote cluster. When
false, it means that the cluster no longer has an open connection to the
remote cluster. It does not necessarily mean that the remote cluster is
down or unavailable, just that at some point a connection was lost.

`initial_connect_timeout`::
The initial connect timeout for remote cluster connections.
Expand Down
20 changes: 17 additions & 3 deletions docs/reference/indices/resolve-cluster.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ with this endpoint.

For each cluster in the index expression, information is returned about:

1. whether the querying ("local") cluster is currently connected to each remote cluster
in the index expression scope
1. whether the querying ("local") cluster was able to connect to each remote cluster
specified in the index expression. Note that this endpoint actively attempts to
contact the remote clusters, unlike the <<cluster-remote-info,remote/info>> endpoint.
2. whether each remote cluster is configured with `skip_unavailable` as `true` or `false`
3. whether there are any indices, aliases or data streams on that cluster that match
the index expression
Expand Down Expand Up @@ -112,11 +113,24 @@ deprecated:[7.16.0]

[discrete]
[[usecases-for-resolve-cluster]]
=== Test availability of remote clusters

The <<cluster-remote-info,remote/info>> endpoint is commonly used to test whether the "local"
cluster (the cluster being queried) is connected to its remote clusters, but it does not
necessarily reflect whether the remote cluster is available or not. The remote cluster may
be available, while the local cluster is not currently connected to it.

You can use this API to force a reconnection (for example with `GET _resolve/cluster/*:*`) and
the `connected` field in the response will indicate whether it was successful or not.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that #119516 means this is no longer true.

Copy link
Contributor Author

@quux00 quux00 Jan 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even with 119516, if the cluster can be connected to it will reset the connected status so that the _remote/info endpoint will report connected: true, right?

So perhaps the phrase "force a reconnection" is misleading. It will attempt a reconnection and if successful, the _remote/info status will now be refreshed. Would that be the better wording for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified the wording in this commit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still isn't true given the (IMO misguided) move to using FAIL_IF_DISCONNECTED. The connected field indicates whether the responding node was connected to the remote cluster at the start of the check. If it's disconnected at that point then we return connected: false and trigger a background connection attempt (unless one was already running) in the hope that some future call will eventually return connected: true.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks David. Pawan and I discussed this last week and came to a similar conclusion. We're going to revert #119516 in favor of another approach. We identified a couple of options and Pawan is going to follow up with the distributed team on some of the options that are thinking of. Once that's decided, I'll revise this PR according to the new model.

If a connection was (re-)established, this will also cause the
<<cluster-remote-info,remote/info>> endpoint to now indicate a connected status.


=== Advantages of using this endpoint before a {ccs}

You may want to exclude a cluster or index from a search when:

1. A remote cluster is not currently connected and is configured with `skip_unavailable`=`false`.
1. A remote cluster could not be connected to and is configured with `skip_unavailable`=`false`.
Executing a {ccs} under those conditions will cause
<<cross-cluster-search-failures,the entire search to fail>>.

Expand Down