Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
51e3da7
initial cleanup for remote clusters
florent-leborgne Feb 13, 2025
25bceed
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 13, 2025
41693ad
refine more
florent-leborgne Feb 13, 2025
567fab3
variables
florent-leborgne Feb 13, 2025
297a4fc
undo some variables
florent-leborgne Feb 13, 2025
6757e61
shorter nav titles
florent-leborgne Feb 17, 2025
12c0e4c
more cleanup + comments
florent-leborgne Feb 17, 2025
9c78a70
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 17, 2025
921d25b
missing list item
florent-leborgne Feb 17, 2025
d17aa90
more review changes
florent-leborgne Feb 17, 2025
bb2b4ff
link
florent-leborgne Feb 17, 2025
0cfb20a
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 18, 2025
05bc37c
apply tags
florent-leborgne Feb 20, 2025
a982cdf
cleanup
shainaraskas Feb 24, 2025
3176a62
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 24, 2025
cb8ec6f
Merge branch 'dm-remote-clusters' of github.com:florent-leborgne/docs…
florent-leborgne Feb 24, 2025
da366d6
fix troubleshooting links
shainaraskas Feb 24, 2025
913b811
little formatting thingies
shainaraskas Feb 24, 2025
e8377d1
Merge branch 'main' into dm-remote-clusters
shainaraskas Feb 24, 2025
4f59eb7
fix"
shainaraskas Feb 24, 2025
a8c53d3
Merge branch 'dm-remote-clusters' of https://github.com/florent-lebor…
shainaraskas Feb 24, 2025
c84e9e1
Merge branch 'main' into dm-remote-clusters
shainaraskas Feb 24, 2025
1bf69c9
Merge branch 'main' into dm-remote-clusters
florent-leborgne Feb 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 42 additions & 3 deletions deploy-manage/remote-clusters.md
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a perfect page????

Original file line number Diff line number Diff line change
@@ -1,9 +1,48 @@
# Remote clusters
---
applies:
stack:
ece:
eck:
hosted:
self:
serverless: unavailable
---

# Remote clusters [remote-clusters]

% What needs to be done: Write from scratch

% GitHub issue: https://github.com/elastic/docs-projects/issues/345

% Scope notes: "Landing page for cross cluster comms, used by CCS and CCR.
We will cover here the raw configuration at Elasticsearch level and the docs to enable remote clusters in ESS / ECE / ECK.
We can include links to the use cases of remote clusters, such as CCR and CCS."
% We will cover here the raw configuration at Elasticsearch level and the docs to enable remote clusters in ESS / ECE / ECK.
% We can include links to the use cases of remote clusters, such as CCR and CCS."

By setting up **remote clusters**, you can connect an {{es}} cluster to other {{es}} clusters. Remote clusters can be located in different data centers, geographic regions, and run on a different type of environment: {{ech}}, {{ece}}, {{eck}}, or self-managed.

Remote clusters are especially useful in two cases:

- **Cross-cluster replication**
With [cross-cluster replication](/deploy-manage/tools/cross-cluster-replication.md), or CCR, you ingest data to an index on a remote cluster. This leader index is replicated to one or more read-only follower indices on your local cluster. Creating a multi-cluster architecture with cross-cluster replication enables you to configure disaster recovery, bring data closer to your users, or establish a centralized reporting cluster to process reports locally.

- **Cross-cluster search**
[Cross-cluster search](/solutions/search/cross-cluster-search.md), or CCS, enables you to run a search request against one or more remote clusters. This capability provides each region with a global view of all clusters, allowing you to send a search request from a local cluster and return results from all connected remote clusters. For full {{ccs}} capabilities, the local and remote cluster must be on the same [subscription level](https://www.elastic.co/subscriptions).

::::{note} about terminology
In the case of remote clusters, the {{es}} cluster or deployment initiating the connection and requests is often referred to as the **local cluster**, while the {{es}} cluster or deployment receiving the requests is referred to as the **remote cluster**.
::::

## Setup

Depending on the environment the local and remote clusters are deployed on and the security model you wish to use, the exact details needed to add a remote cluster vary but generally follow the same path:

1. **Configure trust between clusters.** In the settings of the local deployment or cluster, configure the trust security model that your remote connections will use to access the remote cluster. This step involves specifying API keys or certificates retrieved from the remote clusters.

2. **Establish the connection.** In {{kib}} on the local cluster, finalize the connection by specifying each remote cluster's details.

Find the instructions with details on the supported security models and available connection modes for your specific scenario:

- [Remote clusters with {{ech}}](remote-clusters/ec-enable-ccs.md)
- [Remote clusters with {{ece}}](remote-clusters/ece-enable-ccs.md)
- [Remote clusters with {{eck}}](remote-clusters/eck-remote-clusters.md)
- [Remote clusters with self-managed installations](remote-clusters/remote-clusters-self-managed.md)
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ With this method, you can only remove trusted environments relying exclusively o
:alt: button for deleting a trusted environment
:::

4. In Kibana, go to **Stack Management** > **Remote Clusters**.
4. In {{kib}}, go to **Stack Management** > **Remote Clusters**.
5. In the list of existing remote clusters, delete the ones corresponding to the trusted environment you removed earlier.


Expand All @@ -56,11 +56,11 @@ With this method, you can only remove trusted environments relying exclusively o
This section describes the steps to change the API key used for an existing remote connection. For example, if the previous key expired and you need to rotate it with a new one.

::::{note}
If you need to update the permissions granted by a cross-cluster API key for a remote connection, you only need to update the privileges granted by the API key directly in Kibana.
If you need to update the permissions granted by a cross-cluster API key for a remote connection, you only need to update the privileges granted by the API key directly in {{kib}}.
::::


1. On the deployment you will use as remote, use the [{{es}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) or [Kibana](../api-keys/elasticsearch-api-keys.md) to create a cross-cluster API key with the appropriate permissions. Configure it with access to the indices you want to use for {{ccs}} or {{ccr}}.
1. On the deployment you will use as remote, use the [{{es}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) or [{{kib}}](../api-keys/elasticsearch-api-keys.md) to create a cross-cluster API key with the appropriate permissions. Configure it with access to the indices you want to use for {{ccs}} or {{ccr}}.
2. Copy the encoded key (`encoded` in the response) to a safe location. You will need it in the next steps.
3. Go to the **Security** page of the local deployment and locate the **Remote connections** section.
4. Locate the API key currently used for connecting to the remote cluster, copy its current alias, and delete it.
Expand All @@ -69,14 +69,14 @@ If you need to update the permissions granted by a cross-cluster API key for a r
* For the **Setting name**, enter the same alias that was used for the previous key.

::::{note}
If you use a different alias, you also need to re-create the remote cluster in Kibana with a **Name** that matches the new alias.
If you use a different alias, you also need to re-create the remote cluster in {{kib}} with a **Name** that matches the new alias.
::::

* For the **Secret**, paste the encoded cross-cluster API key.

1. Click **Add** to save the API key to the keystore.

6. Restart the local deployment to reload the keystore with its new setting. To do that, go to the deployment’s main page (named after your deployment’s name), locate the **Actions** menu, and select **Restart Elasticsearch**.<br>
6. Restart the local deployment to reload the keystore with its new setting. To do that, go to the deployment’s main page (named after your deployment’s name), locate the **Actions** menu, and select **Restart {{es}}**.<br>

::::{note}
If the local deployment runs on version 8.13 or greater, you no longer need to perform this step because the keystore is reloaded automatically with the new API keys.
Expand Down
16 changes: 8 additions & 8 deletions deploy-manage/remote-clusters/ec-enable-ccs-for-eck.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,17 @@ mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-enable-ccs-for-eck.html
---

# Enabling CCS/R between Elasticsearch Service and ECK [ec-enable-ccs-for-eck]
# Remote clusters between {{ech}} and ECK [ec-enable-ccs-for-eck]

These steps describe how to configure remote clusters between an {{es}} cluster in Elasticsearch Service and an {{es}} cluster running within [Elastic Cloud on Kubernetes (ECK)](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-overview.html). Once that’s done, you’ll be able to [run CCS queries from {{es}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html) or [set up CCR](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-getting-started-tutorial.html).
These steps describe how to configure remote clusters between an {{es}} cluster in {{ech}} and an {{es}} cluster running within [{{eck}} (ECK)](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-overview.html). Once that’s done, you’ll be able to [run CCS queries from {{es}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html) or [set up CCR](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-getting-started-tutorial.html).


## Establish trust between two clusters [ec_establish_trust_between_two_clusters]
Copy link
Collaborator

@shainaraskas shainaraskas Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 81: confused about what the user is supposed to do on the target page. the establishing trust procedures above also refer to these pages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's what you're referring to: Those page used to be maintained by the ECK team. Only thing I know is that doing it on ECK is very close to how it's done with self-managed, which isn't very concrete information... We can make a note to improve this later.


The first step is to establish trust between the two clusters.


### Establish trust in the Elasticsearch Service cluster [ec_establish_trust_in_the_elasticsearch_service_cluster]
### Establish trust in the {{ech}} cluster [ec_establish_trust_in_the_elasticsearch_service_cluster]

1. Save the ECK CA certificate to a file. For a cluster named `quickstart`, run:

Expand All @@ -22,7 +22,7 @@ The first step is to establish trust between the two clusters.
```


1. Update the trust settings for the Elasticsearch Service deployment. Follow the steps provided in [Access clusters of a self-managed environment](ec-remote-cluster-self-managed.md), and specifically the first three steps in **Specify the deployments trusted to be used as remote clusters** using TLS certificate as security model.
1. Update the trust settings for the {{ech}} deployment. Follow the steps provided in [Access clusters of a self-managed environment](ec-remote-cluster-self-managed.md), and specifically the first three steps in **Specify the deployments trusted to be used as remote clusters** using TLS certificate as security model.

* Use the certificate file saved in the first step.
* Select the {{ecloud}} pattern and enter `default.es.local` for the `Scope ID`.
Expand All @@ -32,7 +32,7 @@ The first step is to establish trust between the two clusters.

### Establish trust in the ECK cluster [ec_establish_trust_in_the_eck_cluster]

1. Upload the Elasticsearch Service certificate (that you downloaded in the last step of the previous section) as a Kubernetes secret.
1. Upload the {{ech}} certificate (that you downloaded in the last step of the previous section) as a Kubernetes secret.

```sh
kubectl create secret generic ce-aws-cert --from-file=<path to certificate file>
Expand Down Expand Up @@ -75,14 +75,14 @@ The first step is to establish trust between the two clusters.

## Setup CCS/R [ec_setup_ccsr]

Now that trust has been established, you can set up CCS/R from the ECK cluster to the Elasticsearch Service cluster or from the Elasticsearch Service cluster to the ECK cluster.
Now that trust has been established, you can set up CCS/R from the ECK cluster to the {{ech}} cluster or from the {{ech}} cluster to the ECK cluster.


### ECK Cluster to Elasticsearch Service cluster [ec_eck_cluster_to_elasticsearch_service_cluster]
### ECK Cluster to {{ech}} cluster [ec_eck_cluster_to_elasticsearch_service_cluster]

Configure the ECK cluster [using certificate based authentication](ec-remote-cluster-self-managed.md).


### Elasticsearch Service cluster to ECK Cluster [ec_elasticsearch_service_cluster_to_eck_cluster]
### {{ech}} cluster to ECK Cluster [ec_elasticsearch_service_cluster_to_eck_cluster]

Follow the steps outlined in the [ECK documentation](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-remote-clusters.html#k8s_configure_the_remote_cluster_connection_through_the_elasticsearch_rest_api).
48 changes: 20 additions & 28 deletions deploy-manage/remote-clusters/ec-enable-ccs.md
Original file line number Diff line number Diff line change
@@ -1,63 +1,55 @@
---
applies:
hosted:
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-enable-ccs.html
---

# Enable cross-cluster search and cross-cluster replication [ec-enable-ccs]
# Remote clusters with {{ech}} [ec-enable-ccs]

[Cross-cluster search (CCS)](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html) allows you to configure multiple remote clusters across different locations and to enable federated search queries across all of the configured remote clusters.
You can configure an {{ech}} deployment to remotely access or (be accessed by) a cluster from:

[Cross-cluster replication (CCR)](https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-ccr.html) allows you to replicate indices across multiple remote clusters regardless of where they’re located. This provides tremendous benefit in scenarios of disaster recovery or data locality.

These remote clusters could be:

* Another {{es}} cluster of your {{ecloud}} organization across any region or cloud provider (AWS, GCP, Azure…​)
* An {{es}} cluster of another {{ecloud}} organization
* An {{es}} cluster in an {{ece}} installation
* Any other self-managed {{es}} cluster
* Another {{ech}} deployment of your {{ecloud}} organization, across any region or cloud provider (AWS, GCP, Azure…​)
* An {{ech}} deployment of another {{ecloud}} organization
* A deployment in an {{ece}} installation
* A deployment in an {{eck}} installation
* A self-managed installation.


## Prerequisites [ec-ccs-ccr-prerequisites]

To use CCS or CCR, your deployments must meet the following criteria:

* Local and remote clusters must be in compatible versions. Review the [{{es}} version compatibility](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters-cert.html#remote-clusters-prerequisites-cert) table.
* The local and remote clusters must run on compatible versions of {{es}}. Review the [version compatibility](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters-cert.html#remote-clusters-prerequisites-cert) table.

The steps, information, and authentication method required to configure CCS and CCR can vary depending on where the clusters you want to use as remote are hosted.

* Connect remotely to other clusters from your Elasticsearch Service deployments
* Connect remotely to other clusters from your {{ech}} deployments

* [Access other deployments of the same Elasticsearch Service organization](ec-remote-cluster-same-ess.md)
* [Access deployments of a different Elasticsearch Service organization](ec-remote-cluster-other-ess.md)
* [Access other deployments of the same {{ecloud}} organization](ec-remote-cluster-same-ess.md)
* [Access deployments of a different {{ecloud}} organization](ec-remote-cluster-other-ess.md)
* [Access deployments of an {{ECE}} environment](ec-remote-cluster-ece.md)
* [Access clusters of a self-managed environment](ec-remote-cluster-self-managed.md)
* [Access deployments of an ECK environment](ec-enable-ccs-for-eck.md)

* Use clusters from your Elasticsearch Service deployments as remote
* Use clusters from your {{ech}} deployments as remote

* [From another deployment of your Elasticsearch Service organization](ec-remote-cluster-same-ess.md)
* [From a deployment of another Elasticsearch Service organization](ec-remote-cluster-other-ess.md)
* [From another deployment of your {{ecloud}} organization](ec-remote-cluster-same-ess.md)
* [From a deployment of another {{ecloud}} organization](ec-remote-cluster-other-ess.md)
* [From an ECE deployment](https://www.elastic.co/guide/en/cloud-enterprise/{{ece-version-link}}/ece-enable-ccs.html)
* [From a self-managed cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters.html)



## Enable CCR and the Remote Clusters UI in Kibana [ec-enable-ccr]

If your deployment was created before February 2021, CCR won’t be enabled by default and you won’t find the Remote Clusters UI in Kibana even though your deployment meets all the [criteria](#ec-ccs-ccr-prerequisites).

To enable these features, go to the **Security** page of your deployment and under **Trust management** select **Enable CCR**.

::::{note}
CCR is not supported for indices used by Enterprise Search.
::::
## Enable Remote clusters in {{kib}} [ec-enable-ccr]

If your deployment was created before February 2021, the Remote clusters page in {kib} must be enabled manually from the **Security** page of your deployment, by selecting **Enable CCR** under **Trust management**.


## Remote clusters and traffic filtering [ec-ccs-ccr-traffic-filtering]

::::{note}
Traffic filtering isn’t supported for cross-cluster operations initiated from an {{ece}} environment to a remote {{ess}} deployment.
Traffic filtering isn’t supported for cross-cluster operations initiated from an {{ece}} environment to a remote {{ech}} deployment.
::::


Expand All @@ -66,7 +58,7 @@ For remote clusters configured using TLS certificate authentication, [traffic fi
Traffic filtering for remote clusters supports 2 methods:

* [Filtering by IP addresses and Classless Inter-Domain Routing (CIDR) masks](../security/ip-traffic-filtering.md)
* Filtering by Organization or Elasticsearch cluster ID with a Remote cluster type filter. You can configure this type of filter from the **Features** > **Traffic filters** page of your organization or using the [Elasticsearch Service API](https://www.elastic.co/docs/api/doc/cloud) and apply it from each deployment’s **Security** page.
* Filtering by Organization or {{es}} cluster ID with a Remote cluster type filter. You can configure this type of filter from the **Features** > **Traffic filters** page of your organization or using the [{{ecloud}} RESTful API](https://www.elastic.co/docs/api/doc/cloud) and apply it from each deployment’s **Security** page.

::::{note}
When setting up traffic filters for a remote connection to an {{ece}} environment, you also need to upload the region’s TLS certificate of the local cluster to the {{ece}} environment’s proxy. You can find that region’s TLS certificate in the Security page of any deployment of the environment initiating the remote connection.
Expand Down
Loading
Loading