Remote clusters missing use cases and structural changes (#4170)

This PR addresses some missing use cases on Remote Clusters with ECK.

Due to the missing use cases and the fact that originally we only had
one doc in the Remote Clusters ECK section we have updated the structure
to contain:

- New landing page with links to the different use cases, including the
ones that reside in ECH or ECE sections.
- Dedicated doc to configure remote clusters within ECK (this is the
original doc)
- Dedicated doc to configure remote clusters belonging to different ECH
environments (comes partially from the old section "Connect from a
cluster running outside the kubernetes cluster" where we have refined +
added API key method).
- Created new doc to connect a self-managed cluster to ECK (comes
partially from the old section "Connect from an Elasticsearch cluster
running outside the Kubernetes cluster" where we have refined + added
API key method).
- Created new doc to connect ECK clusters to external remote clusters
(not managed by ECK operator). The remote could be ECH, ECE or
self-managed.

Addresses the following issues:
- #1471
- elastic/cloud-on-k8s#8502

---------

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
Co-authored-by: Fabrizio Ferri-Benedetti <fabri.ferribenedetti@elastic.co>
Co-authored-by: Michael Morello <michael.morello@elastic.co>

Author: 4 people

deploy-manage/remote-clusters/_snippets/allow-connection-intro.md

@@ -1,3 +1,19 @@
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters.md
+- eck-remote-clusters-to-other-eck.md
+- ece-remote-cluster-self-managed.md
+- ece-remote-cluster-same-ece.md
+- ece-remote-cluster-other-ece.md
+- ece-remote-cluster-ece-ess.md
+- ece-enable-ccs-for-eck.md
+- ec-remote-cluster-self-managed.md
+- ec-remote-cluster-same-ess.md
+- ec-remote-cluster-other-ess.md
+- ec-remote-cluster-ece.md
+- ec-enable-ccs-for-eck.md
+-->
 Before you start, consider the [security model](/deploy-manage/remote-clusters/security-models.md) that you would prefer to use for authenticating remote connections between clusters, and follow the corresponding steps.
 
 API key

deploy-manage/remote-clusters/_snippets/apikeys-create-key.md

@@ -1,2 +1,19 @@
-* On the deployment you will use as remote, use the [{{es}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) or [{{kib}}](/deploy-manage/api-keys/elasticsearch-api-keys.md) to create a cross-cluster API key. Configure it with access to the indices you want to use for {{ccs}} or {{ccr}}.
-* Copy the encoded key (`encoded` in the response) to a safe location. You will need it in the next step.
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- remote-clusters-api-key.md
+- eck-remote-clusters-to-other-eck.md
+- eck-remote-clusters-to-external.md
+- ece-remote-cluster-self-managed.md
+- ece-remote-cluster-same-ece.md
+- ece-remote-cluster-other-ece.md
+- ece-remote-cluster-ece-ess.md
+- ece-enable-ccs-for-eck.md
+- ec-remote-cluster-self-managed.md
+- ec-remote-cluster-same-ess.md
+- ec-remote-cluster-other-ess.md
+- ec-remote-cluster-ece.md
+- ec-enable-ccs-for-eck.md
+-->
+1. On the remote cluster, use the [{{es}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) or [{{kib}}](/deploy-manage/api-keys/elasticsearch-api-keys.md) to create a cross-cluster API key. Configure it to include access to the indices you want to use for {{ccs}} or {{ccr}}.
+2. Copy the encoded key (`encoded` in the response) to a safe location. It is required for the local cluster configuration.

deploy-manage/remote-clusters/_snippets/apikeys-intro.md

@@ -1 +1,18 @@
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters.md
+- eck-remote-clusters-to-other-eck.md
+- eck-remote-clusters-to-external.md
+- ece-remote-cluster-self-managed.md
+- ece-remote-cluster-same-ece.md
+- ece-remote-cluster-other-ece.md
+- ece-remote-cluster-ece-ess.md
+- ece-enable-ccs-for-eck.md
+- ec-remote-cluster-self-managed.md
+- ec-remote-cluster-same-ess.md
+- ec-remote-cluster-other-ess.md
+- ec-remote-cluster-ece.md
+- ec-enable-ccs-for-eck.md
+-->
 Follow these steps to configure the [API key security model](/deploy-manage/remote-clusters/security-models.md#api-key) for remote clusters. If you run into any issues, refer to [Troubleshooting](/troubleshoot/elasticsearch/remote-clusters.md).

deploy-manage/remote-clusters/_snippets/apikeys-local-config-intro.md

@@ -1 +1,17 @@
-The API key created previously will be used by the local deployment to authenticate with the corresponding set of permissions to the remote deployment. For that, you need to add the API key to the local deployment's keystore.
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters-to-other-eck.md
+- eck-remote-clusters-to-external.md
+- ece-remote-cluster-self-managed.md
+- ece-remote-cluster-same-ece.md
+- ece-remote-cluster-other-ece.md
+- ece-remote-cluster-ece-ess.md
+- ece-enable-ccs-for-eck.md
+- ec-remote-cluster-self-managed.md
+- ec-remote-cluster-same-ess.md
+- ec-remote-cluster-other-ess.md
+- ec-remote-cluster-ece.md
+- ec-enable-ccs-for-eck.md
+-->
+The API key created previously is needed by the local {{local_type_generic}} to authenticate with the corresponding set of permissions to the remote {{remote_type_generic}}. To enable this, add the API key to the local {{local_type_generic}}'s keystore.

deploy-manage/remote-clusters/_snippets/apikeys-prerequisites-limitations.md

@@ -1,2 +1,17 @@
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters-to-other-eck.md
+- ece-remote-cluster-self-managed.md
+- ece-remote-cluster-same-ece.md
+- ece-remote-cluster-other-ece.md
+- ece-remote-cluster-ece-ess.md
+- ece-enable-ccs-for-eck.md
+- ec-remote-cluster-self-managed.md
+- ec-remote-cluster-same-ess.md
+- ec-remote-cluster-other-ess.md
+- ec-remote-cluster-ece.md
+- ec-enable-ccs-for-eck.md
+-->
 * The local and remote deployments must be on {{stack}} 8.14 or later.
-* Contrary to the certificate security model, the API key security model does not require that both local and remote clusters trust each other.
+* Unlike the certificate-based security model, the API key model does not require mutual trust between clusters; only the local cluster is required to trust the remote cluster's certificate.

deploy-manage/remote-clusters/_snippets/configure-roles-and-users.md

@@ -1,3 +1,20 @@
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters.md
+- eck-remote-clusters-to-other-eck.md
+- eck-remote-clusters-to-external.md
+- ece-remote-cluster-self-managed.md
+- ece-remote-cluster-same-ece.md
+- ece-remote-cluster-other-ece.md
+- ece-remote-cluster-ece-ess.md
+- ece-enable-ccs-for-eck.md
+- ec-remote-cluster-self-managed.md
+- ec-remote-cluster-same-ess.md
+- ec-remote-cluster-other-ess.md
+- ec-remote-cluster-ece.md
+- ec-enable-ccs-for-eck.md
+-->
 % this will need improvement in a future PR, as the text below is only valid for API key based security model
 
-If you're using the API key based security model, to use a remote cluster for {{ccr}} or {{ccs}}, you need to create user roles with [remote indices privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure.md#roles-remote-indices-priv) on the local cluster. Refer to [Configure roles and users](/deploy-manage/remote-clusters/remote-clusters-api-key.md#remote-clusters-privileges-api-key).
+If you're using the API key–based security model for {{ccr}} or {{ccs}}, you can define user roles with [remote indices privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure.md#roles-remote-indices-priv) on the local cluster to further restrict the permissions granted by the API key. For more details, refer to [Configure roles and users](/deploy-manage/remote-clusters/remote-clusters-api-key.md#remote-clusters-privileges-api-key).

deploy-manage/remote-clusters/_snippets/eck_apikey_secret.md

@@ -0,0 +1,20 @@
+<!--
+This snippet is in use in the following locations:
+- eck-remote-clusters-to-other-eck.md
+- eck-remote-clusters-to-external.md
+-->
+The following command creates a secret containing the encoded API key obtained earlier:
+
+```sh
+cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: Secret
+metadata:
+  name: remote-api-keys
+type: Opaque
+stringData:
+  cluster.remote.<remote-cluster-name>.credentials: <encoded value> <1>
+EOF
+```
+1. For the `<remote-cluster-name>`, enter the alias of your choice. This alias is used when connecting to the remote cluster. It must be lowercase and only contain letters, numbers, dashes, and underscores.
+

deploy-manage/remote-clusters/_snippets/eck_expose_transport.md

@@ -1,15 +1,26 @@
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters-to-other-eck.md
+- ece-enable-ccs-for-eck.md
+- ec-enable-ccs-for-eck.md
+-->
 Expose the transport service (defaults to port `9300`) of your ECK cluster to allow external {{es}} clusters to connect:
 
 ```yaml
 apiVersion: elasticsearch.k8s.elastic.co/v1
 kind: Elasticsearch
 metadata:
-  name: <cluster-name>
+  name: <remote-cluster-name>
 spec:
   transport:
     service:
       spec:
         type: LoadBalancer <1>
 ```
 
-1. On cloud providers which support external load balancers, setting the type field to `LoadBalancer` provisions a load balancer for your service. Alternatively, expose the service `<cluster-name>-es-transport` through one of the Kubernetes Ingress controllers that support TCP services.
+1. On cloud providers which support external load balancers, setting the `type` field to `LoadBalancer` provisions a load balancer for your service. Alternatively, expose the service `<cluster-name>-es-transport` through one of the Kubernetes Ingress controllers that support TCP services.
+
+:::{note}
+If you change the service’s `port` to expose a different port externally, set `targetPort` explicitly to `9300`, which is the default transport service listening port. Otherwise, Kubernetes uses the same value for both fields, resulting in failed connections.
+:::

deploy-manage/remote-clusters/_snippets/eck_rcs_certs_retrieve_ca.md

@@ -0,0 +1,22 @@
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters-to-other-eck.md
+-->
+ECK stores the CA used to issue certificates for the {{es}} transport layer in a secret named `<cluster_name>-es-transport-certs-public`.
+
+For example, to save the transport CA certificate of a cluster named `quickstart` into a local file, run the following command:
+
+```sh
+kubectl get secret quickstart-es-transport-certs-public \
+-o go-template='{{index .data "ca.crt" | base64decode}}' > quickstart_transport_ca.crt
+```
+
+::::{note}
+Beware of copying the source secret as-is into a different namespace. Refer to [Copying secrets with Owner References](/troubleshoot/deployments/cloud-on-k8s/common-problems.md#k8s-common-problems-owner-refs) for more information.
+::::
+
+::::{note}
+CA certificates are automatically rotated after one year by default. You can [configure](/deploy-manage/deploy/cloud-on-k8s/configure-eck.md) this period. Make sure to keep the copy of the certificates secret up-to-date.
+::::
+

deploy-manage/remote-clusters/_snippets/eck_rcs_connect_intro.md

@@ -1,5 +1,19 @@
-On the local deployment, add the remote ECK cluster using {{kib}} or the {{es}} API with the following connection settings:
+<!--
+This snippet is in use in the following locations:
+- self-remote-cluster-eck.md
+- eck-remote-clusters-to-other-eck.md
+- ece-enable-ccs-for-eck.md
+- ec-enable-ccs-for-eck.md
 
-* **Remote address**: Use the FQDN or IP address of the LoadBalancer service, or similar resource, you created to expose the remote cluster server interface (for API key-based authentication) or the transport interface (for TLS certificate-based authentication).
+It requires local_type_generic substitution to be defined
+-->
+On the local {{local_type_generic}}, use {{kib}} or the {{es}} API to add the remote ECK cluster with the following connection settings:
+
+* **Remote address**: Use the FQDN or IP address of the LoadBalancer service or alternative resource that you created to expose the remote cluster.
+
+	* For API key-based authentication, use the server interface address and port.
+	* For TLS certificate-based authentication, use the transport interface address and port.
+
+  If you haven't changed the external listening port of the kubernetes service, the port should be `9443` for API key-based authentication, or `9300` for TLS certificate-based authentication.
 
 * **TLS server name**: You can try leaving this field empty first. If the connection fails, and your environment is presenting the ECK-managed certificates during the TLS handshake, use `<cluster-name>-es-remote-cluster.<namespace>.svc` as the server name. For example, for a cluster named `quickstart` in the `default` namespace, use `quickstart-es-remote-cluster.default.svc`.