[Issue]: Doc about ECH/ECK CCS/CCR is misleading

[jeanfabrice]

Type of issue

Inaccurate

What documentation page is affected

https://www.elastic.co/docs/deploy-manage/remote-clusters/ec-enable-ccs-for-eck

What happened?

Hi Team,

We might have found a few issues regarding this documentation page: https://www.elastic.co/docs/deploy-manage/remote-clusters/ec-enable-ccs-for-eck

Some are structural / important and can lead to a cluster down situation.

Let me try to explain

First part in the documentation, Establish trust in the Elastic Cloud Hosted cluster, is about to make ECH aware of the ECK transport CA and build a trust.yml from the Cloud UI wizard to be installed on the ECK cluster. This file will make the local ECK cluster trust a limited set of SSL certificates based on their subject name. The second step of this part says: Select the Elastic Cloud pattern and enter default.es.local for the Scope ID.

Unfortunately, this is wrong for various reasons.

1. The first reason is that default.es.local is incorrect. It is only valid when the ECK cluster is deployed in the default Kubernetes namespace. If the user has deployed his ECK cluster in a different namespace, this is not valid anymore and should be replaced with <kubernetes-namespace>.es.local
2. The second reason is that the Cloud UI wizard (see screenshot) that helps to build the trust.yml file is 100% ECH/ECE oriented. If you enter the (now) correct scope ID <kubernetes-namespace>.es.local, it will generate a trust.yml file that contains:

trust.subject_name:
  - "*.node.2dc556bb4bd040e89d0135683b66a2f6.cluster.1075708151.account"
  - "*.node.*.cluster.<kubernetes-namespace>.es.local.account" <= this does not follow the certificate subject name generated by ECK

Here, *.node.*.cluster.<kubernetes-namespace>.es.local.account is incorrect:

• It ends with an accountsuffix that is specific to ECH (and maybe ECE, I don't know)
• This format, *.node.*.cluster.<kubernetes-namespace>.es.local.account does not match what is detailed in the Cloud UI wizard ({node_id}.node.{cluster_id}.{scope_id}), and it also doesn't match what is in the subject name of the certificates generated by ECK, which have the form: {node_id}.node.{cluster_name}.{scope_id} where {scope_id} is {kubernetes-namespace}.es.local

As a conclusion:

• The current documentation wrongly suggests to use the trust.yml file generated from the cloud UI wizard, suggesting misleading informations (default.es.local). As soon as the customer uses the generated trust.yml file, his ECK Elasticsearch cluster will stop, due to local node not trusting each other anymore
• The cloud UI wizard does not permit the creation of a valid trust.yml file for ECK as it is 100% Elastic Cloud oriented
• Unfortunately, the Cloud UI wizard does not permit to fall back to another naming convention since it doesn't accept wildcards

As a quick win, from a documentation perspective, we should fix the doc and suggest to edit the trust.yml file manually, adding the ECK subject name wildcard (*.node.<cluster-name>.<kubernetes-namespace>.es.local) while not adding any other from the Cloud UI wizard
As a second step, the cloud team could fix the cloud UI wizard, allowing wildcards or making it compatible with ECK
I'm happy to discuss this further with you

Screenshot:

pinging @eedugon as requested

Additional info

No response

[eedugon]

@pebrc : what do you think of this issue? TLDR, Jean Fabrice is proposing to make two changes to that doc:

• Replacing default.es.local with <kubernetes-namespace>.es.local
• Telling users to edit trust.yml downloaded from Cloud UI wizard and not trying to integrate the scope ID with it.

The reason for the changes is probably a bug in cloud UI wizard.

@jakommo , do you think we should report the second part of the issue to cloud devs so users can really generate a valid trust.yml with that UI?

This format, .node..cluster..es.local.account does not match what is detailed in the Cloud UI wizard ({node_id}.node.{cluster_id}.{scope_id}), and it also doesn't match what is in the subject name of the certificates generated by ECK, which have the form: {node_id}.node.{cluster_name}.{scope_id} where {scope_id} is {kubernetes-namespace}.es.local

In the way I'm understanding this (which might not be ok), all this might have been caused by a bug in the cloud UI wizard, as the generated trust.yml is not really following its own description.

[jakommo]

@jakommo , do you think we should report the second part of the issue to cloud devs so users can really generate a valid trust.yml with that UI?

Thanks for the ping, yes, we should definitely report this to Cloud UI folks and check how we can make it work for ECK @jeanfabrice can you please raise an issue for this and tag me on it?

[jeanfabrice]

@jakommo Sure! Can you suggest me the proper Github repo to raise the issue?

[pebrc]

@eedugon sorry missed the ping here for some reason.

Yes I think documenting the correct trust.yml syntax could be a short term workaround until the cloud UI is made more flexible.

[eedugon]

The YAML example of that page was extremely broken as @naemono had pointed out 🤦‍♂.

Fixing it also on the scope of this issue.

For the kubernetes secret and configmap creation I'm going to suggest ca.crt and trust.yml as the keys to create in the secret, and instead of the otherdirectory I'll propose ech-ca. Then the final YAML won't need modifications or <CA-Certificate-Filename>.

Something like:

Secret:

```sh
kubectl create secret generic ce-aws-cert --from-file=ca.crt=<path to certificate file>
```

CM for trust:

```sh
kubectl create configmap quickstart-trust --from-file=trust.yml=<path to trust.yml>
```

Example of integrating the secret and CM into Elasticsearch nodeSet:

```yaml
spec:
  nodeSets:
  - config:
      xpack.security.transport.ssl.certificate_authorities:
        - /usr/share/elasticsearch/config/ech-ca/ca.crt
      xpack.security.transport.ssl.trust_restrictions.path: /usr/share/elasticsearch/config/trust-filter/trust.yml
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          volumeMounts:
            - mountPath: /usr/share/elasticsearch/config/ech-ca
              name: ce-aws-cert
            - mountPath: /usr/share/elasticsearch/config/trust-filter
              name: quickstart-trust
        volumes:
          - name: ce-aws-cert
            secret:
              secretName: ce-aws-cert
          - name: quickstart-trust
            configMap:
              name: quickstart-trust
```