Skip to content
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions deploy-manage/remote-clusters/remote-clusters-api-key.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ To add a remote cluster using API key authentication:
2. [Establish trust with a remote cluster](#remote-clusters-security-api-key)
3. [Connect to a remote cluster](#remote-clusters-connect-api-key)
4. [Configure roles and users](#remote-clusters-privileges-api-key)
5. (Optional) [Configure remote cluster strong verification](#remote-cluster-strong-verification) for additional security

If you run into any issues, refer to [Troubleshooting](/troubleshoot/elasticsearch/remote-clusters.md).

Expand Down Expand Up @@ -424,3 +425,80 @@ POST /_security/user/cross-search-user
```

Note that you only need to create this user on the local cluster.


## Remote cluster strong verification [remote-cluster-strong-verification]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a lot more settings specified in your reference PR. how does someone know from this procedure when to use those additional settings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I wasn't sure if it made sense to include all settings. This is the most basic configuration to keep it simple, would it make sense to link the other settings here you think?

This is a setup the mirrors our SSL configuration so I could either copy some of the instructions from there or link that too? WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including just some settings in the configuration guide should be ok, but probably we should include all settings we want users to know in the Elasticsearch reference docs. Somewhere like here: https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/remote-clusters, or here: https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/networking-settings#remote-cluster-network-settings (we have settings in 2 places).

We have an opened internal issue to review these settings as some of them (remote_cluster_client) are missing in the reference docs.

For your consideration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I see you have created elastic/elasticsearch#137822! So I'd just wait until the Elasticsearch PR is merged first and then in this PR just link to the reference doc for reference about all available settings, with something like:

For a full list of available strong verification settings for remote clusters, refer to xxxxx.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that the improved functionlity is named "strong verification", while we (I?) generally use "strong identity verification" elsewhere.

Was there a discussion about that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is an oversight from me. I'll update it! Thanks for reviewing!

preview::[]

Cross-cluster API keys can be configured with strong verification to provide an additional layer of security. To enable this feature,
the cross-cluster API key is created with a certificate identity pattern. The local cluster is then required to sign each request
to the remote cluster with its private key and provide a trusted certificate whose subject matches the certificate identity
pattern configured on the cross-cluster API key.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph could be clearer about which cluster each component lives on, and it should mention how trust is established. The phrase "provide a trusted certificate" doesn't explain that the remote cluster validates trust based on its configured certificate authorities.

Something like this:

"Cross-cluster API keys can be configured with strong verification to provide an additional layer of security. To enable this feature, a cross-cluster API key is created on the remote cluster with a certificate identity pattern that specifies which certificates are allowed to use it. The local cluster must then sign each request with its private key and include a certificate whose subject Distinguished Name (DN) matches the pattern. The remote cluster validates both that the certificate is trusted by its configured certificate authorities and that the certificate's subject matches the API key's identity pattern."

This makes it explicit that

  1. The API key lives on the remote cluster
  2. Trust is validated against the remote cluster's CA configuration

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also consider mentioning that this is configured per-connection (each remote cluster alias can have different signing certs) and per-API-key (each API key can have different certificate identity patterns). This helps users understand the granularity of the feature.


### How strong verification works [_how_strong_verification_works]

When a local cluster makes a request to a remote cluster using a cross-cluster API key:

1. The local cluster signs the request headers with its configured private key and sends the signature and certificate chain as header
in the request to the remote cluster
2. The remote cluster verifies that the API key is valid
3. If the API key has a certificate identity pattern configured, the remote cluster extracts the Distinguished Name (DN) from the
certificate chain's leaf certificate and matches it against the certificate identity pattern
4. The remote cluster validates that the provided certificate chain is trusted
5. The remote cluster validates the signature and checks that the certificate is not expired

If any of these validation steps fail, the request is rejected.

### Configure strong verification [_configure_strong_verification]

To use certificate identity validation, the local and remote clusters must be configured to sign request headers and to verify request
headers.

#### On the local cluster [_certificate_identity_local_cluster]

The local cluster must be configured to sign cross-cluster requests with a certificate-private key pair. You can generate a signing
certificate using `elasticsearch-certutil` or use an existing certificate.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if this should link back to the section above that shows how to use elasticsearch-certutil

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also wondering: for ECH if a customer wants to use their own certs, they would need to use the bundles feature. Where would we document instructions for that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good point! I think that should be specifically for the ECH documentation. I'll investigate where we should add that. Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gmjehovich , @jfreden , I've been working lately in all docs related with remote clusters, for all deployment types (ECH, ECE, ECK, self-managed). Feel free to ping me in private and we can review where to add information related with this PR (Strong verification configuration) for other deployment types, and I could do that in another PR.


```yaml
cluster.remote.my_remote_cluster.signing.certificate: "path/to/signing/certificate.crt"
cluster.remote.my_remote_cluster.signing.key: "path/to/signing/key.key"
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would explicitly mention that the private key can be encrypted and the password must be stored securely in the keystore. People may be wondering otherwise.

::::{note}
Replace my_remote_cluster with your remote cluster alias and the paths with actual paths
::::

#### On the remote cluster [_certificate_identity_remote_cluster]

The remote cluster must be configured with a certificate authority that trusts the certificate that was used to sign the request headers.

```yaml
cluster.remote.signing.certificate_authorities: "path/to/signing/certificate_authorities.crt"
```

When creating a cross-cluster API key on the remote cluster, specify a `certificate_identity` pattern that matches the Distinguished
Name (DN) of the local cluster's certificate. Use the [Create Cross-Cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) API:

```console
POST /_security/cross_cluster/api_key
{
"name": "my-cross-cluster-api-key",
"access": {
"search": [
{
"names": ["logs-*"]
}
]
},
"certificate_identity": "CN=local-cluster.example.com,O=Example Corp,C=US"
}
```

The `certificate_identity` field supports regular expressions. For example:

* `"CN=.*.example.com,O=Example Corp,C=US"` matches any certificate with a CN ending in "example.com"
* `"CN=local-cluster.*,O=Example Corp,C=US"` matches any certificate with a CN starting with "local-cluster"
* `"CN=.*"` matches any certificate (not recommended for production)

To verify strong verification is active, check the Elasticsearch logs on the remote cluster for certificate validation messages, or

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if this should be more specific by including the actual output/examples from logs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there isn't one right now, my bad. Thinking more about this, having an info log for every successful request will be too verbose so I'll just remove this.

attempt a connection without proper signing configured (which should fail with a certificate validation error).