Send cross cluster api key signature as header #135674

jfreden · 2025-09-30T08:19:34Z

This PR is a follow up to #134137 and #134893. It adds serialization and verification of the new header:

The signing configurations are now used to generate a signature that's passed as a header for cross cluster api keys.
The signature headers are deserializaed and validated on the server side and auth fails if the validation fails.

This PR does not use the certificate identity that was added in #134604 to verify that the identity in the passed leaf certificate belongs to the signed cross cluster API key by matching it against the API key certificate identity pattern. That will be done in a follow up PR to keep the scope of this PR manageable.

jfreden · 2025-10-03T06:26:22Z

...src/main/java/org/elasticsearch/xpack/core/security/authc/CrossClusterAccessSubjectInfo.java

-            );
-        }
-        return decode(header);
-    }


We need the encoded header for the signature so I moved this to CrossClusterAccessHeaders.

jfreden · 2025-10-03T06:27:42Z

.../java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBWCToRCS2ClusterRestIT.java

            .apply(commonClusterConfig)
            .setting("xpack.security.remote_cluster_client.ssl.enabled", "true")
            .setting("xpack.security.remote_cluster_client.ssl.certificate_authorities", "remote-cluster-ca.crt")
+            .configFile("signing.crt", Resource.fromClasspath("signing/signing.crt"))


This adds a signature config to the query cluster to make sure we don't send it when communicating with older fulfilling clusters.

jfreden · 2025-10-03T06:29:18Z

.../org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityCrossClusterApiKeySigningIT.java

+
+        assertCrossClusterSearchSuccessfulWithoutResult();
+
+        // TODO add test for certificate identity configured for API key but no signature provided (should 401)


When the cross cluster api key with a cert identity is available these test will be implemented here.

I'll do this in a separate PR to keep scope manageable.

jfreden · 2025-10-03T12:59:06Z

server/src/main/resources/transport/upper_bounds/9.3.csv

@@ -1 +1 @@
-esql_plan_with_no_columns,9186000


New transport version needed for make sure we don't serialize the new thread context header for a fulfilling cluster that doesn't support it.

jfreden · 2025-10-03T13:01:53Z

...org/elasticsearch/xpack/security/transport/CrossClusterApiKeySignatureManagerIntegTests.java

    private static final String STATIC_TEST_CLUSTER_ALIAS = "static_test_cluster";

-    public void testSignWithPemKeyConfig() {
+    public void testSignWithPemKeyConfig() throws GeneralSecurityException {


Updated the verifier to move the auth error message building out of the verifier. That way the auth error can be constructed in a way that we can easily separate between cert chain validation error and signature verification error.

jfreden · 2025-10-03T13:02:25Z

...ain/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessAuthenticationService.java

                    assert authentication.isApiKey() : "initial authentication for cross cluster access must be by API key";
                    assert false == authentication.isRunAs() : "initial authentication for cross cluster access cannot be run-as";
+
+                    // TODO ALWAYS check if used api key has a certificate identity and do this verification conditionally based on that


This TODO will be addressed in a follow up.

jfreden · 2025-10-03T13:03:53Z

...security/src/main/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessHeaders.java

+        this(credentialsHeader, crossClusterAccessSubjectInfo, null);
+    }
+
+    private CrossClusterAccessHeaders(


This constructor is only used when we read the headers, the other one when constructing the headers.

jfreden · 2025-10-03T13:06:39Z

...ain/java/org/elasticsearch/xpack/security/transport/CrossClusterAccessSecurityExtension.java

            components.securityContext(),
            this.authenticationService,
-            this.crossClusterApiKeySignatureManager,
+            crossClusterApiKeySignatureManager,


I pass the full manager here and only the verifier above because the manager will issue a new signer per remote cluster while we only have a single verifier per cluster.

jfreden · 2025-10-03T13:11:13Z

.../java/org/elasticsearch/xpack/security/transport/CrossClusterAccessTransportInterceptor.java

+                    if (connection.getTransportVersion().supports(ADD_CROSS_CLUSTER_API_KEY_SIGNATURE)) {
+                        crossClusterAccessHeaders.writeToContext(threadContext, signer);
+                    } else {
+                        crossClusterAccessHeaders.writeToContext(threadContext, null);


Passing a null signer is the same as "do not generate a signature", not very explicit but probably acceptable. Even if a signature is configured there is no use in providing it unless the fulfilling cluster can validate it. I considered failing here but this could actually happen in production while upgrading and we don't want to fail requests during a rolling upgrade for example.

jfreden · 2025-10-03T13:12:13Z

...main/java/org/elasticsearch/xpack/security/transport/CrossClusterApiKeySignatureManager.java

-            } catch (GeneralSecurityException e) {
-                logger.debug("failed certificate validation for Signature [" + signature + "]", e);
-                throw new ElasticsearchSecurityException("Failed to verify signature from [{}]", signature.certificates()[0], e);
+            // Make sure the provided certificate chain is trusted


This change only removes the try-catch to let the caller handle different exceptions however they please.

jfreden · 2025-10-03T13:12:53Z

...urity/src/main/java/org/elasticsearch/xpack/security/transport/X509CertificateSignature.java

    }

-    private String certificateToString(X509Certificate certificate) {
+    public static String certificateToString(X509Certificate certificate) {


This is changed to be static and public to be used in error messages.

jfreden · 2025-10-03T13:14:16Z

...ity/src/test/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessHeadersTests.java

        );
    }

+    private X509Certificate[] getTestCertificates() throws CertificateException, IOException {


Mocking certs is hard and therefore I'm just reading a real one here.

elasticsearchmachine · 2025-10-03T13:27:22Z

Pinging @elastic/es-security (Team:Security)

elasticsearchmachine · 2025-10-03T13:27:23Z

Hi @jfreden, I've created a changelog YAML for you.

jfreden · 2025-10-03T14:23:06Z

...main/java/org/elasticsearch/xpack/security/transport/CrossClusterApiKeySignatureManager.java


    public Signer signerForClusterAlias(String clusterAlias) {
-        return new Signer(clusterAlias);
+        return keyPairByClusterAlias.containsKey(clusterAlias) ? new Signer(clusterAlias) : null;


This has changed to only return a signer if we have a signing config. This makes it a little easier to understand if there is a signing config or not in a caller.

gmjehovich · 2025-10-06T17:24:11Z

...security/src/main/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessHeaders.java

+    public void writeToContext(final ThreadContext ctx, @Nullable CrossClusterApiKeySignatureManager.Signer signer) throws IOException {
+        var encodedSubjectInfo = crossClusterAccessSubjectInfo.encode();
+        if (signer != null) {
+            var signature = signer.sign(encodedSubjectInfo, credentialsHeader);


Does the signed payload include anything request-specific (timestamps, nonces, etc.)? Do we need to worry about replay protection here?

Started an internal thread on this. In summary: replay protection is already handled by TLS.

gmjehovich · 2025-10-06T18:18:18Z

...security/src/main/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessHeaders.java

This is more of a general question for my understanding; do we need to worry about header size limits? Would the new certificate information we are adding have any possibility of breaching them?

Since this is all sent over the transport protocol we don't have any special restrictions on headers. I think a concern could be if it gets expensive to generate, serialize, parse or validate the headers.

gmjehovich · 2025-10-06T18:26:44Z

...security/src/main/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessHeaders.java

+
+        final String subjectInfoHeader = ctx.getHeader(CROSS_CLUSTER_ACCESS_SUBJECT_INFO_HEADER_KEY);
+        if (subjectInfoHeader == null) {
+            throw new IllegalArgumentException(


nit: Wondering if SecurityException would be more appropriate here?

This code actually just moved from the SubjectInfo class because I needed the encoded string before decoding it to create the signature.

I think IllegalArgumentException is appropriate here since we're missing the subject info argument in the API call and it appears to be the convention throughout the RCS implementation. I think SecurityException could make sense too, since you're missing a required auth argument.

gmjehovich

LGTM! Will leave it up to you whether or not you also want Slobodan to look it over

ankit--sethi · 2025-10-07T19:09:49Z

...security/src/main/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessHeaders.java

    }

-    public void writeToContext(final ThreadContext ctx) throws IOException {
+    public void writeToContext(final ThreadContext ctx, @Nullable CrossClusterApiKeySignatureManager.Signer signer) throws IOException {


nit: you could overload the method here:

public void writeToContext(final ThreadContext ctx) throws IOException { return writeToContext(ctx, null); }

which will cut down the footprint of the various writeToContext(threadContext, null); diffs.

Thanks for reviewing! I was actually considering that but eventually decided not to since in production code we always specify the signer (unless we're on a different transport version, but that code will eventually go away).

I don't think we should add overloaded methods only to reduce the footprint of the test code, since it can result in test specific code that doesn't reflect reality.

oh I see! yeah, it's usually a code smell to update src/main merely for the sake of src/test

slobodanadamovic · 2025-10-09T11:02:11Z

...ain/java/org/elasticsearch/xpack/security/authc/CrossClusterAccessAuthenticationService.java

+                    var signature = crossClusterAccessHeaders.signature();
+                    // Always validate a signature if provided
+                    if (signature != null) {
+                        verifySignature(signature, crossClusterAccessHeaders);


I'm not sure if this is a good place to verify signature. Is there a reason why we don't pull verification into ApiKeyService?

What do you think if we extend ApiKeyCredentials with signature and let ApiKeyService verify API key credential, signature and caching?

Another advantage is that we would also have CrossClusterAccessAuthenticationService#authenticateHeaders method be able to do an early verification.

Also, if I'm not mistaken, throwing authentication exception here would cause us to audit the same request twice: once successfully because API keys auth succeeded and second failure because we failed the signature verification.

Well actually, I haven't fully thought about caching bit. Intuitively I'd say we should cache successful auth and signature verification results. But will need to look into this closely.

We talked about this offline, the idea is to follow this up when we do the certificate identity verification.

In summary we'll:

Do the signature and certificate verification up front and extract the certificate identity from the request around here before anything else happens (so before regular auth)

Add a new certificateIdentity field to ApiKeyCredentials here and populate it around here

Only validate that the certificateIdentity against the field from that api key doc in the index later somewhere around here

In terms of cache we agreed that at least initially we won't do any caching for the signature verification since the signature would be unique per user (subject info) and risks being very large when there are a lot of concurrent users. We should revisit this decision in the follow up.

slobodanadamovic · 2025-10-09T11:08:20Z

...java/org/elasticsearch/xpack/security/transport/CrossClusterAccessServerTransportFilter.java

+    static final Set<String> ALLOWED_TRANSPORT_HEADERS = Stream.concat(
+        Stream.of(
+            CROSS_CLUSTER_ACCESS_CREDENTIALS_HEADER_KEY,
+            CROSS_CLUSTER_ACCESS_SUBJECT_INFO_HEADER_KEY,
+            CROSS_CLUSTER_ACCESS_SIGNATURE_HEADER_KEY,
+            AuditUtil.AUDIT_REQUEST_ID,
+            Task.TRACE_STATE
+        ),
+        Task.HEADERS_TO_COPY.stream()
+    ).collect(Collectors.toUnmodifiableSet());


Nice. Alternatively to this would be using Sets.union().

Nice, that's cleaner!

slobodanadamovic

LGTM 👍

…aders

We have to trust the cert we're signing with. Introduced in #135674. Resolves: #136368

We have to trust the cert we're signing with. Introduced in elastic#135674. Resolves: elastic#136368

elasticsearchmachine added v9.2.0 v9.3.0 and removed v9.2.0 labels Sep 30, 2025

jfreden commented Oct 3, 2025

View reviewed changes

jfreden force-pushed the add_signature_to_headers branch 3 times, most recently from da8e485 to 1ad64cf Compare October 3, 2025 12:34

jfreden commented Oct 3, 2025

View reviewed changes

Send cross cluster api key signature as headers

1173919

jfreden force-pushed the add_signature_to_headers branch from 1ad64cf to 1173919 Compare October 3, 2025 13:26

jfreden marked this pull request as ready for review October 3, 2025 13:26

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Oct 3, 2025

jfreden added >enhancement :Security/Security Security issues without another label and removed needs:triage Requires assignment of a team area label labels Oct 3, 2025

jfreden requested review from gmjehovich and slobodanadamovic October 3, 2025 13:27

elasticsearchmachine added the Team:Security Meta label for security team label Oct 3, 2025

Update docs/changelog/135674.yaml

e2c0425

jfreden changed the title ~~Send cross cluster api key signature as headers~~ Send cross cluster api key signature as header Oct 3, 2025

jfreden added 2 commits October 3, 2025 16:17

Fix fips issue in test - different message

ac84479

Fix signer behaviour change in test

f406c8b

jfreden commented Oct 3, 2025

View reviewed changes

elasticsearchmachine and others added 3 commits October 3, 2025 14:24

[CI] Auto commit changes from spotless

10b0281

Merge branch 'main' into add_signature_to_headers

7977629

Merge branch 'main' into add_signature_to_headers

26fe56b

gmjehovich reviewed Oct 6, 2025

View reviewed changes

gmjehovich approved these changes Oct 7, 2025

View reviewed changes

ankit--sethi reviewed Oct 7, 2025

View reviewed changes

slobodanadamovic reviewed Oct 9, 2025

View reviewed changes

jfreden added 3 commits October 9, 2025 15:37

Review comments

c865f50

fixup! Import

113139e

Fix failing test

2670f02

slobodanadamovic approved these changes Oct 9, 2025

View reviewed changes

jfreden added 2 commits October 10, 2025 09:53

Merge remote-tracking branch 'upstream/main' into add_signature_to_he…

717572d

…aders

Changes after merge main

66791a0

jfreden force-pushed the add_signature_to_headers branch from 328e56b to 66791a0 Compare October 10, 2025 08:03

jfreden added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Oct 10, 2025

elasticsearchmachine and others added 2 commits October 10, 2025 08:11

[CI] Auto commit changes from spotless

531d35c

Merge branch 'main' into add_signature_to_headers

f0da020

elasticsearchmachine merged commit 74da8c0 into elastic:main Oct 10, 2025
40 checks passed

jfreden deleted the add_signature_to_headers branch October 10, 2025 10:22

jfreden mentioned this pull request Oct 10, 2025

Fix RCS Signing BWC Tests by Adding CA cert to FC #136384

Merged

elasticsearchmachine pushed a commit that referenced this pull request Oct 13, 2025

Fix RCS Signing BWC Tests by Adding CA cert to FC (#136384)

efd41a4

We have to trust the cert we're signing with. Introduced in #135674. Resolves: #136368

georgewallace pushed a commit to georgewallace/elasticsearch that referenced this pull request Oct 13, 2025

Fix RCS Signing BWC Tests by Adding CA cert to FC (elastic#136384)

e73a935

We have to trust the cert we're signing with. Introduced in elastic#135674. Resolves: elastic#136368

jfreden mentioned this pull request Oct 14, 2025

Validate Certificate Identity provided in Cross Cluster API Key Certificate #136299

Open


		assertCrossClusterSearchSuccessfulWithoutResult();

		// TODO add test for certificate identity configured for API key but no signature provided (should 401)

		@@ -1 +1 @@
		esql_plan_with_no_columns,9186000

Send cross cluster api key signature as header #135674

Send cross cluster api key signature as header #135674

Uh oh!

Conversation

jfreden commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 3, 2025

Uh oh!

elasticsearchmachine commented Oct 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfreden Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmjehovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slobodanadamovic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

jfreden commented Sep 30, 2025 •

edited

Loading

jfreden Oct 7, 2025 •

edited

Loading