Add convenience API key param to remote reindex (#135949)

This adds a `source.remote.api_key` parameter to the reindex API. This
is equivalent to using `source.remote.headers` with `Authorization`
set to `ApiKey <api_key>`. It is a convenience for the user.

Note on docs changes: The example request using an API key now uses an
`applies-switch` to show both the pre-9.3 and post-9.3 / serverless
versions. There are a few drive-by doc bug-fixes and other improvements to
the docs, most notably:

- Adding some subsections, as suggested by a tech writer.

- Fixing the bug which meant that `<OTHER_HOST_URL>` was previously
  not being rendered, because `<` is a control character.

- Repeating the note about using HTTPS for basic auth in the API key
  section, as we wouldn't recommend sending API keys over plain HTTP
  either.

- A nit, but writing `<OTHER_HOST_URL>:9200` is strange, since we have
  a placeholder URL, and the port is part of the URL, so that's fixed.

* Update docs/reference/elasticsearch/rest-apis/reindex-indices.md

Co-authored-by: shainaraskas <[email protected]>

* add cross-reference to SSL settings in docs

* Remove the `When using {{escloud}}` caveat at the start of the API key section, since you can use API keys on pretty much any ES type

---------

Co-authored-by: shainaraskas <[email protected]>

ES-9691 #comment Added `api_key` to reindex API as convenience parameter in #135949

Author: PeteGillinElastic

docs/changelog/135949.yaml

@@ -0,0 +1,5 @@
+pr: 135949
+summary: Add convenience API key param to remote reindex
+area: Indices APIs
+type: enhancement
+issues: []

docs/reference/elasticsearch/rest-apis/reindex-indices.md

@@ -597,7 +597,7 @@ POST _reindex
 {
   "source": {
     "remote": {
-      "host": "<OTHER_HOST_URL>:9200",
+      "host": "&lt;OTHER_HOST_URL>",
       "username": "user",
       "password": "pass"
     },
@@ -619,20 +619,55 @@ POST _reindex
 % TEST[s/"username": "user",/"username": "test_admin",/]
 % TEST[s/"password": "pass"/"password": "x-pack-test-password"/]
 
-The `host` parameter must contain a scheme, host, port (for example, `https://otherhost:9200`), and optional path (for example, `https://otherhost:9200/proxy`).
-The `username` and `password` parameters are optional, and when they are present the reindex API will connect to the remote {{es}} node using basic auth.
-Be sure to use `https` when using basic auth or the password will be sent in plain text. There are a range of settings available to configure the behaviour of the `https` connection.
+The `host` parameter must contain a scheme, host, port (for example, `https://<OTHER_HOST_URL>:9200`), and optional path (for example, `https://<OTHER_HOST_URL>:9200/proxy`).
 
-When using {{ecloud}}, it is also possible to authenticate against the remote cluster through the use of a valid API key:
+### Using basic auth [reindex-basic-auth]
 
+To authenticate with the remote cluster using basic auth, set the `username` and `password` parameters, as in the example above.
+Be sure to use `https` when using basic auth, or the password will be sent in plain text. There are a [range of settings](#reindex-ssl) available to configure the behaviour of the `https` connection.
+
+### Using an API key [reindex-api-key]
+
+It is also possible (and encouraged) to authenticate with the remote cluster through the use of a valid API key:
+
+::::{applies-switch}
+
+:::{applies-item} { "stack": "ga 9.3", "serverless": }
 ```console
 POST _reindex
 {
   "source": {
     "remote": {
-      "host": "<OTHER_HOST_URL>:9200",
+      "host": "&lt;OTHER_HOST_URL>",
+      "api_key": "&lt;API_KEY_VALUE>"
+    },
+    "index": "my-index-000001",
+    "query": {
+      "match": {
+        "test": "data"
+      }
+    }
+  },
+  "dest": {
+    "index": "my-new-index-000001"
+  }
+}
+```
+% TEST[setup:host]
+% TEST[s/^/PUT my-index-000001\n/]
+% TEST[s/otherhost:9200",/\${host}",/]
+% TEST[s/"headers": \{[^}]*\}/"username": "test_admin", "password": "x-pack-test-password"/]
+:::
+
+:::{applies-item} { "stack": "ga 9.0" }
+```console
+POST _reindex
+{
+  "source": {
+    "remote": {
+      "host": "&lt;OTHER_HOST_URL>",
       "headers": {
-        "Authorization": "ApiKey API_KEY_VALUE"
+        "Authorization": "ApiKey &lt;API_KEY_VALUE>"
       }
     },
     "index": "my-index-000001",
@@ -651,15 +686,26 @@ POST _reindex
 % TEST[s/^/PUT my-index-000001\n/]
 % TEST[s/otherhost:9200",/\${host}",/]
 % TEST[s/"headers": \{[^}]*\}/"username": "test_admin", "password": "x-pack-test-password"/]
+:::
+
+::::
+
+
+Be sure to use `https` when using an API key, or it will be sent in plain text. There are a [range of settings](#reindex-ssl) available to configure the behaviour of the `https` connection.
+
+### Whitelisting remote hosts [reindex-remote-whitelist]
 
 Remote hosts have to be explicitly allowed in `elasticsearch.yml` using the `reindex.remote.whitelist` property.
-It can be set to a comma delimited list of allowed remote `host` and `port` combinations. 
+It can be set to a comma-delimited list of allowed remote `host` and `port` combinations.
 Scheme is ignored, only the host and port are used. For example:
 
 ```yaml
 reindex.remote.whitelist: [otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*"]
 ```
-The list of allowed hosts must be configured on any nodes that will coordinate the reindex.
+The list of allowed hosts must be configured on any node that will coordinate the reindex.
+
+### Compatibility [reindex-remote-compatibility]
+
 This feature should work with remote clusters of any version of {{es}} you are likely to find. This should allow you to upgrade from any version of {{es}} to the current version by reindexing from a cluster of the old version.
 ::::{warning}
 {{es}} does not support forward compatibility across major versions. For example, you cannot reindex from a 7.x cluster into a 6.x cluster.
@@ -670,16 +716,18 @@ To enable queries sent to older versions of {{es}} the `query` parameter is sent
 Reindexing from remote clusters does not support manual or automatic slicing.
 ::::
 
+### Tuning parameters [reindex-remote-tuning]
+
 Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb.
-If the remote index includes very large documents you'll need to use a smaller batch size. 
+If the remote index includes very large documents you'll need to use a smaller batch size.
 The example below sets the batch size to `10` which is very, very small.
 
 ```console
 POST _reindex
 {
   "source": {
     "remote": {
-      "host": "<OTHER_HOST_URL>:9200",
+      "host": "&lt;OTHER_HOST_URL>",
       ...
     },
     "index": "source",
@@ -709,7 +757,7 @@ POST _reindex
 {
   "source": {
     "remote": {
-      "host": "<OTHER_HOST_URL>:9200",
+      "host": "&lt;OTHER_HOST_URL>",
       ...,
       "socket_timeout": "1m",
       "connect_timeout": "10s"

server/src/main/java/org/elasticsearch/index/reindex/ReindexRequest.java

@@ -38,6 +38,7 @@
 import java.io.IOException;
 import java.net.URI;
 import java.net.URISyntaxException;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.function.Predicate;
@@ -417,6 +418,11 @@ static RemoteInfo buildRemoteInfo(Map<String, Object> source) throws IOException
         }
 
         Map<String, String> headers = extractStringStringMap(remote, "headers");
+        String apiKey = extractString(remote, "api_key");
+        if (apiKey != null) {
+            headers = headersWithApiKey(headers, apiKey);
+        }
+
         TimeValue socketTimeout = extractTimeValue(remote, "socket_timeout", RemoteInfo.DEFAULT_SOCKET_TIMEOUT);
         TimeValue connectTimeout = extractTimeValue(remote, "connect_timeout", RemoteInfo.DEFAULT_CONNECT_TIMEOUT);
         if (false == remote.isEmpty()) {
@@ -493,4 +499,18 @@ static void setMaxDocsValidateIdentical(AbstractBulkByScrollRequest<?> request,
             request.setMaxDocs(maxDocs);
         }
     }
+
+    /**
+     * Returns a headers map with the {@code Authorization} key set to the value {@code "ApiKey <apiKey>"}. If the original map is a
+     * {@link HashMap}, it is mutated; if not (e.g. it is {@link java.util.Collections#EMPTY_MAP}), it is copied. If the headers already
+     * include an {@code Authorization} key, an {@link IllegalArgumentException} is thrown.
+     */
+    private static Map<String, String> headersWithApiKey(Map<String, String> original, String apiKey) {
+        if (original.keySet().stream().anyMatch(key -> key.equalsIgnoreCase("Authorization"))) {
+            throw new IllegalArgumentException("Cannot specify both [api_key] and [headers] including [Authorization] key");
+        }
+        Map<String, String> updated = (original instanceof HashMap) ? original : new HashMap<>(original);
+        updated.put("Authorization", "ApiKey " + apiKey);
+        return updated;
+    }
 }

server/src/test/java/org/elasticsearch/index/reindex/ReindexRequestTests.java

@@ -325,6 +325,45 @@ public void testBuildRemoteInfoWithAllHostParts() throws IOException {
         assertEquals("[host] must be of the form [scheme]://[host]:[port](/[pathPrefix])? but was [https]", exception.getMessage());
     }
 
+    public void testBuildRemoteInfoWithApiKey() throws IOException {
+        Map<String, Object> remote = new HashMap<>();
+        remote.put("host", "https://example.com:9200");
+        remote.put("api_key", "l3t-m3-1n");
+        Map<String, Object> source = new HashMap<>();
+        source.put("remote", remote);
+        RemoteInfo remoteInfo = ReindexRequest.buildRemoteInfo(source);
+        assertEquals(remoteInfo.getHeaders(), Map.of("Authorization", "ApiKey l3t-m3-1n"));
+    }
+
+    public void testBuildRemoteInfoWithApiKeyAndOtherHeaders() throws IOException {
+        Map<String, Object> originalHeaders = new HashMap<>();
+        originalHeaders.put("X-Routing-Magic", "Abracadabra");
+        originalHeaders.put("X-Tracing-Magic", "12345");
+        Map<String, Object> remote = new HashMap<>();
+        remote.put("host", "https://example.com:9200");
+        remote.put("api_key", "l3t-m3-1n");
+        remote.put("headers", originalHeaders);
+        Map<String, Object> source = new HashMap<>();
+        source.put("remote", remote);
+        RemoteInfo remoteInfo = ReindexRequest.buildRemoteInfo(source);
+        assertEquals(
+            remoteInfo.getHeaders(),
+            Map.of("X-Routing-Magic", "Abracadabra", "X-Tracing-Magic", "12345", "Authorization", "ApiKey l3t-m3-1n")
+        );
+    }
+
+    public void testBuildRemoteInfoWithConflictingApiKeyAndAuthorizationHeader() throws IOException {
+        Map<String, Object> originalHeaders = new HashMap<>();
+        originalHeaders.put("aUtHoRiZaTiOn", "op3n-s3s4m3"); // non-standard capitalization, but HTTP headers are not case-sensitive
+        Map<String, Object> remote = new HashMap<>();
+        remote.put("host", "https://example.com:9200");
+        remote.put("api_key", "l3t-m3-1n");
+        remote.put("headers", originalHeaders);
+        Map<String, Object> source = new HashMap<>();
+        source.put("remote", remote);
+        assertThrows(IllegalArgumentException.class, () -> ReindexRequest.buildRemoteInfo(source));
+    }
+
     public void testReindexFromRemoteRequestParsing() throws IOException {
         BytesReference request;
         try (XContentBuilder b = JsonXContent.contentBuilder()) {