Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 44 additions & 8 deletions reference/fleet/remote-elasticsearch-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ mapped_pages:

# Remote Elasticsearch output [remote-elasticsearch-output]

Beginning in version 8.12.0, you can send {{agent}} data to a remote {{es}} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {{fleet}} to manage the agents.
Remote {{es}} outputs allow you to send {{agent}} data to a remote {{es}} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {{fleet}} to manage the agents.

A remote {{es}} cluster supports the same [output settings](/reference/fleet/es-output-settings.md) as your main {{es}} cluster.

Expand All @@ -18,6 +18,7 @@ A bug has been found that causes {{elastic-defend}} response actions to stop wor
Using a remote {{es}} output with a target cluster that has [traffic filters](/deploy-manage/security/traffic-filtering.md) enabled is not currently supported.
::::

## Configuration

To configure a remote {{es}} cluster for your {{agent}} data:

Expand All @@ -33,7 +34,7 @@ To configure a remote {{es}} cluster for your {{agent}} data:
5. Create a service token to access the remote cluster.

1. Below the **Service Token** field, copy the API request.
2. In the remote cluster, open the {{kib}} menu and go to **Management Dev Tools**.
2. In the remote cluster, open the {{kib}} menu and go to **Management > Dev Tools**.
3. Run the API request.
4. Copy the value for the generated token.
5. Back in your main cluster, paste the value you copied into the output **Service Token** field.
Expand All @@ -42,12 +43,13 @@ To configure a remote {{es}} cluster for your {{agent}} data:
To prevent unauthorized access the {{es}} Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires {{fleet-server}} version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See [Preconfiguration settings](kibana://reference/configuration-reference/fleet-settings.md#_preconfiguration_settings_for_advanced_use_cases) in the {{kib}} Guide to learn more.
::::

6. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agent}}s use this output to send data if no other output is set in the [agent policy](/reference/fleet/agent-policy.md).
7. Select which [performance tuning settings](/reference/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting.
8. Add any [advanced YAML configuration settings](/reference/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output.
9. Click **Save and apply settings**.
6. Choose whether integrations should automatically be synchronized on the remote {{es}} cluster. Refer to [Automatic integrations synchronization](#automatic-integrations-synchronization) below to configure this feature.
7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agents}} use this output to send data if no other output is set in the [agent policy](/reference/fleet/agent-policy.md).
8. Select which [performance tuning settings](/reference/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting.
9. Add any [advanced YAML configuration settings](/reference/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output.
10. Click **Save and apply settings**.

After the output is created, you can update an {{agent}} policy to use the new remote {{es}} cluster:
After the output is created, you can update an {{agent}} policy to use the new output and send data to the remote {{es}} cluster:

1. In {{fleet}}, open the **Agent policies** tab.
2. Click the agent policy to edit it, then click **Settings**.
Expand All @@ -57,4 +59,38 @@ After the output is created, you can update an {{agent}} policy to use the new r

The remote {{es}} cluster is now configured.

As a final step before using the remote {{es}} output, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/fleet/install-uninstall-integration-assets.md) for the steps.
If you have chosen not to automatically synchronize integrations, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/fleet/install-uninstall-integration-assets.md) for the steps.

## Automatic integrations synchronization

```{applies_to}
stack: ga 9.1
```

When enabled, this feature keeps integrations synced between your main {{es}} cluster and remote {{es}} clusters.

### Requirements

This feature requires setting up [{{ccr}}](/deploy-manage/tools/cross-cluster-replication.md), which is available to Platinum and Enterprise [subscriptions](https://www.elastic.co/subscriptions). Remote clusters must be running the same version of {{es}} as the main cluster or a newer version that is compatible with {{ccr}}.

Remote clusters require access to the [{{package-registry}}](/reference/fleet/index.md#package-registry-intro) to install integrations.

### Configuration

1. Configure {{ccr}} on the remote cluster.

1. In the remote cluster, open the {{kib}} menu and go to **Stack Management > Remote Clusters**.
2. Refer to [Remote clusters](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters.html) to add your main cluster (where the remote {{es}} output is configured) as a remote cluster.
3. Go to **Stack Management > Cross-Cluster Replication**.
4. Create a follower index named `fleet-synced-integrations-ccr-<output name>` that replicates the `fleet-synced-integrations` leader index on the main cluster.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add one more step to Resume replication after adding the follower index.

5. Resume replication once the follower index is created.

2. In the main cluster, in the **Remote Kibana URL** field, add the Kibana URL of the remote cluster.

3. Create an API key to access Kibana on the remote cluster.

1. Below the **Remote Kibana API Key** field, copy the API request.
2. In the remote cluster, open the {{kib}} menu and go to **Management > Dev Tools**.
3. Run the API request.
4. Copy the encoded value of the generated API key.
5. Back in the main cluster, paste the value you copied into the **Remote Kibana API Key** field.
Loading