-
Notifications
You must be signed in to change notification settings - Fork 159
[Fleet] Add documentation for remote ES output integrations syncing #952
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jillguyonnet
merged 6 commits into
elastic:main
from
jillguyonnet:fleet-add-integrations-sync-doc
Mar 31, 2025
Merged
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
2475eb0
[Fleet] Add documentation for remote ES output integrations syncing
jillguyonnet e56e4a3
Feedback
jillguyonnet 47929c4
Fix reference links
jillguyonnet b2a4a2d
Remove remote ES output min version
jillguyonnet 1ff83ec
Feedback
jillguyonnet f935672
Merge branch 'main' into fleet-add-integrations-sync-doc
jillguyonnet File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -5,7 +5,7 @@ mapped_pages: | |||||
|
||||||
# Remote Elasticsearch output [remote-elasticsearch-output] | ||||||
|
||||||
Beginning in version 8.12.0, you can send {{agent}} data to a remote {{es}} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {{fleet}} to manage the agents. | ||||||
Remote {{es}} outputs allow you to send {{agent}} data to a remote {{es}} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {{fleet}} to manage the agents. | ||||||
|
||||||
A remote {{es}} cluster supports the same [output settings](/reference/fleet/es-output-settings.md) as your main {{es}} cluster. | ||||||
|
||||||
|
@@ -18,6 +18,7 @@ A bug has been found that causes {{elastic-defend}} response actions to stop wor | |||||
Using a remote {{es}} output with a target cluster that has [traffic filters](/deploy-manage/security/traffic-filtering.md) enabled is not currently supported. | ||||||
:::: | ||||||
|
||||||
## Configuration | ||||||
|
||||||
To configure a remote {{es}} cluster for your {{agent}} data: | ||||||
|
||||||
|
@@ -33,7 +34,7 @@ To configure a remote {{es}} cluster for your {{agent}} data: | |||||
5. Create a service token to access the remote cluster. | ||||||
|
||||||
1. Below the **Service Token** field, copy the API request. | ||||||
2. In the remote cluster, open the {{kib}} menu and go to **Management → Dev Tools**. | ||||||
2. In the remote cluster, open the {{kib}} menu and go to **Management > Dev Tools**. | ||||||
3. Run the API request. | ||||||
4. Copy the value for the generated token. | ||||||
5. Back in your main cluster, paste the value you copied into the output **Service Token** field. | ||||||
|
@@ -42,12 +43,13 @@ To configure a remote {{es}} cluster for your {{agent}} data: | |||||
To prevent unauthorized access the {{es}} Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires {{fleet-server}} version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See [Preconfiguration settings](kibana://reference/configuration-reference/fleet-settings.md#_preconfiguration_settings_for_advanced_use_cases) in the {{kib}} Guide to learn more. | ||||||
:::: | ||||||
|
||||||
6. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agent}}s use this output to send data if no other output is set in the [agent policy](/reference/fleet/agent-policy.md). | ||||||
7. Select which [performance tuning settings](/reference/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting. | ||||||
8. Add any [advanced YAML configuration settings](/reference/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output. | ||||||
9. Click **Save and apply settings**. | ||||||
6. Choose whether integrations should automatically be synchronized on the remote {{es}} cluster. Refer to [Automatic integrations synchronization](#automatic-integrations-synchronization) below to configure this feature. | ||||||
7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agents}} use this output to send data if no other output is set in the [agent policy](/reference/fleet/agent-policy.md). | ||||||
8. Select which [performance tuning settings](/reference/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting. | ||||||
9. Add any [advanced YAML configuration settings](/reference/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output. | ||||||
10. Click **Save and apply settings**. | ||||||
|
||||||
After the output is created, you can update an {{agent}} policy to use the new remote {{es}} cluster: | ||||||
After the output is created, you can update an {{agent}} policy to use the new output and send data to the remote {{es}} cluster: | ||||||
|
||||||
1. In {{fleet}}, open the **Agent policies** tab. | ||||||
2. Click the agent policy to edit it, then click **Settings**. | ||||||
|
@@ -57,4 +59,37 @@ After the output is created, you can update an {{agent}} policy to use the new r | |||||
|
||||||
The remote {{es}} cluster is now configured. | ||||||
|
||||||
As a final step before using the remote {{es}} output, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/fleet/install-uninstall-integration-assets.md) for the steps. | ||||||
If you have chosen not to automatically synchronize integrations, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/fleet/install-uninstall-integration-assets.md) for the steps. | ||||||
|
||||||
## Automatic integrations synchronization | ||||||
|
||||||
```{applies_to} | ||||||
stack: ga 9.1 | ||||||
``` | ||||||
|
||||||
When enabled, this feature keeps integrations synced between your main {{es}} cluster and remote {{es}} clusters. | ||||||
|
||||||
### Requirements | ||||||
|
||||||
This feature requires setting up [{{ccr}}](/deploy-manage/tools/cross-cluster-replication.md), which is available to Platinum and Enterprise [subscriptions](https://www.elastic.co/subscriptions). Remote clusters must be running the same version of {{es}} as the main cluster or a newer version that is compatible with {{ccr}}. | ||||||
|
||||||
Remote clusters require access to the [{{package-registry}}](/reference/fleet/index.md#package-registry-intro) to install integrations. | ||||||
|
||||||
### Configuration | ||||||
|
||||||
1. Configure {{ccr}} on the remote cluster. | ||||||
|
||||||
1. In the remote cluster, open the {{kib}} menu and go to **Stack Management > Remote Clusters**. | ||||||
2. Refer to [Remote clusters](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters.html) to add your main cluster (where the remote {{es}} output is configured) as a remote cluster. | ||||||
3. Go to **Stack Management > Cross-Cluster Replication**. | ||||||
4. Create a follower index named `fleet-synced-integrations-ccr-<output name>` that replicates the `fleet-synced-integrations` leader index on the main cluster. | ||||||
|
||||||
2. In the main cluster, in the **Remote Kibana URL** field, add the Kibana URL of the remote cluster. | ||||||
|
||||||
3. Create an API key to access Kibana on the remote cluster. | ||||||
|
||||||
1. Below the **Remote Kibana API Key** field, copy the API request. | ||||||
2. In the remote cluster, open the {{kib}} menu and go to **Management > Dev Tools**. | ||||||
3. Run the API request. | ||||||
4. Copy the encoded value of the generated API key. | ||||||
5. Back in the main cluster, paste the value you copied into the output **Remote Kibana API Key** field. | ||||||
|
5. Back in the main cluster, paste the value you copied into the output **Remote Kibana API Key** field. | |
5. Back in the main cluster, paste the value you copied into the **Remote Kibana API Key** field. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could add one more step to
Resume replication
after adding the follower index.