diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index cc22d9955b..3694da8857 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -94,3 +94,82 @@ Remote clusters require access to the [{{package-registry}}](/reference/fleet/in 3. Run the API request. 4. Copy the encoded value of the generated API key. 5. Back in the main cluster, paste the value you copied into the **Remote Kibana API Key** field. + +4. Choose whether uninstalled integrations should also be uninstalled on the remote cluster. + +### Troubleshooting + +When integration syncing is enabled for a remote {{es}} output, the current sync status is reported in **{{fleet}} Settings** in the **Outputs** table. To see a detailed breakdown of the integration syncing status, click on the **Integration syncing** status badge. + +You can also use the API to view the list of synced integrations with their sync status: + +1. In the main cluster, go to **{{fleet}} Settings** and edit the remote {{es}} output to check. + +2. Copy the output ID from the address in your browser. + +3. Go to **Management > Dev Tools**. + +4. Run the following query using the copied output ID: + ```sh + GET kbn:/api/fleet/remote_synced_integrations//remote_status + ``` + This should return the list of synced integrations with their sync status. + +::::{note} +Syncing can take up to five minutes after an integration is installed, updated, or removed on the main cluster. +:::: + + +#### Integration syncing status failure + +If integration syncing reports connection errors or fails to report the syncing status, take the following steps to verify your setup. + +1. In the remote cluster, check the integration sync status using the API: + + 1. Go to **Management > Dev Tools**. + 2. Run the following query: + ```sh + GET kbn:/api/fleet/remote_synced_integrations/status + ``` + This should return the list of synced integrations with their sync status. + +2. If the above query returns an error, verify your setup on the remote cluster: + + 1. Go to **Stack Management > Remote Clusters**. + 2. Check that the main cluster is connected as a remote cluster. + 4. Go to **Stack Management > Cross-Cluster Replication**. + 3. Check that {{ccr}} using the main cluster as remote is correctly set up and active. In particular, check that the name of the follower index `fleet-synced-integrations-ccr-` contains the name of the remote {{es}} output on the main cluster. + +3. Verify your setup in the main cluster: + + 1. In {{fleet}}, open the **Settings** tab. + 1. Check that the remote {{es}} output is healthy. In particular, check that the remote host URL matches one of the {{es}} hosts on the remote cluster. + 2. Edit the remote {{es}} output and check that the remote {{kib}} URL is correct, as well as the validity and privileges of the remote {{kib}} API key. Note that an incorrect value in either of these fields will not cause the output to become unhealthy, but will affect integration syncing. + +#### Integrations are not installed on the remote cluster + +1. In the main cluster, look for errors in the integration syncing status of the remote {{es}} output in {{fleet}} **Settings** or using the API as described [previously](#troubleshooting). + +2. Check the contents of the leader index: + + 1. Go to **Management > Dev Tools**. + 2. Run the following query + ```sh + GET fleet-synced-integrations/_search + ``` + The response payload should include the list of integrations with their install status. + +3. In the remote cluster, check the contents of the follower index: + + 1. Go to **Management > Dev Tools** + 2. Run the following query: + ```sh + GET fleet-synced-integrations-ccr-/_search + ``` + The response should match the the contents of the leader index on the main cluster. + +4. If there is a mismatch between the leader and follower index, wait up to five minutes for the next sync to be completed in each cluster. You can check this by inspecting {{kib}} logs and looking for the line: `[SyncIntegrationsTask] runTask ended: success`. + +#### Uninstalled integrations are not uninstalled on the remote cluster + +This can happen when the integration cannot be uninstalled on the remote cluster, for instance if it has integration policies assigned to agent policies. To inspect the reason why an integration failed to be uninstalled in the remote cluster, review the integration syncing status of the remote {{es}} output in {{fleet}} **Settings** or using the API as described [above](#troubleshooting).