From d5f2a0a4b7b19b4def021d1fa363faa0ff5c1f78 Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Wed, 16 Apr 2025 10:44:29 +0200 Subject: [PATCH 1/3] [Add] Add integration syncing troubleshooting section to remote ES output doc --- .../fleet/remote-elasticsearch-output.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index cc22d9955b..b119393c31 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -94,3 +94,82 @@ Remote clusters require access to the [{{package-registry}}](/reference/fleet/in 3. Run the API request. 4. Copy the encoded value of the generated API key. 5. Back in the main cluster, paste the value you copied into the **Remote Kibana API Key** field. + +4. Choose whether uninstalled integrations should also be uninstalled on the remote cluster. + +### Troubleshooting + +When integration syncing is enabled for a remote {{es}} output, the current sync status is reported in **{{fleet}} Settings** in the **Outputs** table. To see a detailed breakdown of the integration syncing status, click on the **Integration syncing** status badge. + +You can also use the API to view the list of synced integrations with their sync status: + +1. In the main cluster, go to **{{fleet}} Settings** and edit the remote {{es}} output to check. + +2. Copy the output ID from the address in your browser. + +3. Go to **Management > Dev Tools**. + +4. Run the following query using the copied output ID: + ```sh + GET kbn:/api/fleet/remote_synced_integrations//remote_status + ``` + This should return the list of synced integrations with their sync status. + +::::{note} +Syncing can take up to 5 minutes after an integration is installed, updated or removed on the main cluster. +:::: + + +#### Integration syncing status failure + +If integration syncing reports connection errors or fails to report the syncing status, take the following steps to verify your setup. + +1. In the remote cluster, check the integration sync status using the API: + + 1. Go to **Management > Dev Tools**. + 2. Run the following query: + ```sh + GET kbn:/api/fleet/remote_synced_integrations/status + ``` + This should return the list of synced integrations with their sync status. + +2. If the above query returns an error, verify your setup on the target cluster: + + 1. Go to **Stack Management > Remote Clusters**. + 2. Check that the main cluster is connected as a remote cluster. + 4. Go to **Stack Management > Cross-Cluster Replication**. + 3. Check that {{ccr}} using the main cluster as remote is correctly set up. In particular, check that the name of the follower index `fleet-synced-integrations-ccr-` contains the name of the remote {{es}} output on the main cluster. + +3. Then verify your setup in the main cluster: + + 1. In {{fleet}}, open the **Settings** tab. + 1. Check that the remote {{es}} output is healthy. + 2. Edit the remote {{es}} output and check that the value of the **Remote Kibana URL** field is correct. Note that an incorrect value will not cause the output to become unhealthy but will affect integration syncing. + +#### Integrations are not installed on the remote cluster + +1. In the main cluster, look for errors in the integration syncing status of the remote {{es}} output in {{fleet}} **Settings** or using the API as described [above](#troubleshooting). + +2. Check the contents of the leader index: + + 1. Go to **Management > Dev Tools**. + 2. Run the following query + ```sh + GET fleet-synced-integrations/_search + ``` + The response payload should include the list of integrations with their install status. + +3. In the remote cluster, check the contents of the follower index: + + 1. Go to **Management > Dev Tools** + 2. Run the following query: + ```sh + GET fleet-synced-integrations-ccr-remote1/_search + ``` + The response should match the the contents of the leader index on the main cluster. + +4. If there is a mismatch between the leader and follower index, wait up to 5 minutes for the next sync to be completed in each cluster. You can check this by inspecting {{kib}} logs and looking for the line: `[SyncIntegrationsTask] runTask ended: success`. + +#### Uninstalled integrations are not uninstalled on the remote cluster + +This can happen when the integration cannot be uninstalled on the remote cluster, for instance if it has integration policies assigned to agent policies. To inspect the reason why an integration failed to be uninstalled in the remote cluster, review the integration syncing status of the remote {{es}} output in {{fleet}} **Settings** or using the API as described [above](#troubleshooting). From e369c4580d7dd4a23bd3aacc0ae292411f4211f6 Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Wed, 16 Apr 2025 14:22:58 +0200 Subject: [PATCH 2/3] Feedback --- reference/fleet/remote-elasticsearch-output.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index b119393c31..6e3f1d9c29 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -133,18 +133,18 @@ If integration syncing reports connection errors or fails to report the syncing ``` This should return the list of synced integrations with their sync status. -2. If the above query returns an error, verify your setup on the target cluster: +2. If the above query returns an error, verify your setup on the remote cluster: 1. Go to **Stack Management > Remote Clusters**. 2. Check that the main cluster is connected as a remote cluster. 4. Go to **Stack Management > Cross-Cluster Replication**. - 3. Check that {{ccr}} using the main cluster as remote is correctly set up. In particular, check that the name of the follower index `fleet-synced-integrations-ccr-` contains the name of the remote {{es}} output on the main cluster. + 3. Check that {{ccr}} using the main cluster as remote is correctly set up and active. In particular, check that the name of the follower index `fleet-synced-integrations-ccr-` contains the name of the remote {{es}} output on the main cluster. 3. Then verify your setup in the main cluster: 1. In {{fleet}}, open the **Settings** tab. - 1. Check that the remote {{es}} output is healthy. - 2. Edit the remote {{es}} output and check that the value of the **Remote Kibana URL** field is correct. Note that an incorrect value will not cause the output to become unhealthy but will affect integration syncing. + 1. Check that the remote {{es}} output is healthy. In particular, check that the remote host URL matches one of the Elasticsearch hosts on the remote cluster. + 2. Edit the remote {{es}} output and check that the remote {{kib}} URL is correct, as well as the validity and privileges of the remote {{kib}} API key. Note that an incorrect value in either of these fields will not cause the output to become unhealthy, but will affect integration syncing. #### Integrations are not installed on the remote cluster @@ -164,7 +164,7 @@ If integration syncing reports connection errors or fails to report the syncing 1. Go to **Management > Dev Tools** 2. Run the following query: ```sh - GET fleet-synced-integrations-ccr-remote1/_search + GET fleet-synced-integrations-ccr-/_search ``` The response should match the the contents of the leader index on the main cluster. From 0dafa1065b1da26b1cd4fad11feee99dd876c152 Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Thu, 17 Apr 2025 10:09:03 +0200 Subject: [PATCH 3/3] Feedback --- reference/fleet/remote-elasticsearch-output.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index 6e3f1d9c29..3694da8857 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -116,7 +116,7 @@ You can also use the API to view the list of synced integrations with their sync This should return the list of synced integrations with their sync status. ::::{note} -Syncing can take up to 5 minutes after an integration is installed, updated or removed on the main cluster. +Syncing can take up to five minutes after an integration is installed, updated, or removed on the main cluster. :::: @@ -140,15 +140,15 @@ If integration syncing reports connection errors or fails to report the syncing 4. Go to **Stack Management > Cross-Cluster Replication**. 3. Check that {{ccr}} using the main cluster as remote is correctly set up and active. In particular, check that the name of the follower index `fleet-synced-integrations-ccr-` contains the name of the remote {{es}} output on the main cluster. -3. Then verify your setup in the main cluster: +3. Verify your setup in the main cluster: 1. In {{fleet}}, open the **Settings** tab. - 1. Check that the remote {{es}} output is healthy. In particular, check that the remote host URL matches one of the Elasticsearch hosts on the remote cluster. + 1. Check that the remote {{es}} output is healthy. In particular, check that the remote host URL matches one of the {{es}} hosts on the remote cluster. 2. Edit the remote {{es}} output and check that the remote {{kib}} URL is correct, as well as the validity and privileges of the remote {{kib}} API key. Note that an incorrect value in either of these fields will not cause the output to become unhealthy, but will affect integration syncing. #### Integrations are not installed on the remote cluster -1. In the main cluster, look for errors in the integration syncing status of the remote {{es}} output in {{fleet}} **Settings** or using the API as described [above](#troubleshooting). +1. In the main cluster, look for errors in the integration syncing status of the remote {{es}} output in {{fleet}} **Settings** or using the API as described [previously](#troubleshooting). 2. Check the contents of the leader index: @@ -168,7 +168,7 @@ If integration syncing reports connection errors or fails to report the syncing ``` The response should match the the contents of the leader index on the main cluster. -4. If there is a mismatch between the leader and follower index, wait up to 5 minutes for the next sync to be completed in each cluster. You can check this by inspecting {{kib}} logs and looking for the line: `[SyncIntegrationsTask] runTask ended: success`. +4. If there is a mismatch between the leader and follower index, wait up to five minutes for the next sync to be completed in each cluster. You can check this by inspecting {{kib}} logs and looking for the line: `[SyncIntegrationsTask] runTask ended: success`. #### Uninstalled integrations are not uninstalled on the remote cluster