You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Repeated snapshot failures are usually an indicator of a problem with your deployment. Continuous failures of automated snapshots can leave a deployment without recovery options in cases of data loss or outages.
19
14
20
-
Elasticsearch keeps track of the number of repeated failures when executing automated snapshots. If an automated snapshot fails too many times without a successful execution, the health API will report a warning. The number of repeated failures before reporting a warning is controlled by the [`slm.health.failed_snapshot_warn_threshold`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-health-failed-snapshot-warn-threshold) setting.
In the event that an automated {{slm}} policy execution is experiencing repeated failures, follow these steps to get more information about the problem:
18
+
{{es}} keeps track of the number of repeated failures when executing automated snapshots with [{{slm}} ({{slm-init}})](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) policies. If an automated snapshot fails too many times without a successful execution, the health API reports a warning. The number of repeated failures before reporting a warning is controlled by the [`slm.health.failed_snapshot_warn_threshold`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-health-failed-snapshot-warn-threshold) setting.
23
19
24
-
:::::::{tab-set}
20
+
## Review snapshot policy failures
25
21
26
-
::::::{tab-item} {{ech}}
27
-
In order to check the status of failing {{slm}} policies we need to go to Kibana and retrieve the [Snapshot Lifecycle Policy information](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle).
22
+
If an automated {{slm-init}} policy execution is experiencing repeated failures, follow these steps to get more information about the problem:
28
23
29
-
**Use {{kib}}**
30
-
31
-
1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body).
32
-
2. On the **Hosted deployments** panel, click the name of your deployment.
24
+
:::::::{tab-set}
33
25
34
-
::::{note}
35
-
If the name of your deployment is disabled your {{kib}} instances might be unhealthy, in which case contact [Elastic Support](https://support.elastic.co). If your deployment doesn’t include {{kib}}, all you need to do is [enable it first](../../deploy-manage/deploy/elastic-cloud/access-kibana.md).
36
-
::::
26
+
::::::{tab-item} Using {{kib}}
27
+
In {{kib}}, you can view all configured {{slm-init}} policies and review their status and execution history. If the UI does not provide sufficient details about the failure, use the Console to retrieve the [snapshot policy information](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) with the {{es}} API.
37
28
38
-
3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to **Dev Tools > Console**.
29
+
1. Go to **Snapshot and Restore > Policies** to see the list of configured policies. You can find the **Snapshot and Restore** management page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
"details" : """{"type":"snapshot_exception","reason":"[daily-snap-2099.06.16-ywe-kgh5rfqfrpnchvsujq] failed to create snapshot successfully, 5 out of 149 total shards failed"}""" <4>
83
-
},
84
-
"stats": {
85
-
"policy": "daily-snapshots",
86
-
"snapshots_taken": 0,
87
-
"snapshots_failed": 0,
88
-
"snapshots_deleted": 0,
89
-
"snapshot_deletion_failures": 0
90
-
},
91
-
"next_execution": "2099-06-17T01:30:00.000Z",
92
-
"next_execution_millis": 4085343000000
93
-
}
94
-
}
95
-
```
96
-
97
-
1. The affected snapshot lifecycle policy.
98
-
2. The information about the last failure for the policy.
99
-
3. The time when the failure occurred in millis. Use the `human=true` request parameter to see a formatted timestamp.
100
-
4. Error details containing the reason for the snapshot failure.
101
-
102
-
103
-
Snapshots can fail for a variety reasons. If the failures are due to configuration errors, consult the documentation for the repository that the automated snapshots are using. Refer to the [guide on managing repositories in ECE](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md) if you are using such a deployment.
104
-
36
+
2. The policies table lists all configured policies. Click on any of the policies to review the details and execution history.
105
37
106
-
One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem.
38
+
3. To get more detailed information about the failure, open {{kib}} **Dev Tools > Console**. You can find the **Console** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
107
39
108
-
In the event that snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information.
40
+
Once the Console is open, execute the steps described in the **Using the {{es}} API** tab to retrieve the affected {{slm-init}} policy information.
109
41
::::::
110
42
111
-
::::::{tab-item} Self-managed
112
-
[Retrieve](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) the {{slm}} policy:
43
+
::::::{tab-item} Using the {{es}} API
44
+
The following step can be run using either [{{kib}} console](/explore-analyze/query-filter/tools/console.md) or direct [{{es}} API](elasticsearch://reference/elasticsearch/rest-apis/index.md) calls.
45
+
46
+
[Retrieve the affected {{slm-init}} policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle):
113
47
114
48
```console
115
49
GET _slm/policy/<affected-policy-name>
116
50
```
117
51
118
-
The response will look like this:
52
+
The response looks like this:
119
53
120
54
```console-result
121
55
{
@@ -166,15 +100,14 @@ The response will look like this:
166
100
3. The time when the failure occurred in millis. Use the `human=true` request parameter to see a formatted timestamp.
167
101
4. Error details containing the reason for the snapshot failure.
168
102
103
+
::::::
169
104
170
-
Snapshots can fail for a variety reasons. If the failures are due to configuration errors, consult the documentation for the repository that the automated snapshots are using.
105
+
:::::::
171
106
172
-
One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem.
107
+
## Possible causes
173
108
174
-
In the event that snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information.
175
-
::::::
109
+
Snapshots can fail for a variety of reasons. If the failures are due to configuration errors, consult the documentation for the repository type that the snapshot policy is using. Refer to the [guide on managing repositories in ECE](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md) if you are using an Elastic Cloud Enterprise deployment.
176
110
177
-
:::::::
111
+
One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem.
0 commit comments