Restore feature states requires restricted permissions (#3857)

👋 howdy, team!

This
- adds in the two most common errors encountered when snapshot restoring
feature states / system indices wrong
- cross-links how to get the needed restricted permissions to restore
system feature states (and removes outdated-since-v8.0 information which
no longer works)
- adds in an example API outline of reading the feature state to its
snapshot appearance to determining which to restore

Related changed pages
- [Snapshot and restore > Feature
states](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore#feature-state)
- Restore a snapshot > [Restore a feature
state](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore/restore-snapshot#restore-feature-state)
and [Restore an entire
cluster](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore/restore-snapshot#restore-entire-cluster)
- [File-based access
recovery](https://www.elastic.co/docs/troubleshoot/elasticsearch/file-based-recovery)

---------

Co-authored-by: Vlada Chirmicci <[email protected]>

Author: stefnestor

deploy-manage/tools/snapshot-and-restore.md

@@ -105,15 +105,26 @@ Snapshots don’t contain or back up:
 
 ### Feature states [feature-state]
 
-A **feature state** contains the indices and data streams used to store configurations, history, and other data for an Elastic feature, such as **{{es}} security** or **Kibana**.
+A **feature state** contains the indices and data streams used to store configurations, history, and other data for an Elastic feature, such as **{{es}} security** or **Kibana**. To retrieve a list of feature states, use the [Features API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-features-get-features).
 
-::::{note}
-To retrieve a list of feature states, use the [Features API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-features-get-features).
-::::
+```console
+GET /_features
+```
 
 A feature state typically includes one or more [system indices or system data streams](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#system-indices). It may also include regular indices and data streams used by the feature. For example, a feature state may include a regular index that contains the feature’s execution history. Storing this history in a regular index lets you more easily search it.
 
-In {{es}} 8.0 and later versions, feature states are the only way to back up and restore system indices and system data streams.
+Starting with {{es}} 8.0 and later versions, feature states are the only way to back up and restore system indices and system data streams. Attempting to restore a system index or data stream outside its feature state is not permitted and will result in the following error:
+
+```
+requested system indices [.example], but system indices can only be restored as part of a feature state
+```
+
+Restoring system indices and data streams will require temporary elevated permissions to edit restricted indices. For more information, refer to [File-based access recovery](/troubleshoot/elasticsearch/file-based-recovery.md). Attempting to restore a system index or data stream without the required temporary elevated permissions will result in the following error:
+
+```
+Indices [.example] use and access is reserved for system operations
+```
+
 
 ## How snapshots work
 

deploy-manage/tools/snapshot-and-restore/restore-snapshot.md

@@ -165,25 +165,75 @@ You can restore a [feature state](../../../deploy-manage/tools/snapshot-and-rest
 
 If you restore a snapshot’s cluster state, the operation restores all feature states in the snapshot by default. Similarly, if you don’t restore a snapshot’s cluster state, the operation doesn’t restore any feature states by default. You can also choose to restore only specific feature states from a snapshot, regardless of the cluster state.
 
-To view a snapshot’s feature states, use the get snapshot API.
+Feature backing indices are version dependent. To see which indices are included within a snapshot's feature state, [list an applicable snapshot](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-get). For example, on {{ech}} you might poll its latest snapshot on its built-in  `cloud-snapshot-policy` SLM policy:
 
 ```console
-GET _snapshot/my_repository/my_snapshot_2099.05.06
+GET /_snapshot/_all/_all?filter_path=snapshots.feature_states&index_names=false&sort=start_time&size=1&order=desc&slm_policy_filter=cloud-snapshot-policy
 ```
 
-The response’s `feature_states` property contains a list of features in the snapshot as well as each feature’s indices.
+The response’s `feature_states` property contains a list of features in the snapshot as well as each feature’s indices. The following is an example of the output that might display for a cluster:
 
-To restore a specific feature state from the snapshot, specify the `feature_name` from the response in the restore snapshot API’s [`feature_states`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) parameter.
+```json
+{
+  "snapshots": [
+    {
+      "feature_states": [
+        {
+          "feature_name": "security",
+          "indices": [".security-tokens-7",".security-7",".security-profile-8"]
+        },
+        {
+          "feature_name": "geoip",
+          "indices": [".geoip_databases"]
+        },
+        {
+          "feature_name": "async_search",
+          "indices": [".async-search"]
+        },
+        {
+          "feature_name": "searchable_snapshots",
+          "indices": [".snapshot-blob-cache"]
+        },
+        {
+          "feature_name": "transform",
+          "indices": [".transform-internal-007"]
+        },
+        {
+          "feature_name": "inference_plugin",
+          "indices": [".secrets-inference",".inference"]
+        },
+        {
+          "feature_name": "kibana",
+          "indices": [
+            ".kibana_usage_counters_9.x.x_001",
+            ".kibana_9.x.x_001",
+            ".apm-custom-link",
+            ".kibana_search_solution_9.x.x_001",
+            ".kibana_task_manager_9.x.x_001",
+            ".apm-agent-configuration",
+            ".kibana_locks-000001",
+            ".kibana_security_session_1",
+            ".kibana_alerting_cases_9.x.x_001",
+            ".kibana_analytics_9.x.x_001",
+            ".kibana_security_solution_9.x.x_001",
+            ".kibana_ingest_9.x.x_001"
+          ]
+        }
+      ]
+    }
+  ]
+}
+```
 
-::::{note}
-When you restore a feature state, {{es}} closes and overwrites the feature’s existing indices.
-::::
+To restore a specific feature state from the snapshot, specify the `feature_name` from the response in the [restore snapshot API’s](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) `feature_states` parameter.
 
+Note that feature state names may match {{kib}} UI sections, but restoring a single feature state may not fully reset that UI. For example, the {{fleet}} UI depends on the `fleet` feature state as well as `kibana` and `security`. When restoring, it's important to include all required feature states in the `feature_states` parameter to achieve the desired reset behavior.
 
 ::::{warning}
-Restoring the `security` feature state overwrites system indices used for authentication. If you use {{ech}}, ensure you have access to the {{ech}} Console before restoring the `security` feature state. If you run {{es}} on your own hardware, [create a superuser in the file realm](../../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-create-file-realm-user) to ensure you’ll still be able to access your cluster.
+Restoring the `security` feature state overwrites system indices used for authentication. If you use {{ech}} or {{ece}}, ensure you have access to the [{{es}} API console](cloud://reference/cloud-hosted/ec-api-console.md) before restoring the `security` feature state. If you run {{es}} on your own hardware or in {{eck}}, [create a temporary user with elevated permissions to edit restricted indices in the file realm](/troubleshoot/elasticsearch/file-based-recovery.md) to ensure you’ll still be able to access your cluster.
 ::::
 
+When you restore a feature state, {{es}} closes and overwrites the feature’s existing indices and data streams. For example, to [snapshot restore](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) the `geoip` feature state, you might use:
 
 ```console
 POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
@@ -282,15 +332,7 @@ If you’re restoring to a different cluster, see [Restore to a different cluste
         }
         ```
 
-3. $$$restore-create-file-realm-user$$$If you use {{es}} security features, log in to a node host, navigate to the {{es}} installation directory, and add a user with the `superuser` role to the file realm using the [`elasticsearch-users`](elasticsearch://reference/elasticsearch/command-line-tools/users-command.md) tool.
-
-    For example, the following command creates a user named `restore_user`.
-
-    ```sh
-    ./bin/elasticsearch-users useradd restore_user -p my_password -r superuser
-    ```
-
-    Use this file realm user to authenticate requests until the restore operation is complete.
+3. $$$restore-create-file-realm-user$$$If you use {{es}} security features, follow [File-based access recovery](/troubleshoot/elasticsearch/file-based-recovery.md) to temporarily create a user with temporary elevated permissions to edit restricted indices. Use this file realm user to authenticate requests until the restore operation is complete.
 
 4. Use the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) to set [`action.destructive_requires_name`](elasticsearch://reference/elasticsearch/configuration-reference/index-management-settings.md#action-destructive-requires-name) to `false`. This lets you delete data streams and indices using wildcards.
 

troubleshoot/elasticsearch/file-based-recovery.md

@@ -236,4 +236,4 @@ You can confirm that the desired `superduperuser` role is applied to your `admin
 curl -X GET -sk -u "admin:changeme" "https://localhost:9200/_security/_authenticate?pretty=true" 
 ```
 
-Now that you have regained recovery access to the cluster, you can investigate and recover the {{stack}} {{security-features}} as needed.
+Now that you have regained recovery access to the cluster, you can investigate and recover the {{stack}} {{security-features}} as needed. For more information, refer to [Restore a feature state](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-feature-state).