Combine 'Index mgmt' with 'Index mgmt in Kibana' (#2010)

As part of the ILM review it was pointed out that the very important
[Index management in
Kibana](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana)
page is kind of hard to find, since it's nested a few levels down in the
"Index Lifecycle Management" section. It's also confusing that we have a
few pages about managing indices.

This PR combines the contents of "Index management in Kibana" into
[Index
Management](https://www.elastic.co/docs/manage-data/data-store/index-basics#index-management)
(a section of the main, very prominent "Index basics" page at the top of
the ToC).

**Notes:**

- For the removed [Index management in
Kibana](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana)
page, I've changed the page to "hidden" in the ToC, replaced the content
with a `This page has moved` note, and created a redirect in the
`redirect.yml` file.
- The original pages differed in that one seemed mainly for Stack, one
for both Serverless and Stack. In combining them I've used tabs to
separate out the steps that differ.
- The "Index management in Kibana" page had a nice tutorial about
setting up index templates. I've tested to make sure that it works on
Serverless.
- We also have a [Manage existing
indices](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices)
page. In a separate PR I've moved that to be nested under "Migrate to
ILM" so it's clear that the page is not really about managing indices in
general.
- I've updated outdated screencaps and also changed them to show the
"Include hidden indices" option enabled, since this came up a few times
in the ILM review.

See: 
* Updated [Index
management](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2010/manage-data/data-store/index-basics#index-management)
* Removed [Index management in
Kibana](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2010/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana)

Closes: #1569

---------

Co-authored-by: Edu González de la Herrán <[email protected]>
Co-authored-by: Liam Thompson <[email protected]>

Author: 3 people

deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md

@@ -72,7 +72,7 @@ $$$ec-restrictions-apis-kibana$$$
 * {{es}} plugins, are not enabled by default for security purposes. Reach out to support if you would like to enable {{es}} plugins support on your account.
 * Some {{es}} plugins do not apply to {{ecloud}}. For example, you won’t ever need to change discovery, as {{ecloud}} handles how nodes discover one another.
 % * In {{es}} 5.0 and later, site plugins are no longer supported. This change does not affect the site plugins {{ecloud}} might provide out of the box, such as Kopf or Head, since these site plugins are serviced by our proxies and not {{es}} itself.
-% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) if you want more detailed information or perform index management actions.
+% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/data-store/index-basics.md#manage-indices) if you want more detailed information or perform index management actions.
 
 
 ## Watcher [ec-restrictions-watcher]

explore-analyze/alerts-cases/watcher/watcher-ui.md

@@ -30,7 +30,7 @@ To use Watcher in {{kib}}, you must have the built-in `kibana_admin` role and ei
 To manage roles, go to the **Roles** management page, or use the [role APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-roles). Watches are shared between all users with the same role.
 
 ::::{note}
-If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) for detailed information.
+If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](/manage-data/data-store/index-basics.md#index-management) for detailed information.
 ::::
 
 ## Create a threshold alert [watcher-create-threshold-alert]

manage-data/data-store/data-streams/logs-data-stream.md

@@ -48,7 +48,7 @@ PUT _index_template/my-index-template
 
 After the index template is created, new indices that use the template will be configured as a logs data stream. You can start indexing data and [using the data stream](use-data-stream.md).
 
-You can also set the index mode and adjust other template settings in [the Elastic UI](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md).
+You can also set the index mode and adjust other template settings in [the Elastic UI](/manage-data/data-store/index-basics.md#index-management-manage-index-templates).
 
 
 ## Synthetic source [logsdb-synthetic-source]

manage-data/data-store/index-basics.md

@@ -68,23 +68,75 @@ Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how
 
 ## Index management
 
-Elastic's index management features are an easy, convenient way to manage your cluster's indices, data streams, index templates, and enrich policies. Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible.
+Elastic's **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](/manage-data/lifecycle/data-stream.md), [templates](/manage-data/data-store/templates.md), and [enrich policies](/manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible.
 
-Go to **{{project-settings}} → {{manage-app}} → {{index-manage-app}}**.
+To use these features:
+ * On {{serverless-full}}, go to **Project settings** > **Management** > **Index Management**.
+ * On {{stack}}, go to **Stack Management** > **Index Management**.
+
+### Required permissions [index-mgm-req-permissions]
+```{applies_to}
+stack: ga
+```
+
+If you use {{es}} {{security-features}}, the following [security privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) are required:
+
+* The `monitor` cluster privilege to access {{kib}}'s **Index Management** features.
+* The `view_index_metadata` and `manage` index privileges to view a data stream or index’s data.
+* The `manage_index_templates` cluster privilege to manage index templates.
+
+To add these privileges, go to **Stack Management > Security > Roles** or use the [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role).
 
 ### Manage indices
 
 Investigate your indices and perform operations from the **Indices** view.
 
+:::::{tab-set}
+:group: stack-serverless
+
+::::{tab-item} {{serverless-short}}
+:sync: serverless
+
 :::{image} /manage-data/images/serverless-index-management-indices.png
 :alt: Index Management indices
 :screenshot:
 :::
 
+* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md).
+
 * To show details and perform operations, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices).
+
+* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze).
+
+* To drill down into the index overivew, [mappings](/manage-data/data-store/mapping.md), and [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index.
+
+::::
+
+::::{tab-item} {{stack}}
+:sync: stack
+
+:::{image} /manage-data/images/elasticsearch-reference-management_index_labels.png
+:alt: Index Management UI
+:screenshot:
+:::
+
+* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md).
+
+* To show details and perform operations such as close, forcemerge, and flush, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices).
+
 * To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze).
-* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index.
-* To create new indices, use the **Create index** wizard.
+
+   You can also use the drop-down menus to filter the list by the [index lifecycle](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md) status or phase.
+
+* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), statistics, and lifecycle details, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index.
+
+    :::{image} /manage-data/images/elasticsearch-reference-management_index_details.png
+    :alt: Index Management UI
+    :screenshot:
+    :::
+
+::::
+:::::
 
 ### Manage data streams
 
@@ -101,8 +153,8 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind
 
 * To view information about the stream's backing indices, click the number in the **Indices** column.
 * A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time.
-* To modify the data retention value, select an index, open the **Manage**  menu, and click **Edit data retention**.
-* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream's name. From this view, you can navigate to **Discover** to further explore data within the data stream.
+* To modify the data retention value, select an index, open the **Manage**  menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is not managed by an ILM policy.
+* To view more information about a data stream including it's lifecycle settings, click the stream's name.
 
 ### Manage index templates [index-management-manage-index-templates]
 
@@ -120,6 +172,123 @@ Create, edit, clone, and delete your index templates in the **Index Templates**
 * Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases.
 * To create new index templates, use the **Create template** wizard.
 
+#### Try it: Create an index template [_try_it_create_an_index_template]
+
+In this tutorial, you’ll create an index template and use it to configure two new indices.
+
+##### Step 1. Add a name and index pattern
+
+1. In the **Index Templates** view, open the **Create template** wizard.
+
+    :::{image} /manage-data/images/elasticsearch-reference-management_index_create_wizard.png
+    :alt: Create wizard
+    :screenshot:
+    :::
+
+2. In the **Name** field, enter `my-index-template`.
+3. Set **Index pattern** to `my-index-*` so the template matches any index with that index pattern.
+4. Leave **Data Stream**, **Priority**, **Version**, and **_meta field** blank or as-is.
+
+##### Step 2. Add settings, mappings, and aliases
+
+When creating an index template, you can define settings, mappings, and aliases directly in the template or include them through one or more component templates.
+
+A [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) is a type of [template](/manage-data/data-store/templates.md) used as a building block for constructing index templates. {{kib}} displays badges indicating whether a component template contains mappings (**M**), index settings (**S**), aliases (**A**), or a combination of the three.
+
+1. Add component templates to your index template.
+
+    Component templates are optional. For this tutorial, do not add any component templates.
+
+    :::{image} /manage-data/images/elasticsearch-reference-management_index_component_template.png
+    :alt: Component templates page
+    :screenshot:
+    :::
+
+2. Define index settings directly in the index template. When used in conjunction with component templates, settings defined directly in the index template override any conflicting settings from the associated component templates.
+
+    This step is optional. For this tutorial, leave this section blank.
+3. Define mappings directly in the index template. When used in conjunction with component templates, these mappings override any conflicting definitions from the associated component templates.
+
+    Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`:
+
+    :::{image} /manage-data/images/elasticsearch-reference-management-index-templates-mappings.png
+    :alt: Mapped fields page
+    :screenshot:
+    :::
+
+    Alternatively, you can click the **Load JSON** link and define the mapping as JSON:
+
+    ```js
+    {
+      "properties": {
+        "geo": {
+          "properties": {
+            "coordinates": {
+              "type": "geo_point"
+            }
+          }
+        }
+      }
+    }
+    ```
+
+    You can create additional mapping configurations in the **Dynamic templates** and **Advanced options** tabs. For this tutorial, do not create any additional mappings.
+
+4. Define an alias named `my-index`:
+
+    ```js
+    {
+      "my-index": {}
+    }
+    ```
+
+5. On the review page, check the summary. If everything looks right, click **Create template**.
+
+##### Step 3. Create new indices
+
+You’re now ready to create new indices using your index template.
+
+1. Index the following documents to create two indices: `my-index-000001` and `my-index-000002`.
+
+    ```console
+    POST /my-index-000001/_doc
+    {
+      "@timestamp": "2019-05-18T15:57:27.541Z",
+      "ip": "225.44.217.191",
+      "extension": "jpg",
+      "response": "200",
+      "geo": {
+        "coordinates": {
+          "lat": 38.53146222,
+          "lon": -121.7864906
+        }
+      },
+      "url": "https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/charles-fullerton.jpg"
+    }
+
+    POST /my-index-000002/_doc
+    {
+      "@timestamp": "2019-05-20T03:44:20.844Z",
+      "ip": "198.247.165.49",
+      "extension": "php",
+      "response": "200",
+      "geo": {
+        "coordinates": {
+          "lat": 37.13189556,
+          "lon": -76.4929875
+        }
+      },
+      "memory": 241720,
+      "url": "https://theacademyofperformingartsandscience.org/people/type:astronauts/name:laurel-b-clark/profile"
+    }
+    ```
+
+2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier.
+
+    ```console
+    GET /my-index-000001,my-index-000002
+    ```
+
 ### Manage component templates
 
 [Component templates](/manage-data/data-store/templates.md) are reusable building blocks that configure mappings, settings, and aliases.
@@ -136,9 +305,15 @@ Create, edit, clone, and delete your component templates in the **Component Temp
 
 ### Manage enrich policies
 
-An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add the right enrich data to the right incoming documents.
+An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add data from your existing indices to incoming documents during ingest. An enrich policy contains:
 
-Add data from your existing indices to incoming documents using the **Enrich Policies** view.
+* The policy type that determines how the policy matches the enrich data to incoming documents
+* The source indices that store enrich data as documents
+* The fields from the source indices used to match incoming documents
+* The enrich fields containing enrich data from the source indices that you want to add to incoming documents
+* An optional [query](elasticsearch://reference/query-languages/query-dsl/query-dsl-match-all-query.md).
+
+Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest.
 
 :::{image} /manage-data/images/serverless-management-enrich-policies.png
 :alt: Enrich policies
@@ -149,6 +324,14 @@ Add data from your existing indices to incoming documents using the **Enrich Pol
 * To perform operations, click the policy name or use the buttons in the **Actions** column.
 * To create new policies, use the **Create enrich policy** wizard.
 
-You must execute a new enrich policy before you can use it with an enrich processor. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents.
+You must execute a new enrich policy before you can use it with an enrich processor or {{esql}} query. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents.
+
+When creating an enrich policy, the UI walks you through the configuration setup and selecting the fields. Before you can use the policy with an enrich processor or {{esql}} query, you must execute the policy.
+
+When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents.
 
+Check out these examples:
 
+* [Example: Enrich your data based on geolocation](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md)
+* [Example: Enrich your data based on exact values](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md)
+* [Example: Enrich your data by matching a value to a range](/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md)

manage-data/data-store/templates.md

@@ -55,7 +55,7 @@ If you use {{fleet}} or {{agent}}, assign your index templates a priority lower
 
 ## Create index template [create-index-templates]
 
-Use the [index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) and [put component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) APIs to create and update index templates. You can also [manage index templates](../lifecycle/index-lifecycle-management/index-management-in-kibana.md) from Stack Management in {{kib}}.
+Use the [put index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) and [put component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) APIs to create and update index templates. You can also [manage index templates](/manage-data/data-store/index-basics.md#index-management) from Stack Management in {{kib}}.
 
 The following requests create two component templates.
 

manage-data/images/elasticsearch-reference-management_index_details.png

@@ -270,7 +270,7 @@ $$$pipeline-custom-logs-index-template$$$
 
 2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs-<dataset-name>-*`.
 
-    You can create this template using {{kib}}'s [**Index Management**](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template).
+    You can create this template using {{kib}}'s [**Index Management**](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template).
 
     For example, the following request creates a template matching `logs-my_app-*`. The template uses a component template that contains the `index.default_pipeline` index setting.