diff --git a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md index 8889eb8b90..9fe2730afa 100644 --- a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md +++ b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md @@ -72,7 +72,7 @@ $$$ec-restrictions-apis-kibana$$$ * {{es}} plugins, are not enabled by default for security purposes. Reach out to support if you would like to enable {{es}} plugins support on your account. * Some {{es}} plugins do not apply to {{ecloud}}. For example, you won’t ever need to change discovery, as {{ecloud}} handles how nodes discover one another. % * In {{es}} 5.0 and later, site plugins are no longer supported. This change does not affect the site plugins {{ecloud}} might provide out of the box, such as Kopf or Head, since these site plugins are serviced by our proxies and not {{es}} itself. -% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) if you want more detailed information or perform index management actions. +% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/data-store/index-basics.md#manage-indices) if you want more detailed information or perform index management actions. ## Watcher [ec-restrictions-watcher] diff --git a/explore-analyze/alerts-cases/watcher/watcher-ui.md b/explore-analyze/alerts-cases/watcher/watcher-ui.md index 946b421d47..c4d119a280 100644 --- a/explore-analyze/alerts-cases/watcher/watcher-ui.md +++ b/explore-analyze/alerts-cases/watcher/watcher-ui.md @@ -30,7 +30,7 @@ To use Watcher in {{kib}}, you must have the built-in `kibana_admin` role and ei To manage roles, go to the **Roles** management page, or use the [role APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-roles). Watches are shared between all users with the same role. ::::{note} -If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) for detailed information. +If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](/manage-data/data-store/index-basics.md#index-management) for detailed information. :::: ## Create a threshold alert [watcher-create-threshold-alert] diff --git a/manage-data/data-store/data-streams/logs-data-stream.md b/manage-data/data-store/data-streams/logs-data-stream.md index bbcf830934..69ac770914 100644 --- a/manage-data/data-store/data-streams/logs-data-stream.md +++ b/manage-data/data-store/data-streams/logs-data-stream.md @@ -48,7 +48,7 @@ PUT _index_template/my-index-template After the index template is created, new indices that use the template will be configured as a logs data stream. You can start indexing data and [using the data stream](use-data-stream.md). -You can also set the index mode and adjust other template settings in [the Elastic UI](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md). +You can also set the index mode and adjust other template settings in [the Elastic UI](/manage-data/data-store/index-basics.md#index-management-manage-index-templates). ## Synthetic source [logsdb-synthetic-source] diff --git a/manage-data/data-store/index-basics.md b/manage-data/data-store/index-basics.md index 9f7aa41a9c..47eb567d84 100644 --- a/manage-data/data-store/index-basics.md +++ b/manage-data/data-store/index-basics.md @@ -68,23 +68,75 @@ Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how ## Index management -Elastic's index management features are an easy, convenient way to manage your cluster's indices, data streams, index templates, and enrich policies. Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. +Elastic's **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](/manage-data/lifecycle/data-stream.md), [templates](/manage-data/data-store/templates.md), and [enrich policies](/manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. -Go to **{{project-settings}} → {{manage-app}} → {{index-manage-app}}**. +To use these features: + * On {{serverless-full}}, go to **Project settings** > **Management** > **Index Management**. + * On {{stack}}, go to **Stack Management** > **Index Management**. + +### Required permissions [index-mgm-req-permissions] +```{applies_to} +stack: ga +``` + +If you use {{es}} {{security-features}}, the following [security privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) are required: + +* The `monitor` cluster privilege to access {{kib}}'s **Index Management** features. +* The `view_index_metadata` and `manage` index privileges to view a data stream or index’s data. +* The `manage_index_templates` cluster privilege to manage index templates. + +To add these privileges, go to **Stack Management > Security > Roles** or use the [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role). ### Manage indices Investigate your indices and perform operations from the **Indices** view. +:::::{tab-set} +:group: stack-serverless + +::::{tab-item} {{serverless-short}} +:sync: serverless + :::{image} /manage-data/images/serverless-index-management-indices.png :alt: Index Management indices :screenshot: ::: +* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md). + * To show details and perform operations, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). + +* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). + +* To drill down into the index overivew, [mappings](/manage-data/data-store/mapping.md), and [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. + +:::: + +::::{tab-item} {{stack}} +:sync: stack + +:::{image} /manage-data/images/elasticsearch-reference-management_index_labels.png +:alt: Index Management UI +:screenshot: +::: + +* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md). + +* To show details and perform operations such as close, forcemerge, and flush, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). + * To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). -* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. -* To create new indices, use the **Create index** wizard. + + You can also use the drop-down menus to filter the list by the [index lifecycle](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md) status or phase. + +* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), statistics, and lifecycle details, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. + + :::{image} /manage-data/images/elasticsearch-reference-management_index_details.png + :alt: Index Management UI + :screenshot: + ::: + +:::: +::::: ### Manage data streams @@ -101,8 +153,8 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind * To view information about the stream's backing indices, click the number in the **Indices** column. * A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. -* To modify the data retention value, select an index, open the **Manage** menu, and click **Edit data retention**. -* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream's name. From this view, you can navigate to **Discover** to further explore data within the data stream. +* To modify the data retention value, select an index, open the **Manage** menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is not managed by an ILM policy. +* To view more information about a data stream including it's lifecycle settings, click the stream's name. ### Manage index templates [index-management-manage-index-templates] @@ -120,6 +172,123 @@ Create, edit, clone, and delete your index templates in the **Index Templates** * Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases. * To create new index templates, use the **Create template** wizard. +#### Try it: Create an index template [_try_it_create_an_index_template] + +In this tutorial, you’ll create an index template and use it to configure two new indices. + +##### Step 1. Add a name and index pattern + +1. In the **Index Templates** view, open the **Create template** wizard. + + :::{image} /manage-data/images/elasticsearch-reference-management_index_create_wizard.png + :alt: Create wizard + :screenshot: + ::: + +2. In the **Name** field, enter `my-index-template`. +3. Set **Index pattern** to `my-index-*` so the template matches any index with that index pattern. +4. Leave **Data Stream**, **Priority**, **Version**, and **_meta field** blank or as-is. + +##### Step 2. Add settings, mappings, and aliases + +When creating an index template, you can define settings, mappings, and aliases directly in the template or include them through one or more component templates. + +A [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) is a type of [template](/manage-data/data-store/templates.md) used as a building block for constructing index templates. {{kib}} displays badges indicating whether a component template contains mappings (**M**), index settings (**S**), aliases (**A**), or a combination of the three. + +1. Add component templates to your index template. + + Component templates are optional. For this tutorial, do not add any component templates. + + :::{image} /manage-data/images/elasticsearch-reference-management_index_component_template.png + :alt: Component templates page + :screenshot: + ::: + +2. Define index settings directly in the index template. When used in conjunction with component templates, settings defined directly in the index template override any conflicting settings from the associated component templates. + + This step is optional. For this tutorial, leave this section blank. +3. Define mappings directly in the index template. When used in conjunction with component templates, these mappings override any conflicting definitions from the associated component templates. + + Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: + + :::{image} /manage-data/images/elasticsearch-reference-management-index-templates-mappings.png + :alt: Mapped fields page + :screenshot: + ::: + + Alternatively, you can click the **Load JSON** link and define the mapping as JSON: + + ```js + { + "properties": { + "geo": { + "properties": { + "coordinates": { + "type": "geo_point" + } + } + } + } + } + ``` + + You can create additional mapping configurations in the **Dynamic templates** and **Advanced options** tabs. For this tutorial, do not create any additional mappings. + +4. Define an alias named `my-index`: + + ```js + { + "my-index": {} + } + ``` + +5. On the review page, check the summary. If everything looks right, click **Create template**. + +##### Step 3. Create new indices + +You’re now ready to create new indices using your index template. + +1. Index the following documents to create two indices: `my-index-000001` and `my-index-000002`. + + ```console + POST /my-index-000001/_doc + { + "@timestamp": "2019-05-18T15:57:27.541Z", + "ip": "225.44.217.191", + "extension": "jpg", + "response": "200", + "geo": { + "coordinates": { + "lat": 38.53146222, + "lon": -121.7864906 + } + }, + "url": "https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/charles-fullerton.jpg" + } + + POST /my-index-000002/_doc + { + "@timestamp": "2019-05-20T03:44:20.844Z", + "ip": "198.247.165.49", + "extension": "php", + "response": "200", + "geo": { + "coordinates": { + "lat": 37.13189556, + "lon": -76.4929875 + } + }, + "memory": 241720, + "url": "https://theacademyofperformingartsandscience.org/people/type:astronauts/name:laurel-b-clark/profile" + } + ``` + +2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier. + + ```console + GET /my-index-000001,my-index-000002 + ``` + ### Manage component templates [Component templates](/manage-data/data-store/templates.md) are reusable building blocks that configure mappings, settings, and aliases. @@ -136,9 +305,15 @@ Create, edit, clone, and delete your component templates in the **Component Temp ### Manage enrich policies -An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add the right enrich data to the right incoming documents. +An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add data from your existing indices to incoming documents during ingest. An enrich policy contains: -Add data from your existing indices to incoming documents using the **Enrich Policies** view. +* The policy type that determines how the policy matches the enrich data to incoming documents +* The source indices that store enrich data as documents +* The fields from the source indices used to match incoming documents +* The enrich fields containing enrich data from the source indices that you want to add to incoming documents +* An optional [query](elasticsearch://reference/query-languages/query-dsl/query-dsl-match-all-query.md). + +Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest. :::{image} /manage-data/images/serverless-management-enrich-policies.png :alt: Enrich policies @@ -149,6 +324,14 @@ Add data from your existing indices to incoming documents using the **Enrich Pol * To perform operations, click the policy name or use the buttons in the **Actions** column. * To create new policies, use the **Create enrich policy** wizard. -You must execute a new enrich policy before you can use it with an enrich processor. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. +You must execute a new enrich policy before you can use it with an enrich processor or {{esql}} query. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. + +When creating an enrich policy, the UI walks you through the configuration setup and selecting the fields. Before you can use the policy with an enrich processor or {{esql}} query, you must execute the policy. + +When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. +Check out these examples: +* [Example: Enrich your data based on geolocation](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md) +* [Example: Enrich your data based on exact values](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md) +* [Example: Enrich your data by matching a value to a range](/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md) \ No newline at end of file diff --git a/manage-data/data-store/templates.md b/manage-data/data-store/templates.md index fba3657786..04a4f26b15 100644 --- a/manage-data/data-store/templates.md +++ b/manage-data/data-store/templates.md @@ -55,7 +55,7 @@ If you use {{fleet}} or {{agent}}, assign your index templates a priority lower ## Create index template [create-index-templates] -Use the [index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) and [put component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) APIs to create and update index templates. You can also [manage index templates](../lifecycle/index-lifecycle-management/index-management-in-kibana.md) from Stack Management in {{kib}}. +Use the [put index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) and [put component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) APIs to create and update index templates. You can also [manage index templates](/manage-data/data-store/index-basics.md#index-management) from Stack Management in {{kib}}. The following requests create two component templates. diff --git a/manage-data/images/elasticsearch-reference-management_index_details.png b/manage-data/images/elasticsearch-reference-management_index_details.png index a975b9952c..1829510599 100644 Binary files a/manage-data/images/elasticsearch-reference-management_index_details.png and b/manage-data/images/elasticsearch-reference-management_index_details.png differ diff --git a/manage-data/images/elasticsearch-reference-management_index_labels.png b/manage-data/images/elasticsearch-reference-management_index_labels.png index caec6d5afc..3e35308f76 100644 Binary files a/manage-data/images/elasticsearch-reference-management_index_labels.png and b/manage-data/images/elasticsearch-reference-management_index_labels.png differ diff --git a/manage-data/images/serverless-index-management-indices.png b/manage-data/images/serverless-index-management-indices.png index 1079f71bca..9adc65a8da 100644 Binary files a/manage-data/images/serverless-index-management-indices.png and b/manage-data/images/serverless-index-management-indices.png differ diff --git a/manage-data/ingest/transform-enrich/ingest-pipelines.md b/manage-data/ingest/transform-enrich/ingest-pipelines.md index 4bd43706aa..d5c1c45b5d 100644 --- a/manage-data/ingest/transform-enrich/ingest-pipelines.md +++ b/manage-data/ingest/transform-enrich/ingest-pipelines.md @@ -270,7 +270,7 @@ $$$pipeline-custom-logs-index-template$$$ 2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. - You can create this template using {{kib}}'s [**Index Management**](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). + You can create this template using {{kib}}'s [**Index Management**](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). For example, the following request creates a template matching `logs-my_app-*`. The template uses a component template that contains the `index.default_pipeline` index setting. diff --git a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md index 89f961a8bf..cd9e147b30 100644 --- a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md +++ b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md @@ -50,7 +50,7 @@ You also can set up [{{beats}}](beats://reference/index.md), such as a [{{filebe ## Create an enrich policy [create-enrich-policy] -After adding enrich data to your source indices, use the [create enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-put-policy) or [Index Management in {{kib}}](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-enrich-policies) to create an enrich policy. +After adding enrich data to your source indices, use the [create enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-put-policy) or [Index Management in {{kib}}](/manage-data/data-store/index-basics.md#manage-enrich-policies) to create an enrich policy. ::::{warning} Once created, you can’t update or change an enrich policy. See [Update an enrich policy](#update-enrich-policies). @@ -61,7 +61,7 @@ Once created, you can’t update or change an enrich policy. See [Update an enri ## Execute the enrich policy [execute-enrich-policy] -Once the enrich policy is created, you need to execute it using the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) or [Index Management in {{kib}}](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-enrich-policies) to create an [enrich index](data-enrichment.md#enrich-index). +Once the enrich policy is created, you need to execute it using the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) or [Index Management in {{kib}}](/manage-data/data-store/index-basics.md#manage-enrich-policies) to create an [enrich index](data-enrichment.md#enrich-index). :::{image} /manage-data/images/elasticsearch-reference-enrich-policy-index.svg :alt: enrich policy index @@ -123,7 +123,7 @@ Once created, you can’t update or change an enrich policy. Instead, you can: 1. Create and [execute](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) a new enrich policy. 2. Replace the previous enrich policy with the new enrich policy in any in-use enrich processors or {{esql}} queries. -3. Use the [delete enrich policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-delete-policy) API or [Index Management in {{kib}}](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-enrich-policies) to delete the previous enrich policy. +3. Use the [delete enrich policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-delete-policy) API or [Index Management in {{kib}}](/manage-data/data-store/index-basics.md#manage-enrich-policies) to delete the previous enrich policy. ## Enrich components [ingest-enrich-components] diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index 1f859da486..c465a6f7e0 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -65,7 +65,7 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elastics ## Create and manage {{ilm-init}} policies -You can create and manage index lifecycle policies through [{{kib}} Management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to: +You can create and manage index lifecycle policies through {{kib}}'s [Index Management](/manage-data/data-store/index-basics.md#index-management) UI or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to: * [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) * [Update a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) diff --git a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md index a78209d4ac..d2198b6a9c 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md @@ -9,204 +9,4 @@ products: # Index management in Kibana [index-mgmt] -{{kib}}'s **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](../../data-store/data-streams.md), [index templates](../../data-store/templates.md), and [enrich policies](../../ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. - -To use these features, go to **Stack Management** > **Index Management**. - - -## Required permissions [index-mgm-req-permissions] - -If you use {{es}} {{security-features}}, the following [security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) are required: - -* The `monitor` cluster privilege to access {{kib}}'s **Index Management** features. -* The `view_index_metadata` and `manage` index privileges to view a data stream or index’s data. -* The `manage_index_templates` cluster privilege to manage index templates. - -To add these privileges, go to **Stack Management > Security > Roles** or use the [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role). - - -## Manage indices [view-edit-indices] - -Investigate your indices and perform operations from the **Indices** view. - -:::{image} /manage-data/images/elasticsearch-reference-management_index_labels.png -:alt: Index Management UI -:screenshot: -::: - -* To show details and perform operations such as close, forcemerge, and flush, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). -* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). -* To drill down into the index [mappings](../../data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. - - :::{image} /manage-data/images/elasticsearch-reference-management_index_details.png - :alt: Index Management UI - :screenshot: - ::: - - - -## Manage data streams [manage-data-streams] - -Investigate your data streams and address lifecycle management needs in the **Data Streams** view. - -The value in the **Indices** column indicates the number of backing indices. Click this number to drill down into details. - -A value in the data retention column indicates that the data stream is managed by a [data stream lifecycle policy](../data-stream.md). This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by Elasticsearch at a later time. - -:::{image} /manage-data/images/elasticsearch-reference-management-data-stream-fields.png -:alt: Data stream details -:screenshot: -::: - -* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream’s name. From this view, you can navigate to **Discover** to further explore data within the data stream. -* [preview]To edit the data retention value, open the **Manage** menu, and then click **Edit data retention**. This action is only available if your data stream is not managed by an ILM policy. - - -## Manage index templates [manage-index-templates] - -Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices. - -:::{image} /manage-data/images/elasticsearch-reference-management-index-templates.png -:alt: Index templates -:screenshot: -::: - - -### Try it: Create an index template [_try_it_create_an_index_template] - -In this tutorial, you’ll create an index template and use it to configure two new indices. - -**Step 1. Add a name and index pattern** - -1. In the **Index Templates** view, open the **Create template** wizard. - - :::{image} /manage-data/images/elasticsearch-reference-management_index_create_wizard.png - :alt: Create wizard - :screenshot: - ::: - -2. In the **Name** field, enter `my-index-template`. -3. Set **Index pattern** to `my-index-*` so the template matches any index with that index pattern. -4. Leave **Data Stream**, **Priority**, **Version**, and **_meta field** blank or as-is. - -**Step 2. Add settings, mappings, and aliases** - -1. Add [component templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) to your index template. - - Component templates are pre-configured sets of mappings, index settings, and aliases you can reuse across multiple index templates. Badges indicate whether a component template contains mappings (**M**), index settings (**S**), aliases (**A**), or a combination of the three. - - Component templates are optional. For this tutorial, do not add any component templates. - - :::{image} /manage-data/images/elasticsearch-reference-management_index_component_template.png - :alt: Component templates page - :screenshot: - ::: - -2. Define index settings. These are optional. For this tutorial, leave this section blank. -3. Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: - - :::{image} /manage-data/images/elasticsearch-reference-management-index-templates-mappings.png - :alt: Mapped fields page - :screenshot: - ::: - - Alternatively, you can click the **Load JSON** link and define the mapping as JSON: - - ```js - { - "properties": { - "geo": { - "properties": { - "coordinates": { - "type": "geo_point" - } - } - } - } - } - ``` - - You can create additional mapping configurations in the **Dynamic templates** and **Advanced options** tabs. For this tutorial, do not create any additional mappings. - -4. Define an alias named `my-index`: - - ```js - { - "my-index": {} - } - ``` - -5. On the review page, check the summary. If everything looks right, click **Create template**. - -**Step 3. Create new indices** - -You’re now ready to create new indices using your index template. - -1. Index the following documents to create two indices: `my-index-000001` and `my-index-000002`. - - ```console - POST /my-index-000001/_doc - { - "@timestamp": "2019-05-18T15:57:27.541Z", - "ip": "225.44.217.191", - "extension": "jpg", - "response": "200", - "geo": { - "coordinates": { - "lat": 38.53146222, - "lon": -121.7864906 - } - }, - "url": "https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/charles-fullerton.jpg" - } - - POST /my-index-000002/_doc - { - "@timestamp": "2019-05-20T03:44:20.844Z", - "ip": "198.247.165.49", - "extension": "php", - "response": "200", - "geo": { - "coordinates": { - "lat": 37.13189556, - "lon": -76.4929875 - } - }, - "memory": 241720, - "url": "https://theacademyofperformingartsandscience.org/people/type:astronauts/name:laurel-b-clark/profile" - } - ``` - -2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier. - - ```console - GET /my-index-000001,my-index-000002 - ``` - - - -## Manage enrich policies [manage-enrich-policies] - -Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest. An enrich policy contains: - -* The policy type that determines how the policy matches the enrich data to incoming documents -* The source indices that store enrich data as documents -* The fields from the source indices used to match incoming documents -* The enrich fields containing enrich data from the source indices that you want to add to incoming documents -* An optional [query](elasticsearch://reference/query-languages/query-dsl/query-dsl-match-all-query.md). - -:::{image} /manage-data/images/elasticsearch-reference-management-enrich-policies.png -:alt: Enrich policies -:screenshot: -::: - -When creating an enrich policy, the UI walks you through the configuration setup and selecting the fields. Before you can use the policy with an enrich processor or {{esql}} query, you must execute the policy. - -When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. - -Check out these examples: - -* [Example: Enrich your data based on geolocation](../../ingest/transform-enrich/example-enrich-data-based-on-geolocation.md) -* [Example: Enrich your data based on exact values](../../ingest/transform-enrich/example-enrich-data-based-on-exact-values.md) -* [Example: Enrich your data by matching a value to a range](../../ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md) - +This page has moved. Refer to [Index management](/manage-data/data-store/index-basics.md#index-management) for guidance on managing indices, data streams, index templates, component templates, and enrich policies in {{kib}}. \ No newline at end of file diff --git a/manage-data/toc.yml b/manage-data/toc.yml index e4eb23c0f5..41e0cd82d9 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -126,7 +126,7 @@ toc: - file: lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md - file: lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md - file: lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md - - file: lifecycle/index-lifecycle-management/index-management-in-kibana.md + - hidden: lifecycle/index-lifecycle-management/index-management-in-kibana.md - file: lifecycle/index-lifecycle-management/migrate-index-management.md children: - file: lifecycle/index-lifecycle-management/manage-existing-indices.md diff --git a/redirects.yml b/redirects.yml index f8aa702bfe..deed1417a6 100644 --- a/redirects.yml +++ b/redirects.yml @@ -212,3 +212,6 @@ redirects: 'solutions/observability/get-started/get-started-with-dashboards.md': 'solutions/observability/get-started.md' # Related to https://github.com/elastic/docs-content/pull/1329 'manage-data/ingest/transform-enrich/ingest-pipelines-serverless.md': 'manage-data/ingest/transform-enrich/ingest-pipelines.md' + +# Related to https://github.com/elastic/docs-content/pull/2010 + 'manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md': 'manage-data/data-store/index-basics.md' \ No newline at end of file diff --git a/solutions/observability/apm/explore-data-in-elasticsearch.md b/solutions/observability/apm/explore-data-in-elasticsearch.md index dacb889fc2..9bf884faba 100644 --- a/solutions/observability/apm/explore-data-in-elasticsearch.md +++ b/solutions/observability/apm/explore-data-in-elasticsearch.md @@ -18,7 +18,7 @@ Elastic APM data is stored in [Data streams](/solutions/observability/apm/data-s The following examples enable you to interact with {{es}}'s REST API. One possible way to do this is using {{kib}}'s [{{dev-tools-app}} console](/explore-analyze/query-filter/tools/console.md). -Data streams, templates, and index-level operations can also be manged via {{kib}}'s [Index management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) panel. +Data streams, templates, and index-level operations can also be manged via {{kib}}'s [Index management](/manage-data/data-store/index-basics.md#index-management) UI. To see an overview of existing data streams, run: diff --git a/solutions/observability/apm/reduce-storage.md b/solutions/observability/apm/reduce-storage.md index 22dc84b700..1d1944296a 100644 --- a/solutions/observability/apm/reduce-storage.md +++ b/solutions/observability/apm/reduce-storage.md @@ -75,7 +75,7 @@ POST /.ds-*-apm*/_delete_by_query ### Delete data with {{kib}} Index Management [apm-delete-data-in-kibana] -{{kib}}'s [Index Management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) allows you to manage your cluster’s indices, data streams, index templates, and much more. +{{kib}}'s [Index management](/manage-data/data-store/index-basics.md#index-management) UI allows you to manage your cluster’s indices, data streams, index templates, and much more. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Select **Data Streams**. Select the data streams you want to delete, and click **Delete data streams**. diff --git a/solutions/observability/synthetics/manage-data-retention.md b/solutions/observability/synthetics/manage-data-retention.md index 0e4d2f3628..0ebbfb5d5f 100644 --- a/solutions/observability/synthetics/manage-data-retention.md +++ b/solutions/observability/synthetics/manage-data-retention.md @@ -43,7 +43,7 @@ To find Synthetics data streams: :::{tab-item} Elastic Stack :sync: stack -1. Navigate to [{{kib}} index management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md). +1. In {{kib}}, navigate to [Index Management](/manage-data/data-store/index-basics.md#index-management). 2. Filter the list of data streams for those containing the term `synthetics`. 1. In the UI there will be three types of browser data streams: `synthetics-browser-*`, `synthetics-browser.network-*`, and `synthetics-browser.screenshot-*`. diff --git a/troubleshoot/observability/troubleshoot-mapping-issues.md b/troubleshoot/observability/troubleshoot-mapping-issues.md index 75dc52c5ee..872f2bf922 100644 --- a/troubleshoot/observability/troubleshoot-mapping-issues.md +++ b/troubleshoot/observability/troubleshoot-mapping-issues.md @@ -27,7 +27,7 @@ It is necessary to stop all {{heartbeat}}/{{elastic-agent}} instances that are t To ensure the mapping is applied to all {{heartbeat}} data going forward, delete all the {{heartbeat}} indices that match the pattern the {{uptime-app}} will use. -There are multiple ways to achieve this. You can read about performing this using the [Index Management UI](../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) or with the [Delete index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete). +There are multiple ways to achieve this. You can read about performing this using the [Index Management UI](/manage-data/data-store/index-basics.md#index-management) or with the [Delete index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete). If using {{elastic-agent}} you will want to fix any issues with custom data stream mappings. We encourage the use of {{fleet}} to eliminate this issue.