diff --git a/deploy-manage/production-guidance/elasticsearch-in-production-environments.md b/deploy-manage/production-guidance/elasticsearch-in-production-environments.md index 5bb9adefb4..b7088699fe 100644 --- a/deploy-manage/production-guidance/elasticsearch-in-production-environments.md +++ b/deploy-manage/production-guidance/elasticsearch-in-production-environments.md @@ -59,7 +59,7 @@ The following topics, covered in other sections of the documentation, offer valu * Consider [data streams](/manage-data/data-store/data-streams.md) and [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md) to manage and retain your data efficiently over time. ::::{tip} - [Elastic integrations](https://www.elastic.co/integrations) provide default index lifecycle policies, and you can [build your own policies for your custom integrations](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md). + [Elastic integrations](https://www.elastic.co/integrations) provide default index lifecycle policies, and you can [build your own policies for your custom integrations](/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md). :::: ### Security and monitoring [security-and-monitoring] diff --git a/manage-data/_snippets/create-lifecycle-policy.md b/manage-data/_snippets/create-lifecycle-policy.md new file mode 100644 index 0000000000..db49fd6a3c --- /dev/null +++ b/manage-data/_snippets/create-lifecycle-policy.md @@ -0,0 +1,65 @@ +A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to five phases: `hot`, `warm`, `cold`, `frozen`, and `delete`. + +For example, you might define a policy named `timeseries_policy` that has the following two phases: + +* A `hot` phase that defines a rollover action to specify that an index rolls over when it reaches either a `max_primary_shard_size` of 50 gigabytes or a `max_age` of 30 days. +* A `delete` phase that sets `min_age` to remove the index 90 days after rollover. + +::::{note} +The `min_age` value is relative to the rollover time, not the index creation time. [Learn more](../../troubleshoot/elasticsearch/index-lifecycle-management-errors.md#min-age-calculation). +:::: + + +You can create the policy in {{kib}} or with the [create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API. + +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To create the policy from {{kib}}, open the menu and go to **Stack Management > Index Lifecycle Policies**. Click **Create policy**. + +:::{image} /manage-data/images/elasticsearch-reference-create-policy.png +:alt: Create policy page +:screenshot: +::: +::: + +:::{tab-item} API +:sync: api +Use the [Create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API to add an ILM policy to the {{es}} cluster: + +```console +PUT _ilm/policy/timeseries_policy +{ + "policy": { + "phases": { + "hot": { <1> + "actions": { + "rollover": { + "max_primary_shard_size": "50GB", <2> + "max_age": "30d" + } + } + }, + "delete": { + "min_age": "90d", <3> + "actions": { + "delete": {} <4> + } + } + } + } +} +``` + +1. The `min_age` defaults to `0ms`, so new indices enter the `hot` phase immediately. +2. Trigger the `rollover` action when either of the conditions are met. +3. Move the index into the `delete` phase 90 days after rollover. +4. Trigger the `delete` action when the index enters the delete phase. + +::: +:::: + +:::{tip} +For more details about default {{ilm-init}} policy settings, refer to [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#ilm-create-policy). +::: \ No newline at end of file diff --git a/manage-data/data-store/aliases.md b/manage-data/data-store/aliases.md index ba3d6783a9..98915a3c20 100644 --- a/manage-data/data-store/aliases.md +++ b/manage-data/data-store/aliases.md @@ -268,7 +268,7 @@ POST _aliases If an alias points to multiple indices or data streams and `is_write_index` isn’t set, the alias rejects write requests. If an index alias points to one index and `is_write_index` isn’t set, the index automatically acts as the write index. Data stream aliases don’t automatically set a write data stream, even if the alias points to one data stream. ::::{tip} -We recommend using data streams to store append-only time series data. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See [Manage time series data without data streams](../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). +We recommend using data streams to store append-only time series data. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md). :::: diff --git a/manage-data/data-store/data-streams.md b/manage-data/data-store/data-streams.md index b7e8f6c6be..98ea0e4d64 100644 --- a/manage-data/data-store/data-streams.md +++ b/manage-data/data-store/data-streams.md @@ -26,7 +26,7 @@ To determine whether you should use a data stream for your data, you should cons * You mostly perform indexing requests, with occasional updates and deletes. * You index documents without an `_id`, or when indexing documents with an explicit `_id` you expect first-write-wins behavior. -For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See documentation for [managing time series data without a data stream](../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams) for more information. +For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) for more information. Keep in mind that some features such as [Time Series Data Streams (TSDS)](../data-store/data-streams/time-series-data-stream-tsds.md) and [data stream lifecycles](../lifecycle/data-stream.md) require a data stream. @@ -106,7 +106,7 @@ Data streams are designed for use cases where existing data is rarely updated. Y If you need to update a larger number of documents in a data stream, you can use the [update by query](data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query) APIs. ::::{tip} -If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See [Manage time series data without data streams](../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). +If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md). :::: diff --git a/manage-data/data-store/data-streams/modify-data-stream.md b/manage-data/data-store/data-streams/modify-data-stream.md index 7b7e62b74a..56ebdb5622 100644 --- a/manage-data/data-store/data-streams/modify-data-stream.md +++ b/manage-data/data-store/data-streams/modify-data-stream.md @@ -344,7 +344,7 @@ Follow these steps: ``` 4. If you do not want to mix new and old data in your new data stream, pause the indexing of new documents. While mixing old and new data is safe, it could interfere with data retention. See [Mixing new and old data in a data stream](../data-streams/modify-data-stream.md#data-stream-mix-new-old-data). -5. If you use {{ilm-init}} to [automate rollover](../../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md), reduce the {{ilm-init}} poll interval. This ensures the current write index doesn’t grow too large while waiting for the rollover check. By default, {{ilm-init}} checks rollover conditions every 10 minutes. +5. If you use {{ilm-init}} to [automate rollover](../../lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md), reduce the {{ilm-init}} poll interval. This ensures the current write index doesn’t grow too large while waiting for the rollover check. By default, {{ilm-init}} checks rollover conditions every 10 minutes. The following [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) request lowers the `indices.lifecycle.poll_interval` setting to `1m` (one minute). diff --git a/manage-data/data-store/data-streams/set-up-data-stream.md b/manage-data/data-store/data-streams/set-up-data-stream.md index 2f99f10938..c17bc3b1cb 100644 --- a/manage-data/data-store/data-streams/set-up-data-stream.md +++ b/manage-data/data-store/data-streams/set-up-data-stream.md @@ -213,7 +213,7 @@ For an example, see [Data stream privileges](../../../deploy-manage/users-roles/ ## Convert an index alias to a data stream [convert-index-alias-to-data-stream] -Prior to {{es}} 7.9, you’d typically use an [index alias with a write index](../../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams) to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md). +Prior to {{es}} 7.9, you’d typically use an [index alias with a write index](../../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md). To convert an index alias with a write index to a data stream with the same name, use the [migrate to data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream). During conversion, the alias’s indices become hidden backing indices for the stream. The alias’s write index becomes the stream’s write index. The stream still requires a matching index template with data stream enabled. diff --git a/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md b/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md new file mode 100644 index 0000000000..0e0cccceb7 --- /dev/null +++ b/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md @@ -0,0 +1,26 @@ +--- +navigation_title: ILM tutorials +applies_to: + stack: ga +products: + - id: elasticsearch +--- + +# {{ilm-cap}} tutorials + +A collection of tutorials is available to help you control the lifecycle of your data using {{ilm}}. + +## Configuring rollover + +When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. This enables you to implement a [hot-warm-cold architecture](../data-tiers.md) to meet the performance requirements for your newest data, control costs over time, enforce retention policies, and still get the most out of your data. + +To simplify index management and automate rollover, select one of the scenarios that best applies to your situation: + +* **Roll over data streams with ILM.** When ingesting write-once, timestamped data that doesn't change, follow the steps in [](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md) for simple, automated data stream rollover. ILM-managed backing indices are automatically created under a single data stream alias. ILM also tracks and transitions the backing indices through the lifecycle automatically. +* **Roll over time series indices with ILM.** Data streams are best suited for [append-only](../../data-store/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use {{ilm-init}} to manage and roll over the alias’s indices. Follow the steps in [](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) for more information. +* **Roll over general content as data streams with ILM.** If some of your indices store data that isn't timestamped, but you would like to get the benefits of automatic rotation when the index reaches a certain size or age, or delete already rotated indices after a certain amount of time, follow the steps in [](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md). These steps include injecting a timestamp field during indexing time to mimic time series data. + +## Customizing a built-in {{ilm-init}} policy + +When your data streams have been set up automatically, for example when you're ingesting data by means of an [Elastic integration](https://docs.elastic.co/en/integrations), the data is typically managed using a built-in {{ilm-init}} policy. To customize the lifecycle policy for managed indices, refer to: [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md). + diff --git a/manage-data/lifecycle/index-lifecycle-management/rollover.md b/manage-data/lifecycle/index-lifecycle-management/rollover.md index 5595522740..fdd6bd5408 100644 --- a/manage-data/lifecycle/index-lifecycle-management/rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/rollover.md @@ -22,6 +22,16 @@ The rollover feature is an important part of how [index lifecycle](../index-life Rollover can be triggered via the [API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover), {{ilm-init}}, or {{dlm-init}}. +:::{tip} +The following tutorials are available to help you configure rollover for your indices, for three different scenarios: +* [](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md) +* [](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) +* [](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md) + +Refer to [](/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md) for an overview of these. + +::: + ## How rollover works in {{ilm-init}} You define a rollover action in the hot phase of an index lifecycle policy. It will run when any of the configured conditions are met and the write index contains at least one document. @@ -70,7 +80,7 @@ Each data stream requires an [index template](../../data-store/templates.md) tha * A configuration that indicates a data stream is used for the index pattern. * Optional: The mappings and settings applied to each backing index when it’s created. -For more information about this approach, refer to the [Manage time series data with data streams](../index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-with-data-streams) tutorial. +For more information about this approach, refer to the [](../index-lifecycle-management/tutorial-time-series-with-data-streams.md) tutorial. :::{tip} Data streams are designed for append-only data, where the data stream name can be used as the operations (read, write, rollover, shrink etc.) target. If your use case requires data to be updated in place, you can perform [update or delete operations directly on the backing indices](../../data-store/data-streams/use-data-stream.md#update-delete-docs-in-a-backing-index). @@ -86,7 +96,7 @@ For more information about the data stream naming pattern, refer to the [Generat ### Rotating your indices with aliases [rollover-with-aliases] - Rotating indices with aliases requires additional configuration steps, including bootstrapping the initial index. For more details about this approach, refer to the [Manage time series data without data streams](../index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams) tutorial. + Rotating indices with aliases requires additional configuration steps, including bootstrapping the initial index. For more details about this approach, refer to the [](../index-lifecycle-management/tutorial-time-series-without-data-streams.md) tutorial. :::{important} The use of aliases for rollover requires meeting certain conditions. Review these considerations before applying this approach: diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md deleted file mode 100644 index 0e56f007b3..0000000000 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md +++ /dev/null @@ -1,485 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html -applies_to: - stack: ga -products: - - id: elasticsearch ---- - -# Tutorial: Automate rollover [getting-started-index-lifecycle-management] - -When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. This enables you to implement a [hot-warm-cold architecture](../data-tiers.md) to meet the performance requirements for your newest data, control costs over time, enforce retention policies, and still get the most out of your data. - -To simplify index management and automate rollover, select one of the scenarios that best applies to your situation: - -* **Roll over data streams with ILM.** When ingesting write-once, timestamped data that doesn't change, follow the steps in [Manage time series data with data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-with-data-streams) for simple, automated data stream rollover. ILM-managed backing indices are automatically created under a single data stream alias. ILM also tracks and transitions the backing indices through the lifecycle automatically. -* **Roll over time series indices with ILM.** Data streams are best suited for [append-only](../../data-store/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use [ILM](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md) to manage and roll over the alias’s indices. Follow the steps in [Manage time series data without data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams) for more information. -* **Roll over general content as data streams with ILM.** If some of your indices store data that isn't timestamped, but you would like to get the benefits of automatic rotation when the index reaches a certain size or age, or delete already rotated indices after a certain amount of time, follow the steps in [Manage general content with data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-general-content-with-data-streams). These steps include injecting a timestamp field during indexing time to mimic time series data. - - -## Manage time series data with data streams [manage-time-series-data-with-data-streams] - -To automate rollover and management of a data stream with {{ilm-init}}, you: - -1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-policy) that defines the appropriate [phases](index-lifecycle.md) and [actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md). -2. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-apply-policy) to [create the data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-the-data-stream) and apply the ILM policy and the indices settings and mappings configurations for the backing indices. -3. [Verify indices are moving through the lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-check-progress) as expected. - -::::{important} -When you enable {{ilm}} for {{beats}} or the {{ls}} {{es}} output plugin, lifecycle policies are set up automatically. You do not need to take any other actions. You can modify the default policies through [{{kib}} Management](tutorial-customize-built-in-policies.md) or the {{ilm-init}} APIs. -:::: - - - -### Create a lifecycle policy [ilm-gs-create-policy] - -A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to five phases: `hot`, `warm`, `cold`, `frozen`, and `delete`. - -For example, you might define a policy named `timeseries_policy` that has the following two phases: - -* A `hot` phase that defines a rollover action to specify that an index rolls over when it reaches either a `max_primary_shard_size` of 50 gigabytes or a `max_age` of 30 days. -* A `delete` phase that sets `min_age` to remove the index 90 days after rollover. - -::::{note} -The `min_age` value is relative to the rollover time, not the index creation time. [Learn more](../../../troubleshoot/elasticsearch/index-lifecycle-management-errors.md#min-age-calculation). - -:::: - - -You can create the policy in {{kib}} or with the [create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API. - -::::{tab-set} -:group: kibana-api -:::{tab-item} {{kib}} -:sync: kibana -To create the policy from {{kib}}, open the menu and go to **Stack Management > Index Lifecycle Policies**. Click **Create policy**. - -:::{image} /manage-data/images/elasticsearch-reference-create-policy.png -:alt: Create policy page -:screenshot: -::: -::: - -:::{tab-item} API -:sync: api -Use the [Create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API to add an ILM policy to the {{es}} cluster: - -```console -PUT _ilm/policy/timeseries_policy -{ - "policy": { - "phases": { - "hot": { <1> - "actions": { - "rollover": { - "max_primary_shard_size": "50GB", <2> - "max_age": "30d" - } - } - }, - "delete": { - "min_age": "90d", <3> - "actions": { - "delete": {} <4> - } - } - } - } -} -``` - -1. The `min_age` defaults to `0ms`, so new indices enter the `hot` phase immediately. -2. Trigger the `rollover` action when either of the conditions are met. -3. Move the index into the `delete` phase 90 days after rollover. -4. Trigger the `delete` action when the index enters the delete phase. - -::: -:::: - -:::{tip} -For more details about default {{ilm-init}} policy settings, refer to [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#ilm-create-policy). -::: - - -### Create an index template to create the data stream and apply the lifecycle policy [ilm-gs-apply-policy] - -To set up a data stream, first create an index template to specify the lifecycle policy. Because the template is for a data stream, it must also include a `data_stream` definition. - -For example, you might create a template named `timeseries_template` and use it for a future data stream named `timeseries`. - -To enable {{ilm-init}} to manage the data stream, the template configures one {{ilm-init}} setting: - -* `index.lifecycle.name` specifies the name of the lifecycle policy that you want to apply to the data stream. - -Use the {{kib}} **Create template** wizard to add a template or the [Create or update index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API to add a template and apply the lifecycle policy to indices matching the template. - -::::{tab-set} -:group: kibana-api -:::{tab-item} {{kib}} -:sync: kibana -To add an index template to a cluster using the wizard, go to **Stack Management > Index Management**. In the **Index Templates** tab, click **Create template**. - -![Create template page](/manage-data/images/elasticsearch-reference-create-index-template.png "") - -This wizard invokes the create or update index template API to create the index template with the options you specify. - -:::{tip} -To learn about which index template options you can specify, refer to [Create an index template to apply the lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#apply-policy-template). -::: -::: - -:::{tab-item} API -:sync: api -Use the API to add an index template to your cluster: - -```console -PUT _index_template/timeseries_template -{ - "index_patterns": ["timeseries"], <1> - "data_stream": { }, - "template": { - "settings": { - "number_of_shards": 1, - "number_of_replicas": 1, - "index.lifecycle.name": "timeseries_policy" <2> - } - } -} -``` - -1. Apply the template when a document is indexed into the `timeseries` target. -2. The name of the {{ilm-init}} policy used to manage the data stream. - -::: -:::: - - - -### Create the data stream [ilm-gs-create-the-data-stream] - -To get things started, index a document into the name or wildcard pattern defined in the `index_patterns` of the [index template](../../data-store/templates.md). As long as an existing data stream, index, or index alias does not already use the name, the index request automatically creates a corresponding data stream with a single backing index. {{es}} automatically indexes the request’s documents into this backing index, which also acts as the stream’s [write index](../../data-store/data-streams.md#data-stream-write-index). - -For example, the following request creates the `timeseries` data stream and the first generation backing index called `.ds-timeseries-2099.03.08-000001`. - -```console -POST timeseries/_doc -{ - "message": "logged the request", - "@timestamp": "1591890611" -} -``` - -When a rollover condition in the lifecycle policy is met, the `rollover` action: - -* Creates the second generation backing index, named `.ds-timeseries-2099.03.08-000002`. Because it is a backing index of the `timeseries` data stream, the configuration from the `timeseries_template` index template is applied to the new index. -* As it is the latest generation index of the `timeseries` data stream, the newly created backing index `.ds-timeseries-2099.03.08-000002` becomes the data stream’s write index. - -This process repeats each time a rollover condition is met. You can search across all of the data stream’s backing indices, managed by the `timeseries_policy`, with the `timeseries` data stream name. Write operations should be sent to the data stream name, which will route them to its current write index. Read operations against the data stream will be handled by all its backing indices. - - -### Check lifecycle progress [ilm-gs-check-progress] - -Use {{kib}} to [view the current status of your managed indices](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md) and details about the ILM policy, or the {{ilm-init}} explain API. Find out things like: - -* What phase an index is in and when it entered that phase. -* The current action and what step is being performed. -* If any errors have occurred or progress is blocked. - -For example, the following request gets information about the `timeseries` data stream’s backing indices: - -```console -GET .ds-timeseries-*/_ilm/explain -``` - -The following response shows the data stream’s first generation backing index is waiting for the `hot` phase’s `rollover` action. It remains in this state and {{ilm-init}} continues to call `check-rollover-ready` until a rollover condition is met. - -```console-result -{ - "indices": { - ".ds-timeseries-2099.03.07-000001": { - "index": ".ds-timeseries-2099.03.07-000001", - "index_creation_date_millis": 1538475653281, - "time_since_index_creation": "30s", <1> - "managed": true, - "policy": "timeseries_policy", <2> - "lifecycle_date_millis": 1538475653281, - "age": "30s", <3> - "phase": "hot", - "phase_time_millis": 1538475653317, - "action": "rollover", - "action_time_millis": 1538475653317, - "step": "check-rollover-ready", <4> - "step_time_millis": 1538475653317, - "phase_execution": { - "policy": "timeseries_policy", - "phase_definition": { <5> - "min_age": "0ms", - "actions": { - "rollover": { - "max_primary_shard_size": "50gb", - "max_age": "30d" - } - } - }, - "version": 1, - "modified_date_in_millis": 1539609701576 - } - } - } -} -``` - -1. The age of the index used for calculating when to rollover the index via the `max_age` -2. The policy used to manage the index -3. The age of the indexed used to transition to the next phase (in this case it is the same with the age of the index). -4. The step {{ilm-init}} is performing on the index -5. The definition of the current phase (the `hot` phase) - - - -## Manage time series data without data streams [manage-time-series-data-without-data-streams] - -Even though [data streams](../../data-store/data-streams.md) are a convenient way to scale and manage time series data, they are designed to be append-only. We recognise there might be use-cases where data needs to be updated or deleted in place and the data streams don’t support delete and update requests directly, so the index APIs would need to be used directly on the data stream’s backing indices. In these cases we still recommend using a data stream. - -If you frequently send multiple documents using the same `_id` expecting last-write-wins, you can use an index alias instead of a data stream to manage indices containing the time series data and periodically roll over to a new index. - -To automate rollover and management of time series indices with {{ilm-init}} using an index alias, you: - -1. Create a lifecycle policy that defines the appropriate phases and actions. See [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-policy) above. -2. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-apply-policy) to apply the policy to each new index. -3. [Bootstrap an index](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-bootstrap) as the initial write index. -4. [Verify indices are moving through the lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-check-progress) as expected. - - -### Create an index template to apply the lifecycle policy [ilm-gs-alias-apply-policy] - -To automatically apply a lifecycle policy to the new write index on rollover, specify the policy in the index template used to create new indices. - -For example, you might create a `timeseries_template` that is applied to new indices whose names match the `timeseries-*` index pattern. - -To enable automatic rollover, the template configures two {{ilm-init}} settings: - -* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to new indices that match the index pattern. -* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over when the rollover action is triggered for an index. - -::::{tab-set} -:group: kibana-api -:::{tab-item} {{kib}} -:sync: kibana -To use the {{kib}} **Create template** wizard to add the template, go to **Stack Management > Index Management**. In the **Index Templates** tab, click **Create template**. - -![Create template page](/manage-data/images/elasticsearch-reference-create-template-wizard.png "") - -:::{tip} -For more information about the available index template options that you can specify, refer to [Create an index template to apply the lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#apply-policy-template). -::: -::: - -:::{tab-item} API -:sync: api -The create template request for the example template looks like this: - -```console -PUT _index_template/timeseries_template -{ - "index_patterns": ["timeseries-*"], <1> - "template": { - "settings": { - "number_of_shards": 1, - "number_of_replicas": 1, - "index.lifecycle.name": "timeseries_policy", <2> - "index.lifecycle.rollover_alias": "timeseries" <3> - } - } -} -``` - -1. Apply the template to a new index if its name starts with `timeseries-`. -2. The name of the lifecycle policy to apply to each new index. -3. The name of the alias used to reference these indices. Required for policies that use the rollover action. - -::: -:::: - -### Bootstrap the initial time series index with a write index alias [ilm-gs-alias-bootstrap] - -To get things started, you need to bootstrap an initial index and designate it as the write index for the rollover alias specified in your index template. The name of this index must match the template’s index pattern and end with a number. On rollover, this value is incremented to generate a name for the new index. - -For example, the following request creates an index called `timeseries-000001` and makes it the write index for the `timeseries` alias. - -```console -PUT timeseries-000001 -{ - "aliases": { - "timeseries": { - "is_write_index": true - } - } -} -``` - -When the rollover conditions are met, the `rollover` action: - -* Creates a new index called `timeseries-000002`. This matches the `timeseries-*` pattern, so the settings from `timeseries_template` are applied to the new index. -* Designates the new index as the write index and makes the bootstrap index read-only. - -This process repeats each time rollover conditions are met. You can search across all of the indices managed by the `timeseries_policy` with the `timeseries` alias. Write operations should be sent towards the alias, which will route them to its current write index. - - -### Check lifecycle progress [ilm-gs-alias-check-progress] - -Retrieving the status information for managed indices is very similar to the data stream case. See the data stream [check progress section](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-check-progress) for more information. The only difference is the indices namespace, so retrieving the progress will entail the following api call: - -```console -GET timeseries-*/_ilm/explain -``` - -## Manage general content with data streams [manage-general-content-with-data-streams] - -Data streams are specifically designed for time series data. -If you want to manage general content (data without timestamps) with data streams, you can set up [ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) to transform and enrich your general content by adding a timestamp field at [ingest](/manage-data/ingest.md) time and get the benefits of time-based data management. - -For example, search use cases such as knowledge base, website content, e-commerce, or product catalog search, might require you to frequently index general content (data without timestamps). As a result, your index can grow significantly over time, which might impact storage requirements, query performance, and cluster health. Following the steps in this procedure (including a timestamp field and moving to ILM-managed data streams) can help you rotate your indices in a simpler way, based on their size or lifecycle phase. - -To roll over your general content from indices to a data stream, you: - -1. [Create an ingest pipeline](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-general-content-with-data-streams-ingest) to process your general content and add a `@timestamp` field. - -1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-general-content-with-data-streams-policy) that meets your requirements. - -1. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-general-content-with-data-streams-template) that uses the created ingest pipeline and lifecycle policy. - -1. [Create a data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-general-content-with-data-streams-create-stream). - -1. *Optional:* If you have an existing, non-managed index and want to migrate your data to the data stream you created, [reindex with a data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-general-content-with-data-streams-reindex). - -1. [Update your ingest endpoint](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-general-content-with-data-streams-endpoint) to target the created data stream. - -1. *Optional:* You can use the [ILM explain API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-explain-lifecycle) to get status information for your managed indices. -For more information, refer to [Check lifecycle progress](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-check-progress). - - -### Create an ingest pipeline to transform your general content [manage-general-content-with-data-streams-ingest] - -Create an ingest pipeline that uses the [`set` enrich processor](elasticsearch://reference/enrich-processor/set-processor.md) to add a `@timestamp` field: - -```console -PUT _ingest/pipeline/ingest_time_1 -{ - "description": "Add an ingest timestamp", - "processors": [ - { - "set": { - "field": "@timestamp", - "value": "{{_ingest.timestamp}}" - } - }] -} -``` - -### Create a lifecycle policy [manage-general-content-with-data-streams-policy] - - In this example, the policy is configured to roll over when the shard size reaches 10 GB: - -```console -PUT _ilm/policy/indextods -{ - "policy": { - "phases": { - "hot": { - "min_age": "0ms", - "actions": { - "set_priority": { - "priority": 100 - }, - "rollover": { - "max_primary_shard_size": "10gb" - } - } - } - } - } -} -``` - -For more information about lifecycle phases and available actions, check [Create a lifecycle policy](configure-lifecycle-policy.md#ilm-create-policy). - - -### Create an index template to apply the ingest pipeline and lifecycle policy [manage-general-content-with-data-streams-template] - -Create an index template that uses the created ingest pipeline and lifecycle policy: - -```console -PUT _index_template/index_to_dot -{ - "template": { - "settings": { - "index": { - "lifecycle": { - "name": "indextods" - }, - "default_pipeline": "ingest_time_1" - } - }, - "mappings": { - "_source": { - "excludes": [], - "includes": [], - "enabled": true - }, - "_routing": { - "required": false - }, - "dynamic": true, - "numeric_detection": false, - "date_detection": true, - "dynamic_date_formats": [ - "strict_date_optional_time", - "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z" - ] - } - }, - "index_patterns": [ - "movetods" - ], - "data_stream": { - "hidden": false, - "allow_custom_routing": false - } -} -``` - -### Create a data stream [manage-general-content-with-data-streams-create-stream] - -Create a data stream using the [_data_stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream): - -```console -PUT /_data_stream/movetods -``` - -You can [view the lifecycle status of your data stream](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md), including details about its associated ILM policy. - -### Optional: Reindex your data with a data stream [manage-general-content-with-data-streams-reindex] - -If you want to copy your documents from an existing index to the data stream you created, reindex with a data stream using the [_reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex): - -```console -POST /_reindex -{ - "source": { - "index": "indextods" - }, - "dest": { - "index": "movetods", - "op_type": "create" - - } -} -``` - -For more information, check [Reindex with a data stream](../../data-store/data-streams/use-data-stream.md#reindex-with-a-data-stream). - -### Update your ingest endpoint to target the created data stream [manage-general-content-with-data-streams-endpoint] - -If you use Elastic clients, scripts, or any other third party tool to ingest data to {{es}}, make sure you update these to use the created data stream. \ No newline at end of file diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md index 15c71750cf..1c0cbe95c1 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md @@ -7,7 +7,7 @@ products: - id: elasticsearch --- -# Tutorial: Customize built-in policies +# Customize built-in policies {{es}} includes a set of built-in {{ilm-init}} policies that govern how managed indices transition as they age. This guide demonstrates how you can customize the lifecycle of a managed index, to adjust how the index transitions across [data tiers](/manage-data/lifecycle/data-tiers.md) and what [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), such as downsampling or shrinking, are performed on the index during each lifecycle phase. diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md new file mode 100644 index 0000000000..9c5f4582f2 --- /dev/null +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md @@ -0,0 +1,156 @@ +--- +applies_to: + stack: ga +products: + - id: elasticsearch +--- + + +# Manage general content with data streams [manage-general-content-with-data-streams] + +Data streams are specifically designed for time series data. +If you want to manage general content (data without timestamps) with data streams, you can set up [ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) to transform and enrich your general content by adding a timestamp field at [ingest](/manage-data/ingest.md) time and get the benefits of time-based data management. + +For example, search use cases such as knowledge base, website content, e-commerce, or product catalog search, might require you to frequently index general content (data without timestamps). As a result, your index can grow significantly over time, which might impact storage requirements, query performance, and cluster health. Following the steps in this procedure (including a timestamp field and moving to ILM-managed data streams) can help you rotate your indices in a simpler way, based on their size or lifecycle phase. + +To roll over your general content from indices to a data stream, you: + +1. [Create an ingest pipeline](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md#manage-general-content-with-data-streams-ingest) to process your general content and add a `@timestamp` field. + +1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md#manage-general-content-with-data-streams-policy) that meets your requirements. + +1. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md#manage-general-content-with-data-streams-template) that uses the created ingest pipeline and lifecycle policy. + +1. [Create a data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md#manage-general-content-with-data-streams-create-stream). + +1. *Optional:* If you have an existing, non-managed index and want to migrate your data to the data stream you created, [reindex with a data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md#manage-general-content-with-data-streams-reindex). + +1. [Update your ingest endpoint](/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md#manage-general-content-with-data-streams-endpoint) to target the created data stream. + +1. *Optional:* You can use the [ILM explain API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-explain-lifecycle) to get status information for your managed indices. +For more information, refer to [Check lifecycle progress](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md#ilm-gs-check-progress). + + +## Create an ingest pipeline to transform your general content [manage-general-content-with-data-streams-ingest] + +Create an ingest pipeline that uses the [`set` enrich processor](elasticsearch://reference/enrich-processor/set-processor.md) to add a `@timestamp` field: + +```console +PUT _ingest/pipeline/ingest_time_1 +{ + "description": "Add an ingest timestamp", + "processors": [ + { + "set": { + "field": "@timestamp", + "value": "{{_ingest.timestamp}}" + } + }] +} +``` + +## Create a lifecycle policy [manage-general-content-with-data-streams-policy] + + In this example, the policy is configured to roll over when the shard size reaches 10 GB: + +```console +PUT _ilm/policy/indextods +{ + "policy": { + "phases": { + "hot": { + "min_age": "0ms", + "actions": { + "set_priority": { + "priority": 100 + }, + "rollover": { + "max_primary_shard_size": "10gb" + } + } + } + } + } +} +``` + +For more information about lifecycle phases and available actions, check [Create a lifecycle policy](configure-lifecycle-policy.md#ilm-create-policy). + + +## Create an index template to apply the ingest pipeline and lifecycle policy [manage-general-content-with-data-streams-template] + +Create an index template that uses the created ingest pipeline and lifecycle policy: + +```console +PUT _index_template/index_to_dot +{ + "template": { + "settings": { + "index": { + "lifecycle": { + "name": "indextods" + }, + "default_pipeline": "ingest_time_1" + } + }, + "mappings": { + "_source": { + "excludes": [], + "includes": [], + "enabled": true + }, + "_routing": { + "required": false + }, + "dynamic": true, + "numeric_detection": false, + "date_detection": true, + "dynamic_date_formats": [ + "strict_date_optional_time", + "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z" + ] + } + }, + "index_patterns": [ + "movetods" + ], + "data_stream": { + "hidden": false, + "allow_custom_routing": false + } +} +``` + +## Create a data stream [manage-general-content-with-data-streams-create-stream] + +Create a data stream using the [_data_stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream): + +```console +PUT /_data_stream/movetods +``` + +You can [view the lifecycle status of your data stream](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md), including details about its associated ILM policy. + +### Optional: Reindex your data with a data stream [manage-general-content-with-data-streams-reindex] + +If you want to copy your documents from an existing index to the data stream you created, reindex with a data stream using the [_reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex): + +```console +POST /_reindex +{ + "source": { + "index": "indextods" + }, + "dest": { + "index": "movetods", + "op_type": "create" + + } +} +``` + +For more information, check [Reindex with a data stream](../../data-store/data-streams/use-data-stream.md#reindex-with-a-data-stream). + +## Update your ingest endpoint to target the created data stream [manage-general-content-with-data-streams-endpoint] + +If you use Elastic clients, scripts, or any other third party tool to ingest data to {{es}}, make sure you update these to use the created data stream. \ No newline at end of file diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md new file mode 100644 index 0000000000..c7b8926e99 --- /dev/null +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md @@ -0,0 +1,160 @@ +--- +applies_to: + stack: ga +products: + - id: elasticsearch +--- + +# Manage time series data with data streams [manage-time-series-data-with-data-streams] + +To automate rollover and management of a data stream with {{ilm-init}}, you: + +1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md#ilm-gs-create-policy) that defines the appropriate [phases](index-lifecycle.md) and [actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md). +2. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md#ilm-gs-apply-policy) to [create the data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md#ilm-gs-create-the-data-stream) and apply the ILM policy and the indices settings and mappings configurations for the backing indices. +3. [Verify that indices are moving through the lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md#ilm-gs-check-progress) as expected. + +::::{important} +When you enable {{ilm}} for {{beats}} or the {{ls}} {{es}} output plugin, lifecycle policies are set up automatically. You do not need to take any other actions. You can modify the default policies through [{{kib}} Management](tutorial-customize-built-in-policies.md) or the {{ilm-init}} APIs. +:::: + + + +## Create a lifecycle policy [ilm-gs-create-policy] + +:::{include} /manage-data/_snippets/create-lifecycle-policy.md +::: + + +## Create an index template to create the data stream and apply the lifecycle policy [ilm-gs-apply-policy] + +To set up a data stream, first create an index template to specify the lifecycle policy. Because the template is for a data stream, it must also include a `data_stream` definition. + +For example, you might create a template named `timeseries_template` and use it for a future data stream named `timeseries`. + +To enable {{ilm-init}} to manage the data stream, the template configures one {{ilm-init}} setting: + +* `index.lifecycle.name` specifies the name of the lifecycle policy that you want to apply to the data stream. + +Use the {{kib}} **Create template** wizard to add a template or the [Create or update index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API to add a template and apply the lifecycle policy to indices matching the template. + +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To add an index template to a cluster using the wizard, go to **Stack Management > Index Management**. In the **Index Templates** tab, click **Create template**. + +![Create template page](/manage-data/images/elasticsearch-reference-create-index-template.png "") + +This wizard invokes the create or update index template API to create the index template with the options you specify. + +:::{tip} +To learn about which index template options you can specify, refer to [Create an index template to apply the lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#apply-policy-template). +::: +::: + +:::{tab-item} API +:sync: api +Use the API to add an index template to your cluster: + +```console +PUT _index_template/timeseries_template +{ + "index_patterns": ["timeseries"], <1> + "data_stream": { }, + "template": { + "settings": { + "number_of_shards": 1, + "number_of_replicas": 1, + "index.lifecycle.name": "timeseries_policy" <2> + } + } +} +``` + +1. Apply the template when a document is indexed into the `timeseries` target. +2. The name of the {{ilm-init}} policy used to manage the data stream. + +::: +:::: + + + +## Create the data stream [ilm-gs-create-the-data-stream] + +To get things started, index a document into the name or wildcard pattern defined in the `index_patterns` of the [index template](../../data-store/templates.md). As long as an existing data stream, index, or index alias does not already use the name, the index request automatically creates a corresponding data stream with a single backing index. {{es}} automatically indexes the request’s documents into this backing index, which also acts as the stream’s [write index](../../data-store/data-streams.md#data-stream-write-index). + +For example, the following request creates the `timeseries` data stream and the first generation backing index called `.ds-timeseries-2099.03.08-000001`. + +```console +POST timeseries/_doc +{ + "message": "logged the request", + "@timestamp": "1591890611" +} +``` + +When a rollover condition in the lifecycle policy is met, the `rollover` action: + +* Creates the second generation backing index, named `.ds-timeseries-2099.03.08-000002`. Because it is a backing index of the `timeseries` data stream, the configuration from the `timeseries_template` index template is applied to the new index. +* As it is the latest generation index of the `timeseries` data stream, the newly created backing index `.ds-timeseries-2099.03.08-000002` becomes the data stream’s write index. + +This process repeats each time a rollover condition is met. You can search across all of the data stream’s backing indices, managed by the `timeseries_policy`, with the `timeseries` data stream name. Write operations should be sent to the data stream name, which will route them to its current write index. Read operations against the data stream will be handled by all its backing indices. + + +## Check lifecycle progress [ilm-gs-check-progress] + +Use {{kib}} to [view the current status of your managed indices](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md) and details about the ILM policy, or the {{ilm-init}} explain API. Find out things like: + +* What phase an index is in and when it entered that phase. +* The current action and what step is being performed. +* If any errors have occurred or progress is blocked. + +For example, the following request gets information about the `timeseries` data stream’s backing indices: + +```console +GET .ds-timeseries-*/_ilm/explain +``` + +The following response shows the data stream’s first generation backing index is waiting for the `hot` phase’s `rollover` action. It remains in this state and {{ilm-init}} continues to call `check-rollover-ready` until a rollover condition is met. + +```console-result +{ + "indices": { + ".ds-timeseries-2099.03.07-000001": { + "index": ".ds-timeseries-2099.03.07-000001", + "index_creation_date_millis": 1538475653281, + "time_since_index_creation": "30s", <1> + "managed": true, + "policy": "timeseries_policy", <2> + "lifecycle_date_millis": 1538475653281, + "age": "30s", <3> + "phase": "hot", + "phase_time_millis": 1538475653317, + "action": "rollover", + "action_time_millis": 1538475653317, + "step": "check-rollover-ready", <4> + "step_time_millis": 1538475653317, + "phase_execution": { + "policy": "timeseries_policy", + "phase_definition": { <5> + "min_age": "0ms", + "actions": { + "rollover": { + "max_primary_shard_size": "50gb", + "max_age": "30d" + } + } + }, + "version": 1, + "modified_date_in_millis": 1539609701576 + } + } + } +} +``` + +1. The age of the index used for calculating when to rollover the index via the `max_age` +2. The policy used to manage the index +3. The age of the indexed used to transition to the next phase (in this case it is the same with the age of the index). +4. The step {{ilm-init}} is performing on the index +5. The definition of the current phase (the `hot` phase) diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md new file mode 100644 index 0000000000..a5e437e7ea --- /dev/null +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md @@ -0,0 +1,109 @@ +--- +applies_to: + stack: ga +products: + - id: elasticsearch +--- + +# Manage time series data without data streams [manage-time-series-data-without-data-streams] + +Even though [data streams](../../data-store/data-streams.md) are a convenient way to scale and manage time series data, they are designed to be append-only. We recognise there might be use-cases where data needs to be updated or deleted in place and the data streams don’t support delete and update requests directly, so the index APIs would need to be used directly on the data stream’s backing indices. In these cases we still recommend using a data stream. + +If you frequently send multiple documents using the same `_id` expecting last-write-wins, you can use an index alias instead of a data stream to manage indices containing the time series data and periodically roll over to a new index. + +To automate rollover and management of time series indices with {{ilm-init}} using an index alias, you: + +1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md#ilm-gs-create-policy) that defines the appropriate phases and actions. +2. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md#ilm-gs-alias-apply-policy) to apply the policy to each new index. +3. [Bootstrap an index](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md#ilm-gs-alias-bootstrap) as the initial write index. +4. [Verify indices are moving through the lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md#ilm-gs-alias-check-progress) as expected. + +## Create a lifecycle policy [ilm-gs-create-policy] + +:::{include} /manage-data/_snippets/create-lifecycle-policy.md +::: + +## Create an index template to apply the lifecycle policy [ilm-gs-alias-apply-policy] + +To automatically apply a lifecycle policy to the new write index on rollover, specify the policy in the index template used to create new indices. + +For example, you might create a `timeseries_template` that is applied to new indices whose names match the `timeseries-*` index pattern. + +To enable automatic rollover, the template configures two {{ilm-init}} settings: + +* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to new indices that match the index pattern. +* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over when the rollover action is triggered for an index. + +::::{tab-set} +:group: kibana-api +:::{tab-item} {{kib}} +:sync: kibana +To use the {{kib}} **Create template** wizard to add the template, go to **Stack Management > Index Management**. In the **Index Templates** tab, click **Create template**. + +![Create template page](/manage-data/images/elasticsearch-reference-create-template-wizard.png "") + +:::{tip} +For more information about the available index template options that you can specify, refer to [Create an index template to apply the lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#apply-policy-template). +::: +::: + +:::{tab-item} API +:sync: api +The create template request for the example template looks like this: + +```console +PUT _index_template/timeseries_template +{ + "index_patterns": ["timeseries-*"], <1> + "template": { + "settings": { + "number_of_shards": 1, + "number_of_replicas": 1, + "index.lifecycle.name": "timeseries_policy", <2> + "index.lifecycle.rollover_alias": "timeseries" <3> + } + } +} +``` + +1. Apply the template to a new index if its name starts with `timeseries-`. +2. The name of the lifecycle policy to apply to each new index. +3. The name of the alias used to reference these indices. Required for policies that use the rollover action. + +::: +:::: + +## Bootstrap the initial time series index with a write index alias [ilm-gs-alias-bootstrap] + +To get things started, you need to bootstrap an initial index and designate it as the write index for the rollover alias specified in your index template. The name of this index must match the template’s index pattern and end with a number. On rollover, this value is incremented to generate a name for the new index. + +For example, the following request creates an index called `timeseries-000001` and makes it the write index for the `timeseries` alias. + +```console +PUT timeseries-000001 +{ + "aliases": { + "timeseries": { + "is_write_index": true + } + } +} +``` + +When the rollover conditions are met, the `rollover` action: + +* Creates a new index called `timeseries-000002`. This matches the `timeseries-*` pattern, so the settings from `timeseries_template` are applied to the new index. +* Designates the new index as the write index and makes the bootstrap index read-only. + +This process repeats each time rollover conditions are met. You can search across all of the indices managed by the `timeseries_policy` with the `timeseries` alias. Write operations should be sent towards the alias, which will route them to its current write index. + + +## Check lifecycle progress [ilm-gs-alias-check-progress] + +Retrieving the status information for managed indices is very similar to the case of [managing time series data with data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md) case, the only difference being the indices namespace. Run the following API request to retrieve the lifecycle progress: + +```console +GET timeseries-*/_ilm/explain +``` + +See [Check lifecycle progress](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md#ilm-gs-check-progress) for more information. \ No newline at end of file diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 62bc0732f3..b520f8a7c3 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -124,7 +124,6 @@ toc: - file: lifecycle/index-lifecycle-management/rollover.md children: - file: lifecycle/index-lifecycle-management/skip-rollover.md - - file: lifecycle/index-lifecycle-management/tutorial-automate-rollover.md - file: lifecycle/index-lifecycle-management/configure-lifecycle-policy.md - file: lifecycle/index-lifecycle-management/policy-apply.md - file: lifecycle/index-lifecycle-management/policy-view-status.md @@ -132,7 +131,12 @@ toc: - file: lifecycle/index-lifecycle-management/policy-updates.md - file: lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md - file: lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md - - file: lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md + - file: lifecycle/index-lifecycle-management/ilm-tutorials.md + children: + - file: lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md + - file: lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md + - file: lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md + - file: lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md - hidden: lifecycle/index-lifecycle-management/index-management-in-kibana.md - file: lifecycle/index-lifecycle-management/migrate-ilm.md children: diff --git a/redirects.yml b/redirects.yml index ef165b48c0..04014b6198 100644 --- a/redirects.yml +++ b/redirects.yml @@ -357,6 +357,10 @@ redirects: # Related to https://github.com/elastic/docs-content/pull/2843 'solutions/search/serverless-elasticsearch-get-started.md': 'solutions/search/get-started.md' +# Related to https://github.com/elastic/docs-content/pull/3032 + 'manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md': 'manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md' + # Remove duplicate reference content 'deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md': 'elasticsearch://reference/elasticsearch/roles.md' 'deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md': 'elasticsearch://reference/elasticsearch/security-privileges.md' + diff --git a/troubleshoot/elasticsearch/index-lifecycle-management-errors.md b/troubleshoot/elasticsearch/index-lifecycle-management-errors.md index fa801d41b0..4890292942 100644 --- a/troubleshoot/elasticsearch/index-lifecycle-management-errors.md +++ b/troubleshoot/elasticsearch/index-lifecycle-management-errors.md @@ -170,7 +170,7 @@ Problems with rollover aliases are a common cause of errors. Consider using [dat ### Rollover alias [x] can point to multiple indices, found duplicated alias [x] in index template [z] [_rollover_alias_x_can_point_to_multiple_indices_found_duplicated_alias_x_in_index_template_z] -The target rollover alias is specified in an index template’s `index.lifecycle.rollover_alias` setting. You need to explicitly configure this alias *one time* when you [bootstrap the initial index](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-alias-bootstrap). The rollover action then manages setting and updating the alias to [roll over](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover#rollover-index-api-desc) to each subsequent index. +The target rollover alias is specified in an index template’s `index.lifecycle.rollover_alias` setting. You need to explicitly configure this alias *one time* when you [bootstrap the initial index](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md#ilm-gs-alias-bootstrap). The rollover action then manages setting and updating the alias to [roll over](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover#rollover-index-api-desc) to each subsequent index. Do not explicitly configure this same alias in the aliases section of an index template. diff --git a/troubleshoot/monitoring/high-memory-pressure.md b/troubleshoot/monitoring/high-memory-pressure.md index 3085f585f7..611d8c90da 100644 --- a/troubleshoot/monitoring/high-memory-pressure.md +++ b/troubleshoot/monitoring/high-memory-pressure.md @@ -55,7 +55,7 @@ By understanding and adjusting the way your data is indexed, retained, and searc ### Data retention [ec_data_retention] -The total amount of data being searched affects search performance. Check the tutorial [Automate rollover with index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md) (ILM) to automate data retention policies. +The total amount of data being searched affects search performance. Check the [{{ilm}} tutorials](/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md) to learn about automating data retention policies. ### Tune for search speed [ec_tune_for_search_speed]