Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,26 @@ For more information, refer to [Check lifecycle progress](/manage-data/lifecycle

## Create an ingest pipeline to transform your general content [manage-general-content-with-data-streams-ingest]

Create an ingest pipeline that uses the [`set` enrich processor](elasticsearch://reference/enrich-processor/set-processor.md) to add a `@timestamp` field:
You can create an ingest pipeline that uses the [`set` enrich processor](elasticsearch://reference/enrich-processor/set-processor.md) to add a `@timestamp` field in Kibana or with the [create or update a pipeline](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) API.

::::{tab-set}
:group: kibana-api
:::{tab-item} {{kib}}
:sync: kibana
To add an ingest pipeline from {{kib}}, go to **Stack Management > Ingest Pipelines**. From the upper right, select **Create pipeline > New pipeline**.

Configure the pipeline with a name, description, and a **Set** processor that adds the `@timestamp` field with a value of `{{_ingest.timestamp}}`.

:::{image} /manage-data/images/elasticsearch-reference-tutorial-ilm-general-content-ingest.png
:alt: Create ingest pipeline
:screenshot:
:::

:::

:::{tab-item} API
:sync: api
Use the API to add an ingest pipeline:

```console
PUT _ingest/pipeline/ingest_time_1
Expand All @@ -48,11 +67,38 @@ PUT _ingest/pipeline/ingest_time_1
}]
}
```
:::
::::

## Create a lifecycle policy [manage-general-content-with-data-streams-policy]

In this example, the policy is configured to roll over when the shard size reaches 10 GB:
A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to five phases: `hot`, `warm`, `cold`, `frozen`, and `delete`.

For example, you might define a policy named `indextods` that is configured to roll over when the shard size reaches 10 GB.

You can create the policy in {{kib}} or with the [create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API.

::::{tab-set}
:group: kibana-api
:::{tab-item} {{kib}}
:sync: kibana
To create the policy from {{kib}}, open the menu and go to **Stack Management > Index Lifecycle Policies**. Click **Create policy**.

In the **Hot phase**, by default, an ILM-managed index [rolls over](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) when either:
* It reaches 30 days of age.
* One or more primary shards reach 50 GB in size.

Disable **Use recommended defaults** to adjust these values and roll over when the primary shard reaches 10GB.

:::{image} /manage-data/images/elasticsearch-reference-tutorial-ilm-rollover-general-content-create-policy.png
:alt: Create policy page
:screenshot:
:::
:::

:::{tab-item} API
:sync: api
Use the API to create a lifecyle policy:
```console
PUT _ilm/policy/indextods
{
Expand All @@ -73,13 +119,22 @@ PUT _ilm/policy/indextods
}
}
```
:::
::::

For more information about lifecycle phases and available actions, check [Create a lifecycle policy](configure-lifecycle-policy.md#ilm-create-policy).
For more information about lifecycle phases and available actions, refer to [Create a lifecycle policy](configure-lifecycle-policy.md#ilm-create-policy).


## Create an index template to apply the ingest pipeline and lifecycle policy [manage-general-content-with-data-streams-template]

Create an index template that uses the created ingest pipeline and lifecycle policy:
To use the created lifecycle policy, you configure an index template that uses it.
When creating the index template, specify the following details:
* the name of the lifecycle policy, which in our example is `indextods`
* the ingest pipeline that enriches the data by adding the `@timestamp` field, which in our example is `ingest_time_1`
* that the template is data stream enabled by including the `data_stream` definition
* the index pattern, which ensures that this template will be applied to matching indices and in our example is `movetods`

Use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) to create an index template that uses the created ingest pipeline and lifecycle policy:

```console
PUT _index_template/index_to_dot
Expand Down Expand Up @@ -121,6 +176,9 @@ PUT _index_template/index_to_dot
}
```

To create an index template in Kibana, open the main menu and go to **Stack Management > Index Management**. In the **Index Templates** view, click **Create template**.
Refer to [Manage index templates](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) for more information about configuring templates in Kibana.

## Create a data stream [manage-general-content-with-data-streams-create-stream]

Create a data stream using the [_data_stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream):
Expand Down
Loading