You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md
+125-4Lines changed: 125 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,7 +33,26 @@ For more information, refer to [Check lifecycle progress](/manage-data/lifecycle
33
33
34
34
## Create an ingest pipeline to transform your general content [manage-general-content-with-data-streams-ingest]
35
35
36
-
Create an ingest pipeline that uses the [`set` enrich processor](elasticsearch://reference/enrich-processor/set-processor.md) to add a `@timestamp` field:
36
+
You can create an ingest pipeline that uses the [`set` enrich processor](elasticsearch://reference/enrich-processor/set-processor.md) to add a `@timestamp` field. Follow these steps in Kibana or using the [create or update a pipeline](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) API.
37
+
38
+
::::{tab-set}
39
+
:group: kibana-api
40
+
:::{tab-item} {{kib}}
41
+
:sync: kibana
42
+
To add an ingest pipeline in {{kib}}, go to **Stack Management > Ingest Pipelines**, and then select **Create pipeline > New pipeline**.
43
+
44
+
Configure the pipeline with a name, description, and a **Set** processor that adds the `@timestamp` field with a value of `{{_ingest.timestamp}}`.
@@ -48,11 +67,38 @@ PUT _ingest/pipeline/ingest_time_1
48
67
}]
49
68
}
50
69
```
70
+
:::
71
+
::::
51
72
52
73
## Create a lifecycle policy [manage-general-content-with-data-streams-policy]
53
74
54
-
In this example, the policy is configured to roll over when the shard size reaches 10 GB:
75
+
A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to five phases: `hot`, `warm`, `cold`, `frozen`, and `delete`.
76
+
77
+
For example, you might define a policy named `indextods` that is configured to roll over when the shard size reaches 10 GB.
78
+
79
+
You can create the policy in {{kib}} or with the [create or update policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle) API.
80
+
81
+
::::{tab-set}
82
+
:group: kibana-api
83
+
:::{tab-item} {{kib}}
84
+
:sync: kibana
85
+
To create the policy in {{kib}}, open the menu and go to **Stack Management > Index Lifecycle Policies**. Click **Create policy**.
86
+
87
+
In the **Hot phase**, by default, an ILM-managed index [rolls over](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) when either:
88
+
* It reaches 30 days of age.
89
+
* One or more primary shards reach 50 GB in size.
90
+
91
+
Disable **Use recommended defaults** to adjust these values and roll over when the primary shard reaches 10GB.
For more information about lifecycle phases and available actions, check[Create a lifecycle policy](configure-lifecycle-policy.md#ilm-create-policy).
125
+
For more information about lifecycle phases and available actions, refer to[Create a lifecycle policy](configure-lifecycle-policy.md#ilm-create-policy).
78
126
79
127
80
128
## Create an index template to apply the ingest pipeline and lifecycle policy [manage-general-content-with-data-streams-template]
81
129
82
-
Create an index template that uses the created ingest pipeline and lifecycle policy:
130
+
To use the created lifecycle policy, you configure an index template that uses it.
131
+
When creating the index template, specify the following details:
132
+
* the name of the lifecycle policy, which in our example is `indextods`
133
+
* the ingest pipeline that enriches the data by adding the `@timestamp` field, which in our example is `ingest_time_1`
134
+
* that the template is data stream enabled by including the `data_stream` definition
135
+
* the index pattern, which ensures that this template will be applied to matching indices and in our example is `movetods`
136
+
137
+
You can create the template in {{kib}} or with the [create or update index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API.
138
+
139
+
::::{tab-set}
140
+
:group: kibana-api
141
+
:::{tab-item} {{kib}}
142
+
:sync: kibana
143
+
To create an index template in Kibana, complete these steps:
144
+
145
+
1. Go to **Stack Management > Index Management**. In the **Index Templates** tab, select **Create template**.
1. Specify the name of the template. For example `index_to_dot`.
151
+
1. Specify a pattern to match the indices you want to manage with the lifecycle policy. For example, `movetodos`.
152
+
1. Turn on the **Create data streams** toggle.
153
+
1. Set the [index mode](elasticsearch://reference/elasticsearch/index-settings/time-series.md) to **Standard**.
154
+
1. Optional: On the **Component templates** page, use the search and filter tools to select any [component templates](/manage-data/data-store/templates.md#component-templates) to include in the index template. The index template will inherit the settings, mappings, and aliases defined in the component templates and apply them to indices when they're created.
155
+
156
+
1. On the **Index settings** page, specify the lifecycle policy and ingest pipeline you want to use. For example, `indextods` and `ingest_time_1`:
157
+
158
+
```json
159
+
{
160
+
"lifecycle": {
161
+
"name": "indextods"
162
+
},
163
+
"default_pipeline": "ingest_time_1"
164
+
}
165
+
```
166
+
167
+
1. On the **Mappings** page, customize the fields and data types used when documents are indexed into {{es}}. For example, select **Load JSON** and include these mappings:
168
+
169
+
```json
170
+
{
171
+
"_source": {
172
+
"excludes": [],
173
+
"includes": [],
174
+
"enabled": true
175
+
},
176
+
"_routing": {
177
+
"required": false
178
+
},
179
+
"dynamic": true,
180
+
"numeric_detection": false,
181
+
"date_detection": true,
182
+
"dynamic_date_formats": [
183
+
"strict_date_optional_time",
184
+
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
185
+
]
186
+
}
187
+
```
188
+
189
+
1. On the **Review** page, confirm your selections. You can check your selected options, as well as both the format of the index template that will be created and the associated API request.
190
+
191
+
The newly created index template will be used for all new indices with names that match the specified pattern, and for each of these, the specified ILM policy will be applied.
192
+
193
+
For more information about configuring templates in Kibana, refer to [Manage index templates](/manage-data/data-store/index-basics.md#index-management-manage-index-templates).
194
+
:::
195
+
196
+
:::{tab-item} API
197
+
:sync: api
198
+
199
+
Use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) to create an index template that specifies the created ingest pipeline and lifecycle policy:
83
200
84
201
```console
85
202
PUT _index_template/index_to_dot
@@ -120,6 +237,10 @@ PUT _index_template/index_to_dot
120
237
}
121
238
}
122
239
```
240
+
:::
241
+
::::
242
+
243
+
123
244
124
245
## Create a data stream [manage-general-content-with-data-streams-create-stream]
0 commit comments