Skip to content

Commit 3e82318

Browse files
committed
Updating the Rollover page
As per internal feedback, we're updating this page. Fixes #1563
1 parent 11ad157 commit 3e82318

File tree

1 file changed

+56
-11
lines changed
  • manage-data/lifecycle/index-lifecycle-management

1 file changed

+56
-11
lines changed

manage-data/lifecycle/index-lifecycle-management/rollover.md

Lines changed: 56 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,33 +9,77 @@ products:
99

1010
# Rollover [index-rollover]
1111

12-
When indexing time series data like logs or metrics, you can’t write to a single index indefinitely. To meet your indexing and search performance requirements and manage resource usage, you write to an index until some threshold is met and then create a new index and start writing to it instead. Using rolling indices enables you to:
12+
In {{es}}, *rollover* replaces your active write index with a new one whenever your index grows too large, too old, or stores too many documents.
13+
This is particularly useful for time-series data, such as logs or metrics, where index growth is continuous and performance can degrade over time.
1314

14-
* Optimize the active index for high ingest rates on high-performance *hot* nodes.
15-
* Optimize for search performance on *warm* nodes.
16-
* Shift older, less frequently accessed data to less expensive *cold* nodes,
17-
* Delete data according to your retention policies by removing entire indices.
15+
Without rollover, a single index can grow until searches slow down, shard relocation takes hours, and nodes run low on disk space.
1816

19-
We recommend using [data streams](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream) to manage time series data. Data streams automatically track the write index while keeping configuration to a minimum.
17+
The rollover feature is an important part of how [index lifecycle](../index-lifecycle-management/index-lifecycle.md) and [data stream lifecycles](../data-stream.md) work to keep your indices fast and manageable. By switching the write target of an index, the rollover action provides the following benefits:
2018

21-
Each data stream requires an [index template](../../data-store/templates.md) that contains:
19+
* **Automation** - works with ILM to remove manual index rotation tasks and allows for granular control over retention cycles
20+
* **Optimized performance** - keeps shard sizes within recommended limits (10-50 GB)
21+
* **Queries run faster** - improves search performance
22+
23+
::::{tip}
24+
Rolling over to a new index based on size, document count, or age is preferable to time-based rollovers. Rolling over at an arbitrary time often results in many small indices, which can have a negative impact on performance and resource usage.
25+
::::
26+
27+
28+
## Recommended approaches
29+
30+
Decide your approach to index rotation based on your use case and requirements.
31+
32+
| Use case | Recommended approach | Benefits |
33+
| ---------------------- | --------------------------------------------------------- | ------------------------------------- |
34+
| Logs, metrics | [Data streams](rollover.md#rollover-data-stream) | Automatic rollover, minimal setup |
35+
| Legacy indexing setup | [Alias-based rollover](rollover.md#rollover-with-aliases) | Granular control over rollover timing |
36+
| Small, static datasets | No rollover | Simpler management |
37+
38+
:::{tip}
39+
For new projects, use data streams. They're simple to manage and handle rollover automatically.
40+
:::
41+
42+
43+
### Rotating your indices with data streams [rollover-data-stream]
44+
45+
We recommend using [data streams](../../data-store/data-streams.md) to manage time series data. Data streams automatically track the write index while keeping configuration to a minimum.
46+
47+
Each data stream requires an [index template](../../data-store/templates.md) that contains the following:
2248

2349
* A name or wildcard (`*`) pattern for the data stream.
2450
* The data stream’s timestamp field. This field must be mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) field data type and must be included in every document indexed to the data stream.
2551
* The mappings and settings applied to each backing index when it’s created.
2652

27-
Data streams are designed for append-only data, where the data stream name can be used as the operations (read, write, rollover, shrink etc.) target. If your use case requires data to be updated in place, you can instead manage your time series data using [index aliases](../../data-store/aliases.md). However, there are a few more configuration steps and concepts:
2853

29-
* An *index template* that specifies the settings for each new index in the series. You optimize this configuration for ingestion, typically using as many shards as you have hot nodes.
30-
* An *index alias* that references the entire set of indices.
31-
* A single index designated as the *write index*. This is the active index that handles all write requests. On each rollover, the new index becomes the write index.
54+
55+
### Rotating your indices with index aliases [rollover-with-aliases]
56+
57+
Data streams are designed for append-only data, where the data stream name can be used as the operations (read, write, rollover, shrink etc.) target. If your use case requires data to be updated in place, you can instead manage your time series data using [index aliases](../../data-store/aliases.md). However, this approach requires additional steps to configure a write alias and templates, such as the following:
58+
59+
* Creating an *index template* that specifies the settings for each new index in the series. You optimize this configuration for ingestion, typically using as many shards as you have hot nodes.
60+
* Creating an *index alias* that references the entire set of indices.
61+
* Configuring a designated index as the *write index*. This is the active index that handles all write requests. On each rollover, the new index becomes the write index.
62+
* Creating an ILM policy with a rollover action and applying the policy to the index template.
3263

3364
::::{note}
3465
When an index is rolled over, the previous index’s age is updated to reflect the rollover time. This date, rather than the index’s `creation_date`, is used in {{ilm}} `min_age` phase calculations. [Learn more](../../../troubleshoot/elasticsearch/index-lifecycle-management-errors.md#min-age-calculation).
3566

3667
::::
3768

3869

70+
## How rollover works in ILM
71+
72+
You can define a rollover action in the hot phase of an ILM policy. It will run when any of the configured thresholds are met.
73+
74+
* **Size** - if an index size reaches a set size, for example 50 GB
75+
* **Age** - if an index reaches a certain age, for example 7 days
76+
* **Document count** - if a shards contains 200 million documents or more.
77+
78+
79+
**Special rules:**
80+
81+
* Empty index rollover is blocked unless you set `min_docs: 0` in rollover conditions.
82+
* Forced rollover occurs if any shard reaches 200 M documents.
3983

4084
## Automatic rollover [ilm-automatic-rollover]
4185

@@ -56,3 +100,4 @@ The rollover action implicitly always rolls over a data stream or alias if one o
56100
::::
57101

58102

103+
[data streams API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream)

0 commit comments

Comments
 (0)