Updating the Rollover page

[yetanothertw]

As per feedback received and collected in #1563 about the Rollover page, we're updating this page to improve the information quality and relevance.

The content has been restructured into the following sections:

• The Rollover page includes a general description/introduction of the concept above the fold.
• The How rollover works in ILM section ties in the concept or rollover in to the context of ILM. Some older admonitions that existed on this page have been restructured and integrated into logical compartments.
• The Recommended approaches section aims to explain (at a very high level) how a users approach to index rotation might differ depending on their use case.
• Rotating your indices with data streams introduces the recommended approach and gives an overview of some of the requirements. It also links to Data streams documentation where naming patterns/generation are explained in more detail. Another important link is that it now logically ties this section in with the child page (the tutorial) that offers step-by-step guidance.
• Rotating your indices with aliases introduces the other use case of using the rollover action and also links it with its respective tutorial steps

Fixes #1563

[github-actions]

🔍 Preview links for changed docs

• manage-data/lifecycle/index-lifecycle-management/rollover.md

[dakrone]

Thanks for taking a look at this, I left quite a few comments.

[yetanothertw]

Thank you so much for reviewing, @dakrone! I've addressed your comments (hopefully correctly) -- that's been very useful.

One thing I think I struggle to understand, and maybe you can help with, is when/under what conditions would a user prefer to rotate their indices using aliases as opposed to data streams? Would that be attributable to legacy reasons mostly or is there a specific problem they would be solving that way? I'm wondering how to provide more context around the when and why one would do that.

The text previously said:

Data streams are designed for append-only data, where the data stream name can be used as the operations (read, write, rollover, shrink etc.) target. If your use case requires data to be updated in place, you can instead manage your time series data using index aliases.

And as you've pointed out, documents can be updated or deleted in the backing index that contain them using the API. So this is not a relevant reason for choosing the aliases approach.

[dakrone]

One thing I think I struggle to understand, and maybe you can help with, is when/under what conditions would a user prefer to rotate their indices using aliases as opposed to data streams? Would that be attributable to legacy reasons mostly or is there a specific problem they would be solving that way?

Certainly, the specific reason that users use aliases and ILM is that they can then avoid rollover. For example, if a user does time-based indices, they can have an alias called mydata backed by the concrete indices mydata-jan-01, mydata-jan-02, mydata-jan-03, etc (some users do daily, some monthly, weekly, etc). They then index into the correct backing index depending on the timestamp of the document.

The main reason that users wish to avoid rollover is to solve the duplicate handling with rollover problem. It's something we plan to address, but it's a real issue for some users.

[kilfoyle]

Suggested change

	In {{es}}, the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) replaces your active write index with a new one whenever your index grows too large, too old, or stores too many documents.
	In {{es}}, the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) replaces your active write index with a new one whenever your index grows beyond a specified size, age, or number of documents.

Just a thought. I think someone might interpret "too large", "too old", etc. as meaning that Elasticsearch struggles to manage the index in these cases. Maybe the phrasing can be more positive, suggesting just that rollover gives the user better control over these things (as a way to optimize for performance, and retain data according to requirements, as you have in the next sentence).

[kilfoyle]

@dakrone I think this and line 19 are based off of your comment. Do you think it's okay for us to introduce "DLM" here as an acronym for "data stream lifecycle management"? I'm a little hesitant since I gather DLM is already in popular use as "data lifecycle management" (e.g. IBM, HP), and our use here specifically to denote "data stream lifecycle" doesn't quite match. That is, to my mind "ILM" and "data stream lifecycle" are two different approaches to DLM.

What would you think of our sticking with:

• index lifecycle management (acronym: "ILM")
• data stream lifecycle (no official acronym)

If we do introduce an acronym for "data stream lifecycle" we should also update this main page about it.

[dakrone]

I don't think we can stick with ILM to refer to the data stream's lifecycle, since they are different (ILM has policies, is attached at the index level, has separate APIs, is not available in Serverless, etc).

The challenge that we've run into with "data stream lifecycle" is that absent an acronym, folks have been calling it "DSL", which is a MUCH more confusing acronym than "DLM", so we've been substituting DLM for the name to differentiate it from ILM, but still indicate it is the "Data (in the data stream) Lifecycle Management".

[kilfoyle]

Roger that. Thanks for the clarification! DLM it is. :-)

[kilfoyle]

Suggested change

	* **Age** - an index will rollover when an index reaches a certain age, for example 7 days.
	* **Age** - an index will rollover when it reaches a certain age, for example 7 days.

[kilfoyle]

Suggested change

	After rollover, indices move to other index lifecycle phases like warm, cold, frozen, and delete. Rollover creates a new write index while the old one continues through the lifecycle phases.
	After rollover, indices move through other configured index lifecycle phases: warm, cold, frozen, and/or delete. Rollover creates a new write index while the old one continues through the lifecycle phases.

Would this be better? I'd probably avoid like since that seems to me to imply that "warm", "cold", etc. are examples, rather than an exhaustive list.

[kilfoyle]

Suggested change

	* Rollover for an empty write index is skipped even if they have an associated `max_age` that would otherwise result in a roll over occurring. A policy can override this behavior if you set `min_docs: 0` in the rollover conditions. This can also be disabled on a cluster-wide basis if you set `indices.lifecycle.rollover.only_if_has_documents` to `false`.
	* Rollover for an empty write index is skipped even if it has an associated `max_age` that would otherwise result in a rollover occurring. A policy can override this behavior if you set `min_docs: 0` in the rollover conditions. This can also be disabled on a cluster-wide basis if you set `indices.lifecycle.rollover.only_if_has_documents` to `false`.

[kilfoyle]

Suggested change

	^1^ Rollover is handled automatically in Serverless projects, therefore configuring rollover timing is abstracted from the user. {applies_to}`serverless: ga`
	^1^ Rollover is handled automatically in {{es-serverless}} projects, therefore configuring rollover timing is abstracted from the user. {applies_to}`serverless: ga`

I think I'd also remove therefore configuring rollover timing is abstracted from the user since we usually avoid speaking about "the user" in favour of addressing users directly.

[kilfoyle]

Suggested change

	We recommend using [data streams](../../data-store/data-streams.md) to manage time series data. When set up to use ILM policies that include rollover, data streams automatically manage the rotation of your indices. This ensures you can write to the data stream without additional configuration.
	We recommend using [data streams](../../data-store/data-streams.md) to manage time series data. When set up to use an ILM policy that includes rollover, a data stream automatically manages the rotation of your indices. This ensures you can write to the data stream without additional configuration.

Maybe use singular for simplicity and to match the next sentence at line 65.

[kilfoyle]

Suggested change

	When targeting a data stream, the new backing index becomes the data stream's writing index. The generation of new backing indices is incremented automatically when it reaches a specified age or size.
	When you use a data stream, each time the current write index reaches a specified age or size, a new backing index is generated, with an incremented number and timestamp, and it becomes the data stream's writing index.

Sorry to belabour this one. Would this phrasing maybe be clearer? Feel free to ignore. :-)

[kilfoyle]

Suggested change

	For more information about data stream naming patterns, refer to the [Generation](../../data-store/data-streams.md#data-streams-generation) section of the Data streams page.
	For more information about the data stream naming pattern, refer to the [Generation](../../data-store/data-streams.md#data-streams-generation) section of the Data streams page.

[kilfoyle]

Is it really ^.-\d+$*? I would think ^.-\d+$. That is, can there be characters in the name following the end of line delimiter ($)?

We could also say something more easily readable like "must match the pattern <INDEX_NAME>-<INDEX_NUMBER>, for example my-index-000001"

[kilfoyle]

Very nice work on this @yetanothertw!