Skip to content

Add practical tips for downsampling. #3340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
+28 −0
Open

Add practical tips for downsampling. #3340

Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ccceb1e
Add practical tips for downsampling
gmarouli Oct 6, 2025
542ef9a
Add ILM specific section
gmarouli Oct 7, 2025
1e32d3c
Fix link to migrate action
gmarouli Oct 7, 2025
69f7ddf
Apply review comments
gmarouli Oct 7, 2025
4dcb367
Merge branch 'main' into downsampling-practical-tips
gmarouli Oct 7, 2025
2c7aac1
Apply suggestions from code review
gmarouli Oct 7, 2025
566abe3
Rearrange ILM tips
gmarouli Oct 7, 2025
6de3d3d
Apply suggestions from code review
gmarouli Oct 8, 2025
3244f09
Tweaks
gmarouli Oct 8, 2025
abf557b
Merge branch 'main' into downsampling-practical-tips
gmarouli Oct 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions manage-data/data-store/data-streams/run-downsampling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,38 @@ Set `fixed_interval` to your preferred level of granularity. The original time s
:::
::::

## Practical tips

Downsampling requires reading and indexing the contents of a backing index. The following guidelines can help you get the most out of it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a note about rollover? To avoid creating backing indices that are too big..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been going back and forth for this. For ILM it's easy because it's part of the policy, for data stream lifecycle, I would suggest that if we really think that it should be less maybe we should set it to something less. Right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean, update the default? We can do that at a later point, but what about older versions, or ILM configurations with existing rollover overrides? It could still help to suggest a best practice here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we could update the default, that would apply on all version unless the user chose to overwrite it. I restructure it a bit so we can have ILM focused recommendations. But if we think it should be reduced, we should consider updating the default for DLM as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's file a tracking issue for this, so that we don't forget.

### Choosing the downsampling interval

When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%.
When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds, downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%.


The same applied when downsampling already downsampled data.

### Downsampling with Index Lifecycle Management

The following tips apply to data streams downsampled by index lifecycle management (ILM).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would delete this heading and intro sentence, and then promote the "Phases and tiers" and "Reduce the index size" headings to level 3, so that they're parallel with "choose an optimal downsampling interval"

(I suggested specific changes, just explaining why I'd delete these lines)

You can just make it clear in the text that those best practices are for ILM only, instead of having an ILM subsection

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Downsampling with Index Lifecycle Management
The following tips apply to data streams downsampled by index lifecycle management (ILM).
### Downsampling with Index Lifecycle Management
The following tips apply to data streams downsampled by index lifecycle management (ILM).


#### Phases and tiers

When using ILM, you can define at most one downsampling round in the following phases:

- `hot` phase: it will execute the downsampling after the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed
- `warm` phase: it will execute the downsampling `min_age` time after the rollover (respecting the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time))
- `cold` phase: it will execute the downsampling `min_age` time after the rollover (respecting the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time))

The phases do not require the respective tiers to exist. However, when a cluster has tiers, ILM automatically migrates the data processed in the phase to the respective tier. This can be disabled by adding the [migrate action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md#ilm-migrate-options) with `enabled: false`.

The migrate action is implicitly enabled, moving downsampling data to the respective tier. The downsampling operation occurs at the same tier as the source index and then the downsampled data gets migrated, allowing downsampling to leverage the resources of the "hotter" tier and move less data to the next tier.

#### Reducing index size

When configuring an ILM policy with downsampling, it is necessary to define the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) in the `hot` phase. The rollover action consists of the conditions that would trigger a rollover hence it determines the size of an index and its shards. The size of an index can influence the impact that downsampling has on a cluster's performance.

The downsampling operation runs over a whole index, so in certain cases downsampling can increase the load on a cluster. One of the ways to reduce that load is to reduce the size of the index; this way you can have smaller downsampling tasks that get better distributed. You can achieve that either by reducing the number of primary shards or by using setting [`max_primary_shard_docs`](https://www.elastic.co/docs/reference/elasticsearch/index-lifecycle-actions/ilm-rollover#ilm-rollover-options) to reduce the number of docs in a single shard. Using a lower value than the default of 200 million is expected to help smoothen load spikes due to downsampling.

## Additional resources

* [](downsampling-concepts.md)
Expand Down
Loading