Skip to content

Add practical tips for downsampling. #3340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
+28 −0
Open

Add practical tips for downsampling. #3340

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ccceb1e
Add practical tips for downsampling
gmarouli Oct 6, 2025
542ef9a
Add ILM specific section
gmarouli Oct 7, 2025
1e32d3c
Fix link to migrate action
gmarouli Oct 7, 2025
69f7ddf
Apply review comments
gmarouli Oct 7, 2025
4dcb367
Merge branch 'main' into downsampling-practical-tips
gmarouli Oct 7, 2025
2c7aac1
Apply suggestions from code review
gmarouli Oct 7, 2025
566abe3
Rearrange ILM tips
gmarouli Oct 7, 2025
6de3d3d
Apply suggestions from code review
gmarouli Oct 8, 2025
3244f09
Tweaks
gmarouli Oct 8, 2025
abf557b
Merge branch 'main' into downsampling-practical-tips
gmarouli Oct 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions manage-data/data-store/data-streams/run-downsampling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,34 @@ Set `fixed_interval` to your preferred level of granularity. The original time s
:::
::::

## Best practices

This section provides some best practices for downsampling.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a note about rollover? To avoid creating backing indices that are too big..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been going back and forth for this. For ILM it's easy because it's part of the policy, for data stream lifecycle, I would suggest that if we really think that it should be less maybe we should set it to something less. Right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean, update the default? We can do that at a later point, but what about older versions, or ILM configurations with existing rollover overrides? It could still help to suggest a best practice here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we could update the default, that would apply on all version unless the user chose to overwrite it. I restructure it a bit so we can have ILM focused recommendations. But if we think it should be reduced, we should consider updating the default for DLM as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's file a tracking issue for this, so that we don't forget.

### Choose an optimal downsampling interval

When choosing the downsampling interval, make sure to consider the original sampling rate of your measurements. Use an interval that reduces the number of documents by a significant percentage. For example, if a sensor sends data every 10 seconds, downsampling to 1 minute would reduce the number of documents by 83%. Downsampling to 5 minutes instead would reduce the number by 96%.

The same applies when downsampling already downsampled data.

### Understand downsampling phases (ILM only)

When using [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md) (ILM), you can define at most one downsampling round in each of the following phases:

- `hot` phase: Runs after the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) passes
- `warm` phase: Runs after the `min_age` time (starting to count after the rollover and respecting the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this read well @marciw ? Considering that we want to stress that rollover marks "the start of time"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say "starting the count" (instead of "to count") but sure, that works! 👍 Feel free to apply this to my other suggestion for this spot

- `cold` phase: Runs after the `min_age` time (starting to count after the rollover and respecting the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time)

Phases don't require matching tiers. If a matching tier exists for the phase, ILM automatically migrates the data to the respective tier. To prevent this, add a [migrate action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md#ilm-migrate-options) and specify `enabled: false`.

If you leave the default migrate action enabled, downsampling runs on the tier of the source index, which typically has more resources. The smaller, downsampled data is then migrated to the next tier.

### Reduce the index size (ILM only)

When configuring an ILM policy with downsampling, use the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) in the `hot` phase to control index size. Using smaller indices helps to minimize the impact of downsampling on a cluster's performance.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's important here to say use and not define. When writing an ILM policy a user needs to define a rollover action no matter what. However, if they are using downsampling, they can consider using this to reduce the size of their index. I want us to be careful to not imply that a user needs to define it only if they are trying to reduce the size. @marciw does this make sense?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I see what you mean. we could try "set the rollover action to run in the hot phase"

("use" doesn't seem idiomatic to me here, so i'm hoping we can find an alternative)


Because the downsampling operation processes an entire index at once, it can increase the load on the cluster. Smaller indices improve task distribution. To reduce the index size, limit the number of primary shards or use [`max_primary_shard_docs`](https://www.elastic.co/docs/reference/elasticsearch/index-lifecycle-actions/ilm-rollover#ilm-rollover-options) to cap documents per shard. Specify a lower value than the default of 200 million, to help prevent load spikes due to downsampling.

## Additional resources

* [](downsampling-concepts.md)
Expand Down
Loading