Skip to content

Commit a78f070

Browse files
authored
[DOCS] updates transforms at scale doc with date rounding. (#109073) (#109076)
1 parent 1889ffb commit a78f070

File tree

1 file changed

+12
-5
lines changed

1 file changed

+12
-5
lines changed

docs/reference/transform/transforms-at-scale.asciidoc

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ relevant considerations in this guide to improve performance. It also helps to
1515
understand how {transforms} work as different considerations apply depending on
1616
whether or not your transform is running in continuous mode or in batch.
1717

18-
In this guide, youll learn how to:
18+
In this guide, you'll learn how to:
1919

2020
* Understand the impact of configuration options on the performance of
2121
{transforms}.
@@ -111,10 +111,17 @@ group of IPs, in order to calculate the total `bytes_sent`. If this second
111111
search matches many shards, then this could be resource intensive. Consider
112112
limiting the scope that the source index pattern and query will match.
113113

114-
Use an absolute time value as a date range filter in your source query (for
115-
example, greater than `2020-01-01T00:00:00`) to limit which historical indices
116-
are accessed. If you use a relative time value (for example, `now-30d`) then
117-
this date range is re-evaluated at the point of each checkpoint execution.
114+
To limit which historical indices are accessed, exclude certain tiers (for
115+
example `"must_not": { "terms": { "_tier": [ "data_frozen", "data_cold" ] } }`
116+
and/or use an absolute time value as a date range filter in your source query
117+
(for example, greater than 2024-01-01T00:00:00). If you use a relative time
118+
value (for example, gte now-30d/d) then ensure date rounding is applied to take
119+
advantage of query caching and ensure that the relative time is much larger than
120+
the largest of `frequency` or `time.sync.delay` or the date histogram bucket,
121+
otherwise data may be missed. Do not use date filters which are less than a date
122+
value (for example, `lt`: less than or `lte`: less than or equal to) as this
123+
conflicts with the logic applied at each checkpoint execution and data may be
124+
missed.
118125

119126
Consider using <<api-date-math-index-names,date math>> in your index names to
120127
reduce the number of indices to resolve in your queries. Add a date pattern

0 commit comments

Comments
 (0)