@@ -15,7 +15,7 @@ relevant considerations in this guide to improve performance. It also helps to
15
15
understand how {transforms} work as different considerations apply depending on
16
16
whether or not your transform is running in continuous mode or in batch.
17
17
18
- In this guide, you’ ll learn how to:
18
+ In this guide, you' ll learn how to:
19
19
20
20
* Understand the impact of configuration options on the performance of
21
21
{transforms}.
@@ -111,10 +111,17 @@ group of IPs, in order to calculate the total `bytes_sent`. If this second
111
111
search matches many shards, then this could be resource intensive. Consider
112
112
limiting the scope that the source index pattern and query will match.
113
113
114
- Use an absolute time value as a date range filter in your source query (for
115
- example, greater than `2020-01-01T00:00:00`) to limit which historical indices
116
- are accessed. If you use a relative time value (for example, `now-30d`) then
117
- this date range is re-evaluated at the point of each checkpoint execution.
114
+ To limit which historical indices are accessed, exclude certain tiers (for
115
+ example `"must_not": { "terms": { "_tier": [ "data_frozen", "data_cold" ] } }`
116
+ and/or use an absolute time value as a date range filter in your source query
117
+ (for example, greater than 2024-01-01T00:00:00). If you use a relative time
118
+ value (for example, gte now-30d/d) then ensure date rounding is applied to take
119
+ advantage of query caching and ensure that the relative time is much larger than
120
+ the largest of `frequency` or `time.sync.delay` or the date histogram bucket,
121
+ otherwise data may be missed. Do not use date filters which are less than a date
122
+ value (for example, `lt`: less than or `lte`: less than or equal to) as this
123
+ conflicts with the logic applied at each checkpoint execution and data may be
124
+ missed.
118
125
119
126
Consider using <<api-date-math-index-names,date math>> in your index names to
120
127
reduce the number of indices to resolve in your queries. Add a date pattern
0 commit comments