|
| 1 | +[role="xpack"] |
| 2 | +[[rollup-migrating-to-downsampling]] |
| 3 | +=== Migrating from {rollup-cap} to downsampling |
| 4 | +++++ |
| 5 | +<titleabbrev>Migrating to downsampling</titleabbrev> |
| 6 | +++++ |
| 7 | + |
| 8 | +Rollup and downsampling are two different features that allow historical metrics to be rolled up. |
| 9 | +From a high level rollup is more flexible compared to downsampling, but downsampling is a more robust and |
| 10 | +easier feature to downsample metrics. |
| 11 | + |
| 12 | +The following aspects of downsampling are easier or more robust: |
| 13 | + |
| 14 | +* No need to schedule jobs. Downsampling is integrated with Index Lifecycle Management (ILM) and Data Stream Lifecycle (DSL). |
| 15 | +* No separate search API. Downsampled indices can be accessed via the search api and es|ql. |
| 16 | +* No separate rollup configuration. Downsampling uses the time series dimension and metric configuration from the mapping. |
| 17 | + |
| 18 | +It isn't possible to migrate all rollup usages to downsampling. The main requirement |
| 19 | +is that the data should be stored in Elasticsearch as <<tsds,time series data stream (TSDS)>>. |
| 20 | +Rollup usages that basically roll the data up by time and all dimensions can migrate to downsampling. |
| 21 | + |
| 22 | +An example rollup usage that can be migrated to downsampling: |
| 23 | + |
| 24 | +[source,console] |
| 25 | +-------------------------------------------------- |
| 26 | +PUT _rollup/job/sensor |
| 27 | +{ |
| 28 | + "index_pattern": "sensor-*", |
| 29 | + "rollup_index": "sensor_rollup", |
| 30 | + "cron": "0 0 * * * *", <1> |
| 31 | + "page_size": 1000, |
| 32 | + "groups": { <2> |
| 33 | + "date_histogram": { |
| 34 | + "field": "timestamp", |
| 35 | + "fixed_interval": "60m" <3> |
| 36 | + }, |
| 37 | + "terms": { |
| 38 | + "fields": [ "node" ] |
| 39 | + } |
| 40 | + }, |
| 41 | + "metrics": [ |
| 42 | + { |
| 43 | + "field": "temperature", |
| 44 | + "metrics": [ "min", "max", "sum" ] <4> |
| 45 | + }, |
| 46 | + { |
| 47 | + "field": "voltage", |
| 48 | + "metrics": [ "avg" ] <4> |
| 49 | + } |
| 50 | + ] |
| 51 | +} |
| 52 | +-------------------------------------------------- |
| 53 | +// TEST[setup:sensor_index] |
| 54 | + |
| 55 | +The equivalent <<tsds,time series data stream (TSDS)>> setup that uses downsampling via DSL: |
| 56 | + |
| 57 | +[source,console] |
| 58 | +-------------------------------------------------- |
| 59 | +PUT _index_template/sensor-template |
| 60 | +{ |
| 61 | + "index_patterns": ["sensor-*"], |
| 62 | + "data_stream": { }, |
| 63 | + "template": { |
| 64 | + "lifecycle": { |
| 65 | + "downsampling": [ |
| 66 | + { |
| 67 | + "after": "1d", <1> |
| 68 | + "fixed_interval": "1h" <3> |
| 69 | + } |
| 70 | + ] |
| 71 | + }, |
| 72 | + "settings": { |
| 73 | + "index.mode": "time_series" |
| 74 | + }, |
| 75 | + "mappings": { |
| 76 | + "properties": { |
| 77 | + "node": { |
| 78 | + "type": "keyword", |
| 79 | + "time_series_dimension": true <2> |
| 80 | + }, |
| 81 | + "temperature": { |
| 82 | + "type": "half_float", |
| 83 | + "time_series_metric": "gauge" <4> |
| 84 | + }, |
| 85 | + "voltage": { |
| 86 | + "type": "half_float", |
| 87 | + "time_series_metric": "gauge" <4> |
| 88 | + }, |
| 89 | + "@timestamp": { <2> |
| 90 | + "type": "date" |
| 91 | + } |
| 92 | + } |
| 93 | + } |
| 94 | + } |
| 95 | +} |
| 96 | +-------------------------------------------------- |
| 97 | +// TEST[continued] |
| 98 | + |
| 99 | +//// |
| 100 | +[source,console] |
| 101 | +---- |
| 102 | +DELETE _index_template/sensor-template |
| 103 | +---- |
| 104 | +// TEST[continued] |
| 105 | +//// |
| 106 | + |
| 107 | +The downsample configuration is included in the above template for a <<tsds,time series data stream (TSDS)>>. |
| 108 | +Only the `downsampling` part is necessary to enable downsampling, which indicates when to downsample to what fixed interval. |
| 109 | + |
| 110 | +<1> In the rollup job, the `cron` field determines when the rollup documents. In the index template, |
| 111 | + the `after` field determines when downsampling will rollup documents (note that this the time after a rollover has been performed). |
| 112 | +<2> In the rollup job, the `groups` field determines all dimensions of the group documents are rolled up to. In the index template, |
| 113 | + the fields with `time_series_dimension` set `true` and the `@timestamp` field determine the group. |
| 114 | +<3> In the rollup job, the `fixed_interval` field determines how timestamps are aggregated as part of the grouping. |
| 115 | + In the index template, the `fixed_interval` field has the same purpose. Note that downsampling does not support calendar intervals. |
| 116 | +<4> In the rollup job, the `metrics` field define the metrics and how to store these metrics. In the index template, |
| 117 | + all fields with a `time_series_metric` are metric fields. If a field has `gauge` as `time_series_metric` attribute |
| 118 | + value, then min, max, sum and value counts are stored for this field in the downsampled index. If a field has |
| 119 | + `counter` as `time_series_metric` attribute value, then only the last value stored for this field in the downsampled |
| 120 | + index. |
0 commit comments