Skip to content

Commit 2b8117d

Browse files
andy-stark-redisandy-stark-redis
committed
DOC-5424 added aggregation and compaction info
1 parent ccd18b7 commit 2b8117d

File tree

1 file changed

+213
-65
lines changed
  • content/develop/data-types/timeseries

1 file changed

+213
-65
lines changed

‎content/develop/data-types/timeseries/_index.md‎

Lines changed: 213 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -350,27 +350,74 @@ OK
350350
```
351351

352352
## Aggregation
353-
It's possible to combine values of one or more timeseries by leveraging aggregation functions:
354-
```
355-
TS.RANGE ... AGGREGATION aggType bucketDuration...
356-
```
357353

358-
For example, to find the average temperature per hour in our `sensor1` series we could run:
359-
```
360-
TS.RANGE sensor1 - + + AGGREGATION avg 3600000
361-
```
354+
A time series can become large if samples are added very frequently. Instead
355+
of dealing with individual samples, it is sometimes useful to split the full
356+
time range of the series into equal-sized "buckets" and represent each
357+
bucket by an aggregate value, such as the average or maximum value.
362358

363-
To achieve the same across multiple sensors from the area with id of 32 we would run:
364-
```
365-
TS.MRANGE - + AGGREGATION avg 3600000 FILTER area_id=32
366-
```
359+
For example, if you expect to collect more than one billion data points in a day, you could aggregate the data using buckets of one minute. Since each bucket is represented by a single value, this reduces
360+
the dataset size to 1,440 data points (24 hours x 60 minutes = 1,440 minutes).
361+
362+
The range query commands let you specify an aggregation function and bucket size.
363+
The available aggregation functions are:
367364

368-
### Aggregation bucket alignment
369-
When doing aggregations, the aggregation buckets will be aligned to 0 as so:
365+
- `avg`: Arithmetic mean of all values
366+
- `sum`: Sum of all values
367+
- `min`: Minimum value
368+
- `max`: Maximum value
369+
- `range`: Difference between the highest and the lowest value
370+
- `count`: Number of values
371+
- `first`: Value with lowest timestamp in the bucket
372+
- `last`: Value with highest timestamp in the bucket
373+
- `std.p`: Population standard deviation of the values
374+
- `std.s`: Sample standard deviation of the values
375+
- `var.p`: Population variance of the values
376+
- `var.s`: Sample variance of the values
377+
- `twa`: Time-weighted average over the bucket's timeframe (since RedisTimeSeries v1.8)
378+
379+
For example, the example below shows an aggregation with the `avg` function over all
380+
five data points in the `rg:2` time series. The bucket size is two days, so there are three
381+
aggregated values with only one value used to calculate the average for the last bucket.
382+
383+
```bash
384+
> TS.RANGE rg:2 - + AGGREGATION avg 2
385+
1) 1) (integer) 0
386+
2) 1.9500000000000002
387+
2) 1) (integer) 2
388+
2) 2.0999999999999996
389+
3) 1) (integer) 4
390+
2) 1.78
370391
```
371-
TS.RANGE sensor3 10 70 + AGGREGATION min 25
392+
393+
### Bucket alignment
394+
395+
The sequence of buckets has a reference timestamp, which is the timestamp where
396+
the first bucket in the sequence starts. By default, the reference timestamp is zero.
397+
For example, the following commands create a time series and apply a `min` aggregation
398+
with a bucket size of 25 milliseconds at the default zero alignment.
399+
400+
```bash
401+
> TS.MADD sensor3 10 1000 sensor3 20 2000 sensor3 30 3000 sensor3 40 4000 sensor3 50 5000 sensor3 60 6000 sensor3 70 7000
402+
1) (integer) 10
403+
2) (integer) 20
404+
3) (integer) 30
405+
4) (integer) 40
406+
5) (integer) 50
407+
6) (integer) 60
408+
7) (integer) 70
409+
> TS.RANGE sensor3 10 70 + AGGREGATION min 25
410+
1) 1) (integer) 0
411+
2) 1000
412+
2) 1) (integer) 25
413+
2) 3000
414+
3) 1) (integer) 50
415+
2) 5000
372416
```
373417

418+
The diagram below shows the aggregation buckets and their alignment to the reference timestamp
419+
at time zero.
420+
374421
```
375422
Value: | (1000) (2000) (3000) (4000) (5000) (6000) (7000)
376423
Timestamp: |-------|10|-------|20|-------|30|-------|40|-------|50|-------|60|-------|70|--->
@@ -380,13 +427,20 @@ Bucket(25ms): |_________________________||_________________________||___________
380427
min(1000, 2000)=1000 min(3000, 4000)=3000 min(5000, 6000, 7000)=5000
381428
```
382429

383-
And we will get the following datapoints: 1000, 3000, 5000.
430+
You can also align the buckets to the start or end of the query range. For example, the following command aligns the buckets to the start of the query range at time 10.
384431

385-
You can choose to align the buckets to the start or end of the queried interval as so:
386-
```
387-
TS.RANGE sensor3 10 70 + AGGREGATION min 25 ALIGN start
432+
```bash
433+
> TS.RANGE sensor3 10 70 + AGGREGATION min 25 ALIGN start
434+
1) 1) (integer) 10
435+
2) 1000
436+
2) 1) (integer) 35
437+
2) 4000
438+
3) 1) (integer) 60
439+
2) 6000
388440
```
389441

442+
The diagram below shows this arrangement of buckets.
443+
390444
```
391445
Value: | (1000) (2000) (3000) (4000) (5000) (6000) (7000)
392446
Timestamp: |-------|10|-------|20|-------|30|-------|40|-------|50|-------|60|-------|70|--->
@@ -395,68 +449,158 @@ Bucket(25ms): |__________________________||_________________________||_
395449
V V V
396450
min(1000, 2000, 3000)=1000 min(4000, 5000)=4000 min(6000, 7000)=6000
397451
```
398-
The result array will contain the following datapoints: 1000, 4000 and 6000
399-
400452

401453
### Aggregation across timeseries
402454

403-
By default, results of multiple timeseries will be grouped by timeseries, but (since v1.6) you can use the `GROUPBY` and `REDUCE` options to group them by label and apply an additional aggregation.
455+
By default, the results from
456+
[`TS.MRANGE`]({{< relref "commands/ts.mrange/" >}}) and
457+
[`TS.MREVRANGE`]({{< relref "commands/ts.mrevrange/" >}}) are grouped by time series. However, you can use the `GROUPBY` and `REDUCE` options to group them by label and apply an aggregation over elements
458+
that have the same timestamp and the same label value (this feature is available from RedisTimeSeries v1.6 onwards).
404459

405-
To find minimum temperature per region, for example, we can run:
460+
For example, the following commands create four time series, two for the UK and two for the US, and add some data points. The first `TS.MRANGE` command groups the results by country and applies a `max` aggregation to find the maximum wind speed in each country at each timestamp. The second `TS.MRANGE` command uses the same grouping, but applies an `avg` aggregation.
406461

407-
```
408-
TS.MRANGE - + FILTER region=(east,west) GROUPBY region REDUCE min
409-
```
462+
```bash
463+
> TS.CREATE wind:1 LABELS country uk
464+
OK
465+
> TS.CREATE wind:2 LABELS country uk
466+
OK
467+
> TS.CREATE wind:3 LABELS country us
468+
OK
469+
> TS.CREATE wind:4 LABELS country us
470+
OK
471+
> TS.MADD wind:1 1 12 wind:2 1 18 wind:3 1 5 wind:4 1 20
472+
1) (integer) 1
473+
2) (integer) 1
474+
3) (integer) 1
475+
4) (integer) 1
476+
> TS.MADD wind:1 2 14 wind:2 2 21 wind:3 2 4 wind:4 2 25
477+
1) (integer) 2
478+
2) (integer) 2
479+
3) (integer) 2
480+
4) (integer) 2
481+
> TS.MADD wind:1 3 10 wind:2 3 24 wind:3 3 8 wind:4 3 18
482+
1) (integer) 3
483+
2) (integer) 3
484+
3) (integer) 3
485+
4) (integer) 3
410486

411-
**Note:** When a sample is deleted, the data in all downsampled timeseries will be recalculated for the specific bucket. If part of the bucket has already been removed though, because it's outside of the retention period, we won't be able to recalculate the full bucket, so in those cases we will refuse the delete operation.
487+
# The result pairs contain the timestamp and the maximum wind speed
488+
# for the country at that timestamp.
489+
> TS.MRANGE - + FILTER country=(us,uk) GROUPBY country REDUCE max
490+
1) 1) "country=uk"
491+
2) (empty array)
492+
3) 1) 1) (integer) 1
493+
2) 18
494+
2) 1) (integer) 2
495+
2) 21
496+
3) 1) (integer) 3
497+
2) 24
498+
2) 1) "country=us"
499+
2) (empty array)
500+
3) 1) 1) (integer) 1
501+
2) 20
502+
2) 1) (integer) 2
503+
2) 25
504+
3) 1) (integer) 3
505+
2) 18
412506

413-
## Compaction
507+
# The result pairs contain the timestamp and the average wind speed
508+
# for the country at that timestamp.
509+
> TS.MRANGE - + FILTER country=(us,uk) GROUPBY country REDUCE avg
510+
1) 1) "country=uk"
511+
2) (empty array)
512+
3) 1) 1) (integer) 1
513+
2) 15
514+
2) 1) (integer) 2
515+
2) 17.5
516+
3) 1) (integer) 3
517+
2) 17
518+
2) 1) "country=us"
519+
2) (empty array)
520+
3) 1) 1) (integer) 1
521+
2) 12.5
522+
2) 1) (integer) 2
523+
2) 14.5
524+
3) 1) (integer) 3
525+
2) 13
526+
```
414527

415-
A time series can become large if samples are added very frequently. Instead
416-
of dealing with individual samples, it is sometimes useful to split the full
417-
time range of the series into equal-sized "buckets" and represent each
418-
bucket by an aggregate value, such as the average or maximum value.
419-
Reducing the number of data points in this way is known as *compaction*.
528+
## Compaction
420529

421-
For example, if you expect to collect more than one billion data points in a day, you could aggregate the data using buckets of one minute. Since each bucket is represented by a single value, this compacts the dataset size to 1,440 data points (24 hours x 60 minutes = 1,440 minutes).
530+
Aggregation queries let you extract the important information from a large data set
531+
into a smaller, more manageable set. If you are continually adding new data to a
532+
time series as it is generated, you may need to run the same aggregation
533+
regularly on the latest data. Instead of running the query manually
534+
each time, you can add a *compaction rule* to a time series to compute an
535+
aggregation incrementally on data as it arrives. The values from the
536+
aggregation buckets are then added to a separate time series, leaving the original
537+
series unchanged.
422538

423539
Use [`TS.CREATERULE`]({{< relref "commands/ts.createrule/" >}}) to create a
540+
compaction rule, specifying the source and destination time series keys, the
541+
aggregation function, and the bucket duration. Note that the destination time
542+
series must already exist when you create the rule and also that the compaction will
543+
only process data that is added to the source series after you create the rule.
424544

425-
new
426-
compacted time series from an existing one, leaving the original series unchanged.
427-
Specify a duration for each bucket and an aggregation function to apply to each bucket.
428-
The available aggregation functions are:
429-
430-
- `avg`: Arithmetic mean of all values
431-
- `sum`: Sum of all values
432-
- `min`: Minimum value
433-
- `max`: Maximum value
434-
- `range`: Difference between the highest and the lowest value
435-
- `count`: Number of values
436-
- `first`: Value with lowest timestamp in the bucket
437-
- `last`: Value with highest timestamp in the bucket
438-
- `std.p`: Population standard deviation of the values
439-
- `std.s`: Sample standard deviation of the values
440-
- `var.p`: Population variance of the values
441-
- `var.s`: Sample variance of the values
442-
- `twa`: Time-weighted average over the bucket's timeframe (since RedisTimeSeries v1.8)
443-
444-
It's important to point out that there is no data rewriting on the original timeseries; the compaction happens in a new series, while the original one stays the same. In order to prevent the original timeseries from growing indefinitely, you can use the retention option, which will trim it down to a certain period of time.
545+
For example, you could use the commands below to create a time series for
546+
[hygrometer](https://en.wikipedia.org/wiki/Hygrometer) readings along with a compaction
547+
rule to find the minimum reading in each three day period.
445548

446-
**NOTE:** You need to create the destination (the compacted) timeseries before creating the rule.
447-
448-
```
449-
TS.CREATERULE sourceKey destKey AGGREGATION aggregationType bucketDuration
549+
```bash
550+
# The source time series.
551+
> TS.CREATE hyg:1
552+
OK
553+
# The destination time series for the compacted data.
554+
> TS.CREATE hyg:compacted
555+
OK
556+
# The compaction rule.
557+
> TS.CREATERULE hyg:1 hyg:compacted AGGREGATION min 3
558+
OK
559+
> TS.INFO hyg:1
560+
.
561+
.
562+
23) rules
563+
24) 1) 1) "hyg:compacted"
564+
2) (integer) 3
565+
3) MIN
566+
4) (integer) 0
567+
.
568+
.
569+
> TS.INFO hyg:compacted
570+
.
571+
.
572+
21) sourceKey
573+
22) "hyg:1"
574+
.
575+
.
450576
```
451577

452-
Example:
578+
Adding data points within the first three days (the first bucket) doesn't
579+
produce any data in the compacted series. However, when you add data for
580+
day 4 (in the second bucket), the compaction rule computes the minimum
581+
value for the first bucket and adds it to the compacted series.
453582

454-
```
455-
TS.CREATE sensor1_compacted # Create the destination timeseries first
456-
TS.CREATERULE sensor1 sensor1_compacted AGGREGATION avg 60000 # Create the rule
583+
```bash
584+
> TS.MADD hyg:1 0 75 hyg:1 1 77 hyg:1 2 78
585+
1) (integer) 0
586+
2) (integer) 1
587+
3) (integer) 2
588+
> ts.range hyg:compacted - +
589+
(empty array)
590+
> TS.ADD hyg:1 3 79
591+
(integer) 3
592+
127.0.0.1:6379> ts.range hyg:compacted - +
593+
1) 1) (integer) 0
594+
2) 75
457595
```
458596

459-
With this creation rule, datapoints added to the `sensor1` timeseries will be grouped into buckets of 60 seconds (60000ms), averaged, and saved in the `sensor1_compacted` timeseries.
597+
The general strategy is that the rule does not add data to the
598+
compaction for the latest bucket in the source series, but will add and
599+
update the compacted data for any previous buckets. This reflects the
600+
typical usage pattern of adding data samples sequentially in real time.
601+
Note that earlier buckets are not "closed" when you add data to a later
602+
bucket. If you add or [delete](#deleting-data-points) data in a bucket before
603+
the latest one, thecompaction rule will update the compacted data for that bucket.
460604

461605
## Deleting data points
462606

@@ -514,10 +658,14 @@ If you want to delete a single timestamp, use it as both the start and end of th
514658

515659
## Using with other metrics tools
516660

517-
In the [RedisTimeSeries](https://github.com/RedisTimeSeries) GitHub organization you can
661+
In the [RedisTimeSeries](https://github.com/RedisTimeSeries) GitHub organization, you can
518662
find projects that help you integrate RedisTimeSeries with other tools, including:
519663

520-
1. [Prometheus](https://github.com/RedisTimeSeries/prometheus-redistimeseries-adapter), read/write adapter to use RedisTimeSeries as backend db.
664+
1. [Prometheus](https://github.com/RedisTimeSeries/prometheus-redistimeseries-adapter), a read/write adapter to use RedisTimeSeries as the backend database.
521665
2. [Grafana 7.1+](https://github.com/RedisTimeSeries/grafana-redis-datasource), using the [Redis Data Source](https://redislabs.com/blog/introducing-the-redis-data-source-plug-in-for-grafana/).
522666
3. [Telegraf](https://github.com/influxdata/telegraf). Download the plugin from [InfluxData](https://portal.influxdata.com/downloads/).
523667
4. StatsD, Graphite exports using graphite protocol.
668+
669+
## More information
670+
671+
The other pages in this section describe RedisTimeSeries concepts in more detail:

0 commit comments

Comments
 (0)