11# tscodec-go
22
3- High-performance timeseries compression codecs for Go, featuring adaptive algorithms optimized for modern CPU architectures.
3+ High-performance timeseries compression codecs for Go, featuring adaptive algorithms optimized for modern CPU
4+ architectures.
45
56## Features
67
78This library implements several state-of-the-art compression algorithms for timeseries data:
89
9- - ** ALP (Adaptive Lossless floating-Point)** - Lossless compression for float64 values using adaptive scaling and bit-packing
10+ - ** ALP (Adaptive Lossless floating-Point)** - Lossless compression for float64 values using adaptive scaling and
11+ bit-packing
1012- ** Delta Encoding** - First-order delta encoding for int32/int64 values
1113- ** Delta-of-Delta (DoD)** - Second-order delta encoding for regular timeseries
1214- ** Bitpacking** - Low-level bit manipulation with architecture-specific optimizations (amd64, arm64)
@@ -33,14 +35,15 @@ alp.Decode(decodedValues[:], compressedValues)
3335
3436Performance comparison vs Gorilla (XOR) compression from Prometheus (Apple M3, 120 samples):
3537
36- | Codec | Operation | Time/op | Throughput | Compressed Size | Allocs |
37- | -------| -----------| ---------| ------------| -----------------| --------|
38- | Gorilla | Encode | 3321 ns/op | - | 982 bytes | 7 allocs/op |
39- | Gorilla | Decode | 1715 ns/op | - | - | 1 allocs/op |
40- | ALP+DoD | Encode | ** 1406 ns/op** | ** 2.4x faster** | ** 840 bytes** | 6 allocs/op |
41- | ALP+DoD | Decode | ** 252 ns/op** | ** 6.8x faster, 3330 MB/s** | - | ** 0 allocs/op** |
38+ | Codec | Operation | Time/op | Throughput | Compressed Size | Allocs |
39+ | --------- | -----------| ---------------- | ------------------------------- | -----------------| --------- --------|
40+ | Gorilla | Encode | 3321 ns/op | 801.99 MB/s | 982 bytes | 7 allocs/op |
41+ | Gorilla | Decode | 1715 ns/op | 588.06 MB/s | - | 1 allocs/op |
42+ | ALP+DoD | Encode | ** 1406 ns/op** | ** 2.4x faster, 1390.80 MB/s ** | ** 840 bytes** | 6 allocs/op |
43+ | ALP+DoD | Decode | ** 252 ns/op** | ** 6.8x faster, 3330 MB/s** | - | ** 0 allocs/op** |
4244
4345Run benchmarks:
46+
4447``` bash
4548cd benchmarks
4649go test -bench=BenchmarkFloats -benchmem
@@ -62,24 +65,24 @@ go get github.com/fpetkovski/tscodec-go
6265package main
6366
6467import (
65- " fmt"
66- " github.com/fpetkovski/tscodec-go/alp"
68+ " fmt"
69+ " github.com/fpetkovski/tscodec-go/alp"
6770)
6871
6972func main () {
70- // Original data
71- data := []float64 {1.1 , 2.2 , 3.3 , 4.4 , 5.5 }
72- // Compress
73+ // Original data
74+ data := []float64 {1.1 , 2.2 , 3.3 , 4.4 , 5.5 }
75+ // Compress
7376 compressed := make ([]byte , 10 )
7477 compressed = alp.Encode (compressed, data)
7578
76- // Decompress
77- decompressed := make ([]float64 , len (data))
78- alp.Decode (decompressed, compressed)
79+ // Decompress
80+ decompressed := make ([]float64 , len (data))
81+ alp.Decode (decompressed, compressed)
7982
80- // Calculate compression ratio
81- ratio := alp.CompressionRatio (len (data), len (compressed))
82- fmt.Printf (" Compression ratio: %.2f%% \n " , ratio*100 )
83+ // Calculate compression ratio
84+ ratio := alp.CompressionRatio (len (data), len (compressed))
85+ fmt.Printf (" Compression ratio: %.2f%% \n " , ratio*100 )
8386}
8487```
8588
@@ -89,20 +92,20 @@ func main() {
8992package main
9093
9194import (
92- " github.com/fpetkovski/tscodec-go/delta"
95+ " github.com/fpetkovski/tscodec-go/delta"
9396)
9497
9598func main () {
96- // Original data
97- data := []int64 {1000 , 1001 , 1002 , 1003 , 1004 }
99+ // Original data
100+ data := []int64 {1000 , 1001 , 1002 , 1003 , 1004 }
98101
99- // Compress
100- compressed := make ([]byte , 0 )
101- compressed = delta.EncodeInt64 (compressed, data)
102+ // Compress
103+ compressed := make ([]byte , 0 )
104+ compressed = delta.EncodeInt64 (compressed, data)
102105
103- // Decompress
104- decompressed := make ([]int64 , len (data))
105- delta.DecodeInt64 (decompressed, compressed)
106+ // Decompress
107+ decompressed := make ([]int64 , len (data))
108+ delta.DecodeInt64 (decompressed, compressed)
106109}
107110```
108111
@@ -117,6 +120,7 @@ ALP achieves high compression ratios for float64 data through:
1171203 . ** Bit-packing** - Packs values using minimal bit width
118121
119122** Best for:**
123+
120124- Sensor data with limited precision
121125- Price data with 2-4 decimal places
122126- Sequential patterns
@@ -129,6 +133,7 @@ ALP achieves high compression ratios for float64 data through:
129133Encodes differences between consecutive values instead of absolute values.
130134
131135** Best for:**
136+
132137- Monotonically increasing sequences (timestamps, counters)
133138- Values with small differences between consecutive elements
134139
@@ -137,6 +142,7 @@ Encodes differences between consecutive values instead of absolute values.
137142Applies delta encoding twice, encoding the difference of differences.
138143
139144** Best for:**
145+
140146- Highly regular timeseries (e.g., evenly-spaced timestamps)
141147- Data with constant or near-constant rate of change
142148
0 commit comments