Skip to content

Commit ce1f0ea

Browse files
yeya24yeya24
authored
Add proposal for parquet storage (#6712)
* add proposal for parquet storage Signed-off-by: Ben Ye <[email protected]> * typo fix Signed-off-by: Ben Ye <[email protected]> * address comments and update bucket index Signed-off-by: Ben Ye <[email protected]> * mention how chunks work Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]>
1 parent f9ad30b commit ce1f0ea

File tree

1 file changed

+139
-0
lines changed

1 file changed

+139
-0
lines changed

‎docs/proposals/parquet-storage.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
---
2+
title: "Parquet-based Storage"
3+
linkTitle: "Parquet-based Storage"
4+
weight: 1
5+
slug: parquet-storage
6+
---
7+
8+
- Author: [Alan Protasio](https://github.com/alanprot), [Ben Ye](https://github.com/yeya24)
9+
- Date: April 2025
10+
- Status: Proposed
11+
12+
## Background
13+
14+
Since the introduction of Block Storage in Cortex, TSDB format and Store Gateway is the de-facto way to query long term data on object storage. However, it presents several significant challenges:
15+
16+
### TSDB Format Limitations
17+
18+
TSDB format, while efficient for write-heavy workloads on local SSDs, is not designed for object storage:
19+
- Index relies heavily on random reads to serve queries, where each random read becomes a request to object store
20+
- In order to reduce requests to object store, requests needs to be merged, leading to higher overfetch
21+
- Index relies on postings, which can be a huge bottleneck for high cardinality data
22+
23+
### Store Gateway Operational Challenges
24+
25+
Store Gateway is originally introduced in [Thanos](https://thanos.io/). Both Cortex and Thanos community have been collaborating to add a lot of optimizations to Store Gateway. However, it has its own problems related to the design.
26+
27+
1. Resource Intensive
28+
- Requires significant local disk space to store index headers
29+
- High memory utilization due to index header mmap
30+
- Often needs over-provisioning to handle query spikes
31+
32+
2. State Management and Scaling Difficulties
33+
- Requires complex data sharding when scaling. Often causing issues such as consistency check failure. Hard to configure for users
34+
- Initial sync causes long startup time. This affects service availability on both scaling and failure recovery scenario
35+
36+
3. Query Inefficiencies
37+
- Attempts to minimize storage requests often lead to overfetching, causing high bandwidth usage
38+
- Complex caching logic with varying effectiveness. Latency varies a lot when cache miss
39+
- Processes single block with one goroutine, leading to high latency for large blocks and cannot scale without complex data partitioning
40+
41+
### Why Parquet?
42+
43+
[Apache Parquet](https://parquet.apache.org/) is a columnar storage format designed specifically for efficient data storage and retrieval from object storage systems. It offers several key advantages that directly address the problems we face with TSDB and Store Gateway:
44+
45+
- Data organized by columns rather than rows, reduces number of requests to object storage as only limited IO is required to fetch the whole column
46+
- Rich file metadata and index, no local state like index header required to query the data, making it stateless
47+
- Advanced compression techniques reduce storage costs and improve query performance
48+
- Parallel processing friendly using Parquet Row Group
49+
50+
There are other benefits of Parquet formats, but they are not directly related to the proposal:
51+
52+
- Wide ecosystem and tooling support
53+
- Column pruning opportunity using projection pushdown
54+
55+
## Out of Scope
56+
57+
- Allow Ingester and Compactor to create Parquet files instead of TSDB blocks directly. This could be in the future roadmap but this proposal only focuses on converting and querying Parquet files.
58+
59+
## Proposed Design
60+
61+
### Components
62+
63+
There are 2 new Cortex components/modules introduced in this design.
64+
65+
#### 1. Parquet Converter
66+
67+
Parquet Converter is a new component that converts TSDB blocks on object store to Parquet file format.
68+
69+
It is similar to compactor, however, it only converts single block. The converted Parquet files will be stored in the same TSDB block folder so that the lifecycle of Parquet file will be managed together with the block.
70+
71+
Only certain blocks can be configured to convert to Parquet file and it can be block duration based, for example we only convert if block duration is >= 12h.
72+
73+
#### 2. Parquet Queryable
74+
75+
Similar to the existing `distributorQueryable` and `blockStorageQueryable`, Parquet queryable is a queryable implementation which allows Cortex to query parquet files and can be used in both Cortex Querier and Ruler.
76+
77+
If Parquet queryable is enabled, block storage queryable will be disabled and Cortex querier will not query Store Gateway anymore. `distributorQueryable` remains unchanged so it still queries Ingesters.
78+
79+
Parquet queryable uses bucket index to discovers parquet files in object storage. The bucket index is the same as the existing TSDB bucket index file, but using a different name `bucket-index-parquet.json.gz`. It is updated periodically by Cortex Compactor/Parquet Converter if parquet storage is enabled.
80+
81+
Cortex querier remains a stateless component when Parquet queryable is enabled.
82+
83+
### Architecture
84+
85+
```
86+
┌──────────┐ ┌─────────────┐ ┌──────────────┐
87+
│ Ingester │───>│ TSDB │───>│ Parquet │
88+
└──────────┘ │ Blocks │ │ Converter │
89+
└─────────────┘ └──────────────┘
90+
│
91+
v
92+
┌──────────┐ ┌─────────────┐ ┌──────────────┐
93+
│ Query │───>│ Parquet │───>│ Parquet │
94+
│ Frontend │ │ Querier │ │ Files │
95+
└──────────┘ └─────────────┘ └──────────────┘
96+
```
97+
98+
### Data Format
99+
100+
Parquet file is converted from TSDB block so it follows the same time range constraint.
101+
102+
If the largest block is 1 day then parquet file can go up to 1 day. Max block range is configurable in Cortex but default value is 24h. So following schema will use 24h as example.
103+
104+
#### Schema Overview
105+
106+
The Parquet format consists of two types of files:
107+
108+
1. **Labels Parquet File**
109+
- Each row represents a unique time series
110+
- Each column corresponds to a label name (e.g., `__name__`, `label1`, ..., `labelN`)
111+
- Row groups are sorted by `__name__` alphabetically in ascending order
112+
113+
2. **Chunks Parquet File**
114+
- Maintains row and row group order matching the Labels file
115+
- Contains multiple chunk columns for time-series data ordered by time. With 3 chunk columns for example, each column covers 8h of chunks: 0-8h, 8h-16h, 16-24h. It is possible that a single TSDB chunk spans over time ranges of 2 columns and Parquet file writer needs to split and re-encode chunks for each chunk column.
116+
117+
#### Column Specifications
118+
119+
| Column Name | Description | Type | Encoding/Compression/skipPageBounds | Required |
120+
|------------|-------------|------|-----------------------------------|-----------|
121+
| `s_hash` | Hash of all labels | INT64 | None/Zstd/Yes | No |
122+
| `s_col_indexes` | Bitmap indicating which columns store the label set for this row (series) | ByteArray (bitmap) | DeltaByteArray/Zstd/Yes | Yes |
123+
| `s_lbl_{labelName}` | Values for a given label name. Rows are sorted by metric name | ByteArray (string) | RLE_DICTIONARY/Zstd/No | Yes |
124+
| `s_data_{n}` | Chunks columns (0 to data_cols_count). Each column contains data from `[n*duration, (n+1)*duration]` where duration is `24h/data_cols_count` | ByteArray (encoded chunks) | DeltaByteArray/Zstd/Yes | Yes |
125+
126+
data_cols_count will be a parquet file metadata and its value is default to 3 but it can be configurable to adjust for different usecases.
127+
128+
## Open Questions
129+
130+
1. Should we use Parquet Gateway to replace Store Gateway
131+
- Separate query engine and storage
132+
- We can make Parquet Gateway semi-stateful like data locality for better performance
133+
134+
## Acknowledgement
135+
136+
We'd like to give huge credits for people from the Thanos community who started this initiative.
137+
138+
- [Filip Petkovski](https://github.com/fpetkovski) and his initial [talk about Parquet](https://www.youtube.com/watch?v=V8Y4VuUwg8I)
139+
- [Michael Hoffmann](https://github.com/MichaHoffmann) and his great work of [parquet poc](https://github.com/cloudflare/parquet-tsdb-poc)

0 commit comments

Comments
 (0)