Skip to content

Commit c2d3e1e

Browse files
committed
feat(dwh): update
1 parent 30af050 commit c2d3e1e

File tree

2 files changed

+126
-6
lines changed

2 files changed

+126
-6
lines changed

pages/data-warehouse/how-to/manage-storage.mdx

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,119 @@ dates:
77
posted: 2025-10-16
88
---
99

10+
## Overview
11+
12+
Scaleway Data Warehouse for ClickHouse® uses ClickHouse®'s native storage policies to decide where table data lives: on Scaleway Block storage (local PVC), and/or on Object Storage (Amazon S3). You can choose between three policies for your deployment:
13+
14+
- `tiered` (default in Scaleway Data Warehouse for ClickHouse®)
15+
- `s3_cache` (cache mode)
16+
- `default` (local-only mode)
17+
18+
The table below provides an overview of each storage policy's behavior and use case.
19+
20+
| Storage policy | Behavior | When to use |
21+
|------|---------|------------|
22+
| `tiered` | Writes land on Block storage first (hot volume). When the local disk reaches ~90% fullness, parts are automatically moved to Object storage (cold volume). This is controlled by `move_factor: 0.1` (move when only ~10% free remains). | General purpose setting for fast local writes and transparent spillover to Object Storage as the dataset grows. |
23+
| `s3_cache` | Data is mainly stored on Object Storage. A local cache layer (on Block storage) keeps frequently-read parts to accelerate repeated reads. | Large datasets that mostly live in Object Storage, with repeated reads on hot subsets that benefit from local caching. |
24+
| `default` | Data is stored on Block storage only. | Small datasets where lowest latency for reads/writes on local disk is desired and capacity fits the PVC. |
25+
26+
Under the hood, our ClickHouse config defines three disks (default = local PVC, s3 = object storage, s3_cache = S3 + local cache) and three policies (tiered, s3_cache, default).
27+
Cluster default (for new tables when not specified): tiered.
28+
29+
How clients choose a policy (per table)
30+
31+
The storage policy is chosen at table creation time via SETTINGS storage_policy = '...'.
32+
33+
## Tiered storage policy
34+
35+
**Tiered** (Block Storage first, spill to Object Storage) is the cluster default policy. If you omit the `storage_policy` setting; ClickHouse® will automatically apply this policy.
36+
37+
<Tabs id="tiered-explicit-implicit">
38+
<TabsTab label="Explicit">
39+
40+
Storage policy is explicitly defined as `tiered`.
41+
42+
```sql
43+
CREATE TABLE tiered_table
44+
(
45+
id UInt64,
46+
ts DateTime,
47+
value Float32
48+
)
49+
ENGINE = MergeTree
50+
PARTITION BY toYYYYMM(ts)
51+
ORDER BY id
52+
SETTINGS storage_policy = 'tiered';
53+
```
54+
55+
</TabsTab>
56+
<TabsTab label="Implicit">
57+
58+
Storage policy is not defined, ClickHouse® applies the `tiered` storage policy by default.
59+
60+
```sql
61+
CREATE TABLE dw.tiered_table_implicit
62+
(
63+
id UInt64,
64+
ts DateTime,
65+
value Float32
66+
)
67+
ENGINE = MergeTree
68+
PARTITION BY toYYYYMM(ts)
69+
ORDER BY id;
70+
```
71+
</TabsTab>
72+
</Tabs>
73+
74+
S3 with local cache:
75+
76+
```sql
77+
CREATE TABLE cache_table
78+
(
79+
id UInt64,
80+
ts DateTime,
81+
value Float32
82+
)
83+
ENGINE = MergeTree
84+
PARTITION BY toYYYYMM(ts)
85+
ORDER BY id
86+
SETTINGS storage_policy = 's3_cache';
87+
```
88+
89+
Local-only:
90+
91+
```sql
92+
CREATE TABLE local_table
93+
(
94+
id UInt64,
95+
ts DateTime,
96+
value Float32
97+
)
98+
ENGINE = MergeTree
99+
PARTITION BY toYYYYMM(ts)
100+
ORDER BY id
101+
SETTINGS storage_policy = 'default';
102+
```
103+
104+
Moving data with the tiered policy
105+
106+
With tiered, you can manually move partitions between volumes:
107+
108+
Move specific partitions to cold (S3):
109+
110+
```sql
111+
ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'cold';
112+
```
113+
114+
For non-partitioned tables (tuple()):
115+
116+
```sql
117+
ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold';
118+
```
119+
120+
Move back to hot (local):
121+
122+
```sql
123+
ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot';
124+
```
125+

pages/data-warehouse/menu.ts

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@ export const dataWarehouseMenu = {
3030
label: 'Connect to a deployment',
3131
slug: 'connect-applications',
3232
},
33+
{
34+
label: 'Manage databases in a deployment',
35+
slug: 'manage-databases',
36+
},
37+
{
38+
label: 'Manage users in a deployment',
39+
slug: 'manage-users',
40+
},
3341
{
3442
label: 'Connect a deployment with BI tools',
3543
slug: 'connect-bi-tools',
@@ -43,12 +51,8 @@ export const dataWarehouseMenu = {
4351
slug: 'edit-autoscaling',
4452
},
4553
{
46-
label: 'Manage databases in a deployment',
47-
slug: 'manage-databases',
48-
},
49-
{
50-
label: 'Manage users in a deployment',
51-
slug: 'manage-users',
54+
label: 'Manage the storage policy of a deployment',
55+
slug: 'manage-storage',
5256
},
5357
],
5458
label: 'How to',

0 commit comments

Comments
 (0)