Skip to content

Commit e4c7217

Browse files
SamyOubouazizSamyOubouaziz
committed
feat(dwh): update
1 parent c2d3e1e commit e4c7217

File tree

1 file changed

+51
-35
lines changed

1 file changed

+51
-35
lines changed

‎pages/data-warehouse/how-to/manage-storage.mdx‎

Lines changed: 51 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,16 @@ dates:
77
posted: 2025-10-16
88
---
99

10+
import Requirements from '@macros/iam/requirements.mdx'
11+
12+
<Requirements />
13+
14+
- A Scaleway account logged into the [console](https://console.scaleway.com)
15+
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
16+
1017
## Overview
1118

12-
Scaleway Data Warehouse for ClickHouse® uses ClickHouse®'s native storage policies to decide where table data lives: on Scaleway Block storage (local PVC), and/or on Object Storage (Amazon S3). You can choose between three policies for your deployment:
19+
Scaleway Data Warehouse for ClickHouse® uses ClickHouse®'s native storage policies to decide where table data lives: on Scaleway Block storage, and/or on Object Storage (Amazon S3). Policies are applied at table-level during table creation, using the settings presented below. You can choose between three policies for your deployment:
1320

1421
- `tiered` (default in Scaleway Data Warehouse for ClickHouse®)
1522
- `s3_cache` (cache mode)
@@ -19,20 +26,17 @@ The table below provides an overview of each storage policy's behavior and use c
1926

2027
| Storage&nbsp;policy | Behavior | When to use |
2128
|------|---------|------------|
22-
| `tiered` | Writes land on Block storage first (hot volume). When the local disk reaches ~90% fullness, parts are automatically moved to Object storage (cold volume). This is controlled by `move_factor: 0.1` (move when only ~10% free remains). | General purpose setting for fast local writes and transparent spillover to Object Storage as the dataset grows. |
23-
| `s3_cache` | Data is mainly stored on Object Storage. A local cache layer (on Block storage) keeps frequently-read parts to accelerate repeated reads. | Large datasets that mostly live in Object Storage, with repeated reads on hot subsets that benefit from local caching. |
24-
| `default` | Data is stored on Block storage only. | Small datasets where lowest latency for reads/writes on local disk is desired and capacity fits the PVC. |
29+
| `tiered` | Writes on Block Storage first, and parts are automatically moved to Object Storage when disk is 90% full. | General purpose setting for fast local writes and transparent spillover to Object Storage as the dataset grows. |
30+
| `s3_cache` | Data is stored on Object Storage. A local Block Storage cache layer keeps frequently-read parts to accelerate repeated reads. | Large datasets that mostly live in Object Storage, with repeated reads on smaller subsets that benefit from local caching. |
31+
| `default` | Data is stored on Block storage only. | Small datasets where lowest latency for reads/writes on local disk is desired and capacity fits the Block Storage volume. |
2532

26-
Under the hood, our ClickHouse config defines three disks (default = local PVC, s3 = object storage, s3_cache = S3 + local cache) and three policies (tiered, s3_cache, default).
27-
Cluster default (for new tables when not specified): tiered.
28-
29-
How clients choose a policy (per table)
33+
## Tiered storage policy
3034

31-
The storage policy is chosen at table creation time via SETTINGS storage_policy = '...'.
35+
### Applying the tiered policy
3236

33-
## Tiered storage policy
37+
**Tiered** is the deployment's default policy. Data is stored on the `hot` volume (Block Storage) until it reaches 90% of its capacity, then data is moved to the `cold` volume. This mechanism is controlled by `move_factor:0.1`.
3438

35-
**Tiered** (Block Storage first, spill to Object Storage) is the cluster default policy. If you omit the `storage_policy` setting; ClickHouse® will automatically apply this policy.
39+
[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to apply the `tiered` storage policy.
3640

3741
<Tabs id="tiered-explicit-implicit">
3842
<TabsTab label="Explicit">
@@ -71,8 +75,40 @@ The storage policy is chosen at table creation time via SETTINGS storage_policy
7175
</TabsTab>
7276
</Tabs>
7377

74-
S3 with local cache:
78+
<Message type="note">
79+
If you omit the `storage_policy` setting, ClickHouse® will automatically apply the `tiered` policy.
80+
</Message>
81+
82+
### Moving data with the tiered policy
83+
84+
When using the `tiered` storage policy, you can manually move partitions between `hot` (Block Storage) and `cold` (Object Storage) volumes using the SQL queries below.
85+
86+
- Moving specific partitions to the `cold` volume (Object Storage):
87+
88+
```sql
89+
ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'cold';
90+
```
91+
92+
- Moving a non-partitioned table (`tuple()`) to the `cold` volume (Object Storage):
93+
94+
```sql
95+
ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold';
96+
```
97+
98+
- Move a partition to the `hot` volume (Block Storage):
99+
100+
```sql
101+
ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot';
102+
```
103+
- Moving a non-partitioned table (`tuple()`) to the `hot` volume (Block Storage):
75104

105+
```sql
106+
ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'hot';
107+
```
108+
109+
## S3 cache policy
110+
111+
[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `s3_cache` policy enabled.
76112
```sql
77113
CREATE TABLE cache_table
78114
(
@@ -86,7 +122,9 @@ ORDER BY id
86122
SETTINGS storage_policy = 's3_cache';
87123
```
88124

89-
Local-only:
125+
## Local-only policy
126+
127+
[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `default` (Block Storage only) policy enabled.
90128

91129
```sql
92130
CREATE TABLE local_table
@@ -101,25 +139,3 @@ ORDER BY id
101139
SETTINGS storage_policy = 'default';
102140
```
103141

104-
Moving data with the tiered policy
105-
106-
With tiered, you can manually move partitions between volumes:
107-
108-
Move specific partitions to cold (S3):
109-
110-
```sql
111-
ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'cold';
112-
```
113-
114-
For non-partitioned tables (tuple()):
115-
116-
```sql
117-
ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold';
118-
```
119-
120-
Move back to hot (local):
121-
122-
```sql
123-
ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot';
124-
```
125-

0 commit comments

Comments
 (0)