Skip to content

Commit cb2c71c

Browse files
BohuTANGBohuTANG
authored
docs: clarify spill config after databend#19088 (#3026)
Update query-config spill section to match databend#19088 (reverted structured spill config): default remote spill prefix, local disk spill with fallback, and optional dedicated spill backend.
1 parent bdb9bae commit cb2c71c

File tree

2 files changed

+72
-94
lines changed

2 files changed

+72
-94
lines changed

‎docs/cn/guides/10-deploy/04-references/02-node-config/02-query-config.md‎

Lines changed: 36 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -272,39 +272,53 @@ cn=
272272

273273
## [spill] 部分
274274

275-
Databend 支持溢出存储（Spill Storage）以处理超出可用内存的大型查询。统一配置格式为所有存储类型提供一致模式并支持自动检测。
275+
Databend 支持将中间计算数据落盘（Spill），从而降低内存压力。
276276

277-
### [spill.storage] 部分
277+
### 默认溢出存储（零配置）
278278

279-
| 参数 | 描述 |
280-
| --------- | ----------------------------------------------------------------------------------------------------------- |
281-
| type | 存储类型，可选：`fs`（文件系统）、`s3`、`azblob`、`gcs`、`oss`、`cos` 等。 |
279+
- **存储介质**：与主数据存储（即 `[storage]` 配置）一致。
280+
- **路径前缀**：`_query_spill/`（位于配置的 `root` 目录下）。
281+
- **完整路径**：对于对象存储（S3/Azure/GCS 等），路径为 `<bucket>/<root>/_query_spill/`。
282+
- **作为回退**：当启用本地磁盘溢出时，会自动作为回退存储使用。
282283

283-
### [spill.storage.fs] 部分（文件系统存储）
284+
### 本地磁盘溢出（支持自动降级）
284285

285-
| 参数 | 描述 |
286-
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
287-
| data_path | 溢出数据在本地文件系统的存储目录路径。 |
288-
| reserved_space_percentage | 为系统保留、不用于溢出的磁盘空间百分比，防止磁盘被写满，确保系统稳定。默认值：`30`。 |
289-
| max_bytes | 允许溢出到本地文件系统的最大字节数。达到限制后，新溢出操作自动回退到主数据存储（远程存储），保证查询不中断。默认值：无限制。 |
286+
- 优先将溢出数据写入本地磁盘。
287+
- 当本地磁盘配额耗尽或空间不足时，自动降级至溢出存储后端（默认存储或 `[spill.storage]`）。
290288

291-
**示例（文件系统存储）：**
289+
| 参数 | 描述 |
290+
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
291+
| spill_local_disk_path | 本地磁盘上用于存储溢出数据的目录路径。留空则表示禁用本地磁盘溢出功能。 |
292+
| spill_local_disk_reserved_space_percentage | 本地磁盘保留空间百分比（不用于溢出）。用于防止磁盘被写满，保障系统稳定性。默认值：`10`（即保留 10%）。 |
293+
| spill_local_disk_max_bytes | 允许溢出到本地磁盘的最大字节数。当达到此上限时，后续的溢出操作会自动降级至溢出存储后端（默认存储或 `[spill.storage]`），确保查询任务不会中断。默认值：无限制（`unlimited`）。 |
292294

293-
```toml
294-
[spill.storage]
295-
type = "fs"
295+
**示例（本地磁盘溢出）：**
296296

297-
[spill.storage.fs]
298-
data_path = "/fast-ssd/spill"
299-
reserved_space_percentage = 25.0
300-
max_bytes = 107374182400 # 100GB
297+
```toml
298+
[spill]
299+
spill_local_disk_path = "/data/spill"
300+
spill_local_disk_reserved_space_percentage = 10.0
301+
spill_local_disk_max_bytes = 53687091200
301302
```
302303

304+
### 独立溢出存储配置（与主数据分离）
305+
306+
- 通过 `[spill.storage]` 进行配置。
307+
- 仅影响溢出文件（含本地溢出的回退目标）；**不改变**表数据的存储位置。
308+
- 依然使用 `_query_spill/` 作为路径前缀（位于该配置的 `root` 目录下）。
309+
310+
| 参数 | 描述 |
311+
| ---- | --------------------------------------------------------------------------------------- |
312+
| type | 存储类型。可选值：`fs`、`s3`、`azblob`、`gcs`、`oss`、`cos` 等（与 `[storage]` 部分一致）。 |
313+
303314
### [spill.storage.s3] 部分（S3 远程存储）
304315

305-
S3 溢出存储使用与 [storage.s3 部分](#storages3-部分) 相同的参数。
316+
参数配置与 [storage.s3 部分](#storages3-section) 完全一致。
317+
318+
- **前缀**：`_query_spill/`（位于 `root` 下）。
319+
- **AWS S3**：溢出对象将使用 `STANDARD` 存储类型。
306320

307-
**示例（S3 存储）：**
321+
**示例（独立 S3 溢出存储后端）：**
308322

309323
```toml
310324
[spill.storage]
@@ -315,30 +329,5 @@ bucket = "my-spill-bucket"
315329
region = "us-west-2"
316330
access_key_id = "your-access-key"
317331
secret_access_key = "your-secret-key"
332+
root = "spill/"
318333
```
319-
320-
<DetailsWrap>
321-
<details>
322-
<summary>旧格式（向后兼容）</summary>
323-
324-
:::note
325-
旧格式仅用于向后兼容。若 Databend 版本早于 v1.2.901，请使用此格式；新部署请使用上述统一格式。
326-
:::
327-
328-
| 参数 | 描述 |
329-
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
330-
| spill_local_disk_path | 溢出数据在本地磁盘的存储目录路径。 |
331-
| spill_local_disk_reserved_space_percentage | 为系统保留、不用于溢出的磁盘空间百分比，防止磁盘被写满，确保系统稳定。默认值：`30`。 |
332-
| spill_local_disk_max_bytes | 允许溢出到本地磁盘的最大字节数。达到限制后，新溢出操作自动回退到主数据存储（远程存储），保证查询不中断。默认值：无限制。 |
333-
334-
**示例（旧格式）：**
335-
336-
```toml
337-
[spill]
338-
spill_local_disk_path = "/data/spill"
339-
spill_local_disk_reserved_space_percentage = 30.0
340-
spill_local_disk_max_bytes = 53687091200
341-
```
342-
343-
</details>
344-
</DetailsWrap>

‎docs/en/guides/10-deploy/04-references/02-node-config/02-query-config.md‎

Lines changed: 36 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -272,39 +272,53 @@ The following is a list of the parameters available within the [cache.disk] sect
272272

273273
## [spill] Section
274274

275-
Databend supports spill storage to handle large queries that exceed available memory. The unified configuration format provides consistent patterns across all storage types with auto-detection capabilities.
275+
Databend can spill intermediate data to reduce memory pressure.
276276

277-
### [spill.storage] Section
277+
### Default Spill Storage Backend (No Extra Config)
278278

279-
| Parameter | Description |
280-
| --------- | ----------------------------------------------------------------------------------------------------------- |
281-
| type | Specifies the storage type. Available options: `fs` (filesystem), `s3`, `azblob`, `gcs`, `oss`, `cos`, etc. |
279+
- Backend: same as the main [storage].
280+
- Prefix: `_query_spill/` (under the configured `root`).
281+
- Location: for object storage (S3/Azure/GCS/...), `<bucket>/<root>/_query_spill/`.
282+
- Also used as fallback when local disk spill is enabled.
282283

283-
### [spill.storage.fs] Section (Filesystem Storage)
284+
### Local Disk Spill (With Fallback)
284285

285-
| Parameter | Description |
286-
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
287-
| data_path | Specifies the directory path where spilled data will be stored on the local filesystem. |
288-
| reserved_space_percentage | Defines the percentage of disk space that will be reserved and not used for spill. This prevents the spill operations from completely filling the disk and ensures system stability. Default: `30`. |
289-
| max_bytes | Sets the maximum number of bytes allowed for spilling data to the local filesystem. When this limit is reached, new spill operations will automatically fallback to the main data storage (remote storage), ensuring queries continue without interruption. Default: unlimited. |
286+
- Writes spill files to local disk first.
287+
- Falls back to the spill storage backend (default or [spill.storage]) when local quota is exhausted or disk is too full.
290288

291-
**Example (Filesystem Storage):**
289+
| Parameter | Description |
290+
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
291+
| spill_local_disk_path | Specifies the directory path where spilled data will be stored on the local disk. Leave it empty to disable local disk spill. |
292+
| spill_local_disk_reserved_space_percentage | Defines the percentage of disk space that will be reserved and not used for spill. This prevents spill operations from completely filling the disk and ensures system stability. Default: `10`. |
293+
| spill_local_disk_max_bytes | Sets the maximum number of bytes allowed for spilling data to the local disk. When the local spill quota is exhausted, new spill operations automatically fall back to the spill storage backend (default or [spill.storage]), ensuring queries continue without interruption. Default: unlimited. |
292294

293-
```toml
294-
[spill.storage]
295-
type = "fs"
295+
**Example (Local Disk Spill):**
296296

297-
[spill.storage.fs]
298-
data_path = "/fast-ssd/spill"
299-
reserved_space_percentage = 25.0
300-
max_bytes = 107374182400 # 100GB
297+
```toml
298+
[spill]
299+
spill_local_disk_path = "/data/spill"
300+
spill_local_disk_reserved_space_percentage = 10.0
301+
spill_local_disk_max_bytes = 53687091200
301302
```
302303

304+
### Dedicated Spill Storage Backend (Separate From Data)
305+
306+
- Configure via [spill.storage].
307+
- Affects spill files only (including local-disk fallback target); does not change where table data is stored.
308+
- Still uses the `_query_spill/` prefix under the configured `root`.
309+
310+
| Parameter | Description |
311+
| --------- | ------------------------------------------------------------------------------------------------------------------------- |
312+
| type | Specifies the storage type. Available options: `fs`, `s3`, `azblob`, `gcs`, `oss`, `cos`, etc. (same as the [storage] section). |
313+
303314
### [spill.storage.s3] Section (S3 Remote Storage)
304315

305-
For S3-based spill storage, use the same parameters as defined in the [storage.s3 Section](#storages3-section).
316+
Same parameters as [storage.s3 Section](#storages3-section).
317+
318+
- Prefix: `_query_spill/` (under `root`).
319+
- AWS S3: spill objects use `STANDARD` storage class.
306320

307-
**Example (S3 Storage):**
321+
**Example (Dedicated S3 Spill Backend):**
308322

309323
```toml
310324
[spill.storage]
@@ -315,30 +329,5 @@ bucket = "my-spill-bucket"
315329
region = "us-west-2"
316330
access_key_id = "your-access-key"
317331
secret_access_key = "your-secret-key"
332+
root = "spill/"
318333
```
319-
320-
<DetailsWrap>
321-
<details>
322-
<summary>Legacy Format (Backward Compatible)</summary>
323-
324-
:::note
325-
The legacy format is maintained for backward compatibility. If your Databend version is older than v1.2.901, use this format. New deployments should use the unified format above.
326-
:::
327-
328-
| Parameter | Description |
329-
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
330-
| spill_local_disk_path | Specifies the directory path where spilled data will be stored on the local disk. |
331-
| spill_local_disk_reserved_space_percentage | Defines the percentage of disk space that will be reserved and not used for spill. This prevents the spill operations from completely filling the disk and ensures system stability. Default: `30`. |
332-
| spill_local_disk_max_bytes | Sets the maximum number of bytes allowed for spilling data to the local disk. When this limit is reached, new spill operations will automatically fallback to the main data storage (remote storage), ensuring queries continue without interruption. Default: unlimited. |
333-
334-
**Example (Legacy Format):**
335-
336-
```toml
337-
[spill]
338-
spill_local_disk_path = "/data/spill"
339-
spill_local_disk_reserved_space_percentage = 30.0
340-
spill_local_disk_max_bytes = 53687091200
341-
```
342-
343-
</details>
344-
</DetailsWrap>

0 commit comments

Comments
 (0)