You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| spill_local_disk_path | Specifies the directory path where spilled data will be stored on the local disk. |
267
+
| spill_local_disk_reserved_space_percentage | Defines the percentage of disk space that will be reserved and not used for spill. The default value is `30`. |
268
+
| spill_local_disk_max_bytes | Sets the maximum number of bytes allowed for spilling data to the local disk. Defaults to unlimited. |
269
+
270
+
### [spill.storage] Section
271
+
272
+
The following is a list of the parameters available within the [spill.storage] section:
| type | Specifies the storage type for remote spilling, for example, `s3`. |
277
+
278
+
To specify a specific storage, use the parameters in the [storage Section](#storage-section). For examples, see [Configuring Spill Storage](/guides/data-management/data-recycle#configuring-spill-storage).
Copy file name to clipboardExpand all lines: docs/en/guides/57-data-management/04-data-recycle.md
+48Lines changed: 48 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,54 @@ There are two types of data:
12
12
13
13
If the data size is significant, you can run several commands ([Enterprise Edition Features](/guides/products/dee/enterprise-features)) to delete these data and free up storage space.
14
14
15
+
## Spill Data Storage
16
+
17
+
Self-hosted Databend supports spilling intermediate query results to disk when memory usage exceeds available limits. Users can configure where spill data is stored, choosing between local disk storage and a remote S3-compatible bucket.
18
+
19
+
### Spill Storage Options
20
+
21
+
Databend provides the following spill storage configurations:
22
+
23
+
- Local Disk Storage: Spilled data is written to a specified local directory in the query node. Please note that local disk storage is supported only for [Windows Functions](/sql/sql-functions/window-functions/).
24
+
- Remote S3-Compatible Storage: Spilled data is stored in an external bucket.
25
+
- Default Storage: If no spill storage is configured, Databend spills data to the default storage bucket along with your table data.
26
+
27
+
### Spill Priority
28
+
29
+
If both local and S3-compatible spill storage are configured, Databend follows this order:
30
+
31
+
1. Spill to local disk first (if configured).
32
+
2. Spill to remote S3-compatible storage when local disk space is insufficient.
33
+
3. Spill to Databend’s default storage bucket if neither local nor external S3-compatible storage is configured.
34
+
35
+
### Configuring Spill Storage
36
+
37
+
To configure spill storage, update the [databend-query.toml](https://github.com/databendlabs/databend/blob/main/scripts/distribution/configs/databend-query.toml) configuration file.
38
+
39
+
This example sets Databend to use up to 1 TB of local disk space for spill operations, while reserving 40% of the disk for system use:
0 commit comments