Skip to content

Commit 9e56337

Browse files
soyeric128soyeric128
authored
docs: bendsave (#1792)
* Update 02-data-recovery.md * Update 02-data-recovery.md * Update 02-data-recovery.md * bendsave * ee feature
1 parent e0f4e1a commit 9e56337

File tree

4 files changed

+300
-3
lines changed

4 files changed

+300
-3
lines changed

‎docs/en/guides/00-products/01-dee/10-enterprise-features.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ This page provides an updated list of available enterprise features. To access t
2222
| [Masking Policy](/sql/sql-commands/ddl/mask-policy/) | Security | Enhance your data security with role-based masking feature:<br/>- Safeguard sensitive information through customizable data masking.<br/>- Preserve data usability while reinforcing security. |
2323
| Storage Encryption | Security | Enhance the security of your server-side data encryption, safeguarding your data from unauthorized access by the storage vendor:<br/>- Choose encryption through service-managed keys, KMS managed keys, or customer-managed keys. Options may vary by storage type.<br/>- Currently supported on Alibaba Cloud OSS.<br/>See the [deploy guide](../../10-deploy/01-deploy/01-non-production/01-deploying-databend.md) for encryption parameters for each storage vendor. |
2424
| [Fail-Safe](/guides/security/fail-safe) | Security | Recover table data from S3-compatible object storage. |
25+
| [BendSave](/guides/data-management/data-recovery#bendsave) | Recovery | BendSave is a command-line tool for backing up and restoring both metadata and actual data files in Databend. |
2526

2627
## Databend Community vs. Enterprise
2728

‎docs/en/guides/57-data-management/02-data-recovery.md‎

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
---
2-
title: Data Recovery in Databend Cloud
3-
sidebar_label: Data Recovery
2+
title: Data Recovery
43
---
4+
import EEFeature from '@site/src/components/EEFeature';
55

6-
This topic explains how to restore changed or deleted data using Databend Cloud.
6+
<EEFeature featureName='BendSave'/>
7+
8+
This topic explains how to back up and restore data in Databend.
79

810
## Time Travel: Easy Access to Past Data
911

@@ -64,3 +66,77 @@ Fail-safe includes:
6466
- Recovery times can vary from a few hours to several days, depending on the situation.
6567

6668
:::
69+
70+
71+
## BendSave
72+
73+
BendSave is a command-line tool for backing up and restoring both metadata and actual data files in Databend. It stores backups in S3-compatible object storage, making it ideal for disaster recovery.
74+
75+
### Downloading BendSave
76+
77+
The BendSave binary is distributed as part of the [Databend release packages](https://github.com/databendlabs/databend/releases).
78+
79+
To download:
80+
81+
1. Go to the latest [Databend Releases](https://github.com/databendlabs/databend/releases).
82+
83+
2. Select the release that matches your currently running `databend-query` version.
84+
85+
3. Download and extract the release package.
86+
87+
4. Inside the extracted archive, locate the **bin** directory and find the **databend-bendsave** binary.
88+
89+
### Command Reference
90+
91+
To back up the metadata of a Databend cluster:
92+
93+
```bash
94+
databend-bendsave backup \
95+
--from <query-config-path> \
96+
--to <backup-destination>
97+
```
98+
99+
| Parameter | Description |
100+
|-----------|-------------------------------------------------------------------------|
101+
| from | Path to the `databend-query.toml` configuration file. |
102+
| to | Backup destination, e.g.,`s3://backup?endpoint=http://127.0.0.1:9900&access_key_id=xxx&secret_access_key=xxx`.<br/>- It is recommended to use environment variables such as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to provide credentials.|
103+
104+
To restore the metadata to a Databend cluster:
105+
106+
```bash
107+
databend-bendsave restore \
108+
--from <backup-source> \
109+
--to-query <query-config-path> \
110+
--to-meta <meta-config-path> \
111+
--confirm
112+
```
113+
114+
| Parameter | Description |
115+
|-----------|-----------------------------------------------------------------------|
116+
| from | Backup source path. |
117+
| to-query | Path to the restored `databend-query.toml` configuration file. |
118+
| to-meta | Path to the restored `databend-meta.toml` configuration file. |
119+
| confirm | Required flag to confirm restoration and avoid accidental overwrites. |
120+
121+
#### Examples
122+
123+
```bash
124+
export AWS_ACCESS_KEY_ID=minioadmin
125+
export AWS_SECRET_ACCESS_KEY=minioadmin
126+
127+
# Backup
128+
./databend-bendsave backup \
129+
--from ../configs/databend-query.toml \
130+
--to 's3://backupbucket?endpoint=http://127.0.0.1:9000/&region=us-east-1'
131+
132+
# Restore
133+
./databend-bendsave restore \
134+
--from "s3://backupbucket?endpoint=http://127.0.0.1:9000/&region=us-east-1" \
135+
--to-query ../configs/databend-query.toml \
136+
--to-meta ../configs/databend-meta.toml \
137+
--confirm
138+
```
139+
140+
### Tutorials
141+
142+
- [Backing Up and Restoring Data with BendSave](/tutorials/recovery/bendsave)

‎docs/en/tutorials/recovery/_category_.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"label": "Data Recovery"
3+
}

‎docs/en/tutorials/recovery/bendsave.md‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
---
2+
title: Backing Up and Restoring Data with BendSave
3+
---
4+
5+
This tutorial walks you through how to back up and restore data using BendSave. We'll use a local MinIO instance as both the S3-compatible storage backend for Databend and the destination for storing backups.
6+
7+
## Before You Start
8+
9+
Before you start, ensure you have the following prerequisites in place:
10+
11+
- A Linux machine (x86_64 or aarch64 architecture): In this tutorial, we'll deploy Databend on a Linux machine. You can use a local machine, a virtual machine, or a cloud instance such as AWS EC2.
12+
- [Docker](https://www.docker.com/): Used to deploy a local MinIO instance.
13+
- [AWS CLI](https://aws.amazon.com/cli/): Used to manage buckets in MinIO.
14+
- If you are on AWS EC2, make sure your security group allows inbound traffic on port `8000`, as this is required for BendSQL to connect to Databend.
15+
16+
- BendSQL is installed on your local machine. See [Installing BendSQL](/guides/sql-clients/bendsql/#installing-bendsql) for instructions on how to install BendSQL using various package managers.
17+
18+
## Step 1: Launch MinIO in Docker
19+
20+
1. Start a MinIO container on your Linux machine. The following command launches a MinIO container named **minio**, with ports `9000` (for the API) and `9001` (for the web console) exposed:
21+
22+
```bash
23+
docker run -d --name minio \
24+
-e "MINIO_ACCESS_KEY=minioadmin" \
25+
-e "MINIO_SECRET_KEY=minioadmin" \
26+
-p 9000:9000 \
27+
-p 9001:9001 \
28+
minio/minio server /data \
29+
--address :9000 \
30+
--console-address :9001
31+
```
32+
33+
2. Set your MinIO credentials as environment variables, then use the AWS CLI to create two buckets: one for storing backups (**backupbucket**) and another for Databend data (**databend**):
34+
35+
```bash
36+
export AWS_ACCESS_KEY_ID=minioadmin
37+
export AWS_SECRET_ACCESS_KEY=minioadmin
38+
39+
aws --endpoint-url http://127.0.0.1:9000/ s3 mb s3://backupbucket
40+
aws --endpoint-url http://127.0.0.1:9000/ s3 mb s3://databend
41+
```
42+
43+
## Step 2: Set up Databend
44+
45+
1. Download the latest Databend release and extract it to get the necessary binaries:
46+
47+
```bash
48+
wget https://github.com/databendlabs/databend/releases/download/v1.2.719-nightly/databend-dbg-v1.2.719-nightly-x86_64-unknown-linux-gnu.tar.gz
49+
50+
tar -xzvf databend-dbg-v1.2.719-nightly-x86_64-unknown-linux-gnu.tar.gz
51+
```
52+
53+
2. Configure the **databend-query.toml** configuration file in the **configs** folder.
54+
55+
```bash
56+
vi configs/databend-query.toml
57+
```
58+
59+
The following shows the key configuration required for this tutorial:
60+
61+
```toml
62+
...
63+
[[query.users]]
64+
name = "root"
65+
auth_type = "no_password"
66+
...
67+
# Storage config.
68+
[storage]
69+
# fs | s3 | azblob | gcs | oss | cos | hdfs | webhdfs
70+
type = "s3"
71+
...
72+
# To use an Amazon S3-like storage service, uncomment this block and set your values.
73+
[storage.s3]
74+
bucket = "databend"
75+
endpoint_url = "http://127.0.0.1:9000"
76+
access_key_id = "minioadmin"
77+
secret_access_key = "minioadmin"
78+
enable_virtual_host_style = false
79+
```
80+
81+
3. Use the following commands to start the Meta and Query services:
82+
83+
```bash
84+
./databend-meta -c ../configs/databend-meta.toml > meta.log 2>&1 &
85+
```
86+
87+
```bash
88+
./databend-query -c ../configs/databend-query.toml > query.log 2>&1 &
89+
```
90+
91+
After launching the services, verify they are running by checking their health endpoints. A successful response should return HTTP status 200 OK.
92+
93+
```bash
94+
curl -I http://127.0.0.1:28101/v1/health
95+
96+
curl -I http://127.0.0.1:8080/v1/health
97+
```
98+
99+
4. Connect to your Databend instance from your local machine with BendSQL, then apply your Databend Enterprise license, create a table, and insert some sample data.
100+
101+
```bash
102+
bendsql -h <your-linux-host>
103+
```
104+
105+
```sql
106+
SET GLOBAL enterprise_license='<your-license-key>';
107+
```
108+
109+
```sql
110+
CREATE TABLE books (
111+
id BIGINT UNSIGNED,
112+
title VARCHAR,
113+
genre VARCHAR DEFAULT 'General'
114+
);
115+
116+
INSERT INTO books(id, title) VALUES(1, 'Invisible Stars');
117+
```
118+
119+
5. Back on your Linux machine, verify that the table data has been stored in your Databend bucket:
120+
121+
```bash
122+
aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://databend/ --recursive
123+
```
124+
125+
```bash
126+
2025-04-07 15:27:06 748 1/169/_b/h0196160323247b1cab49be6060d42df8_v2.parquet
127+
2025-04-07 15:27:06 646 1/169/_sg/h0196160323247c5eb0a1a860a6442c70_v4.mpk
128+
2025-04-07 15:27:06 550 1/169/_ss/h019610dcc72474adb32ef43698db2a09_v4.mpk
129+
2025-04-07 15:27:06 143 1/169/last_snapshot_location_hint_v2
130+
```
131+
132+
## Step 3: Back up with BendSave
133+
134+
1. Run the following command to back up your Databend data to the **backupbucket** in MinIO:
135+
136+
```bash
137+
export AWS_ACCESS_KEY_ID=minioadmin
138+
export AWS_SECRET_ACCESS_KEY=minioadmin
139+
140+
./databend-bendsave backup \
141+
--from ../configs/databend-query.toml \
142+
--to 's3://backupbucket?endpoint=http://127.0.0.1:9000/&region=us-east-1'
143+
<jemalloc>: Number of CPUs detected is not deterministic. Per-CPU arena disabled.
144+
Backing up from ../configs/databend-query.toml to s3://backupbucket?endpoint=http://127.0.0.1:9000/&region=us-east-1
145+
```
146+
147+
2. After the backup completes, you can verify that the files were written to the backupbucket by listing its contents:
148+
149+
```bash
150+
aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://backupbucket/ --recursive
151+
```
152+
153+
```bash
154+
2025-04-07 15:44:29 748 1/169/_b/h0196160323247b1cab49be6060d42df8_v2.parquet
155+
2025-04-07 15:44:29 646 1/169/_sg/h0196160323247c5eb0a1a860a6442c70_v4.mpk
156+
2025-04-07 15:44:29 550 1/169/_ss/h019610dcc72474adb32ef43698db2a09_v4.mpk
157+
2025-04-07 15:44:29 143 1/169/last_snapshot_location_hint_v2
158+
2025-04-07 15:44:29 344781 databend_meta.db
159+
```
160+
161+
## Step 4: Restore with BendSave
162+
163+
1. Remove all the file in the **databend** bucket:
164+
165+
```bash
166+
aws --endpoint-url http://127.0.0.1:9000 s3 rm s3://databend/ --recursive
167+
```
168+
169+
2. After the removal, you can verify using BendSQL that querying the table in Databend fails:
170+
171+
```sql
172+
SELECT * FROM books;
173+
```
174+
175+
```bash
176+
error: APIError: QueryFailed: [3001]NotFound (persistent) at read, context: { uri: http://127.0.0.1:9000/databend/1/169/_ss/h019610dcc72474adb32ef43698db2a09_v4.mpk, response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "content-length": "423", "content-type": "application/xml", "server": "MinIO", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "Origin", "vary": "Accept-Encoding", "x-amz-id-2": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8", "x-amz-request-id": "18342C51C209C7E9", "x-content-type-options": "nosniff", "x-ratelimit-limit": "144", "x-ratelimit-remaining": "144", "x-xss-protection": "1; mode=block", "date": "Mon, 07 Apr 2025 23:14:45 GMT"} }, service: s3, path: 1/169/_ss/h019610dcc72474adb32ef43698db2a09_v4.mpk, range: 0- } => S3Error { code: "NoSuchKey", message: "The specified key does not exist.", resource: "/databend/1/169/_ss/h019610dcc72474adb32ef43698db2a09_v4.mpk", request_id: "18342C51C209C7E9" }
177+
```
178+
179+
3. Run the following command to restore your Databend data to the **databend** bucket in MinIO:
180+
181+
```bash
182+
./databend-bendsave restore \
183+
--from "s3://backupbucket?endpoint=http://127.0.0.1:9000/&region=us-east-1" \
184+
--to-query ../configs/databend-query.toml \
185+
--to-meta ../configs/databend-meta.toml \
186+
--confirm
187+
<jemalloc>: Number of CPUs detected is not deterministic. Per-CPU arena disabled.
188+
Restoring from s3://backupbucket?endpoint=http://127.0.0.1:9000/&region=us-east-1 to query ../configs/databend-query.toml and meta ../configs/databend-meta.toml with confirmation
189+
```
190+
191+
4. After the restore completes, you can verify that the files were written back to the **databend** bucket by listing its contents:
192+
193+
```bash
194+
aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://databend/ --recursive
195+
```
196+
197+
```bash
198+
2025-04-07 23:21:39 748 1/169/_b/h0196160323247b1cab49be6060d42df8_v2.parquet
199+
2025-04-07 23:21:39 646 1/169/_sg/h0196160323247c5eb0a1a860a6442c70_v4.mpk
200+
2025-04-07 23:21:39 550 1/169/_ss/h019610dcc72474adb32ef43698db2a09_v4.mpk
201+
2025-04-07 23:21:39 143 1/169/last_snapshot_location_hint_v2
202+
2025-04-07 23:21:39 344781 databend_meta.db
203+
```
204+
205+
5. Query the table again using BendSQL, and you will see that the query now succeeds:
206+
207+
```sql
208+
SELECT * FROM books;
209+
```
210+
211+
```sql
212+
┌────────────────────────────────────────────────────────┐
213+
│ id │ title │ genre │
214+
├──────────────────┼──────────────────┼──────────────────┤
215+
│ 1 │ Invisible Stars │ General │
216+
└────────────────────────────────────────────────────────┘
217+
```

0 commit comments

Comments
 (0)