Skip to content

Commit 1b099a4

Browse files
soyeric128soyeric128
authored
docs: deploy on docker (#1821)
* Delete 02-deploying-databend-with-hdfs.md * Update 00-deploying-local.md
1 parent ea6a470 commit 1b099a4

File tree

2 files changed

+115
-407
lines changed

2 files changed

+115
-407
lines changed

‎docs/en/guides/10-deploy/01-deploy/01-non-production/00-deploying-local.md‎

Lines changed: 115 additions & 138 deletions
Original file line numberDiff line numberDiff line change
@@ -2,36 +2,18 @@
22
title: Deploying on Docker
33
---
44

5-
import StepsWrap from '@site/src/components/StepsWrap';
6-
import StepContent from '@site/src/components/Steps/step-content';
7-
8-
In this guide, you will deploy Databend along with [MinIO](https://min.io/) using [Docker](https://www.docker.com/) for a containerized setup on an [Amazon EC2](https://aws.amazon.com/ec2/) Ubuntu virtual machine.
5+
This guide walks you through deploying Databend with [MinIO](https://min.io/) using [Docker](https://www.docker.com/) for a fully containerized setup on your local machine.
96

107
:::note non-production use only
118
The MinIO deployment covered in this guide is only suitable for development and demonstration. Due to the limited resources in a single-machine environment, it is not recommended for production environments or performance testing.
129
:::
1310

14-
<StepsWrap>
15-
<StepContent number="1">
16-
17-
### Set up Environment
18-
19-
Before you start, launch an instance on Amazon EC2 and install the Docker engine.
20-
21-
1. Log into the [Amazon EC2 console](https://console.aws.amazon.com/ec2/), and launch an Ubuntu instance with a memory capacity of at least 8 GiB. Once the instance is up, you can find the public IP address and the private IP address assigned to the instance on the instance details page.
22-
23-
![Alt text](/img/deploy/docker-instance.png)
11+
### Before You Start
2412

25-
2. Create a security group, and add an inbound rule to allow access to your instance through port `9001`, then add the security group to the instance.
13+
Before you start, ensure you have the following prerequisites in place:
2614

27-
![Alt text](/img/deploy/docker-create-sg.png)
28-
29-
3. Connect to your instance. There are many ways to connect to your instance from a local machine. For more information, see [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-to-linux-instance.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-to-linux-instance.html).
30-
31-
4. Follow the [Docker User Manual](https://docs.docker.com/engine/install/ubuntu/) to install the Docker engine on your instance.
32-
33-
</StepContent>
34-
<StepContent number="2">
15+
- Ensure that [Docker](https://www.docker.com/) is installed on your local machine.
16+
- Ensure that BendSQL is installed on your machine. See [Installing BendSQL](/guides/sql-clients/bendsql/#installing-bendsql) for instructions on how to install BendSQL using various package managers.
3517

3618
### Deploy MinIO
3719

@@ -44,167 +26,170 @@ Before you start, launch an instance on Amazon EC2 and install the Docker engine
4426
:::
4527

4628
```shell
47-
mkdir -p ${HOME}/minio/data
48-
49-
docker run -d \
50-
--name minio \
51-
--user $(id -u):$(id -g) \
52-
--net=host \
53-
-e "MINIO_ROOT_USER=ROOTUSER" \
54-
-e "MINIO_ROOT_PASSWORD=CHANGEME123" \
55-
-v ${HOME}/minio/data:/data \
56-
minio/minio server /data --console-address ":9001"
29+
docker run -d --name minio \
30+
-e "MINIO_ACCESS_KEY=ROOTUSER" \
31+
-e "MINIO_SECRET_KEY=CHANGEME123" \
32+
-p 9000:9000 \
33+
-p 9001:9001 \
34+
minio/minio server /data \
35+
--address :9000 \
36+
--console-address :9001
5737
```
5838

5939
2. Run the command `docker logs minio` to find the MinIO API and console (WebUI) addresses in the log message:
6040

6141
```shell
6242
docker logs minio
43+
```
6344

64-
Formatting 1st pool, 1 set(s), 1 drives per set.
65-
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
45+
```
46+
INFO: WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
47+
Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
48+
INFO: Formatting 1st pool, 1 set(s), 1 drives per set.
49+
INFO: WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
6650
MinIO Object Storage Server
67-
Copyright: 2015-2024 MinIO, Inc.
68-
License: GNU AGPLv3 [https://www.gnu.org/licenses/agpl-3.0.html](https://www.gnu.org/licenses/agpl-3.0.html)
69-
Version: RELEASE.2024-04-06T05-26-02Z (go1.21.9 linux/amd64)
70-
71-
// highlight-next-line
72-
API: http://172.31.15.63:9000 http://172.17.0.1:9000 http://127.0.0.1:9000
73-
// highlight-next-line
74-
WebUI: http://172.31.15.63:9001 http://172.17.0.1:9001 http://127.0.0.1:9001
75-
76-
Docs: https://min.io/docs/minio/linux/index.html
77-
Status: 1 Online, 0 Offline.
78-
STARTUP WARNINGS:
79-
- The standard parity is set to 0. This can lead to data loss.
51+
Copyright: 2015-2025 MinIO, Inc.
52+
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
53+
Version: RELEASE.2025-04-03T14-56-28Z (go1.24.2 linux/arm64)
54+
55+
API: http://172.17.0.2:9000 http://127.0.0.1:9000
56+
WebUI: http://172.17.0.2:9001 http://127.0.0.1:9001
57+
58+
Docs: https://docs.min.io
59+
INFO:
60+
You are running an older version of MinIO released 5 days before the latest release
61+
Update: Run `mc admin update ALIAS`
8062
```
8163

82-
3. Open your web browser on your local machine and visit the MinIO console using the WebUI address shown in the logs above (replace the IP address with your instance's public IP address). For example, if your instance's public IP address is `3.142.131.212`, then your MinIO console address would be `http://3.142.131.212:9001`.
64+
3. Open your web browser on your local machine and visit the MinIO console using the WebUI address shown (`http://127.0.0.1:9001`) in the logs above.
8365

8466
![Alt text](/img/deploy/docker-minio.png)
8567

8668
4. Log into the MinIO console with the credentials `ROOTUSER`/`CHANGEME123`, and create a bucket named `databend`.
8769

8870
![Alt text](/img/deploy/docker-bucket.png)
8971

90-
</StepContent>
91-
92-
<StepContent number="3">
93-
9472
### Deploy Databend
9573

9674
1. Pull and run the Databend image as a container with the following command:
9775

9876
:::note
9977

100-
- Replace the `AWS_S3_ENDPOINT` value with your MinIO API address as shown in the MinIO log message returned by `docker logs minio`.
10178
- When starting the Databend Docker container, you can specify the username and password using the environment variables `QUERY_DEFAULT_USER` and `QUERY_DEFAULT_PASSWORD`. If these variables are not provided, a default root user will be created without a password.
10279
- The command below also creates a SQL user (`databend`/`databend`) which you will need to use to connect to Databend later. If you make changes to the SQL user at this point, ensure that you maintain consistency throughout the entire process.
10380
:::
10481

10582
```shell
10683
docker run -d \
107-
--name databend \
108-
--net=host \
109-
-v meta_storage_dir:/var/lib/databend/meta \
110-
-v log_dir:/var/log/databend \
111-
-e QUERY_DEFAULT_USER=databend \
112-
-e QUERY_DEFAULT_PASSWORD=databend \
113-
-e QUERY_STORAGE_TYPE=s3 \
114-
-e AWS_S3_ENDPOINT=http://172.31.15.63:9000 \
115-
-e AWS_S3_BUCKET=databend \
116-
-e AWS_ACCESS_KEY_ID=ROOTUSER \
117-
-e AWS_SECRET_ACCESS_KEY=CHANGEME123 \
118-
datafuselabs/databend
84+
--name databend \
85+
-p 3307:3307 \
86+
-p 8000:8000 \
87+
-p 8124:8124 \
88+
-p 8900:8900 \
89+
-e QUERY_DEFAULT_USER=databend \
90+
-e QUERY_DEFAULT_PASSWORD=databend \
91+
-e QUERY_STORAGE_TYPE=s3 \
92+
-e AWS_S3_ENDPOINT=http://host.docker.internal:9000 \
93+
-e AWS_S3_BUCKET=databend \
94+
-e AWS_ACCESS_KEY_ID=ROOTUSER \
95+
-e AWS_SECRET_ACCESS_KEY=CHANGEME123 \
96+
datafuselabs/databend
11997
```
12098

12199
2. Run the command `docker logs databend` to check the Databend log message and ensure the Databend container has started successfully:
122100

123101
```shell
124102
docker logs databend
103+
```
125104

105+
```shell
126106
==> QUERY_CONFIG_FILE is not set, using default: /etc/databend/query.toml
127107
==> /tmp/std-meta.log <==
128-
<jemalloc>: Number of CPUs detected is not deterministic. Per-CPU arena disabled.
129108
Databend Metasrv
130109

131-
Version: v1.2.410-4b8cd16f0c-simd(1.77.0-nightly-2024-04-08T12:27:32.972822624Z)
132-
Working DataVersion: V002(2023-07-22: Store snapshot in a file)
110+
Version: v1.2.697-d40f88cc51-simd(1.85.0-nightly-2025-02-14T11:57:01.874747910Z)
111+
Working DataVersion: V004(2024-11-11: WAL based raft-log)
133112

134113
Raft Feature set:
135-
Server Provide: { append:v0, install_snapshot:v0, install_snapshot:v1, vote:v0 }
136-
Client Require: { append:v0, install_snapshot:v0, vote:v0 }
137-
138-
On Disk Data:
139-
Dir: /var/lib/databend/meta
140-
DataVersion: V001(2023-05-15: Get rid of compat, use only openraft v08 data types)
141-
In-Upgrading: None
142-
143-
Log:
144-
File: enabled=true, level=INFO, dir=/var/log/databend, format=json
145-
Stderr: enabled=false(To enable: LOG_STDERR_ON=true or RUST_LOG=info), level=WARN, format=text
146-
OTLP: enabled=false, level=INFO, endpoint=http://127.0.0.1:4317, labels=
147-
Tracing: enabled=false, capture_log_level=INFO, otlp_endpoint=http://127.0.0.1:4317
148-
Id: 0
149-
Raft Cluster Name: foo_cluster
150-
Raft Dir: /var/lib/databend/meta
151-
Raft Status: single
152-
153-
HTTP API
154-
listening at 127.0.0.1:28002
155-
gRPC API
156-
listening at 127.0.0.1:9191
157-
advertise: -
158-
Raft API
159-
listening at 127.0.0.1:28004
160-
advertise: ip-172-31-15-63:28004
114+
Server Provide: { append:v0, install_snapshot:v1, install_snapshot:v3, vote:v0 }
115+
Client Require: { append:v0, install_snapshot:v3, vote:v0 }
161116

117+
Disk Data: V002(2023-07-22: Store snapshot in a file); Upgrading: None
118+
Dir: /var/lib/databend/meta
119+
120+
Log File: enabled=true, level='Warn,databend_=Info,openraft=Info', dir=/var/log/databend, format=json, limit=48
121+
Stderr: enabled=false(To enable: LOG_STDERR_ON=true or RUST_LOG=info), level=WARN, format=text
122+
Raft Id: 0; Cluster: foo_cluster
123+
Dir: /var/lib/databend/meta
124+
Status: single
125+
126+
HTTP API listen at: 127.0.0.1:28002
127+
gRPC API listen at: 127.0.0.1:9191 advertise: -
128+
Raft API listen at: 127.0.0.1:28004 advertise: 055b9e5d09a9:28004
129+
130+
Upgrade ondisk data if out of date: V002
131+
Find and clean previous unfinished upgrading
162132
Upgrade on-disk data
163-
From: V001(2023-05-15: Get rid of compat, use only openraft v08 data types)
164-
To: V002(2023-07-22: Store snapshot in a file)
165-
Begin upgrading: version: V001, upgrading: V002
166-
Write header: version: V001, upgrading: V002
167-
tree raft_state not found
168-
tree raft_log not found
169-
Found state machine trees: []
170-
Found min state machine id: 18446744073709551615
171-
No state machine tree, skip upgrade
172-
Finished upgrading: version: V002, upgrading: None
173-
Write header: version: V002, upgrading: None
133+
From: V002(2023-07-22: Store snapshot in a file)
134+
To: V003(2024-06-27: Store snapshot in rotbl)
135+
No V002 snapshot, skip upgrade
136+
Finished upgrading: V003
137+
Upgrade on-disk data
138+
From: V003(2024-06-27: Store snapshot in rotbl)
139+
To: V004(2024-11-11: WAL based raft-log)
140+
Upgrade V003 raft log in sled db to V004
141+
Clean upgrading: V003 -> V004 (cleaning)
142+
Remove V003 log from sled db
143+
Removing sled tree: header
144+
Removing sled tree: raft_log
145+
Removing sled tree: raft_state
146+
Done: Remove V003 log from sled db
147+
Removing: /var/lib/databend/meta/heap
148+
Removing: /var/lib/databend/meta/conf
149+
Removing: /var/lib/databend/meta/db
150+
Removing: /var/lib/databend/meta/DO_NOT_USE_THIS_DIRECTORY_FOR_ANYTHING
151+
Finished upgrading: V004
152+
Upgrade ondisk data finished: V004
174153
Wait for 180s for active leader...
175154
Leader Id: 0
176-
Metrics: id=0, Leader, term=1, last_log=Some(3), last_applied=Some(1-0-3), membership={log_id:Some(1-0-3), voters:[{0:{EmptyNode}}], learners:[]}
155+
Metrics: id=0, Leader, term=1, last_log=Some(3), last_applied=Some(T1-N0.3), membership={log_id:Some(T1-N0.3), {voters:[{0:EmptyNode}], learners:[]}}
177156

178-
Register this node: {id=0 raft=ip-172-31-15-63:28004 grpc=}
157+
Register this node: {id=0 raft=055b9e5d09a9:28004 grpc=}
179158

180159
Register-node: Ok
181160

182161
Databend Metasrv started
183162

184163
==> /tmp/std-query.log <==
185-
<jemalloc>: Number of CPUs detected is not deterministic. Per-CPU arena disabled.
186164
Databend Query
187165

188-
Version: v1.2.410-4b8cd16f0c(rust-1.77.0-nightly-2024-04-08T12:20:44.288903419Z)
166+
Version: v1.2.697-d40f88cc51(rust-1.85.0-nightly-2025-02-14T11:30:59.842308760Z)
189167

190168
Logging:
191-
file: enabled=true, level=INFO, dir=/var/log/databend, format=json
169+
file: enabled=true, level='INFO', dir=/var/log/databend, format=json, limit=48
192170
stderr: enabled=false(To enable: LOG_STDERR_ON=true or RUST_LOG=info), level=WARN, format=text
193-
otlp: enabled=false, level=INFO, endpoint=http://127.0.0.1:4317, labels=
194-
query: enabled=false, dir=, otlp_endpoint=, labels=
195-
tracing: enabled=false, capture_log_level=INFO, otlp_endpoint=http://127.0.0.1:4317
171+
196172
Meta: connected to endpoints [
197173
"0.0.0.0:9191",
198174
]
175+
199176
Memory:
200177
limit: unlimited
201178
allocator: jemalloc
202179
config: percpu_arena:percpu,oversize_threshold:0,background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000
180+
203181
Cluster: standalone
204-
Storage: s3 | bucket=databend,root=,endpoint=http://172.31.15.63:9000
205-
Cache: none
182+
183+
Storage: s3 | bucket=databend,root=,endpoint=http://host.docker.internal:9000
184+
Disk cache:
185+
storage: none
186+
path: DiskCacheConfig { max_bytes: 21474836480, path: "./.databend/_cache", sync_data: true }
187+
reload policy: reset
188+
206189
Builtin users: databend
207190

191+
Builtin UDFs:
192+
208193
Admin
209194
listened at 0.0.0.0:8080
210195
MySQL
@@ -219,42 +204,34 @@ Databend HTTP
219204
usage: curl -u${USER} -p${PASSWORD}: --request POST '0.0.0.0:8000/v1/query/' --header 'Content-Type: application/json' --data-raw '{"sql": "SELECT avg(number) FROM numbers(100000000)"}'
220205
```
221206
222-
</StepContent>
223-
224-
<StepContent number="4">
225-
226207
### Connect to Databend
227208
228-
In this step, you'll connect to Databend using [BendSQL](../../../30-sql-clients/00-bendsql/index.md) from your local machine.
229-
230-
1. Install BendSQL to your local machine. For instructions, see [Installing BendSQL](../../../30-sql-clients/00-bendsql/index.md#installing-bendsql).
231-
232-
2. Launch a terminal on your local machine, then run the command `bendsql -h <instance_public_ip> -u databend -p databend` to establish a connection with Databend. For example, if your instance's public IP address is `3.142.131.212`, the command would be `bendsql -h 3.142.131.212 -u databend -p databend`.
209+
Launch a terminal on your local machine, then run the following command to connect to Databend:
233210
234211
```shell
235-
bendsql -h 3.142.131.212 -u databend -p databend
212+
bendsql -u databend -p databend
213+
```
236214
237-
Welcome to BendSQL 0.16.0-homebrew.
238-
Connecting to 3.142.131.212:8000 as user databend.
239-
Connected to Databend Query v1.2.410-4b8cd16f0c(rust-1.77.0-nightly-2024-04-08T12:20:44.288903419Z)
215+
```shell
216+
Welcome to BendSQL 0.24.1-f1f7de0(2024-12-04T12:31:18.526234000Z).
217+
Connecting to localhost:8000 as user databend.
218+
Connected to Databend Query v1.2.697-d40f88cc51(rust-1.85.0-nightly-2025-02-14T11:30:59.842308760Z)
219+
Loaded 1411 auto complete keywords from server.
220+
Started web server at 127.0.0.1:8080
240221
```
241222
242223
You're all set! Now, you can execute a simple query to verify the deployment:
243224
244225
```sql
245-
databend@3.142.131.212:8000/default> select now();
226+
databend@localhost:8000/default> SELECT NOW();
246227
247-
SELECT
248-
NOW()
228+
SELECT NOW()
249229
250230
┌────────────────────────────┐
251231
│ now() │
252232
│ Timestamp │
253233
├────────────────────────────┤
254-
│ 2024-04-17 17:53:56.307155 │
234+
│ 2025-04-10 03:14:06.778815 │
255235
└────────────────────────────┘
256-
1 row read in 0.178 sec. Processed 1 row, 1 B (5.62 row/s, 5 B/s)
257-
```
258-
259-
</StepContent>
260-
</StepsWrap>
236+
1 row read in 0.003 sec. Processed 1 row, 1 B (333.33 rows/s, 333 B/s)
237+
```

0 commit comments

Comments
 (0)