Skip to content

Commit f0518a3

Browse files
committed
add docker-quickstart doc
Signed-off-by: JaredforReal <[email protected]>
1 parent e441546 commit f0518a3

File tree

3 files changed

+113
-3
lines changed

3 files changed

+113
-3
lines changed

docker/README.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ Why: `bert_model.model_id` in `config/config.yaml` points to a remote model (`se
7979

8080
Fix options:
8181

82-
- Allow network access in the container (recommended):
82+
- Allow network access in the container (online):
8383
- Ensure your host can resolve DNS, or add DNS servers to the `semantic-router` service in `docker-compose.yml`:
8484

8585
```yaml
@@ -109,5 +109,3 @@ Fix options:
109109
### 2) Port already in use
110110

111111
Make sure 8801, 50051, 19000 are not bound by other processes. Adjust ports in `docker-compose.yml` if needed.
112-
113-
```
Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Docker Compose Quick Start
2+
3+
Run Semantic Router + Envoy locally using Docker Compose v2.
4+
5+
## Prerequisites
6+
7+
- Docker Engine and Docker Compose v2 (use the `docker compose` command, not the legacy `docker-compose`)
8+
- Ensure ports 8801, 50051, 19000 are free
9+
10+
## Install and Run with Docker Compose v2
11+
12+
1) Clone the repo and move into it (from your workspace root):
13+
14+
```bash
15+
git clone https://github.com/vllm-project/semantic-router.git
16+
cd semantic-router
17+
```
18+
19+
2) Download required models (classification models):
20+
21+
```bash
22+
make download-models
23+
```
24+
25+
This downloads the classification models used by the router:
26+
27+
- Category classifier (ModernBERT-base)
28+
- PII classifier (ModernBERT-base)
29+
- Jailbreak classifier (ModernBERT-base)
30+
31+
Note: The BERT similarity model defaults to a remote Hugging Face model. See Troubleshooting for offline/local usage.
32+
33+
3) Start the services with Docker Compose v2:
34+
35+
```bash
36+
# Start core services (semantic-router + envoy)
37+
docker compose up --build
38+
39+
# Or run in background (recommended)
40+
docker compose up --build -d
41+
42+
# With testing profile (includes mock vLLM)
43+
docker compose --profile testing up --build
44+
```
45+
46+
4) Verify
47+
48+
- Semantic Router (gRPC): localhost:50051
49+
- Envoy Proxy: http://localhost:8801
50+
- Envoy Admin: http://localhost:19000
51+
52+
## Common Operations
53+
54+
```bash
55+
# View service status
56+
docker compose ps
57+
58+
# Follow logs for the router service
59+
docker compose logs -f semantic-router
60+
61+
# Exec into the router container
62+
docker compose exec semantic-router bash
63+
64+
# Stop and clean up containers
65+
docker compose down
66+
```
67+
68+
## Troubleshooting
69+
70+
### 1) Router exits immediately with a Hugging Face DNS/download error
71+
72+
Symptoms (from `docker compose logs -f semantic-router`):
73+
74+
```
75+
Failed to initialize BERT: request error: https://huggingface.co/... Dns Failed: resolve dns name 'huggingface.co:443'
76+
```
77+
78+
Why: `bert_model.model_id` in `config/config.yaml` points to a remote model (`sentence-transformers/all-MiniLM-L12-v2`). If the container cannot resolve or reach the internet, startup fails.
79+
80+
Fix options:
81+
82+
- Allow network access in the container (online):
83+
- Ensure your host can resolve DNS, or add DNS servers to the `semantic-router` service in `docker-compose.yml`:
84+
85+
```yaml
86+
services:
87+
semantic-router:
88+
# ...
89+
dns:
90+
- 1.1.1.1
91+
- 8.8.8.8
92+
```
93+
94+
- If behind a proxy, set `http_proxy/https_proxy/no_proxy` env vars for the service.
95+
96+
- Use a local copy of the model (offline):
97+
1. Download `sentence-transformers/all-MiniLM-L12-v2` to `./models/sentence-transformers/all-MiniLM-L12-v2/` on the host.
98+
2. Update `config/config.yaml` to use the local path (mounted into the container at `/app/models`):
99+
100+
```yaml
101+
bert_model:
102+
model_id: "models/sentence-transformers/all-MiniLM-L12-v2"
103+
threshold: 0.6
104+
use_cpu: true
105+
```
106+
107+
3. Recreate services: `docker compose up -d --build`
108+
109+
### 2) Port already in use
110+
111+
Make sure 8801, 50051, 19000 are not bound by other processes. Adjust ports in `docker-compose.yml` if needed.

website/sidebars.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ const sidebars = {
4040
'getting-started/installation',
4141
'getting-started/reasoning',
4242
'getting-started/configuration',
43+
'getting-started/docker-quickstart',
4344
],
4445
},
4546
{

0 commit comments

Comments
 (0)