Skip to content

Commit c6fa181

Browse files
JaredforRealJaredforReal
authored
docs: add docker compose quickstart (#181)
* upgrade docker/README.md with docker compose V2 quick start guide Signed-off-by: JaredforReal <[email protected]> * add docker-quickstart doc Signed-off-by: JaredforReal <[email protected]> * remove docker dir & rerank docker-quickstart in sidebar Signed-off-by: JaredforReal <[email protected]> * add install docker compose v2 section Signed-off-by: JaredforReal <[email protected]> * change title Signed-off-by: JaredforReal <[email protected]> --------- Signed-off-by: JaredforReal <[email protected]>
1 parent 5d90816 commit c6fa181

File tree

3 files changed

+135
-58
lines changed

3 files changed

+135
-58
lines changed

‎docker/README.md‎

Lines changed: 0 additions & 58 deletions
This file was deleted.

‎website/docs/getting-started/docker-quickstart.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Install with Docker Compose
2+
3+
Run Semantic Router + Envoy locally using Docker Compose v2.
4+
5+
## Prerequisites
6+
7+
- Docker Engine and Docker Compose v2 (use the `docker compose` command, not the legacy `docker-compose`)
8+
9+
```bash
10+
# Verify
11+
docker compose version
12+
```
13+
14+
Install Docker Compose v2 for Ubuntu(if missing), see more in [Docker Compose Plugin Installation](https://docs.docker.com/compose/install/linux/#install-using-the-repository)
15+
16+
```bash
17+
# Remove legacy v1 if present (optional)
18+
sudo apt-get remove -y docker-compose || true
19+
20+
sudo apt-get update
21+
sudo apt-get install -y ca-certificates curl gnupg
22+
sudo install -m 0755 -d /etc/apt/keyrings
23+
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --yes --dearmor -o /etc/apt/keyrings/docker.gpg
24+
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
25+
sudo apt-get update
26+
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
27+
28+
docker compose version
29+
```
30+
31+
- Ensure ports 8801, 50051, 19000 are free
32+
33+
## Install and Run with Docker Compose v2
34+
35+
1) Clone the repo and move into it (from your workspace root):
36+
37+
```bash
38+
git clone https://github.com/vllm-project/semantic-router.git
39+
cd semantic-router
40+
```
41+
42+
2) Download required models (classification models):
43+
44+
```bash
45+
make download-models
46+
```
47+
48+
This downloads the classification models used by the router:
49+
50+
- Category classifier (ModernBERT-base)
51+
- PII classifier (ModernBERT-base)
52+
- Jailbreak classifier (ModernBERT-base)
53+
54+
Note: The BERT similarity model defaults to a remote Hugging Face model. See Troubleshooting for offline/local usage.
55+
56+
3) Start the services with Docker Compose v2:
57+
58+
```bash
59+
# Start core services (semantic-router + envoy)
60+
docker compose up --build
61+
62+
# Or run in background (recommended)
63+
docker compose up --build -d
64+
65+
# With testing profile (includes mock vLLM)
66+
docker compose --profile testing up --build
67+
```
68+
69+
4) Verify
70+
71+
- Semantic Router (gRPC): localhost:50051
72+
- Envoy Proxy: http://localhost:8801
73+
- Envoy Admin: http://localhost:19000
74+
75+
## Common Operations
76+
77+
```bash
78+
# View service status
79+
docker compose ps
80+
81+
# Follow logs for the router service
82+
docker compose logs -f semantic-router
83+
84+
# Exec into the router container
85+
docker compose exec semantic-router bash
86+
87+
# Stop and clean up containers
88+
docker compose down
89+
```
90+
91+
## Troubleshooting
92+
93+
### 1) Router exits immediately with a Hugging Face DNS/download error
94+
95+
Symptoms (from `docker compose logs -f semantic-router`):
96+
97+
```
98+
Failed to initialize BERT: request error: https://huggingface.co/... Dns Failed: resolve dns name 'huggingface.co:443'
99+
```
100+
101+
Why: `bert_model.model_id` in `config/config.yaml` points to a remote model (`sentence-transformers/all-MiniLM-L12-v2`). If the container cannot resolve or reach the internet, startup fails.
102+
103+
Fix options:
104+
105+
- Allow network access in the container (online):
106+
- Ensure your host can resolve DNS, or add DNS servers to the `semantic-router` service in `docker-compose.yml`:
107+
108+
```yaml
109+
services:
110+
semantic-router:
111+
# ...
112+
dns:
113+
- 1.1.1.1
114+
- 8.8.8.8
115+
```
116+
117+
- If behind a proxy, set `http_proxy/https_proxy/no_proxy` env vars for the service.
118+
119+
- Use a local copy of the model (offline):
120+
1. Download `sentence-transformers/all-MiniLM-L12-v2` to `./models/sentence-transformers/all-MiniLM-L12-v2/` on the host.
121+
2. Update `config/config.yaml` to use the local path (mounted into the container at `/app/models`):
122+
123+
```yaml
124+
bert_model:
125+
model_id: "models/sentence-transformers/all-MiniLM-L12-v2"
126+
threshold: 0.6
127+
use_cpu: true
128+
```
129+
130+
3. Recreate services: `docker compose up -d --build`
131+
132+
### 2) Port already in use
133+
134+
Make sure 8801, 50051, 19000 are not bound by other processes. Adjust ports in `docker-compose.yml` if needed.

‎website/sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ const sidebars = {
3838
label: 'Getting Started',
3939
items: [
4040
'getting-started/installation',
41+
'getting-started/docker-quickstart',
4142
'getting-started/reasoning',
4243
'getting-started/configuration',
4344
],

0 commit comments

Comments
 (0)