From e4415467c8dc084556098fbaf0524b6514d0a9f7 Mon Sep 17 00:00:00 2001 From: JaredforReal Date: Sat, 20 Sep 2025 13:58:12 +0800 Subject: [PATCH 1/5] upgrade docker/README.md with docker compose V2 quick start guide Signed-off-by: JaredforReal --- docker/README.md | 129 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 92 insertions(+), 37 deletions(-) diff --git a/docker/README.md b/docker/README.md index 22b6ec95..e9fc858e 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,58 +1,113 @@ -# Docker Compose Quick Start Guide +# Docker Compose (v2) Quick Start Guide -This Docker Compose configuration allows you to quickly run Semantic Router + Envoy proxy locally. +Run Semantic Router + Envoy locally using Docker Compose v2. ## Prerequisites -- Docker and Docker Compose -- Ensure ports 8801, 50051, 19000 are not in use +- Docker Engine and Docker Compose v2 (use the `docker compose` command, not the legacy `docker-compose`) +- Ensure ports 8801, 50051, 19000 are free -## Install in Docker Compose +## Install and Run with Docker Compose v2 -1. **Clone the repository and navigate to the project directory** +1) Clone the repo and move into it (from your workspace root): - ```bash - git clone https://github.com/vllm-project/semantic-router.git - cd semantic_router - ``` +```bash +git clone https://github.com/vllm-project/semantic-router.git +cd semantic-router +``` -2. **Download required models** (if not already present): +2) Download required models (classification models): + +```bash +make download-models +``` - ```bash - make download-models - ``` +This downloads the classification models used by the router: - This will download the necessary ML models for classification: - - Category classifier (ModernBERT-base) - - PII classifier (ModernBERT-base) - - Jailbreak classifier (ModernBERT-base) +- Category classifier (ModernBERT-base) +- PII classifier (ModernBERT-base) +- Jailbreak classifier (ModernBERT-base) -3. **Start the services using Docker Compose** +Note: The BERT similarity model defaults to a remote Hugging Face model. See Troubleshooting for offline/local usage. - ```bash - # Start core services (semantic-router + envoy) - docker-compose up --build +3) Start the services with Docker Compose v2: - # Or run in background - docker-compose up --build -d +```bash +# Start core services (semantic-router + envoy) +docker compose up --build - # Start with testing services (includes mock vLLM) - docker-compose --profile testing up --build - ``` +# Or run in background (recommended) +docker compose up --build -d -4. **Verify the installation** - - Semantic Router: http://localhost:50051 (gRPC service) - - Envoy Proxy: http://localhost:8801 (main endpoint) - - Envoy Admin: http://localhost:19000 (admin interface) +# With testing profile (includes mock vLLM) +docker compose --profile testing up --build +``` -## Quick Start +4) Verify -### 1. Build and Start Services +- Semantic Router (gRPC): localhost:50051 +- Envoy Proxy: http://localhost:8801 +- Envoy Admin: http://localhost:19000 + +## Common Operations ```bash -# Start core services (semantic-router + envoy) -docker-compose up --build +# View service status +docker compose ps + +# Follow logs for the router service +docker compose logs -f semantic-router + +# Exec into the router container +docker compose exec semantic-router bash + +# Stop and clean up containers +docker compose down +``` + +## Troubleshooting + +### 1) Router exits immediately with a Hugging Face DNS/download error + +Symptoms (from `docker compose logs -f semantic-router`): + +``` +Failed to initialize BERT: request error: https://huggingface.co/... Dns Failed: resolve dns name 'huggingface.co:443' +``` + +Why: `bert_model.model_id` in `config/config.yaml` points to a remote model (`sentence-transformers/all-MiniLM-L12-v2`). If the container cannot resolve or reach the internet, startup fails. + +Fix options: + +- Allow network access in the container (recommended): + - Ensure your host can resolve DNS, or add DNS servers to the `semantic-router` service in `docker-compose.yml`: + + ```yaml + services: + semantic-router: + # ... + dns: + - 1.1.1.1 + - 8.8.8.8 + ``` + + - If behind a proxy, set `http_proxy/https_proxy/no_proxy` env vars for the service. + +- Use a local copy of the model (offline): + 1. Download `sentence-transformers/all-MiniLM-L12-v2` to `./models/sentence-transformers/all-MiniLM-L12-v2/` on the host. + 2. Update `config/config.yaml` to use the local path (mounted into the container at `/app/models`): + + ```yaml + bert_model: + model_id: "models/sentence-transformers/all-MiniLM-L12-v2" + threshold: 0.6 + use_cpu: true + ``` + + 3. Recreate services: `docker compose up -d --build` + +### 2) Port already in use + +Make sure 8801, 50051, 19000 are not bound by other processes. Adjust ports in `docker-compose.yml` if needed. -# Or run in background -docker-compose up --build -d ``` From f0518a3b6a84ecffc46a81d420e76d81a57fd402 Mon Sep 17 00:00:00 2001 From: JaredforReal Date: Sat, 20 Sep 2025 14:33:31 +0800 Subject: [PATCH 2/5] add docker-quickstart doc Signed-off-by: JaredforReal --- docker/README.md | 4 +- .../docs/getting-started/docker-quickstart.md | 111 ++++++++++++++++++ website/sidebars.js | 1 + 3 files changed, 113 insertions(+), 3 deletions(-) create mode 100644 website/docs/getting-started/docker-quickstart.md diff --git a/docker/README.md b/docker/README.md index e9fc858e..1a65895c 100644 --- a/docker/README.md +++ b/docker/README.md @@ -79,7 +79,7 @@ Why: `bert_model.model_id` in `config/config.yaml` points to a remote model (`se Fix options: -- Allow network access in the container (recommended): +- Allow network access in the container (online): - Ensure your host can resolve DNS, or add DNS servers to the `semantic-router` service in `docker-compose.yml`: ```yaml @@ -109,5 +109,3 @@ Fix options: ### 2) Port already in use Make sure 8801, 50051, 19000 are not bound by other processes. Adjust ports in `docker-compose.yml` if needed. - -``` diff --git a/website/docs/getting-started/docker-quickstart.md b/website/docs/getting-started/docker-quickstart.md new file mode 100644 index 00000000..27c20757 --- /dev/null +++ b/website/docs/getting-started/docker-quickstart.md @@ -0,0 +1,111 @@ +# Docker Compose Quick Start + +Run Semantic Router + Envoy locally using Docker Compose v2. + +## Prerequisites + +- Docker Engine and Docker Compose v2 (use the `docker compose` command, not the legacy `docker-compose`) +- Ensure ports 8801, 50051, 19000 are free + +## Install and Run with Docker Compose v2 + +1) Clone the repo and move into it (from your workspace root): + +```bash +git clone https://github.com/vllm-project/semantic-router.git +cd semantic-router +``` + +2) Download required models (classification models): + +```bash +make download-models +``` + +This downloads the classification models used by the router: + +- Category classifier (ModernBERT-base) +- PII classifier (ModernBERT-base) +- Jailbreak classifier (ModernBERT-base) + +Note: The BERT similarity model defaults to a remote Hugging Face model. See Troubleshooting for offline/local usage. + +3) Start the services with Docker Compose v2: + +```bash +# Start core services (semantic-router + envoy) +docker compose up --build + +# Or run in background (recommended) +docker compose up --build -d + +# With testing profile (includes mock vLLM) +docker compose --profile testing up --build +``` + +4) Verify + +- Semantic Router (gRPC): localhost:50051 +- Envoy Proxy: http://localhost:8801 +- Envoy Admin: http://localhost:19000 + +## Common Operations + +```bash +# View service status +docker compose ps + +# Follow logs for the router service +docker compose logs -f semantic-router + +# Exec into the router container +docker compose exec semantic-router bash + +# Stop and clean up containers +docker compose down +``` + +## Troubleshooting + +### 1) Router exits immediately with a Hugging Face DNS/download error + +Symptoms (from `docker compose logs -f semantic-router`): + +``` +Failed to initialize BERT: request error: https://huggingface.co/... Dns Failed: resolve dns name 'huggingface.co:443' +``` + +Why: `bert_model.model_id` in `config/config.yaml` points to a remote model (`sentence-transformers/all-MiniLM-L12-v2`). If the container cannot resolve or reach the internet, startup fails. + +Fix options: + +- Allow network access in the container (online): + - Ensure your host can resolve DNS, or add DNS servers to the `semantic-router` service in `docker-compose.yml`: + + ```yaml + services: + semantic-router: + # ... + dns: + - 1.1.1.1 + - 8.8.8.8 + ``` + + - If behind a proxy, set `http_proxy/https_proxy/no_proxy` env vars for the service. + +- Use a local copy of the model (offline): + 1. Download `sentence-transformers/all-MiniLM-L12-v2` to `./models/sentence-transformers/all-MiniLM-L12-v2/` on the host. + 2. Update `config/config.yaml` to use the local path (mounted into the container at `/app/models`): + + ```yaml + bert_model: + model_id: "models/sentence-transformers/all-MiniLM-L12-v2" + threshold: 0.6 + use_cpu: true + ``` + + 3. Recreate services: `docker compose up -d --build` + +### 2) Port already in use + +Make sure 8801, 50051, 19000 are not bound by other processes. Adjust ports in `docker-compose.yml` if needed. diff --git a/website/sidebars.js b/website/sidebars.js index edf8296d..83587120 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -40,6 +40,7 @@ const sidebars = { 'getting-started/installation', 'getting-started/reasoning', 'getting-started/configuration', + 'getting-started/docker-quickstart', ], }, { From dd5f514b464832aa9665e964668b11fce9a6e5d5 Mon Sep 17 00:00:00 2001 From: JaredforReal Date: Sat, 20 Sep 2025 14:55:23 +0800 Subject: [PATCH 3/5] remove docker dir & rerank docker-quickstart in sidebar Signed-off-by: JaredforReal --- docker/README.md | 111 -------------------------------------------- website/sidebars.js | 2 +- 2 files changed, 1 insertion(+), 112 deletions(-) delete mode 100644 docker/README.md diff --git a/docker/README.md b/docker/README.md deleted file mode 100644 index 1a65895c..00000000 --- a/docker/README.md +++ /dev/null @@ -1,111 +0,0 @@ -# Docker Compose (v2) Quick Start Guide - -Run Semantic Router + Envoy locally using Docker Compose v2. - -## Prerequisites - -- Docker Engine and Docker Compose v2 (use the `docker compose` command, not the legacy `docker-compose`) -- Ensure ports 8801, 50051, 19000 are free - -## Install and Run with Docker Compose v2 - -1) Clone the repo and move into it (from your workspace root): - -```bash -git clone https://github.com/vllm-project/semantic-router.git -cd semantic-router -``` - -2) Download required models (classification models): - -```bash -make download-models -``` - -This downloads the classification models used by the router: - -- Category classifier (ModernBERT-base) -- PII classifier (ModernBERT-base) -- Jailbreak classifier (ModernBERT-base) - -Note: The BERT similarity model defaults to a remote Hugging Face model. See Troubleshooting for offline/local usage. - -3) Start the services with Docker Compose v2: - -```bash -# Start core services (semantic-router + envoy) -docker compose up --build - -# Or run in background (recommended) -docker compose up --build -d - -# With testing profile (includes mock vLLM) -docker compose --profile testing up --build -``` - -4) Verify - -- Semantic Router (gRPC): localhost:50051 -- Envoy Proxy: http://localhost:8801 -- Envoy Admin: http://localhost:19000 - -## Common Operations - -```bash -# View service status -docker compose ps - -# Follow logs for the router service -docker compose logs -f semantic-router - -# Exec into the router container -docker compose exec semantic-router bash - -# Stop and clean up containers -docker compose down -``` - -## Troubleshooting - -### 1) Router exits immediately with a Hugging Face DNS/download error - -Symptoms (from `docker compose logs -f semantic-router`): - -``` -Failed to initialize BERT: request error: https://huggingface.co/... Dns Failed: resolve dns name 'huggingface.co:443' -``` - -Why: `bert_model.model_id` in `config/config.yaml` points to a remote model (`sentence-transformers/all-MiniLM-L12-v2`). If the container cannot resolve or reach the internet, startup fails. - -Fix options: - -- Allow network access in the container (online): - - Ensure your host can resolve DNS, or add DNS servers to the `semantic-router` service in `docker-compose.yml`: - - ```yaml - services: - semantic-router: - # ... - dns: - - 1.1.1.1 - - 8.8.8.8 - ``` - - - If behind a proxy, set `http_proxy/https_proxy/no_proxy` env vars for the service. - -- Use a local copy of the model (offline): - 1. Download `sentence-transformers/all-MiniLM-L12-v2` to `./models/sentence-transformers/all-MiniLM-L12-v2/` on the host. - 2. Update `config/config.yaml` to use the local path (mounted into the container at `/app/models`): - - ```yaml - bert_model: - model_id: "models/sentence-transformers/all-MiniLM-L12-v2" - threshold: 0.6 - use_cpu: true - ``` - - 3. Recreate services: `docker compose up -d --build` - -### 2) Port already in use - -Make sure 8801, 50051, 19000 are not bound by other processes. Adjust ports in `docker-compose.yml` if needed. diff --git a/website/sidebars.js b/website/sidebars.js index 83587120..97f19c64 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -38,9 +38,9 @@ const sidebars = { label: 'Getting Started', items: [ 'getting-started/installation', + 'getting-started/docker-quickstart', 'getting-started/reasoning', 'getting-started/configuration', - 'getting-started/docker-quickstart', ], }, { From 59a575c04ac714757cff577dc59745084e1a2818 Mon Sep 17 00:00:00 2001 From: JaredforReal Date: Sat, 20 Sep 2025 15:09:38 +0800 Subject: [PATCH 4/5] add install docker compose v2 section Signed-off-by: JaredforReal --- .../docs/getting-started/docker-quickstart.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/website/docs/getting-started/docker-quickstart.md b/website/docs/getting-started/docker-quickstart.md index 27c20757..35db6bf2 100644 --- a/website/docs/getting-started/docker-quickstart.md +++ b/website/docs/getting-started/docker-quickstart.md @@ -5,6 +5,29 @@ Run Semantic Router + Envoy locally using Docker Compose v2. ## Prerequisites - Docker Engine and Docker Compose v2 (use the `docker compose` command, not the legacy `docker-compose`) + + ```bash + # Verify + docker compose version + ``` + + Install Docker Compose v2 for Ubuntu(if missing), see more in [Docker Compose Plugin Installation](https://docs.docker.com/compose/install/linux/#install-using-the-repository) + + ```bash + # Remove legacy v1 if present (optional) + sudo apt-get remove -y docker-compose || true + + sudo apt-get update + sudo apt-get install -y ca-certificates curl gnupg + sudo install -m 0755 -d /etc/apt/keyrings + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --yes --dearmor -o /etc/apt/keyrings/docker.gpg + echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + sudo apt-get update + sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin + + docker compose version + ``` + - Ensure ports 8801, 50051, 19000 are free ## Install and Run with Docker Compose v2 From d3ebe329c6867e5d851d5fffdc6d5e3ee0eef1a9 Mon Sep 17 00:00:00 2001 From: JaredforReal Date: Sat, 20 Sep 2025 15:20:09 +0800 Subject: [PATCH 5/5] change title Signed-off-by: JaredforReal --- website/docs/getting-started/docker-quickstart.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/getting-started/docker-quickstart.md b/website/docs/getting-started/docker-quickstart.md index 35db6bf2..e06bed44 100644 --- a/website/docs/getting-started/docker-quickstart.md +++ b/website/docs/getting-started/docker-quickstart.md @@ -1,4 +1,4 @@ -# Docker Compose Quick Start +# Install with Docker Compose Run Semantic Router + Envoy locally using Docker Compose v2.