| title | Local Installation |
|---|---|
| sidebar-title | Local Installation |
| description | Install Dynamo on a local machine or VM with containers or PyPI |
import { InstallSelector } from "@/components/InstallSelector";
This guide walks through installing Dynamo on a local machine or VM with one or more GPUs.
For production multi-node clusters, see the Kubernetes Deployment Guide. To build from source for development, see Building from Source.
| Requirement | Supported |
|---|---|
| GPU | NVIDIA Ampere, Ada Lovelace, Hopper, Blackwell |
| OS | Ubuntu 22.04, Ubuntu 24.04 |
| Architecture | x86_64, ARM64 (ARM64 requires Ubuntu 24.04) |
| CUDA | 12.9+ or 13.0+ (B300/GB300 require CUDA 13) |
| Python | 3.10, 3.12 |
| Driver | 575.51.03+ (CUDA 12) or 580.00.03+ (CUDA 13) |
TensorRT-LLM does not support Python 3.11.
For the full compatibility matrix including backend framework versions, see the Support Matrix.
Before installing Dynamo, make sure the host has the following. The GPU driver is the only piece that must live on the host in every case — the prebuilt container bundles CUDA and all framework dependencies, so the CUDA toolkit is only needed for the Python virtual environment path.
| Prerequisite | Container path | Python venv path | Notes |
|---|---|---|---|
| NVIDIA GPU driver | ✅ Required | ✅ Required | The kernel driver cannot be containerized. Use the version from System Requirements above and verify with nvidia-smi. |
| Docker Engine | ✅ Required | — | Runs the runtime image and the NATS/etcd Compose stack. |
| NVIDIA Container Toolkit | ✅ Required | — | Makes --gpus all work. Without it, the container starts but cannot see the GPU. |
CUDA toolkit (nvcc, headers) |
Bundled in container | ✅ Required | Only the host-install path needs it; the container already includes CUDA. |
HF_TOKEN |
For gated models | For gated models | Export it before launch and accept the model license on the Hugging Face model page: export HF_TOKEN=hf_... |
Verify the driver is installed and the GPU is visible:
nvidia-smi```bash
git clone https://github.com/ai-dynamo/dynamo.git
cd dynamo
```
```bash
docker compose -f dev/docker-compose.yml up -d
```
Leave these running for the rest of the guide.
<Note>
NATS and etcd are **not strictly required** for a single-machine setup — you can pass `--discovery-backend file` to the frontend and worker to skip them entirely (discovery falls back to the local filesystem and events to ZMQ). They are **recommended** because they match the multi-node, production, and Kubernetes deployment path. See [Discovery and events on a single machine](#discovery-and-events-on-a-single-machine) below for the trade-offs.
</Note>
- **Prebuilt container (recommended)** — all dependencies are pre-installed and the repository contents are bundled at `/workspace`. Because the frontend and worker run *inside* this container, you'll need to `docker exec` into it to run subsequent commands.
- **Python virtual environment (PyPI)** — install the `ai-dynamo` wheel for your backend directly on the host.
Pick a backend, release channel, backend version, and install form. The default selection starts the latest stable SGLang container.
<InstallSelector hardware="nvidia" />
The container command drops you into a shell with Dynamo and its backend dependencies installed. `--network host` lets the container reach NATS and etcd on the host.
Before using a wheel command, create and activate a Python virtual environment:
```bash
# Install uv (recommended Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Create and activate the virtual environment
uv venv venv
source venv/bin/activate
uv pip install pip
```
Install the host packages required by your backend before running the selected wheel command:
<Tabs>
<Tab title="SGLang" language="sglang">
```bash
sudo apt install python3-dev
```
For CUDA 13 (B300/GB300), the container is recommended. See the [SGLang install docs](https://docs.sglang.io/docs/get-started/install) for details.
</Tab>
<Tab title="TensorRT-LLM" language="trtllm">
TensorRT-LLM is available through the prebuilt container. The selector does not offer a wheel command for this backend. See the [TensorRT-LLM backend guide](../../developer-guide/knowledge-base/modular-components/backends/tensorrt-llm/overview.md) for details.
</Tab>
<Tab title="vLLM" language="vllm">
```bash
sudo apt install python3-dev libxcb1
```
</Tab>
</Tabs>
See [Release Artifacts](../../reference/general/release-artifacts.mdx) for the complete artifact inventory.
```bash
docker exec -it <container_id> bash
```
</Note>
Verify the CLI is installed and callable:
```bash
python3 -m dynamo.frontend --help
```
You can also run additional system checks from the repository root:
```bash
python3 dev/sanity_check.py
```
Dynamo is a distributed runtime: the frontend (the OpenAI-compatible HTTP server) and the worker (the engine running your model) are separate processes that have to find each other. Two planes make that happen:
- Discovery plane — how the frontend learns which workers exist. Workers register their endpoints on startup; the frontend watches for them and routes requests accordingly.
- Event plane — how components exchange KV-cache events and worker metrics, which power features like KV-aware routing and the planner.
On a single machine you can run either of two configurations:
--discovery-backend file |
etcd + NATS (this guide) | |
|---|---|---|
| Extra services | None | etcd + NATS (via Docker Compose) |
| Discovery | Local filesystem | etcd |
| Events | ZMQ (in-process) | NATS |
| Scope | Single machine only | Single or multi-node |
| Failure cleanup | None | Lease-based auto-cleanup (default 10s TTL) — a crashed worker's endpoints are removed automatically and the frontend reroutes |
| KV events / routing / planner | Not available | Available |
| Matches production / Kubernetes | No | Yes |
The file mode is the lightest way to get a model answering with no prerequisites — ideal for a quick local smoke test. This guide uses etcd + NATS instead because it is the same discovery path Dynamo uses for multi-node and Kubernetes deployments, so what you learn here carries forward to production.
To use the zero-dependency mode instead, skip the infrastructure step above and pass --discovery-backend file to both the frontend and each worker.
CUDA/driver version mismatch
Run nvidia-smi to check your driver version. Dynamo requires driver 575.51.03+ for CUDA 12 or 580.00.03+ for CUDA 13. B300/GB300 GPUs require CUDA 13. See the Support Matrix for full requirements.
Python 3.11 with TensorRT-LLM
TensorRT-LLM does not support Python 3.11. If you see installation failures with TensorRT-LLM, check your Python version with python3 --version. Use Python 3.10 or 3.12 instead.
Container runs but GPU not detected
Ensure you passed --gpus all to docker run. Without this flag, the container won't have access to GPUs:
# Correct
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0
# Wrong -- no GPU access
docker run --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0- Backend Guides -- Backend-specific configuration and features
- Disaggregated Serving -- Scale prefill and decode independently
- KV Cache Aware Routing -- Smart request routing
- Kubernetes Deployment -- Production multi-node deployments