Speculative decoding is an optimization technique that uses a smaller "draft" model to predict multiple tokens, which are then verified by the main model in parallel. This can significantly reduce latency for autoregressive generation.
| Backend | Status | Notes |
|---|---|---|
| vLLM | ✅ | Eagle3 draft model support |
| SGLang | 🚧 | Not yet documented |
| TensorRT-LLM | 🚧 | Not yet documented |
Speculative decoding works by:
- Draft phase: A smaller, faster model generates candidate tokens
- Verify phase: The main model verifies these candidates in a single forward pass
- Accept/reject: Tokens are accepted if they match what the main model would have generated
This approach trades off additional compute for lower latency, as multiple tokens can be generated per forward pass of the main model.
This guide walks through deploying Meta-Llama-3.1-8B-Instruct with Eagle3 speculative decoding on a single GPU with at least 16GB VRAM.
- Start infrastructure services:
docker compose -f deploy/docker-compose.yml up -d- Build and run the vLLM container:
./container/build.sh --framework VLLM
./container/run.sh -it --framework VLLM --mount-workspace- Set up Hugging Face access (Meta-Llama-3.1-8B-Instruct is gated):
export HUGGING_FACE_HUB_TOKEN="your_token_here"
export HF_TOKEN=$HUGGING_FACE_HUB_TOKENcd examples/backends/vllm
bash launch/agg_spec_decoding.shcurl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"messages": [
{"role": "user", "content": "Write a poem about why Sakura trees are beautiful."}
],
"max_tokens": 250
}'| Backend | Guide |
|---|---|
| vLLM | speculative_decoding_vllm.md |
- vLLM Backend - Full vLLM deployment guide
- Disaggregated Serving - Alternative optimization approach
- Meta-Llama-3.1-8B-Instruct on Hugging Face