Nano-vLLM

A lightweight vLLM implementation built from scratch.

Key Features

• 🚀 Fast offline inference - Comparable inference speeds to vLLM
• 📖 Readable codebase - Clean implementation in ~ 1,200 lines of Python code
• ⚡ Optimization Suite - Prefix caching, Tensor Parallelism, Torch compilation, CUDA graph, etc.

Installation

For standard installs with FlashInfer:

pip install "nano-vllm[flashinfer] @ git+https://github.com/GeeeekExplorer/nano-vllm.git"

Model Download

To download the model weights manually, use the following command:

huggingface-cli download --resume-download Qwen/Qwen3-0.6B \
  --local-dir ~/huggingface/Qwen3-0.6B/ \
  --local-dir-use-symlinks False

Quick Start

See example.py for usage. The API mirrors vLLM's interface with minor differences in the LLM.generate method:

from nanovllm import LLM, SamplingParams
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)
prompts = ["Hello, Nano-vLLM."]
outputs = llm.generate(prompts, sampling_params)
outputs[0]["text"]

Benchmark

See bench.py for benchmark.

For a TurboQuant KV-cache comparison across this repo and upstream vLLM, use:

python3 scripts/benchmark_turboquant_compare.py \
  --model ~/huggingface/Qwen3-0.6B/ \
  --nano-backends default,turboquant \
  --vllm-kv-cache-dtypes auto,turboquant_4bit_nc \
  --json-out turboquant_results.json

The comparison script runs each candidate in a fresh subprocess, seeds a shared prefix for the cache-hit phase, then reports cold/cache-hit wall time, throughput, prefix-cache hit ratio where available, and peak CUDA memory.

Test Configuration:

• Hardware: RTX 4070 Laptop (8GB)
• Model: Qwen3-0.6B
• Total Requests: 256 sequences
• Input Length: Randomly sampled between 100–1024 tokens
• Output Length: Randomly sampled between 100–1024 tokens

Performance Results:

Inference Engine	Output Tokens	Time (s)	Throughput (tokens/s)
vLLM	133,966	98.37	1361.84
Nano-vLLM	133,966	93.41	1434.13

Star History