Skip to content

MIGRATION_v2.md

Latest commit

 

History

History
161 lines (106 loc) · 5.21 KB

MIGRATION_v2.md

File metadata and controls

161 lines (106 loc) · 5.21 KB
Raw

Migrating from Shimmy v1.x to v2.0

This guide covers the breaking changes and new defaults introduced in Shimmy v2.0, which replaces the llama.cpp inference backend with the Airframe WebGPU engine.

TL;DR

What changed v1.x v2.0
Default inference engine llama.cpp Airframe (WGSL/WebGPU)
GPU acceleration CUDA / Vulkan / OpenCL / MLX WebGPU via wgpu (auto-selected)
Model configuration --model-path or hardcoded path SHIMMY_BASE_GGUF env var or --model-path
--gpu-backend cuda/vulkan/opencl Worked Ignored (Airframe selects adapter via wgpu)
MoE models Supported (default path) Requires --legacy flag
cargo install shimmy Broken (publish = false) Works (installs huggingface engine)
Binary distribution Pre-built binaries Pre-built binaries (Airframe engine included)

What Is Airframe?

Airframe is a pure-Rust WebGPU inference engine. It replaces the C++ llama.cpp library with:

  • WGSL compute shaders compiled at runtime by wgpu
  • F32 precision throughout (no quantized-on-GPU approximations)
  • YaRN RoPE scaling for extended context windows
  • No C++ toolchain, no CUDA toolkit, no Vulkan SDK required

The llama.cpp code path is historically parked — it still works via --legacy but receives no new features.

Breaking Changes

1. Default engine is now Airframe

The default inference path changed. No flag is needed for the new default.

# v1.x: implicit llama.cpp
shimmy serve --model-path /path/to/model.gguf

# v2.0: same command, now uses Airframe
shimmy serve --model-path /path/to/model.gguf

# v2.0: preferred way — set model via env var
SHIMMY_BASE_GGUF=/path/to/model.gguf shimmy serve

2. SHIMMY_BASE_GGUF is now required (or --model-path)

In v1.x, some builds had a hardcoded default model path. v2.0 requires an explicit model path.

# Required: set the model path
export SHIMMY_BASE_GGUF=/path/to/TinyLlama-1.1B-Chat-v1.0.Q4_0.gguf
shimmy serve

# Or use the flag:
shimmy serve --model-path /path/to/model.gguf

If neither is set, the server will fail to start with a clear error message.

3. --gpu-backend cuda/vulkan/opencl flags are ignored

In v1.x these flags selected the llama.cpp GPU backend. In v2.0, Airframe uses wgpu's adapter enumeration — the GPU is always auto-selected. The --gpu-backend flag is silently ignored by the Airframe engine.

# v1.x: forced CUDA
shimmy serve --gpu-backend cuda

# v2.0: all --gpu-backend values are ignored; wgpu selects the best available adapter
shimmy serve      # Airframe auto-selects GPU adapter via wgpu

To see which GPU adapter was selected:

shimmy gpu-info

4. MoE models require --legacy

Mixture-of-Experts models (e.g., Mixtral) require the llama.cpp backend. Airframe does not yet support MoE routing.

# v1.x: MoE worked by default
shimmy serve --cpu-moe --n-cpu-moe 8

# v2.0: must use --legacy for MoE
shimmy serve --legacy --cpu-moe --n-cpu-moe 8

# Or set via environment:
SHIMMY_ENGINE_BACKEND=llama shimmy serve --cpu-moe --n-cpu-moe 8

What Stayed the Same

  • OpenAI API (/v1/chat/completions, /v1/models, etc.) — 100% compatible, no client changes needed
  • Model format — GGUF files work as before; Airframe reads the same GGUF metadata
  • Port and bind configuration — --bind, SHIMMY_PORT, auto-allocation all work identically
  • Model discovery — Hugging Face cache, Ollama directory, ./models/, SHIMMY_BASE_GGUF
  • Streaming — SSE and WebSocket streaming are unchanged
  • Template routing — Chat templates are applied identically
  • --legacy flag — Restores full llama.cpp behavior for any workflow that needs it

Opting Back to v1.x Behavior

If you need the old llama.cpp engine for any reason:

# Per-invocation
shimmy serve --legacy --model-path /path/to/model.gguf

# Permanently via environment
export SHIMMY_ENGINE_BACKEND=llama
shimmy serve --model-path /path/to/model.gguf

The --legacy path supports all v1.x flags: --gpu-backend cuda/vulkan/opencl, --cpu-moe, --n-cpu-moe, etc.

Extended Context (New in v2.0)

Airframe supports extended context via YaRN RoPE scaling. Set SHIMMY_MAX_CTX to the desired context length:

SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=8192 shimmy serve

Supported values: 2048 (default), 4096, 8192, 16384, 32768. YaRN scaling activates automatically when SHIMMY_MAX_CTX exceeds the model's base context length.

Upgrading

Binary upgrade

Download the latest binary for your platform from GitHub Releases. Replace your existing binary.

From cargo install

cargo install shimmy

Note: cargo install shimmy installs the huggingface engine variant (no GPU). For the Airframe GPU engine, use the GitHub Releases binary.

Getting Help