MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) and Omni Models (VLMs with audio and video support) on your Mac using MLX.
Some models have detailed documentation with prompt formats, examples, and best practices:
| Model | Documentation |
|---|---|
| DeepSeek-OCR | Docs |
| DeepSeek-OCR-2 | Docs |
| GLM-OCR | Docs |
The easiest way to get started is to install the mlx-vlm package using pip:
pip install -U mlx-vlmGenerate output from a model using the CLI:
# Text generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"
# Image generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg
# Audio generation (New)
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav
# Multi-modal generation (Image + Audio)
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wavLaunch a chat interface using Gradio:
mlx_vlm.chat_ui --model mlx-community/Qwen2-VL-2B-Instruct-4bitHere's an example of how to use MLX-VLM in a Python script:
import mlx.core as mx
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
# Load the model
model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = load_config(model_path)
# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
# image = [Image.open("...")] can also be used with PIL.Image.Image objects
prompt = "Describe this image."
# Apply chat template
formatted_prompt = apply_chat_template(
processor, config, prompt, num_images=len(image)
)
# Generate output
output = generate(model, processor, formatted_prompt, image, verbose=False)
print(output)from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
# Load model with audio support
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
# Prepare audio input
audio = ["/path/to/audio1.wav", "/path/to/audio2.mp3"]
prompt = "Describe what you hear in these audio files."
# Apply chat template with audio
formatted_prompt = apply_chat_template(
processor, config, prompt, num_audios=len(audio)
)
# Generate output with audio
output = generate(model, processor, formatted_prompt, audio=audio, verbose=False)
print(output)from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
# Load multi-modal model
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
# Prepare inputs
image = ["/path/to/image.jpg"]
audio = ["/path/to/audio.wav"]
prompt = ""
# Apply chat template
formatted_prompt = apply_chat_template(
processor, config, prompt,
num_images=len(image),
num_audios=len(audio)
)
# Generate output
output = generate(model, processor, formatted_prompt, image, audio=audio, verbose=False)
print(output)Start the server:
mlx_vlm.server --port 8080
# With trust remote code enabled (required for some models)
mlx_vlm.server --trust-remote-code--host: Host address (default:0.0.0.0)--port: Port number (default:8080)--trust-remote-code: Trust remote code when loading models from Hugging Face Hub
You can also set trust remote code via environment variable:
MLX_TRUST_REMOTE_CODE=true mlx_vlm.serverThe server provides multiple endpoints for different use cases and supports dynamic model loading/unloading with caching (one model at a time).
/models- List models available locally/chat/completions- OpenAI-compatible chat-style interaction endpoint with support for images, audio, and text/responses- OpenAI-compatible responses endpoint/health- Check server status/unload- Unload current model from memory
curl "http://localhost:8080/models"curl -X POST "http://localhost:8080/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
"messages": [
{
"role": "user",
"content": "Hello, how are you",
}
],
"stream": true,
"max_tokens": 100
}'curl -X POST "http://localhost:8080/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Qwen2.5-VL-32B-Instruct-8bit",
[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": This is today's chart for energy demand in California. Can you provide an analysis of the chart and comment on the implications for renewable energy in California?"
},
{
"type": "input_image",
"image_url": "/path/to/repo/examples/images/renewables_california.png"
}
]
}
],
"stream": true,
"max_tokens": 1000
}'curl -X POST "http://localhost:8080/generate" \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/gemma-3n-E2B-it-4bit",
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "Describe what you hear in these audio files" },
{"type": "input_audio", "input_audio": "/path/to/audio1.wav"}
{"type": "input_audio", "input_audio": "https://example.com/audio2.mp3"}
]
}
],
"stream": true,
"max_tokens": 500
}'curl -X POST "http://localhost:8080/generate" \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/gemma-3n-E2B-it-4bit",
"messages": [
{
"role": "user",
"content": [
{"type": "input_image", "image_url": "/path/to/image.jpg"},
{"type": "input_audio", "input_audio": "/path/to/audio.wav"}
]
}
],
"max_tokens": 100
}'curl -X POST "http://localhost:8080/responses" \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
"messages": [
{
"role": "user",
"content": [
{"type": "input_text", "text": "What is in this image?"},
{"type": "input_image", "image_url": "/path/to/image.jpg"}
]
}
],
"max_tokens": 100
}'model: Model identifier (required)messages: Chat messages for chat/OpenAI endpointsmax_tokens: Maximum tokens to generatetemperature: Sampling temperaturetop_p: Top-p sampling parameterstream: Enable streaming responses
MLX-VLM supports analyzing multiple images simultaneously with select models. This feature enables more complex visual reasoning tasks and comprehensive analysis across multiple images in a single conversation.
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = model.config
images = ["path/to/image1.jpg", "path/to/image2.jpg"]
prompt = "Compare these two images."
formatted_prompt = apply_chat_template(
processor, config, prompt, num_images=len(images)
)
output = generate(model, processor, formatted_prompt, images, verbose=False)
print(output)mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Compare these images" --image path/to/image1.jpg path/to/image2.jpgMLX-VLM also supports video analysis such as captioning, summarization, and more, with select models.
The following models support video chat:
- Qwen2-VL
- Qwen2.5-VL
- Idefics3
- LLaVA
With more coming soon.
mlx_vlm.video_generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Describe this video" --video path/to/video.mp4 --max-pixels 224 224 --fps 1.0These examples demonstrate how to use multiple images with MLX-VLM for more complex visual reasoning tasks.
MLX-VLM supports fine-tuning models with LoRA and QLoRA.
To learn more about LoRA, please refer to the LoRA.md file.
