|
| 1 | +# Outlines |
| 2 | + |
| 3 | +## Install |
| 4 | + |
| 5 | +As Outlines is a library allowing you to run models from various different providers, it does not include the necessary dependencies for any provider by default. As a result, to use the [`OutlinesModel`][pydantic_ai.models.OutlinesModel], you must install `pydantic-ai-slim` with an optional group composed of outlines, a dash, and the name of the specific model provider you would use through Outlines. For instance: |
| 6 | + |
| 7 | +```bash |
| 8 | +pip/uv-add "pydantic-ai-slim[outlines-transformers]" |
| 9 | +``` |
| 10 | + |
| 11 | +Or |
| 12 | + |
| 13 | +```bash |
| 14 | +pip/uv-add "pydantic-ai-slim[outlines-mlxlm]" |
| 15 | +``` |
| 16 | + |
| 17 | +There are 5 optional groups for the 5 model providers supported through Outlines: |
| 18 | + |
| 19 | +- `outlines-transformers` |
| 20 | +- `outlines-llamacpp` |
| 21 | +- `outlines-mlxlm` |
| 22 | +- `outlines-sglang` |
| 23 | +- `outlines-vllm-offline` |
| 24 | + |
| 25 | +## Model Initialization |
| 26 | + |
| 27 | +As Outlines is not an inference provider, but instead a library allowing you to run both local and API-based models, instantiating the model is a bit different from the other models available on Pydantic AI. |
| 28 | + |
| 29 | +To initialize the `OutlinesModel` through the `__init__` method, the first argument you must provide has to be an `outlines.Model` or an `outlines.AsyncModel` instance. |
| 30 | + |
| 31 | +For instance: |
| 32 | + |
| 33 | +```python {test="skip"} |
| 34 | +import outlines |
| 35 | +from transformers import AutoModelForCausalLM, AutoTokenizer |
| 36 | + |
| 37 | +from pydantic_ai.models.outlines import OutlinesModel |
| 38 | + |
| 39 | +outlines_model = outlines.from_transformers( |
| 40 | + AutoModelForCausalLM.from_pretrained('erwanf/gpt2-mini'), |
| 41 | + AutoTokenizer.from_pretrained('erwanf/gpt2-mini') |
| 42 | +) |
| 43 | +model = OutlinesModel(outlines_model) |
| 44 | +``` |
| 45 | + |
| 46 | +As you already providing an Outlines model instance, there is no need to provide an `OutlinesProvider` yourself. |
| 47 | + |
| 48 | +### Model Loading Methods |
| 49 | + |
| 50 | +Alternatively, you can use some `OutlinesModel` class methods made to load a specific type of Outlines model directly. To do so, you must provide as arguments the same arguments you would have given to the associated Outlines model loading function (except in the case of SGLang). |
| 51 | + |
| 52 | +There are methods for the 5 Outlines models that are officially supported in the integration into Pydantic AI: |
| 53 | + |
| 54 | +- [`from_transformers`][pydantic_ai.models.OutlinesModel.from_transformers] |
| 55 | +- [`from_llamacpp`][pydantic_ai.models.OutlinesModel.from_llamacpp] |
| 56 | +- [`from_mlxlm`][pydantic_ai.models.OutlinesModel.from_mlxlm] |
| 57 | +- [`from_sglang`][pydantic_ai.models.OutlinesModel.from_sglang] |
| 58 | +- [`from_vllm_offline`][pydantic_ai.models.OutlinesModel.from_vllm_offline] |
| 59 | + |
| 60 | +#### Transformers |
| 61 | + |
| 62 | +```python {test="skip"} |
| 63 | +from transformers import AutoModelForCausalLM, AutoTokenizer |
| 64 | + |
| 65 | +from pydantic_ai.models.outlines import OutlinesModel |
| 66 | + |
| 67 | +model = OutlinesModel.from_transformers( |
| 68 | + AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-mini-4k-instruct'), |
| 69 | + AutoTokenizer.from_pretrained('microsoft/Phi-3-mini-4k-instruct') |
| 70 | +) |
| 71 | +``` |
| 72 | + |
| 73 | +#### LlamaCpp |
| 74 | + |
| 75 | +```python {test="skip"} |
| 76 | +from llama_cpp import Llama |
| 77 | + |
| 78 | +from pydantic_ai.models.outlines import OutlinesModel |
| 79 | + |
| 80 | +model = OutlinesModel.from_llamacpp( |
| 81 | + Llama.from_pretrained( |
| 82 | + repo_id='TheBloke/Mistral-7B-Instruct-v0.2-GGUF', |
| 83 | + filename='mistral-7b-instruct-v0.2.Q5_K_M.gguf', |
| 84 | + ) |
| 85 | +) |
| 86 | +``` |
| 87 | + |
| 88 | +#### MLXLM |
| 89 | + |
| 90 | +```python {test="skip"} |
| 91 | +from mlx_lm import load |
| 92 | + |
| 93 | +from pydantic_ai.models.outlines import OutlinesModel |
| 94 | + |
| 95 | +model = OutlinesModel.from_mlxlm( |
| 96 | + *load('mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit') |
| 97 | +) |
| 98 | +``` |
| 99 | + |
| 100 | +#### SGLang |
| 101 | + |
| 102 | +```python {test="skip"} |
| 103 | +from pydantic_ai.models.outlines import OutlinesModel |
| 104 | + |
| 105 | +model = OutlinesModel.from_sglang( |
| 106 | + 'http://localhost:11434', |
| 107 | + 'api_key', |
| 108 | + 'meta-llama/Llama-3.1-8B' |
| 109 | +) |
| 110 | +``` |
| 111 | + |
| 112 | +#### vLLM Offline |
| 113 | + |
| 114 | +```python {test="skip"} |
| 115 | +from vllm import LLM |
| 116 | + |
| 117 | +from pydantic_ai.models.outlines import OutlinesModel |
| 118 | + |
| 119 | +model = OutlinesModel.from_vllm_offline( |
| 120 | + LLM('microsoft/Phi-3-mini-4k-instruct') |
| 121 | +) |
| 122 | +``` |
| 123 | + |
| 124 | +## Running the model |
| 125 | + |
| 126 | +Once you have initialized an `OutlinesModel`, you can use it with an Agent as with all other Pydantic AI models. |
| 127 | + |
| 128 | +As Outlines is focused on structured output, this provider supports the `output_type` component through the `NativeOutput` format. |
| 129 | + |
| 130 | +```python {test="skip"} |
| 131 | +from pydantic import BaseModel |
| 132 | +from transformers import AutoModelForCausalLM, AutoTokenizer |
| 133 | + |
| 134 | +from pydantic_ai import Agent |
| 135 | +from pydantic_ai.models.outlines import OutlinesModel |
| 136 | +from pydantic_ai.settings import ModelSettings |
| 137 | + |
| 138 | +class Box(BaseModel): |
| 139 | + width: int |
| 140 | + height: int |
| 141 | + depth: int |
| 142 | + units: str |
| 143 | + |
| 144 | +model = OutlinesModel.from_transformers( |
| 145 | + AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-mini-4k-instruct'), |
| 146 | + AutoTokenizer.from_pretrained('microsoft/Phi-3-mini-4k-instruct') |
| 147 | +) |
| 148 | +agent = Agent(model, output_type=Box) |
| 149 | + |
| 150 | +result = agent.run_sync( |
| 151 | + 'Give me the dimensions of a box', |
| 152 | + model_settings=ModelSettings(extra_body={'max_new_tokens': 100}) |
| 153 | +) |
| 154 | +print(result.output) # width=20 height=30 depth=40 units='cm' |
| 155 | +``` |
| 156 | + |
| 157 | +Outlines does not support tools yet, but support for that feature will be added in the near future. |
0 commit comments