|
| 1 | +--- |
| 2 | +meta: |
| 3 | + title: How to use structured outputs |
| 4 | + description: Learn how to interact with structured outputs using Scaleway's Chat Completions API service. |
| 5 | +content: |
| 6 | + h1: How to use structured outputs |
| 7 | + paragraph: Learn how to interact with powerful text models using Scaleway's Chat Completions API service. |
| 8 | +tags: chat-completitions-api |
| 9 | +dates: |
| 10 | + validation: 2024-09-17 |
| 11 | + posted: 2024-09-17 |
| 12 | +--- |
| 13 | + |
| 14 | + |
| 15 | +Structured outputs allow users to get consistent, machine-readable JSON format responses from language models. |
| 16 | +JSON, as a widely-used format, enables seamless integration with a variety of platforms and applications. Its interoperability is crucial for developers aiming to incorporate AI functionality into their current systems with minimal adjustments. |
| 17 | + |
| 18 | +By specifying a response format when using the [Chat Completions API](/ai-data/generative-apis/api-cli/using-chat-api/), you can ensure that responses are returned in a JSON structure. |
| 19 | +There are two main modes for generating JSON: **Object Mode** (schemaless) and **Schema Mode** (deterministic, structured output). |
| 20 | + |
| 21 | +You can interact with text models in several ways: |
| 22 | +- Via the Scaleway [console](https://console.scaleway.com), which will soon provide a complete [playground](/ai-data/generative-apis/how-to/query-text-models/#accessing-the-playground), aiming to test models, adapt parameters, and observe how these changes affect the output in real-time. |
| 23 | +- Via the [Chat API](/ai-data/generative-apis/how-to/query-text-models/#querying-text-models-via-api) |
| 24 | + |
| 25 | +<Macro id="requirements" /> |
| 26 | + |
| 27 | +- Access to Generative APIs. |
| 28 | + While in beta, the service is restricted to invited users. You can request access by filling out a form on Scaleway's [Betas page](https://www.scaleway.com/en/betas/#generative-apis). |
| 29 | +- A Scaleway account logged into the [console](https://console.scaleway.com) |
| 30 | +- [Owner](/identity-and-access-management/iam/concepts/#owner) status or [IAM permissions](/identity-and-access-management/iam/concepts/#permission) allowing you to perform actions in the intended Organization |
| 31 | +- A valid [API key](/identity-and-access-management/iam/how-to/create-api-keys/) for API authentication |
| 32 | +- Python 3.7+ installed on your system |
| 33 | + |
| 34 | +## Types of structured outputs |
| 35 | + |
| 36 | +- **JSON mode** (schemaless): |
| 37 | + - Type: `{"type": "json_object"}` |
| 38 | + - This mode is non-deterministic and allows the model to output a JSON object without strict validation. |
| 39 | + - Useful for flexible outputs when you expect the model to infer a reasonable structure based on your prompt. |
| 40 | + - JSON mode is older and has been used by developers since early API implementations. |
| 41 | + |
| 42 | +- **Structured outputs (schema mode)** (deterministic/structured): |
| 43 | + - Type `{"type": "json_schema"}` |
| 44 | + - This mode enforces a strict schema format, where the output adheres to the predefined structure. |
| 45 | + - Supports complex types and validation mechanisms as per the [JSON schema specification](https://json-schema.org/specification/). |
| 46 | + - Structured outputs is a newer feature implemented by OpenAI in 2024 to enable stricter, schema-based response formatting. |
| 47 | + |
| 48 | +<Message type="note"> |
| 49 | + - All LLMs on the Scaleway library support **JSON mode** and **Structured outputs**, however, the quality of results will vary in the schemaless JSON mode. |
| 50 | + - JSON mode: It is important to explicitly ask the model to generate a JSON output either in system prompt or user prompt. To prevent infinite generations, model providers most often encourage to ask the model for short JSON objects. |
| 51 | + - Structured outputs: Scaleway supports the [JSON schema specification](https://json-schema.org/specification/) including nested schemas composition (`anyOf`, `allOf`, `oneOf` etc), `$ref`, `all` types, and regular expressions. |
| 52 | +</Message> |
| 53 | + |
| 54 | +## Code examples |
| 55 | + |
| 56 | +<Message type="tip"> |
| 57 | + Before diving into the code examples, ensure you have the necessary libraries installed: |
| 58 | + ```bash |
| 59 | + pip install openai pydantic |
| 60 | + ``` |
| 61 | +</Message> |
| 62 | + |
| 63 | +The following Python examples demonstrate how to use both **JSON mode** and **Structured outputs** to generate structured responses. |
| 64 | + |
| 65 | +We will send to our LLM a voice note transcript in order to structure it. |
| 66 | +Below is our base code: |
| 67 | + |
| 68 | +```python |
| 69 | +import json |
| 70 | +from openai import OpenAI |
| 71 | +from pydantic import BaseModel, Field |
| 72 | + |
| 73 | +# Set your preferred model |
| 74 | +MODEL = "llama-3.1-8b-instruct" |
| 75 | + |
| 76 | +# Set your API key |
| 77 | +API_KEY = "<SCW_API_KEY>" |
| 78 | + |
| 79 | +client = OpenAI( |
| 80 | + base_url="https://api.scaleway.ai/v1", |
| 81 | + api_key=API_KEY, |
| 82 | +) |
| 83 | + |
| 84 | +# Define the schema for the output using Pydantic |
| 85 | +class VoiceNote(BaseModel): |
| 86 | + title: str = Field(description="A title for the voice note") |
| 87 | + summary: str = Field(description="A short one sentence summary of the voice note.") |
| 88 | + actionItems: list[str] = Field(description="A list of action items from the voice note") |
| 89 | + |
| 90 | +# Transcript to use for the output |
| 91 | +TRANSCRIPT = ( |
| 92 | + "Good evening! It's 6:30 PM, and I'm just getting home from work. I have a few things to do " |
| 93 | + "before I can relax. First, I'll need to water the plants in the garden since they've been in the sun all day. " |
| 94 | + "Then, I'll start preparing dinner. I think a simple pasta dish with some garlic bread should be good. " |
| 95 | + "While that's cooking, I'll catch up on a couple of phone calls I missed earlier." |
| 96 | +) |
| 97 | +``` |
| 98 | + |
| 99 | +### Using JSON mode (schemaless) |
| 100 | + |
| 101 | +In JSON mode, you can prompt the model to output a JSON object without enforcing a strict schema. |
| 102 | + |
| 103 | +```python |
| 104 | +extract = client.chat.completions.create( |
| 105 | + messages=[ |
| 106 | + { |
| 107 | + "role": "system", |
| 108 | + "content": "The following is a voice message transcript. Only answer in JSON.", |
| 109 | + }, |
| 110 | + { |
| 111 | + "role": "user", |
| 112 | + "content": TRANSCRIPT, |
| 113 | + }, |
| 114 | + ], |
| 115 | + model=MODEL, |
| 116 | + response_format={ |
| 117 | + "type": "json_object", |
| 118 | + }, |
| 119 | +) |
| 120 | +output = json.loads(extract.choices[0].message.content) |
| 121 | +print(json.dumps(output, indent=2)) |
| 122 | +``` |
| 123 | + |
| 124 | +Output example: |
| 125 | +```json |
| 126 | +{ |
| 127 | + "current_time": "6:30 PM", |
| 128 | + "tasks": [ |
| 129 | + { |
| 130 | + "task": "water the plants in the garden", |
| 131 | + "priority": "high" |
| 132 | + }, |
| 133 | + { |
| 134 | + "task": "prepare dinner (pasta with garlic bread)", |
| 135 | + "priority": "high" |
| 136 | + }, |
| 137 | + { |
| 138 | + "task": "catch up on phone calls", |
| 139 | + "priority": "medium" |
| 140 | + } |
| 141 | + ] |
| 142 | +} |
| 143 | +``` |
| 144 | + |
| 145 | +### Using structured outputs with JSON schema (Pydantic) |
| 146 | + |
| 147 | +Using [Pydantic](https://docs.pydantic.dev/latest/concepts/models/), users can define the schema as a Python class and enforce the model to return results adhering to this schema. |
| 148 | + |
| 149 | +```python |
| 150 | +extract = client.chat.completions.create( |
| 151 | + messages=[ |
| 152 | + { |
| 153 | + "role": "system", |
| 154 | + "content": "The following is a voice message transcript. Only answer in JSON.", |
| 155 | + }, |
| 156 | + { |
| 157 | + "role": "user", |
| 158 | + "content": TRANSCRIPT, |
| 159 | + }, |
| 160 | + ], |
| 161 | + model=MODEL, |
| 162 | + response_format={ |
| 163 | + "type": "json_schema", |
| 164 | + "json_schema": { |
| 165 | + "schema": VoiceNote.model_json_schema(), |
| 166 | + } |
| 167 | + }, |
| 168 | +) |
| 169 | +output = json.loads(extract.choices[0].message.content) |
| 170 | +print(json.dumps(output, indent=2)) |
| 171 | +``` |
| 172 | + |
| 173 | +Output example: |
| 174 | +```json |
| 175 | +{ |
| 176 | + "title": "To-Do List", |
| 177 | + "summary": "Returning from work, need to complete tasks before relaxing", |
| 178 | + "actionItems": [ |
| 179 | + "Water garden", |
| 180 | + "Prepare dinner: pasta dish with garlic bread", |
| 181 | + "Catch up on missed phone calls" |
| 182 | + ] |
| 183 | +} |
| 184 | +``` |
| 185 | + |
| 186 | +### Using structured outputs with JSON schema (manual definition) |
| 187 | + |
| 188 | +Alternatively, users can manually define the JSON schema inline when calling the model. |
| 189 | + |
| 190 | +```python |
| 191 | +extract = client.chat.completions.create( |
| 192 | + messages=[ |
| 193 | + { |
| 194 | + "role": "system", |
| 195 | + "content": "The following is a voice message transcript. Only answer in JSON.", |
| 196 | + }, |
| 197 | + { |
| 198 | + "role": "user", |
| 199 | + "content": TRANSCRIPT, |
| 200 | + }, |
| 201 | + ], |
| 202 | + model=MODEL, |
| 203 | + response_format={ |
| 204 | + "type": "json_schema", |
| 205 | + "json_schema": { |
| 206 | + "schema": { |
| 207 | + "type": "object", |
| 208 | + "properties": { |
| 209 | + "title": {"type": "string"}, |
| 210 | + "summary": {"type": "string"}, |
| 211 | + "actionItems": { |
| 212 | + "type": "array", |
| 213 | + "items": {"type": "string"} |
| 214 | + } |
| 215 | + }, |
| 216 | + "required": ["title", "summary", "actionItems"] |
| 217 | + } |
| 218 | + } |
| 219 | + } |
| 220 | +) |
| 221 | +output = json.loads(extract.choices[0].message.content) |
| 222 | +print(json.dumps(output, indent=2)) |
| 223 | +``` |
| 224 | + |
| 225 | +Output example: |
| 226 | +```json |
| 227 | +{ |
| 228 | + "title": "Evening Routine", |
| 229 | + "actionItems": [ |
| 230 | + "Water the plants", |
| 231 | + "Cook dinner (pasta and garlic bread)", |
| 232 | + "Make phone calls" |
| 233 | + ], |
| 234 | + "summary": "Made a list of tasks to accomplish before relaxing tonight" |
| 235 | +} |
| 236 | +``` |
| 237 | + |
| 238 | +## Conclusion |
| 239 | + |
| 240 | +Using structured outputs with LLMs can significantly enhance data handling in your applications. |
| 241 | +By choosing between JSON mode and Structured outputs with JSON schema, you control the consistency and structure of the model's responses to suit your specific needs. |
| 242 | + |
| 243 | +- **JSON mode** is flexible but less predictable. |
| 244 | +- **Structured outputs** provide strict adherence to a predefined schema, ensuring consistency. |
| 245 | + |
| 246 | +Experiment with both methods to determine which best fits your application's requirements. |
0 commit comments