You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/ai-foundry/model-inference/includes/use-structured-outputs/python.md
+73-69Lines changed: 73 additions & 69 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -239,24 +239,72 @@ except ValidationError as e:
239
239
print(f"Validation error: {e}")
240
240
```
241
241
242
-
##Structured outputs in images
242
+
### Specifying an schema
243
243
244
-
You can use structured outputs with multi-modal models to extract information from data like images. Let's consider the following chart:
244
+
There are some limitations that models may place in schemas definitions. Such limitations may vary per-model. **We recommend reviewing the documentation from the model provider** to verify that your schemas are valid.
245
245
246
-
:::image type="content" source="../../media/use-structured-outputs/example_graph_treecover.png" alt-text="An example image showing a chart with the annual loss in thousand square kilometers of global tree cover across different climate zones." lightbox="../../media/use-structured-outputs/example_graph_treecover.png":::
246
+
The following guidelines applies to the majority of models:
247
+
248
+
#### Optional fields
247
249
248
-
Let's load this image:
250
+
Some models may require all the fields to be in the `required` section of the schema. If you need to use optional fields, use unions with null types to express that a given field can be optional.
249
251
250
252
```python
251
-
from azure.ai.inference.models import ImageContentItem, ImageUrl
Let's define a generic schema that can be use to encode the information contained in a chart. We use [Pyndatic objects](#use-pydantic-objects) as described before.
263
+
#### Nested types
264
+
265
+
Models may support indicating nesting types. You can compose complex structures as needed:
Verify the level of nesting supported by the model you are working with.
298
+
299
+
## Structured outputs in images
300
+
301
+
You can use structured outputs with multi-modal models to extract information from data like images.
302
+
303
+
Let's consider the following chart:
304
+
305
+
:::image type="content" source="../../media/use-structured-outputs/example_graph_treecover.png" alt-text="An example image showing a chart with the annual loss in thousand square kilometers of global tree cover across different climate zones." lightbox="../../media/use-structured-outputs/example_graph_treecover.png":::
306
+
307
+
We can define a generic schema that can be use to encode the information contained in the chart and then use it for further analysis. We use [Pyndatic objects](#use-pydantic-objects) as described before.
260
308
261
309
```python
262
310
from pydantic import BaseModel
@@ -275,7 +323,18 @@ class Graph(BaseModel):
275
323
data: list[DataPoint]
276
324
```
277
325
278
-
Let's use structured outputs to extract the information:
326
+
We can load the image as follows to pass it to the model:
327
+
328
+
```python
329
+
from azure.ai.inference.models import ImageContentItem, ImageUrl
330
+
331
+
image_graph = ImageUrl.load(
332
+
image_file="example_graph_treecover.png",
333
+
image_format="png"
334
+
)
335
+
```
336
+
337
+
Use structured outputs to extract the information:
279
338
280
339
```python
281
340
response = client.complete(
@@ -297,7 +356,7 @@ response = client.complete(
297
356
)
298
357
```
299
358
300
-
Let's check the outputs:
359
+
It's always a good practice to validate the outputs and schemas:
We can see how much information the model was able to capture by plotting the data using `matplotlib`:
344
403
345
404
```python
346
405
import matplotlib.pyplot as plt
@@ -372,59 +431,4 @@ plt.show()
372
431
373
432
:::image type="content" source="../../media/use-structured-outputs/graph_treecover_plot.png" alt-text="The resulting plot of the data contained in the structured output generated by the model." lightbox="../../media/use-structured-outputs/graph_treecover_plot.png":::
374
433
375
-
376
-
### Supported schemas
377
-
378
-
There are some limitations that models may place in how schemas are defined. When using structure outputs, consider the following limitations. Notice that limitations may vary per model.
379
-
380
-
#### Optional fields
381
-
382
-
Some models may require all the fields to be in the `required` section of the schema. If you need to use optional fields, use unions with null types to express that a given field can be optional.
0 commit comments