images

Author: santiagxf

articles/ai-foundry/model-inference/includes/use-structured-outputs/python.md

@@ -239,6 +239,140 @@ except ValidationError as e:
     print(f"Validation error: {e}")
 ```
 
+## Structured outputs in images
+
+You can use structured outputs with multi-modal models to extract information from data like images. Let's consider the following chart:
+
+:::image type="content" source="../../media/use-structured-outputs/example_graph_treecover.png" alt-text="An example image showing a chart with the annual loss in thousand square kilometers of global tree cover across different climate zones." lightbox="../../media/use-structured-outputs/example_graph_treecover.png":::
+
+Let's load this image:
+
+```python
+from azure.ai.inference.models import ImageContentItem, ImageUrl
+
+image_graph = ImageUrl.load(
+    image_file="example_graph_treecover.png",
+    image_format="png"
+)
+```
+
+Let's define a generic schema that can be use to encode the information contained in a chart. We use [Pyndatic objects](#use-pydantic-objects) as described before.
+
+```python
+from pydantic import BaseModel
+
+class DataPoint(BaseModel):
+    x: float
+    y: float
+    serie: str
+
+class Graph(BaseModel):
+    title: str
+    description: str
+    x_axis: str
+    y_axis: str
+    legend: list[str]
+    data: list[DataPoint]
+```
+
+Let's use structured outputs to extract the information:
+
+```python
+response = client.complete(
+    response_format=JsonSchemaFormat(
+        name="graph_schema",
+        schema=Graph.model_json_schema(),
+        description="Describes the data in the graph",
+        strict=False,
+    ),
+    messages=[
+        SystemMessage("""
+                      Extract the information from the graph. Extrapolate the values of the x axe to ensure you have the correct number
+                      of data points for each of the years from 2001 to 2023. Scale the values of the y axes to account for the values
+                      being stacked.
+                      """
+        ),
+        UserMessage(content=[ImageContentItem(image_url=image_graph)]),
+    ],
+)
+```
+
+Let's check the outputs:
+
+```python
+import json
+
+json_response_message = json.loads(response.choices[0].message.content)
+data = Graph.model_validate(json_response_message)
+print(json.dumps(json_response_message, indent=4))
+```
+
+```output
+{
+    "title": "Global tree cover: annual loss",
+    "description": "Annual loss in thousand square kilometers of global tree cover across different climate zones.",
+    "x_axis": "Year",
+    "y_axis": "Thousand square kilometers",
+    "legend": [
+        "Boreal",
+        "Temperate",
+        "Subtropical",
+        "Tropical"
+    ],
+    "data": [
+        {
+            "x": 2001,
+            "y": -35,
+            "serie": "Boreal"
+        },
+        {
+            "x": 2001,
+            "y": -10,
+            "serie": "Temperate"
+        },
+        {
+            "x": 2001,
+            "y": -55,
+...
+            "serie": "Tropical"
+        }
+    ]
+}
+```
+
+Let's use `matplotlib` and try to plot the data:
+
+```python
+import matplotlib.pyplot as plt
+import pandas as pd
+
+# Convert data to a DataFrame for easier manipulation
+df = pd.DataFrame(data.model_dump()["data"])
+
+# Pivot the data to prepare for stacked bar chart
+pivot_df = df.pivot(index="x", columns="serie", values="y").fillna(0)
+
+# Plotting
+fig, ax = plt.subplots(figsize=(10, 6))
+
+# Stacked bar chart
+pivot_df.plot(kind="bar", stacked=True, ax=ax, color=["#114488", "#3CB371", "#1188AA", "#006400"])
+
+# Chart customization
+ax.set_title(data.title, fontsize=16)
+ax.set_xlabel(data.x_axis, fontsize=12)
+ax.set_ylabel(data.y_axis, fontsize=12)
+ax.legend(title=data.legend, fontsize=10)
+ax.grid(axis="y", linestyle="--", alpha=0.7)
+
+# Show the plot
+plt.tight_layout()
+plt.show()
+```
+
+:::image type="content" source="../../media/use-structured-outputs/graph_treecover_plot.png" alt-text="The resulting plot of the data contained in the structured output generated by the model." lightbox="../../media/use-structured-outputs/graph_treecover_plot.png":::
+
+
 ### Supported schemas
 
 There are some limitations that models may place in how schemas are defined. When using structure outputs, consider the following limitations. Notice that limitations may vary per model.