Skip to content

Commit 7c55369

Browse files
santiagxfsantiagxf
committed
fix: languages
1 parent 6c8b479 commit 7c55369

File tree

3 files changed

+640
-17
lines changed

3 files changed

+640
-17
lines changed

‎articles/ai-foundry/model-inference/includes/use-structured-outputs/javascript.md

Lines changed: 356 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,359 @@ ms.reviewer: fasantia
1313
zone_pivot_groups: azure-ai-inference-samples
1414
---
1515

16-
> [!NOTE]
17-
> This example is not available in the selected language.
16+
[!INCLUDE [intro](intro.md)]
17+
18+
[!INCLUDE [how-to-prerequisites-javascript](../how-to-prerequisites-javascript.md)]
19+
20+
* Initialize a client to consume the model:
21+
22+
```javascript
23+
const client = ModelClient(
24+
"https://<resource>.services.ai.azure.com/models",
25+
new AzureKeyCredential(process.env.AZURE_INFERENCE_CREDENTIAL)
26+
);
27+
```
28+
29+
## How to use structured outputs
30+
31+
Structured outputs use JSON schemas to enforce output structure. JSON schemas describe the shape of the JSON object including expected values, types, and which ones are required. Those JSON objects are encoded as a string within the response of the model.
32+
33+
### Example
34+
35+
To exemplify the scenario, let's try to parse the attributes of a GitHub Issue from its description.
36+
37+
```python
38+
const url = 'https://api.github.com/repos/Azure-Samples/azure-search-openai-demo/issues/2231';
39+
40+
async function getIssueBody() {
41+
try {
42+
const response = await fetch(url);
43+
const data = await response.json();
44+
const issueBody = data.body;
45+
return issueBody;
46+
} catch (error) {
47+
console.error('Error fetching issue:', error);
48+
}
49+
}
50+
51+
issueBody = await getIssueBody();
52+
53+
```
54+
55+
The output of `issueBody` looks as follows:
56+
57+
```output
58+
'<!--\r\nIF SUFFICIENT INFORMATION IS NOT PROVIDED VIA THE FOLLOWING TEMPLATE THE ISSUE MIGHT BE CLOSED WITHOUT FURTHER CONSIDERATION OR INVESTIGATION\r\n-->\r\n> Please provide us with the following information:\r\n> ---------------------------------------------------------------\r\n\r\n### This issue is for a: (mark with an `x`)\r\n```\r\n- [x] bug report -> please search issues before submitting\r\n- [ ] feature request\r\n- [ ] documentation issue or request\r\n- [ ] regression (a behavior that used to work and stopped in a new release)\r\n```\r\n\r\n### Minimal steps to reproduce\r\n> Deploy the app with auth and acl´s turned on, configure the acls file, run all the scripts needed.\r\n\r\n### Any log messages given by the failure\r\n> None\r\n\r\n### Expected/desired behavior\r\n> groups field to be filled the the groups id\'s that have permissions to "view the file"\r\n\r\n### OS and Version?\r\n> win 10\r\n### azd version?\r\n> azd version 1.11.0\r\n\r\n### Versions\r\n>\r\n\r\n### Mention any other details that might be useful\r\n\r\nAfter configuring the json with the perms all the scripts (`adlsgen2setup.py` and `prepdocs.ps1`) everything goes well but the groups metadata tag never gets to have any groups.\r\n\r\n![image](https://github.com/user-attachments/assets/40f1eb09-2c21-4244-98b5-adfb3fa16955)\r\n\r\n\r\n> ---------------------------------------------------------------\r\n> Thanks! We\'ll be in touch soon.\r\n'
59+
```
60+
61+
### Define the schema
62+
63+
The following JSON schema defines the schema of a GitHub issue:
64+
65+
__github_issue_schema.json__
66+
67+
```json
68+
{
69+
"title": "github_issue",
70+
"type": "object",
71+
"properties": {
72+
"title": {
73+
"title": "Title",
74+
"type": "string"
75+
},
76+
"description": {
77+
"title": "Description",
78+
"type": "string"
79+
},
80+
"type": {
81+
"enum": ["Bug", "Feature", "Documentation", "Regression"],
82+
"title": "Type",
83+
"type": "string"
84+
},
85+
"operating_system": {
86+
"title": "Operating System",
87+
"type": "string"
88+
}
89+
},
90+
"required": ["title", "description", "type", "operating_system"],
91+
"additionalProperties": false
92+
}
93+
```
94+
95+
When defining schemas, follow these recommendations:
96+
97+
> [!div class="checklist"]
98+
> * Use clear and expressive keys.
99+
> * Use `_` if you need to separate words to convey meaning.
100+
> * Create clear titles and descriptions for important keys in your structure.
101+
> * Evaluate multiple structures until finding the one that works best for your use case.
102+
> * Take into account limitations when indicating schemas, which may vary per model.
103+
104+
Let's load this schema:
105+
106+
```javascript
107+
const fs = require('fs');
108+
109+
const data = fs.readFileSync('./github_issue_schema.json', 'utf-8');
110+
const gitHubIssueSchema = JSON.parse(data);```
111+
```
112+
113+
### Use structure outputs
114+
115+
We can use structure outputs with the defined schema as follows:
116+
117+
118+
```javascript
119+
var messages = [
120+
{ role: "system", content: `
121+
Extract structured information from GitHub issues opened in our project.
122+
123+
Provide
124+
- The title of the issue
125+
- A 1-2 sentence description of the project
126+
- The type of issue (Bug, Feature, Documentation, Regression)
127+
- The operating system the issue was reported on
128+
- Whether the issue is a duplicate of another issue`
129+
},
130+
{ role: "user", content: issueBody },
131+
];
132+
133+
var response = await client.path("/chat/completions").post({
134+
body: {
135+
model: "gpt-4o",
136+
messages: messages,
137+
response_format: {
138+
type: "json_schema",
139+
json_schema: {
140+
name: "github_issue",
141+
schema: gitHubIssueSchema,
142+
description: "Describes a GitHub issue",
143+
strict: true,
144+
},
145+
}
146+
}
147+
});
148+
```
149+
150+
Let's see how this works:
151+
152+
```javascript
153+
const rawContent = response.choices[0].message.content;
154+
const jsonResponseMessage = JSON.parse(rawContent);
155+
156+
console.log(JSON.stringify(jsonResponseMessage, null, 4));
157+
```
158+
159+
```output
160+
{
161+
"title": "Metadata tags issue on access control lists - ADLSgen2 setup",
162+
"description": "Our project seems to have an issue with the metadata tag for groups when deploying the application with access control lists and necessary settings.",
163+
"type": "Bug",
164+
"operating_system": "Windows 10"
165+
}
166+
```
167+
168+
## Structured outputs in images
169+
170+
You can use structured outputs with multi-modal models to extract information from data like images.
171+
172+
Let's consider the following chart:
173+
174+
:::image type="content" source="../../media/use-structured-outputs/example_graph_treecover.png" alt-text="An example image showing a chart with the annual loss in thousand square kilometers of global tree cover across different climate zones." lightbox="../../media/use-structured-outputs/example_graph_treecover.png":::
175+
176+
We can define a generic schema that can be used to encode the information contained in the chart and then use it for further analysis.
177+
178+
__graph_schema.json__
179+
180+
```json
181+
{
182+
"$defs": {
183+
"DataPoint": {
184+
"properties": {
185+
"x": {
186+
"title": "X",
187+
"type": "number"
188+
},
189+
"y": {
190+
"title": "Y",
191+
"type": "number"
192+
},
193+
"serie": {
194+
"title": "Serie",
195+
"type": "string"
196+
}
197+
},
198+
"required": [
199+
"x",
200+
"y",
201+
"serie"
202+
],
203+
"title": "DataPoint",
204+
"type": "object"
205+
}
206+
},
207+
"title": "Graph",
208+
"type": "object",
209+
"properties": {
210+
"title": {
211+
"title": "Title",
212+
"type": "string"
213+
},
214+
"description": {
215+
"title": "Description",
216+
"type": "string"
217+
},
218+
"x_axis": {
219+
"title": "X Axis",
220+
"type": "string"
221+
},
222+
"y_axis": {
223+
"title": "Y Axis",
224+
"type": "string"
225+
},
226+
"legend": {
227+
"items": {
228+
"type": "string"
229+
},
230+
"title": "Legend",
231+
"type": "array"
232+
},
233+
"data": {
234+
"items": {
235+
"$ref": "#/$defs/DataPoint"
236+
},
237+
"title": "Data",
238+
"type": "array"
239+
}
240+
},
241+
"required": [
242+
"title",
243+
"description",
244+
"x_axis",
245+
"y_axis",
246+
"legend",
247+
"data"
248+
],
249+
"additionalProperties": false
250+
}
251+
```
252+
253+
Let's load this schema:
254+
255+
```javascript
256+
const fs = require('fs');
257+
258+
const data = fs.readFileSync('./graph_schema.json', 'utf-8');
259+
const graphSchema = JSON.parse(data);```
260+
```
261+
262+
We can load the image as follows to pass it to the model:
263+
264+
```javascript
265+
/**
266+
* Get the data URL of an image file.
267+
* @param {string} imageFile - The path to the image file.
268+
* @param {string} imageFormatType - The format of the image file. For example: "jpeg", "png".
269+
* @returns {string} The data URL of the image.
270+
*/
271+
function getImageDataUrl(imageFile, imageFormatType) {
272+
try {
273+
const imageBuffer = fs.readFileSync(imageFile);
274+
const imageBase64 = imageBuffer.toString("base64");
275+
return `data:image/${imageFormatType};base64,${imageBase64}`;
276+
} catch (error) {
277+
console.error(`Could not read '${imageFile}'.`);
278+
console.error("Set the correct path to the image file before running this sample.");
279+
process.exit(1);
280+
}
281+
}
282+
```
283+
284+
Use structured outputs to extract the information:
285+
286+
```javascript
287+
var messages = [
288+
{
289+
role: "system",
290+
content: `
291+
Extract the information from the graph. Extrapolate the values of the x axe to ensure you have the correct number
292+
of data points for each of the years from 2001 to 2023. Scale the values of the y axes to account for the values
293+
being stacked.`
294+
},
295+
{
296+
role: "user",
297+
content: {
298+
type: "image_url",
299+
image_url: {
300+
url: getImageDataUrl("example_graph_treecover.png", "png"),
301+
},
302+
}
303+
},
304+
}
305+
];
306+
307+
const response = await client.path("/chat/completions").post({
308+
body: {
309+
messages: messages,
310+
response_format: {
311+
type: "json_schema",
312+
json_schema: {
313+
name: "graph_schema",
314+
schema: graphSchema,
315+
description: "Describes the data in the graph",
316+
strict: true,
317+
},
318+
}
319+
model: "gpt-4o",
320+
},
321+
});
322+
```
323+
324+
325+
Let's inspect the output:
326+
327+
```javascript
328+
var rawContent = response.choices[0].message.content;
329+
var jsonResponseMessage = JSON.parse(rawContent);
330+
331+
console.log(JSON.stringify(jsonResponseMessage, null, 4));
332+
```
333+
334+
```output
335+
{
336+
"title": "Global tree cover: annual loss",
337+
"description": "Annual loss in thousand square kilometers of global tree cover across different climate zones.",
338+
"x_axis": "Year",
339+
"y_axis": "Thousand square kilometers",
340+
"legend": [
341+
"Boreal",
342+
"Temperate",
343+
"Subtropical",
344+
"Tropical"
345+
],
346+
"data": [
347+
{
348+
"x": 2001,
349+
"y": -35,
350+
"serie": "Boreal"
351+
},
352+
{
353+
"x": 2001,
354+
"y": -10,
355+
"serie": "Temperate"
356+
},
357+
{
358+
"x": 2001,
359+
"y": -55,
360+
...
361+
"serie": "Tropical"
362+
}
363+
]
364+
}
365+
```
366+
367+
To see how much information the model was able to capture, we can try to plot the data:
368+
369+
:::image type="content" source="../../media/use-structured-outputs/graph_treecover_plot.png" alt-text="The resulting plot of the data contained in the structured output generated by the model." lightbox="../../media/use-structured-outputs/graph_treecover_plot.png":::
370+
371+
While the information isn't perfect, we can see the model was able to capture a good amount of information from the original chart.

0 commit comments

Comments
 (0)