Skip to content

Commit fc77ea0

Browse files
trangevitrangevi
authored
[Azure.AI.Inference] Beta 3 (Azure#48288)
* Feature updates for beta 3 ### Features Added - Added new `ImageEmbeddingsClient`, to provide support for generating embeddings with model input. See sample for more information. - Added support for Chat Completions with audio input, for supported models. - Added support for Chat Completions with structured output. - Added support for providing a "Developer" message to models which support it. ### Breaking Changes - `ChatCompletionsResponseFormatJSON` has been renamed to `ChatCompletionsResponseFormatJsonObject`. ### Bugs Fixed - Fixed an issue where `usage` wasn't being properly included in chat responses. --------- Signed-off-by: trangevi <[email protected]>
1 parent cbe5dc8 commit fc77ea0

File tree

91 files changed

+5687
-598
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+5687
-598
lines changed

‎sdk/ai/Azure.AI.Inference/CHANGELOG.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,18 @@
44

55
### Features Added
66

7-
- Exposed `JsonModelWriteCore` for model serialization procedure.
7+
- Added new `ImageEmbeddingsClient`, to provide support for generating embeddings with model input. See sample for more information.
8+
- Added support for Chat Completions with audio input, for supported models.
9+
- Added support for Chat Completions with structured output.
10+
- Added support for providing a "Developer" message to models which support it.
811

912
### Breaking Changes
1013

14+
- `ChatCompletionsResponseFormatJSON` has been renamed to `ChatCompletionsResponseFormatJsonObject`.
15+
1116
### Bugs Fixed
1217

13-
### Other Changes
18+
- Fixed an issue where `usage` wasn't being properly included in chat responses.
1419

1520
## 1.0.0-beta.2 (2024-10-24)
1621

‎sdk/ai/Azure.AI.Inference/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ With some minor adjustments, this client library can also be configured to do in
2525
* An [Azure subscription](https://azure.microsoft.com/free).
2626
* An [AI Model from the catalog](https://ai.azure.com/explore/models) deployed through Azure AI Foundry.
2727
* To construct the client library, you will need to pass in the endpoint URL. The endpoint URL has the form `https://your-host-name.your-azure-region.inference.ai.azure.com`, where `your-host-name` is your unique model deployment host name and `your-azure-region` is the Azure region where the model is deployed (e.g. `eastus2`).
28-
* Depending on your model deployment and authentication preference, you either need a key to authenticate against the service, or Entra ID credentials. The key is a 32-character string.
28+
* Depending on your model deployment and authentication preference, you either need a key to authenticate against the service, or Entra ID credentials.
2929

3030
### Install the package
3131

‎sdk/ai/Azure.AI.Inference/api/Azure.AI.Inference.net8.0.cs‎

Lines changed: 118 additions & 29 deletions
Large diffs are not rendered by default.

‎sdk/ai/Azure.AI.Inference/api/Azure.AI.Inference.netstandard2.0.cs‎

Lines changed: 118 additions & 29 deletions
Large diffs are not rendered by default.

‎sdk/ai/Azure.AI.Inference/assets.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@
22
"AssetsRepo": "Azure/azure-sdk-assets",
33
"AssetsRepoPrefixPath": "net",
44
"TagPrefix": "net/ai/Azure.AI.Inference",
5-
"Tag": "net/ai/Azure.AI.Inference_4a960298fb"
5+
"Tag": "net/ai/Azure.AI.Inference_bda2b3e222"
66
}

‎sdk/ai/Azure.AI.Inference/samples/Sample10_EmbeddingsWithAoai.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Set these two environment variables before running the sample:
88

99
1. AZURE_OPENAI_EMBEDDINGS_ENDPOINT - Your endpoint URL, in the form `https://your-deployment-name.your-azure-region.inference.ai.azure.com` where `your-deployment-name` is your unique AI Model deployment name, and `your-azure-region` is the Azure region where your model is deployed.
1010

11-
2. AZURE_OPENAI_EMBEDDINGS_KEY - Your model key (a 32-character string). Keep it secret.
11+
2. AZURE_OPENAI_EMBEDDINGS_KEY - Your model key. Keep it secret.
1212

1313
In order to target AOAI, the auth key must currently be provided as a separate header. This can be done by creating a `HttpPipelinePolicy` like below:
1414

‎sdk/ai/Azure.AI.Inference/samples/Sample11_ImageEmbeddings.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Simple Embeddings
2+
3+
This sample demonstrates how to get embeddings for a provided image. Here we use the service default of returning embeddings as a list of floating point values.
4+
5+
This sample assumes the AI model is hosted on a Serverless API or Managed Compute endpoint. For GitHub Models or Azure OpenAI endpoints, the client constructor needs to be modified. See package documentation for details.
6+
7+
## Usage
8+
9+
Set these two environment variables before running the sample:
10+
11+
1. AZURE_AI_IMAGE_EMBEDDINGS_ENDPOINT - Your endpoint URL, in the form `https://your-deployment-name.your-azure-region.inference.ai.azure.com` where `your-deployment-name` is your unique AI Model deployment name, and `your-azure-region` is the Azure region where your model is deployed.
12+
13+
2. AZURE_AI_IMAGE_EMBEDDINGS_KEY - Your model key. Keep it secret.
14+
15+
```C# Snippet:Azure_AI_Inference_BasicImageEmbedding
16+
var endpoint = new Uri(System.Environment.GetEnvironmentVariable("AZURE_AI_IMAGE_EMBEDDINGS_ENDPOINT"));
17+
var credential = new AzureKeyCredential(System.Environment.GetEnvironmentVariable("AZURE_AI_IMAGE_EMBEDDINGS_KEY"));
18+
19+
var client = new ImageEmbeddingsClient(endpoint, credential, new AzureAIInferenceClientOptions());
20+
21+
List<ImageEmbeddingInput> input = new List<ImageEmbeddingInput>
22+
{
23+
ImageEmbeddingInput.Load(imageFilePath:"sampleImage.png", imageFormat:"png")
24+
};
25+
26+
var requestOptions = new ImageEmbeddingsOptions(input);
27+
28+
Response<EmbeddingsResult> response = client.Embed(requestOptions);
29+
foreach (EmbeddingItem item in response.Value.Data)
30+
{
31+
List<float> embedding = item.Embedding.ToObjectFromJson<List<float>>();
32+
Console.WriteLine($"Index: {item.Index}, Embedding: <{string.Join(", ", embedding)}>");
33+
}
34+
```
35+
36+
An `async` option is also available.
37+
38+
```C# Snippet:Azure_AI_Inference_BasicImageEmbeddingAsync
39+
var endpoint = new Uri(System.Environment.GetEnvironmentVariable("AZURE_AI_IMAGE_EMBEDDINGS_ENDPOINT"));
40+
var credential = new AzureKeyCredential(System.Environment.GetEnvironmentVariable("AZURE_AI_IMAGE_EMBEDDINGS_KEY"));
41+
42+
var client = new ImageEmbeddingsClient(endpoint, credential, new AzureAIInferenceClientOptions());
43+
44+
List<ImageEmbeddingInput> input = new List<ImageEmbeddingInput>
45+
{
46+
ImageEmbeddingInput.Load(imageFilePath:"sampleImage.png", imageFormat:"png")
47+
};
48+
49+
var requestOptions = new ImageEmbeddingsOptions(input);
50+
51+
Response<EmbeddingsResult> response = await client.EmbedAsync(requestOptions);
52+
foreach (EmbeddingItem item in response.Value.Data)
53+
{
54+
List<float> embedding = item.Embedding.ToObjectFromJson<List<float>>();
55+
Console.WriteLine($"Index: {item.Index}, Embedding: <{string.Join(", ", embedding)}>");
56+
}
57+
```
58+
59+
### Alternative Response Type
60+
61+
It is also possible to request embeddings as base64 encoded strings, instead of the service default of lists of floats.
62+
63+
```C# Snippet:Azure_AI_Inference_Base64ImageEmbedding
64+
List<ImageEmbeddingInput> input = new List<ImageEmbeddingInput>
65+
{
66+
ImageEmbeddingInput.Load(imageFilePath:"sampleImage.png", imageFormat:"png")
67+
};
68+
69+
var requestOptions = new ImageEmbeddingsOptions(input)
70+
{
71+
EncodingFormat = EmbeddingEncodingFormat.Base64,
72+
};
73+
74+
Response<EmbeddingsResult> response = client.Embed(requestOptions);
75+
foreach (EmbeddingItem item in response.Value.Data)
76+
{
77+
string embedding = item.Embedding.ToObjectFromJson<string>();
78+
Console.WriteLine($"Index: {item.Index}, Embedding: <{embedding}>");
79+
}
80+
```

‎sdk/ai/Azure.AI.Inference/samples/Sample12_ChatCompletionsWithAudio.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Chat Completions with Audio Input
2+
3+
This sample demonstrates how to get a chat completions response from the service using a synchronous call. It shows different ways to include audio in the input chat messages.
4+
5+
This sample will only work on AI models that support audio input.
6+
7+
## Usage
8+
9+
Set these two environment variables before running the sample:
10+
11+
1. AZURE_AI_CHAT_ENDPOINT - Your endpoint URL, in the form `https://your-deployment-name.your-azure-region.inference.ai.azure.com` where `your-deployment-name` is your unique AI Model deployment name, and `your-azure-region` is the Azure region where your model is deployed.
12+
13+
2. AZURE_AI_CHAT_KEY - Your model key. Keep it secret.
14+
15+
An audio input can be provided using a URI pointer to an audio file:
16+
17+
```C# Snippet:Azure_AI_Inference_AudioUrlInput
18+
var endpoint = new Uri(System.Environment.GetEnvironmentVariable("AZURE_AI_CHAT_ENDPOINT"));
19+
var credential = new AzureKeyCredential(System.Environment.GetEnvironmentVariable("AZURE_AI_CHAT_KEY"));
20+
21+
var client = new ChatCompletionsClient(endpoint, credential, new AzureAIInferenceClientOptions());
22+
23+
var requestOptions = new ChatCompletionsOptions()
24+
{
25+
Messages =
26+
{
27+
new ChatRequestSystemMessage("You are a helpful assistant that helps provide translations."),
28+
new ChatRequestUserMessage(
29+
new ChatMessageTextContentItem("Translate this audio for me"),
30+
new ChatMessageAudioContentItem(new Uri("https://example.com/audio.mp3"))),
31+
},
32+
};
33+
34+
Response<ChatCompletions> response = client.Complete(requestOptions);
35+
System.Console.WriteLine(response.Value.Content);
36+
```
37+
38+
Alternatively, you can provide a pointer to a file on disk:
39+
40+
```C# Snippet:Azure_AI_Inference_AudioDataFileInput
41+
var endpoint = new Uri(System.Environment.GetEnvironmentVariable("AZURE_AI_CHAT_ENDPOINT"));
42+
var credential = new AzureKeyCredential(System.Environment.GetEnvironmentVariable("AZURE_AI_CHAT_KEY"));
43+
44+
ChatMessageAudioContentItem audioContentItem = new ChatMessageAudioContentItem("sample_audio.mp3", AudioContentFormat.Mp3);
45+
46+
var client = new ChatCompletionsClient(endpoint, credential, new AzureAIInferenceClientOptions());
47+
48+
var requestOptions = new ChatCompletionsOptions()
49+
{
50+
Messages =
51+
{
52+
new ChatRequestSystemMessage("You are a helpful assistant that helps provide translations."),
53+
new ChatRequestUserMessage(
54+
new ChatMessageTextContentItem("Translate this audio for me"),
55+
new ChatMessageAudioContentItem(new Uri("https://example.com/audio.mp3"))),
56+
},
57+
};
58+
59+
Response<ChatCompletions> response = client.Complete(requestOptions);
60+
System.Console.WriteLine(response.Value.Content);
61+
```
62+
63+
Either method supports an `async` option, with the only difference being how the client is invoked:
64+
65+
```C# Snippet:Azure_AI_Inference_AudioInputAsync
66+
Response<ChatCompletions> response = await client.CompleteAsync(requestOptions);
67+
System.Console.WriteLine(response.Value.Content);
68+
```

‎sdk/ai/Azure.AI.Inference/samples/Sample13_ChatCompletionsWithStructuredOutputs.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# Chat Completions with JSON Structured Output
2+
3+
This sample demonstrates how to get a chat completions response with the output structured to match a provided JSON format.
4+
5+
## Usage
6+
7+
Set these two environment variables before running the sample:
8+
9+
1. AZURE_AI_CHAT_ENDPOINT - Your endpoint URL, in the form `https://your-deployment-name.your-azure-region.inference.ai.azure.com` where `your-deployment-name` is your unique AI Model deployment name, and `your-azure-region` is the Azure region where your model is deployed.
10+
11+
2. AZURE_AI_CHAT_KEY - Your model key. Keep it secret, keep it safe.
12+
13+
In order to instruct the model that you want the output to follow a specific structure, you need to build out the JSON schema that you want the model to use. The schema is then given a name and passed to the `ChatCompletionsResponseFormat.CreateJsonFormat` method, which can then be provided to the `ResponseFormat` property of the request options object.
14+
15+
```C# Snippet:Azure_AI_Inference_SampleStructuredOutput
16+
var endpoint = new Uri(System.Environment.GetEnvironmentVariable("AZURE_AI_CHAT_ENDPOINT"));
17+
var credential = new AzureKeyCredential(System.Environment.GetEnvironmentVariable("AZURE_AI_CHAT_KEY"));
18+
19+
var client = new ChatCompletionsClient(endpoint, credential, new AzureAIInferenceClientOptions());
20+
21+
var messages = new List<ChatRequestMessage>()
22+
{
23+
new ChatRequestSystemMessage("You are a helpful assistant."),
24+
new ChatRequestUserMessage("Please give me directions and ingredients to bake a chocolate cake."),
25+
};
26+
27+
var requestOptions = new ChatCompletionsOptions(messages);
28+
29+
Dictionary<string, BinaryData> jsonSchema = new Dictionary<string, BinaryData>
30+
{
31+
{ "type", BinaryData.FromString("\"object\"") },
32+
{ "properties", BinaryData.FromString("""
33+
{
34+
"ingredients": {
35+
"type": "array",
36+
"items": {
37+
"type": "string"
38+
}
39+
},
40+
"steps": {
41+
"type": "array",
42+
"items": {
43+
"type": "object",
44+
"properties": {
45+
"ingredients": {
46+
"type": "array",
47+
"items": {
48+
"type": "string"
49+
}
50+
},
51+
"directions": {
52+
"type": "string"
53+
}
54+
}
55+
}
56+
},
57+
"prep_time": {
58+
"type": "string"
59+
},
60+
"bake_time": {
61+
"type": "string"
62+
}
63+
}
64+
""") },
65+
{ "required", BinaryData.FromString("[\"ingredients\", \"steps\", \"bake_time\"]") },
66+
{ "additionalProperties", BinaryData.FromString("false") }
67+
};
68+
69+
requestOptions.ResponseFormat = ChatCompletionsResponseFormat.CreateJsonFormat("cakeBakingDirections", jsonSchema);
70+
71+
Response<ChatCompletions> response = client.Complete(requestOptions);
72+
```
73+
74+
When the response is returned, it can then be parsed into the expected JSON format.
75+
76+
```C# Snippet:Azure_AI_Inference_SampleStructuredOutputParseJson
77+
using JsonDocument structuredJson = JsonDocument.Parse(result.Content);
78+
structuredJson.RootElement.TryGetProperty("ingredients", out var ingredients);
79+
structuredJson.RootElement.TryGetProperty("steps", out var steps);
80+
structuredJson.RootElement.TryGetProperty("bake_time", out var bakeTime);
81+
```
82+
83+
Printing the output to the console can also show that the output met the expected structure which was requested.
84+
85+
```C# Snippet:Azure_AI_Inference_SampleStructuredOutputPrintOutput
86+
var options = new JsonSerializerOptions
87+
{
88+
WriteIndented = true
89+
};
90+
Console.WriteLine($"Ingredients: {System.Text.Json.JsonSerializer.Serialize(ingredients, options)}");
91+
Console.WriteLine($"Steps: {System.Text.Json.JsonSerializer.Serialize(steps, options)}");
92+
Console.WriteLine($"Bake time: {System.Text.Json.JsonSerializer.Serialize(bakeTime, options)}");
93+
```
94+
95+
```Text
96+
Ingredients: [
97+
"2 cups of all-purpose flour",
98+
"2 cups of sugar",
99+
"3/4 cup of unsweetened cocoa powder",
100+
"2 teaspoons of baking powder",
101+
"1 1/2 teaspoons of baking soda",
102+
"1 teaspoon of salt",
103+
"1 teaspoon of instant coffee powder (optional, for enhancing chocolate flavor)",
104+
"1 cup of milk",
105+
"1/2 cup of vegetable oil",
106+
"2 large eggs",
107+
"2 teaspoons of vanilla extract",
108+
"1 cup of boiling water"
109+
]
110+
Steps: [
111+
{
112+
"ingredients": [
113+
"all-purpose flour",
114+
"sugar",
115+
"unsweetened cocoa powder",
116+
"baking powder",
117+
"baking soda",
118+
"salt",
119+
"instant coffee powder if using"
120+
],
121+
"directions": "Preheat your oven to 350°F (177°C) and grease and lightly flour two 9-inch round baking pans or line them with parchment paper. In a large bowl sift together the dry ingredients."
122+
},
123+
{
124+
"ingredients": [
125+
"milk",
126+
"vegetable oil",
127+
"eggs",
128+
"vanilla extract"
129+
],
130+
"directions": "Add the milk, vegetable oil, eggs, and vanilla to the bowl of dry ingredients and mix until smooth and well combined."
131+
},
132+
{
133+
"ingredients": [
134+
"boiling water"
135+
],
136+
"directions": "Slowly add the cup of boiling water to the mixture (the batter will be thin). Stir until well mixed."
137+
},
138+
{
139+
"ingredients": [],
140+
"directions": "Divide the batter evenly between the two prepared pans and bake in the preheated oven for 30-35 minutes, or until a toothpick inserted in the center comes out clean."
141+
},
142+
{
143+
"ingredients": [],
144+
"directions": "Remove the cakes from the oven and allow them to cool in the pans for about 10 minutes, then transfer them to a wire rack to cool completely."
145+
}
146+
]
147+
Bake time: "30-35 minutes"
148+
```

‎sdk/ai/Azure.AI.Inference/samples/Sample1_ChatCompletions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Set these two environment variables before running the sample:
88

99
1. AZURE_AI_CHAT_ENDPOINT - Your endpoint URL, in the form `https://your-deployment-name.your-azure-region.inference.ai.azure.com` where `your-deployment-name` is your unique AI Model deployment name, and `your-azure-region` is the Azure region where your model is deployed.
1010

11-
2. AZURE_AI_CHAT_KEY - Your model key (a 32-character string). Keep it secret, keep it safe.
11+
2. AZURE_AI_CHAT_KEY - Your model key. Keep it secret, keep it safe.
1212

1313
```C# Snippet:Azure_AI_Inference_HelloWorldScenario
1414
var endpoint = new Uri(System.Environment.GetEnvironmentVariable("AZURE_AI_CHAT_ENDPOINT"));

0 commit comments

Comments
 (0)