Merge pull request #3552 from msakande/update-chat-completions-audio-and-image

Update chat completions audio and image

Author: prmerger-automator[bot]

articles/ai-foundry/model-inference/includes/use-chat-completions/csharp.md

@@ -7,7 +7,7 @@ author: mopeakande
 reviewer: santiagxf
 ms.service: azure-ai-model-inference
 ms.topic: how-to
-ms.date: 1/21/2025
+ms.date: 03/20/2025
 ms.author: mopeakande
 ms.reviewer: fasantia
 ms.custom: references_regions, tool_generated
@@ -26,7 +26,7 @@ To use chat completion models in your application, you need:
 
 [!INCLUDE [how-to-prerequisites-csharp](../how-to-prerequisites-csharp.md)]
 
-* A chat completions model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
+* A chat completions model deployment. If you don't have one, read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
 
     * This example uses `mistral-large-2407`.
 
@@ -42,7 +42,7 @@ ChatCompletionsClient client = new ChatCompletionsClient(
 );
 ```
 
-If you have configured the resource to with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
+If you've configured the resource with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
 
 
 ```csharp
@@ -181,7 +181,7 @@ response = client.Complete(requestOptions);
 Console.WriteLine($"Response: {response.Value.Content}");
 ```
 
-Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs are not guaranteed to be valid JSON.
+Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs aren't guaranteed to be valid JSON.
 
 If you want to pass a parameter that isn't in the list of supported parameters, you can pass it to the underlying model using *extra parameters*. See [Pass extra parameters to the model](#pass-extra-parameters-to-the-model).
 
@@ -400,69 +400,3 @@ catch (RequestFailedException ex)
 
 > [!TIP]
 > To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
-
-## Use chat completions with images
-
-Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
-
-> [!IMPORTANT]
-> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
-
-To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
-
-
-```csharp
-string imageUrl = "https://news.microsoft.com/source/wp-content/uploads/2024/04/The-Phi-3-small-language-models-with-big-potential-1-1900x1069.jpg";
-string imageFormat = "jpeg";
-HttpClient httpClient = new HttpClient();
-httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0");
-byte[] imageBytes = httpClient.GetByteArrayAsync(imageUrl).Result;
-string imageBase64 = Convert.ToBase64String(imageBytes);
-string dataUrl = $"data:image/{imageFormat};base64,{imageBase64}";
-```
-
-Visualize the image:
-
-:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
-
-Now, create a chat completion request with the image:
-
-
-```csharp
-ChatCompletionsOptions requestOptions = new ChatCompletionsOptions()
-{
-    Messages = {
-        new ChatRequestSystemMessage("You are an AI assistant that helps people find information."),
-        new ChatRequestUserMessage([
-            new ChatMessageTextContentItem("Which conclusion can be extracted from the following chart?"),
-            new ChatMessageImageContentItem(new Uri(dataUrl))
-        ]),
-    },
-    MaxTokens=2048,
-    Model = "phi-3.5-vision-instruct",
-};
-
-var response = client.Complete(requestOptions);
-Console.WriteLine(response.Value.Content);
-```
-
-The response is as follows, where you can see the model's usage statistics:
-
-
-```csharp
-Console.WriteLine($"{response.Value.Role}: {response.Value.Content}");
-Console.WriteLine($"Model: {response.Value.Model}");
-Console.WriteLine("Usage:");
-Console.WriteLine($"\tPrompt tokens: {response.Value.Usage.PromptTokens}");
-Console.WriteLine($"\tTotal tokens: {response.Value.Usage.TotalTokens}");
-Console.WriteLine($"\tCompletion tokens: {response.Value.Usage.CompletionTokens}");
-```
-
-```console
-ASSISTANT: The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.
-Model: phi-3.5-vision-instruct
-Usage: 
-  Prompt tokens: 2380
-  Completion tokens: 126
-  Total tokens: 2506
-```

articles/ai-foundry/model-inference/includes/use-chat-completions/java.md

@@ -7,7 +7,7 @@ author: mopeakande
 reviewer: santiagxf
 ms.service: azure-ai-model-inference
 ms.topic: how-to
-ms.date: 1/21/2025
+ms.date: 03/20/2025
 ms.author: mopeakande
 ms.reviewer: fasantia
 ms.custom: references_regions, tool_generated
@@ -26,7 +26,7 @@ To use chat completion models in your application, you need:
 
 [!INCLUDE [how-to-prerequisites-java](../how-to-prerequisites-java.md)]
 
-* A chat completions model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
+* A chat completions model deployment. If you don't have one, read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
 
     * This example uses `mistral-large-2407`.
 
@@ -41,7 +41,7 @@ ChatCompletionsClient client = new ChatCompletionsClientBuilder()
     .buildClient();
 ```
 
-If you have configured the resource to with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
+If you've configured the resource with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
 
 ```java
 TokenCredential defaultCredential = new DefaultAzureCredentialBuilder().build();
@@ -120,7 +120,7 @@ client.completeStream(new ChatCompletionsOptions(chatMessages))
 #### Explore more parameters supported by the inference client
 
 Explore other parameters that you can specify in the inference client. For a full list of all the supported parameters and their corresponding documentation, see [Azure AI Model Inference API reference](https://aka.ms/azureai/modelinference).
-Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs are not guaranteed to be valid JSON.
+Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs aren't guaranteed to be valid JSON.
 
 If you want to pass a parameter that isn't in the list of supported parameters, you can pass it to the underlying model using *extra parameters*. See [Pass extra parameters to the model](#pass-extra-parameters-to-the-model).
 

articles/ai-foundry/model-inference/includes/use-chat-completions/javascript.md

@@ -7,7 +7,7 @@ author: mopeakande
 reviewer: santiagxf
 ms.service: azure-ai-model-inference
 ms.topic: how-to
-ms.date: 1/21/2025
+ms.date: 03/20/2025
 ms.author: mopeakande
 ms.reviewer: fasantia
 ms.custom: references_regions, tool_generated
@@ -26,7 +26,7 @@ To use chat completion models in your application, you need:
 
 [!INCLUDE [how-to-prerequisites-javascript](../how-to-prerequisites-javascript.md)]
 
-* A chat completions model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
+* A chat completions model deployment. If you don't have one, read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
 
 ## Use chat completions
 
@@ -44,7 +44,7 @@ const client = new ModelClient(
 );
 ```
 
-If you have configured the resource to with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
+If you've configured the resource with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
 
 
 ```javascript
@@ -177,7 +177,7 @@ var response = await client.path("/chat/completions").post({
 });
 ```
 
-Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs are not guaranteed to be valid JSON.
+Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs aren't guaranteed to be valid JSON.
 
 If you want to pass a parameter that isn't in the list of supported parameters, you can pass it to the underlying model using *extra parameters*. See [Pass extra parameters to the model](#pass-extra-parameters-to-the-model).
 
@@ -388,82 +388,3 @@ catch (error) {
 > [!TIP]
 > To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
 
-## Use chat completions with images
-
-Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
-
-> [!IMPORTANT]
-> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
-
-To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
-
-
-```javascript
-const image_url = "https://news.microsoft.com/source/wp-content/uploads/2024/04/The-Phi-3-small-language-models-with-big-potential-1-1900x1069.jpg";
-const image_format = "jpeg";
-
-const response = await fetch(image_url, { headers: { "User-Agent": "Mozilla/5.0" } });
-const image_data = await response.arrayBuffer();
-const image_data_base64 = Buffer.from(image_data).toString("base64");
-const data_url = `data:image/${image_format};base64,${image_data_base64}`;
-```
-
-Visualize the image:
-
-
-```javascript
-const img = document.createElement("img");
-img.src = data_url;
-document.body.appendChild(img);
-```
-
-:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
-
-Now, create a chat completion request with the image:
-
-
-```javascript
-var messages = [
-    { role: "system", content: "You are a helpful assistant that can generate responses based on images." },
-    { role: "user", content: 
-        [
-            { type: "text", text: "Which conclusion can be extracted from the following chart?" },
-            { type: "image_url", image:
-                {
-                    url: data_url
-                }
-            } 
-        ] 
-    }
-];
-
-var response = await client.path("/chat/completions").post({
-    body: {
-        messages: messages,
-        temperature: 0,
-        top_p: 1,
-        max_tokens: 2048,
-    }
-});
-```
-
-The response is as follows, where you can see the model's usage statistics:
-
-
-```javascript
-console.log(response.body.choices[0].message.role + ": " + response.body.choices[0].message.content);
-console.log("Model:", response.body.model);
-console.log("Usage:");
-console.log("\tPrompt tokens:", response.body.usage.prompt_tokens);
-console.log("\tCompletion tokens:", response.body.usage.completion_tokens);
-console.log("\tTotal tokens:", response.body.usage.total_tokens);
-```
-
-```console
-ASSISTANT: The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.
-Model: mistral-large-2407
-Usage: 
-  Prompt tokens: 2380
-  Completion tokens: 126
-  Total tokens: 2506
-```

articles/ai-foundry/model-inference/includes/use-chat-completions/python.md

@@ -7,7 +7,7 @@ author: mopeakande
 reviewer: santiagxf
 ms.service: azure-ai-model-inference
 ms.topic: how-to
-ms.date: 1/21/2025
+ms.date: 03/20/2025
 ms.author: mopeakande
 ms.reviewer: fasantia
 ms.custom: references_regions, tool_generated
@@ -26,7 +26,7 @@ To use chat completion models in your application, you need:
 
 [!INCLUDE [how-to-prerequisites-python](../how-to-prerequisites-python.md)]
 
-* A chat completions model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
+* A chat completions model deployment. If you don't have one, read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
 
     * This example uses `mistral-large-2407`.
 
@@ -46,7 +46,7 @@ client = ChatCompletionsClient(
 )
 ```
 
-If you have configured the resource to with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
+If you've configured the resource with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
 
 
 ```python
@@ -166,7 +166,7 @@ response = client.complete(
 )
 ```
 
-Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs are not guaranteed to be valid JSON.
+Some models don't support JSON output formatting. You can always prompt the model to generate JSON outputs. However, such outputs aren't guaranteed to be valid JSON.
 
 If you want to pass a parameter that isn't in the list of supported parameters, you can pass it to the underlying model using *extra parameters*. See [Pass extra parameters to the model](#pass-extra-parameters-to-the-model).
 
@@ -373,76 +373,3 @@ except HttpResponseError as ex:
 > [!TIP]
 > To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
 
-## Use chat completions with images
-
-Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
-
-> [!IMPORTANT]
-> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
-
-To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
-
-
-```python
-from urllib.request import urlopen, Request
-import base64
-
-image_url = "https://news.microsoft.com/source/wp-content/uploads/2024/04/The-Phi-3-small-language-models-with-big-potential-1-1900x1069.jpg"
-image_format = "jpeg"
-
-request = Request(image_url, headers={"User-Agent": "Mozilla/5.0"})
-image_data = base64.b64encode(urlopen(request).read()).decode("utf-8")
-data_url = f"data:image/{image_format};base64,{image_data}"
-```
-
-Visualize the image:
-
-
-```python
-import requests
-import IPython.display as Disp
-
-Disp.Image(requests.get(image_url).content)
-```
-
-:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
-
-Now, create a chat completion request with the image:
-
-
-```python
-from azure.ai.inference.models import TextContentItem, ImageContentItem, ImageUrl
-response = client.complete(
-    messages=[
-        SystemMessage("You are a helpful assistant that can generate responses based on images."),
-        UserMessage(content=[
-            TextContentItem(text="Which conclusion can be extracted from the following chart?"),
-            ImageContentItem(image=ImageUrl(url=data_url))
-        ]),
-    ],
-    temperature=0,
-    top_p=1,
-    max_tokens=2048,
-)
-```
-
-The response is as follows, where you can see the model's usage statistics:
-
-
-```python
-print(f"{response.choices[0].message.role}:\n\t{response.choices[0].message.content}\n")
-print("Model:", response.model)
-print("Usage:")
-print("\tPrompt tokens:", response.usage.prompt_tokens)
-print("\tCompletion tokens:", response.usage.completion_tokens)
-print("\tTotal tokens:", response.usage.total_tokens)
-```
-
-```console
-ASSISTANT: The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.
-Model: mistral-large-2407
-Usage: 
-  Prompt tokens: 2380
-  Completion tokens: 126
-  Total tokens: 2506
-```