Merge pull request #3401 from MicrosoftDocs/main

3/7/2025 AM Publish

Author: Taojunshen

articles/ai-foundry/model-inference/concepts/models.md

@@ -139,6 +139,8 @@ Phi is a family of lightweight, state-of-the-art open models. These models were
 | [Phi-3-small-128k-instruct](https://ai.azure.com/explore/models/Phi-3-small-128k-instruct/version/4/registry/azureml) | chat-completion | Global standard | - **Input:** text (131,072 tokens) <br /> - **Output:**  (4,096 tokens) <br /> - **Languages:** en <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
 | [Phi-3.5-mini-instruct](https://ai.azure.com/explore/models/Phi-3.5-mini-instruct/version/6/registry/azureml) | chat-completion | Global standard | - **Input:** text (131,072 tokens) <br /> - **Output:**  (4,096 tokens) <br /> - **Languages:** en, ar, zh, cs, da, nl, fi, fr, de, he, hu, it, ja, ko, no, pl, pt, ru, es, sv, th, tr, and uk <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
 | [Phi-4](https://ai.azure.com/explore/models/Phi-4/version/2/registry/azureml) | chat-completion | Global standard | - **Input:** text (16,384 tokens) <br /> - **Output:**  (16,384 tokens) <br /> - **Languages:** en, ar, bn, cs, da, de, el, es, fa, fi, fr, gu, ha, he, hi, hu, id, it, ja, jv, kn, ko, ml, mr, nl, no, or, pa, pl, ps, pt, ro, ru, sv, sw, ta, te, th, tl, tr, uk, ur, vi, yo, and zh  - **Tool calling:** No <br /> - **Response formats:** Text |
+| [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) | chat-completion | Global standard | - **Input:** text (131,072 tokens) <br /> - **Output:**  (4,096 tokens) <br /> - **Languages:** `ar`, `zh`, `cs`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `he`, `hu`, `it`, `ja`, `ko`, `no`, `pl`, `pt`, `ru`, `es`, `sv`, `th`, `tr`, and `uk` <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
+| [Phi-4-multimodal-instruct](https://ai.azure.com/explore/models/Phi-4-multimodal-instruct/version/1/registry/azureml) | chat-completion | Global standard | - **Input:** text, images, and audio (131,072 tokens) <br /> - **Output:**  (4,096 tokens) <br /> - **Languages:** `ar`, `zh`, `cs`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `he`, `hu`, `it`, `ja`, `ko`, `no`, `pl`, `pt`, `ru`, `es`, `sv`, `th`, `tr`, and `uk` <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
 
 
 

articles/ai-foundry/model-inference/how-to/use-chat-multi-modal.md

@@ -0,0 +1,54 @@
+---
+title: How to use image and audio in chat completions with Azure AI model inference
+titleSuffix: Azure AI Foundry
+description: Learn how to process audio and images with chat completions models with Azure AI model inference
+manager: scottpolly
+author: msakande
+reviewer: santiagxf
+ms.service: azure-ai-model-inference
+ms.topic: how-to
+ms.date: 1/21/2025
+ms.author: mopeakande
+ms.reviewer: fasantia
+ms.custom: generated
+zone_pivot_groups: azure-ai-inference-samples
+---
+
+# How to use image and audio in chat completions with Azure AI model inference
+
+
+::: zone pivot="programming-language-python"
+
+[!INCLUDE [python](../includes/use-chat-multi-modal/python.md)]
+::: zone-end
+
+
+::: zone pivot="programming-language-javascript"
+
+[!INCLUDE [javascript](../includes/use-chat-multi-modal/javascript.md)]
+::: zone-end
+
+
+::: zone pivot="programming-language-java"
+
+[!INCLUDE [java](../includes/use-chat-multi-modal/java.md)]
+::: zone-end
+
+
+::: zone pivot="programming-language-csharp"
+
+[!INCLUDE [csharp](../includes/use-chat-multi-modal/csharp.md)]
+::: zone-end
+
+
+::: zone pivot="programming-language-rest"
+
+[!INCLUDE [rest](../includes/use-chat-multi-modal/rest.md)]
+::: zone-end
+
+## Related content
+
+* [Use embeddings models](use-embeddings.md)
+* [Use image embeddings models](use-image-embeddings.md)
+* [Use reasoning models](use-chat-reasoning.md)
+* [Azure AI Model Inference API](.././reference/reference-model-inference-api.md)

articles/ai-foundry/model-inference/includes/how-to-prerequisites-csharp.md

@@ -0,0 +1,20 @@
+---
+manager: nitinme
+ms.service: azure-ai-model-inference
+ms.topic: include
+ms.date: 1/21/2025
+ms.author: fasantia
+author: santiagxf
+---
+
+* Install the [Azure AI inference package](https://aka.ms/azsdk/azure-ai-inference/python/reference) with the following command:
+
+    ```bash
+    dotnet add package Azure.AI.Inference --prerelease
+    ```
+    
+* If you are using Entra ID, you also need the following package:
+
+    ```bash
+    dotnet add package Azure.Identity
+    ```

articles/ai-foundry/model-inference/includes/how-to-prerequisites-java.md

@@ -0,0 +1,44 @@
+2---
+manager: nitinme
+ms.service: azure-ai-model-inference
+ms.topic: include
+ms.date: 1/21/2025
+ms.author: fasantia
+author: santiagxf
+---
+
+* Add the [Azure AI inference package](https://aka.ms/azsdk/azure-ai-inference/java/reference) to your project:
+
+  ```xml
+  <dependency>
+      <groupId>com.azure</groupId>
+      <artifactId>azure-ai-inference</artifactId>
+      <version>1.0.0-beta.1</version>
+  </dependency>
+  ```
+  
+* If you are using Entra ID, you also need the following package:
+
+  ```xml
+  <dependency>
+      <groupId>com.azure</groupId>
+      <artifactId>azure-identity</artifactId>
+      <version>1.13.3</version>
+  </dependency>
+  ```
+
+* Import the following namespace:
+  
+  ```java
+  package com.azure.ai.inference.usage;
+  
+  import com.azure.ai.inference.EmbeddingsClient;
+  import com.azure.ai.inference.EmbeddingsClientBuilder;
+  import com.azure.ai.inference.models.EmbeddingsResult;
+  import com.azure.ai.inference.models.EmbeddingItem;
+  import com.azure.core.credential.AzureKeyCredential;
+  import com.azure.core.util.Configuration;
+  
+  import java.util.ArrayList;
+  import java.util.List;
+  ```

articles/ai-foundry/model-inference/includes/how-to-prerequisites-javascript.md

@@ -0,0 +1,14 @@
+---
+manager: nitinme
+ms.service: azure-ai-model-inference
+ms.topic: include
+ms.date: 1/21/2025
+ms.author: fasantia
+author: santiagxf
+---
+
+* Install the [Azure Inference library for JavaScript](https://aka.ms/azsdk/azure-ai-inference/javascript/reference) with the following command:
+
+  ```bash
+  npm install @azure-rest/ai-inference
+  ```

articles/ai-foundry/model-inference/includes/how-to-prerequisites-python.md

@@ -0,0 +1,14 @@
+---
+manager: nitinme
+ms.service: azure-ai-model-inference
+ms.topic: include
+ms.date: 1/21/2025
+ms.author: fasantia
+author: santiagxf
+---
+
+* Install the [Azure AI inference package for Python](https://aka.ms/azsdk/azure-ai-inference/python/reference) with the following command:
+
+  ```bash
+  pip install -U azure-ai-inference
+  ```

articles/ai-foundry/model-inference/includes/use-chat-completions/javascript.md

@@ -24,13 +24,9 @@ To use chat completion models in your application, you need:
 
 [!INCLUDE [how-to-prerequisites](../how-to-prerequisites.md)]
 
-* A chat completions model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
-
-* Install the [Azure Inference library for JavaScript](https://aka.ms/azsdk/azure-ai-inference/javascript/reference) with the following command:
+[!INCLUDE [how-to-prerequisites-javascript](../how-to-prerequisites-javascript.md)]
 
-  ```bash
-  npm install @azure-rest/ai-inference
-  ```
+* A chat completions model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
 
 ## Use chat completions
 

articles/ai-foundry/model-inference/includes/use-chat-completions/python.md

@@ -24,14 +24,9 @@ To use chat completion models in your application, you need:
 
 [!INCLUDE [how-to-prerequisites](../how-to-prerequisites.md)]
 
-* A chat completions model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add a chat completions model to your resource.
-
-* Install the [Azure AI inference package for Python](https://aka.ms/azsdk/azure-ai-inference/python/reference) with the following command:
-
-  ```bash
-  pip install -U azure-ai-inference
-  ```
+[!INCLUDE [how-to-prerequisites-python](../how-to-prerequisites-python.md)]
 
+
 ## Use chat completions
 
 First, create the client to consume the model. The following code uses an endpoint URL and key that are stored in environment variables.
@@ -179,15 +174,13 @@ Some models can create JSON outputs. Set `response_format` to `json_object` to e
 
 
 ```python
-from azure.ai.inference.models import ChatCompletionsResponseFormatJSON
-
 response = client.complete(
     messages=[
         SystemMessage(content="You are a helpful assistant that always generate responses in JSON format, using."
                       " the following format: { ""answer"": ""response"" }."),
         UserMessage(content="How many languages are in the world?"),
     ],
-    response_format={ "type": ChatCompletionsResponseFormatJSON() }
+    response_format="json_object"
 )
 ```
 
@@ -218,9 +211,9 @@ The following code example creates a tool definition that is able to look from f
 
 
 ```python
-from azure.ai.inference.models import FunctionDefinition, ChatCompletionsFunctionToolDefinition
+from azure.ai.inference.models import FunctionDefinition, ChatCompletionsToolDefinition
 
-flight_info = ChatCompletionsFunctionToolDefinition(
+flight_info = ChatCompletionsToolDefinition(
     function=FunctionDefinition(
         name="get_flight_info",
         description="Returns information about the next flight between two cities. This includes the name of the airline, flight number and the date and time of the next flight",

articles/ai-foundry/model-inference/includes/use-chat-completions/rest.md

@@ -28,7 +28,7 @@ To use chat completion models in your application, you need:
 
 ## Use chat completions
 
-To use the text embeddings, use the route `/chat/completions` appended to the base URL along with your credential indicated in `api-key`. `Authorization` header is also supported with the format `Bearer <key>`.
+To use chat completions API, use the route `/chat/completions` appended to the base URL along with your credential indicated in `api-key`. `Authorization` header is also supported with the format `Bearer <key>`.
 
 ```http
 POST https://<resource>.services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview
@@ -553,7 +553,7 @@ Some models can reason across text and images and generate text completions base
 To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
 
 > [!TIP]
-> You will need to construct the data URL using a scripting or programming language. This tutorial use [this sample image](../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg) in JPEG format. A data URL has a format as follows: `data:image/jpg;base64,0xABCDFGHIJKLMNOPQRSTUVWXYZ...`.
+> You will need to construct the data URL using a scripting or programming language. This tutorial uses [this sample image](../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg) in JPEG format. A data URL has a format as follows: `data:image/jpg;base64,0xABCDFGHIJKLMNOPQRSTUVWXYZ...`.
 
 Visualize the image: