Merge branch 'main' into release-azure-search

Author: v-alje

articles/ai-services/openai/api-version-deprecation.md

@@ -5,7 +5,7 @@ services: cognitive-services
 manager: nitinme
 ms.service: azure-ai-openai
 ms.topic: conceptual 
-ms.date: 09/05/2024
+ms.date: 10/01/2024
 author: mrbullwinkle
 ms.author: mbullwin
 recommendations: false
@@ -23,7 +23,7 @@ This article is to help you understand the support lifecycle for the Azure OpenA
 
 Azure OpenAI API latest release:
 
-- Inference: [2024-08-01-preview](https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2024-08-01-preview/inference.json)
+- Inference: [2024-09-01-preview](https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2024-09-01-preview/inference.json)
 - Authoring: [2024-08-01-preview](https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/authoring/preview/2024-08-01-preview/azureopenai.json)
 
 This version contains support for the latest Azure OpenAI features including:
@@ -38,6 +38,13 @@ This version contains support for the latest Azure OpenAI features including:
 - [Function calling](./how-to/function-calling.md)  [**Added in 2023-07-01-preview**]
 - [Retrieval augmented generation with your data feature](./use-your-data-quickstart.md).  [**Added in 2023-06-01-preview**]
 
+## Changes between 2024-09-01-preview and 2024-08-01-preview
+
+- `max_completion_tokens` added to support `o1-preview` and `o1-mini` models. `max_tokens` does not work with the **o1 series** models.
+- `parallel_tool_calls` added.
+- `completion_tokens_details` & `reasoning_tokens` added.
+- `stream_options` & `include_usage` added.
+
 ## Changes between 2024-07-01-preview and 2024-08-01-preview API specification
 
 - [Structured outputs support](./how-to/structured-outputs.md).

articles/ai-services/openai/how-to/audio-real-time.md

@@ -5,10 +5,11 @@ description: Learn how to use GPT-4o Realtime API for speech and audio with Azur
 manager: nitinme
 ms.service: azure-ai-openai
 ms.topic: how-to
-ms.date: 10/1/2024
+ms.date: 10/3/2024
 author: eric-urban
 ms.author: eur
 ms.custom: references_regions
+zone_pivot_groups: openai-studio-js
 recommendations: false
 ---
 
@@ -54,14 +55,35 @@ You can deploy the model from the [Azure AI Studio model catalog](../../../ai-st
 1. Modify other default settings depending on your requirements.
 1. Select **Deploy**. You land on the deployment details page. 
 
-Now that you have a deployment of the `gpt-4o-realtime-preview` model, you can use the Realtime API to interact with it in real time.
+Now that you have a deployment of the `gpt-4o-realtime-preview` model, you can use the AI Studio **Real-time audio** playground or Realtime API to interact with it in real time.
 
-## Use the GPT-4o Realtime API
+## Use the GPT-4o real-time audio
 
 > [!TIP]
-> A playground for GPT-4o real-time audio is coming soon to [Azure AI Studio](https://ai.azure.com). You can already use the API directly in your application.
+> Right now, the fastest way to get started development with the GPT-4o Realtime API is to download the sample code from the [Azure OpenAI GPT-4o real-time audio repository on GitHub](https://github.com/azure-samples/aoai-realtime-audio-sdk).
 
-Right now, the fastest way to get started with the GPT-4o Realtime API is to download the sample code from the [Azure OpenAI GPT-4o real-time audio repository on GitHub](https://github.com/azure-samples/aoai-realtime-audio-sdk).
+::: zone pivot="programming-language-ai-studio"
+
+To chat with your deployed `gpt-4o-realtime-preview` model in the [Azure AI Studio](https://ai.azure.com) **Real-time audio** playground, follow these steps:
+
+1. Go to your project in [Azure AI Studio](https://ai.azure.com). 
+1. Select **Playgrounds** > **Real-time audio** from the left pane.
+1. Select your deployed `gpt-4o-realtime-preview` model from the **Deployment** dropdown. 
+1. Select **Enable microphone** to allow the browser to access your microphone. If you already granted permission, you can skip this step.
+
+    :::image type="content" source="../media/how-to/real-time/real-time-playground.png" alt-text="Screenshot of the real-time audio playground with the deployed model selected." lightbox="../media/how-to/real-time/real-time-playground.png":::
+
+1. Optionally you can edit contents in the **Give the model instructions and context** text box. Give the model instructions about how it should behave and any context it should reference when generating a response. You can describe the assistant's personality, tell it what it should and shouldn't answer, and tell it how to format responses.
+1. Optionally, change settings such as threshold, prefix padding, and silence duration.
+1. Select **Start listening** to start the session. You can speak into the microphone to start a chat.
+
+    :::image type="content" source="../media/how-to/real-time/real-time-playground-start-listening.png" alt-text="Screenshot of the real-time audio playground with the start listening button and microphone access enabled." lightbox="../media/how-to/real-time/real-time-playground-start-listening.png":::
+
+1. You can interrupt the chat at any time by speaking. You can end the chat by selecting the **Stop listening** button.
+
+::: zone-end
+
+::: zone pivot="programming-language-javascript"
 
 The JavaScript web sample demonstrates how to use the GPT-4o Realtime API to interact with the model in real time. The sample code includes a simple web interface that captures audio from the user's microphone and sends it to the model for processing. The model responds with text and audio, which the sample code renders in the web interface.
 
@@ -103,6 +125,8 @@ You can run the sample code locally on your machine by following these steps. Re
 1. You should see a `<< Session Started >>` message in the main output. Then you can speak into the microphone to start a chat.
 1. You can interrupt the chat at any time by speaking. You can end the chat by selecting the **Stop** button.
 
+::: zone-end
+
 ## Related content
 
 * Learn more about Azure OpenAI [deployment types](./deployment-types.md)

articles/ai-services/openai/includes/api-surface.md

@@ -22,8 +22,8 @@ Each API surface/specification encapsulates a different set of Azure OpenAI capa
 | API | Latest preview release | Latest GA release | Specifications | Description |
 |:---|:----|:----|:----|:---|
 | **Control plane** | [`2024-06-01-preview`](/rest/api/aiservices/accountmanagement/operation-groups?view=rest-aiservices-accountmanagement-2024-06-01-preview&preserve-view=true) | [`2023-05-01`](/rest/api/aiservices/accountmanagement/deployments/create-or-update?view=rest-aiservices-accountmanagement-2023-05-01&tabs=HTTP&preserve-view=true) | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/resource-manager/Microsoft.CognitiveServices) | Azure OpenAI shares a common control plane with all other Azure AI Services. The control plane API is used for things like [creating Azure OpenAI resources](/rest/api/aiservices/accountmanagement/accounts/create?view=rest-aiservices-accountmanagement-2023-05-01&tabs=HTTP&preserve-view=true), [model deployment](/rest/api/aiservices/accountmanagement/deployments/create-or-update?view=rest-aiservices-accountmanagement-2023-05-01&tabs=HTTP&preserve-view=true), and other higher level resource management tasks. The control plane also governs what is possible to do with capabilities like Azure Resource Manager, Bicep, Terraform, and Azure CLI.|
-| **Data plane - authoring** | `2024-08-01-preview` | [`2024-06-01`](/rest/api/azureopenai/operation-groups?view=rest-azureopenai-2024-06-01&preserve-view=true) | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI/authoring) | The data plane authoring API controls [fine-tuning](/rest/api/azureopenai/fine-tuning?view=rest-azureopenai-2024-08-01-preview&preserve-view=true), [file-upload](/rest/api/azureopenai/files/upload?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [ingestion jobs](/rest/api/azureopenai/ingestion-jobs/create?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [batch](/rest/api/azureopenai/batch?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true) and certain [model level queries](/rest/api/azureopenai/models/get?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true)
-| **Data plane - inference** | [`2024-08-01-preview`](/azure/ai-services/openai/reference-preview#data-plane-inference) | [`2024-06-01`](/azure/ai-services/openai/reference#data-plane-inference) | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference) | The data plane inference API provides the inference capabilities/endpoints for features like completions, chat completions, embeddings, speech/whisper, on your data, Dall-e, assistants, etc. |
+| **Data plane - authoring** | [`2024-08-01-preview`](/rest/api/azureopenai/operation-groups?view=rest-azureopenai-2024-08-01-preview&preserve-view=true) | [`2024-06-01`](/rest/api/azureopenai/operation-groups?view=rest-azureopenai-2024-06-01&preserve-view=true) | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI/authoring) | The data plane authoring API controls [fine-tuning](/rest/api/azureopenai/fine-tuning?view=rest-azureopenai-2024-08-01-preview&preserve-view=true), [file-upload](/rest/api/azureopenai/files/upload?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [ingestion jobs](/rest/api/azureopenai/ingestion-jobs/create?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true), [batch](/rest/api/azureopenai/batch?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true) and certain [model level queries](/rest/api/azureopenai/models/get?view=rest-azureopenai-2024-08-01-preview&tabs=HTTP&preserve-view=true)
+| **Data plane - inference** | [`2024-09-01-preview`](/azure/ai-services/openai/reference-preview#data-plane-inference) | [`2024-06-01`](/azure/ai-services/openai/reference#data-plane-inference) | [Spec files](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference) | The data plane inference API provides the inference capabilities/endpoints for features like completions, chat completions, embeddings, speech/whisper, on your data, Dall-e, assistants, etc. |
 
 ## Authentication