Merge pull request #275990 from MicrosoftDocs/main

5/22 11:00 AM IST Publish

Author: PhilKang0704

.openpublishing.redirection.azure-monitor.json

@@ -6649,9 +6649,14 @@
             "redirect_url": "/azure/azure-functions/functions-monitoring",
             "redirect_document_id": false
         },
+        {
+            "source_path_from_root": "/articles/azure-monitor/app/app-insights-azure-ad-api.md",
+            "redirect_url": "/azure/azure-monitor/app/azure-ad-authentication",
+            "redirect_document_id": false
+        },
         {
             "source_path_from_root": "/articles/azure-monitor/app/resources-roles-access-control.md",
-            "redirect_url": "/azure/azure-monitor//roles-permissions-security",
+            "redirect_url": "/azure/azure-monitor/roles-permissions-security",
             "redirect_document_id": false
         },
         {

articles/ai-services/openai/concepts/models.md

@@ -282,6 +282,9 @@ The following Embeddings models are available with [Azure Government](/azure/azu
 | `gpt-35-turbo` (0613) | East US2 <br> North Central US <br> Sweden Central <br> Switzerland West | 4,096 | Sep 2021 |
 | `gpt-35-turbo` (1106) | East US2 <br> North Central US <br> Sweden Central <br> Switzerland West | Input: 16,385<br> Output: 4,096 |  Sep 2021|
 | `gpt-35-turbo` (0125)  | East US2 <br> North Central US <br> Sweden Central <br> Switzerland West | 16,385 | Sep 2021 |
+| `gpt-4` (0613) <sup>**1**<sup> | North Central US <br> Sweden Central | 8192 | Sep 2021 |
+
+**<sup>1<sup>** GPT-4 fine-tuning is currently in public preview. See our [GPT-4 fine-tuning safety evaluation guidance](/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython-new&pivots=programming-language-python#safety-evaluation-gpt-4-fine-tuning---public-preview) for more information.
 
 ### Whisper models
 

articles/ai-services/openai/includes/assistants-javascript.md

@@ -7,7 +7,7 @@ author: mrbullwinkle
 ms.author: mbullwin
 ms.service: azure-ai-openai
 ms.topic: include
-ms.date: 04/10/2024
+ms.date: 05/21/2024
 ms.custom: passwordless-js, devex-track-javascript
 ---
 
@@ -38,7 +38,7 @@ For passwordless authentication, you need to
 1. Install the OpenAI Assistants client library for JavaScript with:
 
     ```console
-    npm install @azure/openai-assistants
+    npm install openai
     ```
 
 2. For the **recommended** passwordless authentication:
@@ -47,12 +47,6 @@ For passwordless authentication, you need to
     npm install @azure/identity
     ```
 
-    Or to use the service key connection:
-
-    ```console
-    npm install @azure/core-auth
-    ```
-
 ## Retrieve key and endpoint
 
 To successfully make a call against the Azure OpenAI service, you'll need the following:

articles/ai-services/openai/includes/fine-tuning-openai-in-ai-studio.md

@@ -35,9 +35,12 @@ The following models support fine-tuning:
 - `gpt-35-turbo` (0613)
 - `gpt-35-turbo` (1106)
 - `gpt-35-turbo` (0125)
+- `gpt-4` (0613)
 
 Consult the [models page](../concepts/models.md#fine-tuning-models) to check which regions currently support fine-tuning.
 
+If you plan to use `gpt-4` for fine-tuning, please refer to the [GPT-4 public preview safety evaluation guidance](#safety-evaluation-gpt-4-fine-tuning---public-preview)
+
 ## Review the workflow for Azure AI Studio
 
 Take a moment to review the fine-tuning workflow for using Azure AI Studio:
@@ -243,6 +246,10 @@ You can also view the data in your results.csv file as plots in Azure AI Studio
 
 Look for your loss to decrease over time, and your accuracy to increase. If you see a divergence between your training and validation data that may indicate that you are overfitting. Try training with fewer epochs, or a smaller learning rate multiplier.
 
+## Safety evaluation GPT-4 fine-tuning - public preview
+
+[!INCLUDE [Safety evaluation](../includes/safety-evaluation.md)]
+
 ## Deploy a fine-tuned model
 
 Once your model is fine-tuned, you can deploy the model and can use it in your own application. You can't deploy a fine-tuned model from the deployments page or the playground page in Azure AI Studio. The only way, currently, to deploy a fine-tuned model is from the model details page for that model.

articles/ai-services/openai/includes/fine-tuning-python.md

@@ -35,6 +35,9 @@ The following models support fine-tuning:
 - `gpt-35-turbo` (0613)
 - `gpt-35-turbo` (1106)
 - `gpt-35-turbo` (0125)
+- `gpt-4` (0613)
+
+If you plan to use `gpt-4` for fine-tuning, please refer to the [GPT-4 public preview safety evaluation guidance](#safety-evaluation-gpt-4-fine-tuning---public-preview)
 
 Or you can fine tune a previously fine-tuned model, formatted as base-model.ft-{jobid}.
 
@@ -372,6 +375,10 @@ This command isn't available in the 0.28.1 OpenAI Python library. Upgrade to the
 
 ---
 
+## Safety evaluation GPT-4 fine-tuning - public preview
+
+[!INCLUDE [Safety evaluation](../includes/safety-evaluation.md)]
+
 ## Deploy a customized model
 
 When the fine-tuning job succeeds, the value of the `fine_tuned_model` variable in the response body is set to the name of your customized model. Your model is now also available for discovery from the [list Models API](/rest/api/azureopenai/models/list). However, you can't issue completion calls to your customized model until your customized model is deployed. You must deploy your customized model to make it available for use with completion calls.

articles/ai-services/openai/includes/fine-tuning-rest.md

@@ -33,9 +33,12 @@ The following models support fine-tuning:
 - `gpt-35-turbo` (0613)
 - `gpt-35-turbo` (1106)
 - `gpt-35-turbo` (0125)
+- `gpt-4` (0613)
 
 Consult the [models page](../concepts/models.md#fine-tuning-models) to check which regions currently support fine-tuning.
 
+If you plan to use `gpt-4` for fine-tuning, please refer to the [GPT-4 public preview safety evaluation guidance](#safety-evaluation-gpt-4-fine-tuning---public-preview).
+
 ## Review the workflow for the REST API
 
 Take a moment to review the fine-tuning workflow for using the REST APIS and Python with Azure OpenAI:
@@ -245,6 +248,10 @@ curl -X POST $AZURE_OPENAI_ENDPOINT/openai/fine_tuning/jobs/{fine_tuning_job_id}
   -H "api-key: $AZURE_OPENAI_API_KEY" 
 ```
 
+## Safety evaluation GPT-4 fine-tuning - public preview
+
+[!INCLUDE [Safety evaluation](../includes/safety-evaluation.md)]
+
 ## Deploy a customized model
 
 [!INCLUDE [Fine-tuning deletion](fine-tune.md)]

articles/ai-services/openai/includes/fine-tuning-studio.md

@@ -32,9 +32,12 @@ The following models support fine-tuning:
 - `gpt-35-turbo` (0613)
 - `gpt-35-turbo` (1106)
 - `gpt-35-turbo` (0125)
+- `gpt-4` (0613)
 
 Consult the [models page](../concepts/models.md#fine-tuning-models) to check which regions currently support fine-tuning.
 
+If you plan to use `gpt-4` for fine-tuning, please refer to the [GPT-4 public preview safety evaluation guidance](#safety-evaluation-gpt-4-fine-tuning---public-preview)
+
 ## Review the workflow for Azure OpenAI Studio
 
 Take a moment to review the fine-tuning workflow for using Azure OpenAI Studio:
@@ -162,6 +165,7 @@ You can create a custom model from one of the following available base models:
 - `gpt-35-turbo` (0613)
 - `gpt-35-turbo` (1106)
 - `gpt-35-turbo` (0125)
+- `gpt-4` (0613)
 
 - Or you can fine tune a previously fine-tuned model, formatted as base-model.ft-{jobid}.
 
@@ -313,6 +317,10 @@ Here are some of the tasks you can do on the **Models** pane:
 
 :::image type="content" source="../media/fine-tuning/studio-model-details.png" alt-text="Screenshot of the Models pane in Azure OpenAI Studio, with a custom model displayed." lightbox="../media/fine-tuning/studio-models-job-running.png":::
 
+## Safety evaluation GPT-4 fine-tuning - public preview
+
+[!INCLUDE [Safety evaluation](../includes/safety-evaluation.md)]
+
 ## Deploy a custom model
 
 When the fine-tuning job succeeds, you can deploy the custom model from the **Models** pane. You must deploy your custom model to make it available for use with completion calls.

articles/ai-services/openai/includes/safety-evaluation.md

@@ -0,0 +1,49 @@
+---
+title: 'Safety evaluation - GPT-4 fine tuning only public preview'
+titleSuffix: Azure OpenAI
+description: Learn about how to perform fine-tuning with GPT-4 models.
+manager: nitinme
+ms.service: azure-ai-openai
+ms.topic: include
+ms.date: 05/21/2024
+author: mrbullwinkle
+ms.author: mbullwin
+---
+
+GPT-4 is our most advanced model that can be fine-tuned to your needs. As with Azure OpenAI models generally, the advanced capabilities of fine-tuned models come with increased responsible AI challenges related to harmful content, manipulation, human-like behavior, privacy issues, and more. Learn more about risks, capabilities, and limitations in the [Overview of Responsible AI practices](/legal/cognitive-services/openai/overview?context=%2Fazure%2Fai-services%2Fopenai%2Fcontext%2Fcontext) and [Transparency Note](/legal/cognitive-services/openai/transparency-note?context=%2Fazure%2Fcognitive-services%2Fopenai%2Fcontext%2Fcontext&tabs=text). To help mitigate the risks associated with GPT-4 fine-tuned models, we have implemented additional evaluation steps to help detect and prevent harmful content in the training and outputs of fine-tuned models. These steps are grounded in the [Microsoft Responsible AI Standard](https://www.microsoft.com/ai/responsible-ai) and [Azure OpenAI Service content filtering](/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new).
+
+- Evaluations are conducted in dedicated, customer specific, private workspaces;
+- Evaluation endpoints are in the same geography as the Azure OpenAI resource;
+- Training data is not stored in connection with performing evaluations; only the final model assessment (deployable or not deployable) is persisted; and
+
+GPT-4 fine-tuned model evaluation filters are set to predefined thresholds and cannot be modified by customers; they aren't tied to any custom content filtering configuration you may have created.
+
+### Data evaluation
+
+Before training starts, your data is evaluated for potentially harmful content (violence, sexual, hate, and fairness, self-harm – see category definitions [here](/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new#risk-categories)). If harmful content is detected above the specified severity level, your training job will fail, and you'll receive a message informing you of the categories of failure.
+
+**Sample message:**
+
+```output
+The provided training data failed RAI checks for harm types: [hate_fairness, self_harm, violence]. Please fix the data and try again.
+```
+
+Your training data is evaluated automatically within your data import job as part of providing the fine-tuning capability.
+
+If the fine-tuning job fails due to the detection of harmful content in training data, you won't be charged.
+
+### Model evaluation
+
+After training completes but before the fine-tuned model is available for deployment, the resulting model is evaluated for potentially harmful responses using Azure’s built-in [risk and safety metrics](/azure/ai-studio/concepts/evaluation-metrics-built-in?tabs=warning#risk-and-safety-metrics). Using the same approach to testing that we use for the base large language models, our evaluation capability simulates a conversation with your fine-tuned model to assess the potential to output harmful content, again using specified harmful content [categories](/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new#risk-categories) (violence, sexual, hate, and fairness, self-harm).  
+
+If a model is found to generate output containing content detected as harmful at above an acceptable rate, you'll be informed that your model isn't available for deployment, with information about the specific categories of harm detected:
+
+**Sample Message**:
+
+```output
+This model is unable to be deployed. Model evaluation identified that this fine tuned model scores above acceptable thresholds for [Violence, Self Harm]. Please review your training data set and resubmit the job.
+```
+
+   :::image type="content" source="../media/fine-tuning/failure.png" alt-text="Screenshot of a failed fine-tuning job due to safety evaluation" lightbox="../media/fine-tuning/failure.png":::
+
+As with data evaluation, the model is evaluated automatically within your fine-tuning job as part of providing the fine-tuning capability. Only the resulting assessment (deployable or not deployable) is logged by the service. If deployment of the fine-tuned model fails due to the detection of harmful content in model outputs, you won't be charged for the training run.

articles/ai-services/openai/media/fine-tuning/failure.png

@@ -31,7 +31,7 @@ You can now create messages with the [assistant](.//assistants-reference-message
 * Support for `temperature`, `top_p`, `response_format` [parameters](./assistants-reference.md#create-an-assistant).
 * Streaming and polling support. You can use the helper functions in our Python SDK to create runs and stream responses. We have also added polling SDK helpers to share object status updates without the need for polling. 
 * Experiment with Logic Apps and Function Calling using Azure OpenAI Studio. Import your REST APIs implemented in Logic Apps as functions and the studio invokes the function (as a Logic Apps workflow) automatically based on the user prompt.
-* AutoGen by Microsoft Research provides a multi-agent conversation framework to enable convenient building of Large Language Model (LLM) workflows across a wide range of applications. Azure OpenAI assistants are now integrated into AutoGen via `GPTAssistantAgent`, a new experimental agent that lets you seamlessly add Assistants into AutoGen-based multi-agent workflows. This enables multiple Azure OpenAI assistants, that could be task or domain specialized, to collaborate and tackle complex tasks.
+* AutoGen by Microsoft Research provides a multi-agent conversation framework to enable convenient building of Large Language Model (LLM) workflows across a wide range of applications. Azure OpenAI assistants are now integrated into AutoGen via `GPTAssistantAgent`, a new experimental agent that lets you seamlessly add Assistants into AutoGen-based multi-agent workflows. This enables multiple Azure OpenAI assistants that could be task or domain specialized to collaborate and tackle complex tasks.
 * Support for fine-tuned `gpt-3.5-turbo-0125` [models](./concepts/models.md#assistants-preview) in the following regions:
     * East US 2
     * Sweden Central
@@ -56,6 +56,11 @@ Global deployments are available in the same Azure OpenAI resources as non-globa
 
 For more information, see the [deployment types guide](https://aka.ms/aoai/docs/deployment-types).
 
+### Fine-tuning updates
+
+- GPT-4 fine-tuning is [now available in public preview](./concepts/models.md#fine-tuning-models).
+- Added support for [seed](/azure/ai-services/openai/tutorials/fine-tune?tabs=python-new%2Ccommand-line#begin-fine-tuning), [events](/azure/ai-services/openai/tutorials/fine-tune?tabs=python-new%2Ccommand-line#list-fine-tuning-events), [full validation statistics](/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython-new&pivots=programming-language-python#analyze-your-customized-model), and [checkpoints](/azure/ai-services/openai/tutorials/fine-tune?tabs=python-new%2Ccommand-line#list-checkpoints) as part of the `2024-05-01-preview` API release.
+
 ### DALL-E and GPT-4 Turbo Vision GA configurable content filters
 
 Create custom content filters for your DALL-E 2 and 3 and GPT-4 Turbo with Vision GA (gpt-4-turbo-2024-04-09) deployments. [Content filtering](/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new#configurability-preview)