Merge pull request #5636 from MicrosoftDocs/main

6/19/2025 PM Publish

Author: Taojunshen

articles/ai-foundry/agents/how-to/tools/bing-code-samples.md

@@ -129,8 +129,32 @@ if run.status == "failed":
 messages = project_client.agents.messages.list(thread_id=thread.id)
 for message in messages:
     print(f"Role: {message.role}, Content: {message.content}")
+```
+
+## Optionally output the run steps used by the agent
+```python
+run_steps = project_client.agents.run_steps.list(thread_id=thread.id, run_id=run.id)
+for step in run_steps:
+    print(f"Step {step['id']} status: {step['status']}")
+
+    # Check if there are tool calls in the step details
+    step_details = step.get("step_details", {})
+    tool_calls = step_details.get("tool_calls", [])
+
+    if tool_calls:
+        print("  Tool calls:")
+        for call in tool_calls:
+            print(f"    Tool Call ID: {call.get('id')}")
+            print(f"    Type: {call.get('type')}")
+
+            function_details = call.get("function", {})
+            if function_details:
+                print(f"    Function name: {function_details.get('name')}")
+    print()  # add an extra newline between steps
+```
 
-# Delete the agent when done
+## Delete the agent when done
+```python
 project_client.agents.delete_agent(agent.id)
 print("Deleted agent")
 ```

articles/ai-foundry/concepts/model-benchmarks.md

@@ -20,14 +20,14 @@ author: lgayhardt
 
 Model leaderboards (preview) in Azure AI Foundry portal allow you to streamline the model selection process in the Azure AI Foundry [model catalog](../how-to/model-catalog-overview.md). The model leaderboards, backed by industry-standard benchmarks can help you to find the best model for your custom AI solution. From the model leaderboards section of the model catalog, you can [browse leaderboards](https://aka.ms/model-leaderboards) to compare available models as follows:
 
-- [Quality, cost, and performance leaderboards](../how-to/benchmark-model-in-catalog.md#access-model-leaderboards) to quickly identify the model leaders along a single metric (quality, cost, or throughput);
+- [Quality, safety, cost, and performance leaderboards](../how-to/benchmark-model-in-catalog.md#access-model-leaderboards) to quickly identify the model leaders along a single metric (quality, safety, cost, or throughput);
 - [Trade-off charts](../how-to/benchmark-model-in-catalog.md#compare-models-in-the-trade-off-charts) to see how models perform on one metric versus another, such as quality versus cost;
 - [Leaderboards by scenario](../how-to/benchmark-model-in-catalog.md#view-leaderboards-by-scenario) to find the best leaderboards that suite your scenario.
 
 Whenever you find a model to your liking, you can select it and zoom into the **Detailed benchmarking results** of the model within the model catalog. If satisfied with the model, you can deploy it, try it in the playground, or evaluate it on your data. The leaderboards support benchmarking across text language models (large language models (LLMs) and small language models (SLMs)) and embedding models.
 
 
-Model benchmarks assess LLMs and SLMs across the following categories: quality, performance, and cost. In addition, we assess the quality of embedding models using standard benchmarks. The leaderboards are updated regularly as better and more unsaturated benchmarks are onboarded, and as new models are added to the model catalog.
+Model benchmarks assess LLMs and SLMs across the following categories: quality, safety, cost, and throughput. In addition, we assess the quality of embedding models using standard benchmarks. The leaderboards are updated regularly as better and more unsaturated benchmarks are onboarded, and as new models are added to the model catalog.
 
 
 ## Quality benchmarks of language models
@@ -40,7 +40,7 @@ Azure AI assesses the quality of LLMs and SLMs using accuracy scores from standa
 
 Quality index is provided on a scale of zero to one. Higher values of quality index are better. The datasets included in quality index are: 
 
-| Dataset Name       | Leaderboard Category |
+| Dataset Name       | Leaderboard Scenario |
 |--------------------|----------------------|
 | arena_hard        | QA                   |
 | bigbench_hard     | Reasoning            |
@@ -62,6 +62,45 @@ See more details in accuracy scores:
 Accuracy scores are provided on a scale of zero to one. Higher values are better.
 
 
+## Safety benchmarks of language models
+
+To guide the selection of safety benchmarks for evaluation, we apply a structured filtering and validation process designed to ensure both relevance and rigor. A benchmark qualifies for onboarding if it addresses high-priority risks. For safety leaderboards, we look at different benchmarks that can be considered reliable enough to provide some signals on certain topics of interest as they relate to safety. We select [HarmBench](https://github.com/centerforaisafety/HarmBench) to proxy model safety, and organize scenario leaderboards as follows: 
+
+| Dataset Name       | Leaderboard Scenario |    Metric   | Interpretation   |
+|--------------------|----------------------|----------------------|----------------------|
+| HarmBench (standard)        | Standard harmful behaviors                   |  Attack Success Rate | Lower values means better robustness against attacks designed to illicit standard harmful content   |
+| HarmBench (contextual)    | Contextually harmful behaviors            | Attack Success Rate | Lower values means better robustness against attacks designed to illicit contextually harmful content |
+| HarmBench (copyright violations)             | Copyright violations                   | Attack Success Rate |  Lower values means better robustness against attacks designed to illicit copyright violations|
+| WMDP     | Knowledge in sensitive domains               | Accuracy |  Higher values denotes more knowledge in sensitive domains (cybersecurity, biosecurity, and chemical security) |
+| Toxigen            | Ability to detect toxic content            | F1 Score |  Higher values means better ability to detect toxic content |
+
+### Model harmful behaviors 
+The [HarmBench](https://github.com/centerforaisafety/HarmBench) benchmark measures model harmful behaviors and includes prompts to illicit harmful behavior from model. As it relates to safety, the benchmark covers 7 semantic categories of behavior: 
+- Cybercrime & Unauthorized Intrusion
+- Chemical & Biological Weapons/Drugs
+- Copyright Violations
+- Misinformation & Disinformation
+- Harassment & Bullying
+- Illegal Activities
+- General Harm
+  
+These 7 categories can be summarized into 3 functional categories
+- standard harmful behaviors
+- contextually harmful behaviors
+- copyright violations
+
+Each functional category is featured in a separate scenario leaderboard. We use direct prompts from HarmBench (no attacks) and HarmBench evaluators to calculate Attack Success Rate (ASR). Lower ASR values means safer models. We do not explore any attack strategy for evaluation, and model benchmarking is performed with Azure AI Content Safety Filter turned off. 
+
+
+### Model ability to detect toxic content
+[Toxigen](https://github.com/microsoft/TOXIGEN) is a large-scale machine-generated dataset for adversarial and implicit hate speech detection. It contains implicitly toxic and benign sentences mentioning 13 minority groups. We use the annotated samples from Toxigen for evaluation and calculate F1 scores to measure classification performance. Scoring higher on this dataset means that a model is better at detecting toxic content. Model benchmarking is performed with Azure AI Content Safety Filter turned off.
+
+### Model knowledge in sensitive domains
+The [Weapons of Mass Destruction Proxy](https://github.com/centerforaisafety/wmdp) (WMDP) benchmark measures model knowledge of in sensitive domains including biosecurity, cybersecurity, and chemical security. The leaderboard uses average accuracy scores across cybersecurity, biosecurity, and chemical security. A higher WMDP accuracy score denotes more knowledge of dangerous capabilities (worse behavior from a safety standpoint). Model benchmarking is performed with the default Azure AI Content Safety filters on. These safety filters detect and block content harm in violence, self-harm, sexual, hate and unfairness, but don't target categories in cybersecurity, biosecurity, and chemical security.
+
+### Limitations of safety benchmarks
+We understand and acknowledge that safety is a complex topic and has several dimensions. No single current open-source benchmarks can test or represent the full safety of a system in different scenarios. Additionally, most of these benchmarks suffer from saturation, or misalignment between benchmark design and the risk definition, can lack clear documentation on how the target risks are conceptualized and operationalized, making it difficult to assess whether the benchmark accurately captures the nuances of the risks. This limitation can lead to either overestimating or underestimating model performance in real-world safety scenarios. 
+
 ## Performance benchmarks of language models
 
 Performance metrics are calculated as an aggregate over 14 days, based on 24 trails (two requests per trail) sent daily with a one-hour interval between every trail. The following default parameters are used for each request to the model endpoint:

articles/ai-foundry/concepts/model-lifecycle-retirement.md

@@ -123,6 +123,22 @@ The following tables list the timelines for models that are on track for retirem
 | [Meta-Llama-3-8B-Instruct](https://ai.azure.com/explore/models/Meta-Llama-3-8B-Instruct/version/9/registry/azureml-meta) | February 28, 2025 | March 31, 2025 | June 30, 2025 | [Meta-Llama-3.1-8B-Instruct](https://ai.azure.com/explore/models/Meta-Llama-3.1-8B-Instruct/version/4/registry/azureml-meta) |
 | [Meta-Llama-3.1-70B-Instruct](https://ai.azure.com/explore/models/Meta-Llama-3.1-70B-Instruct/version/4/registry/azureml-meta) | February 28, 2025 | March 31, 2025 | June 30, 2025 | [Llama-3.3-70B-Instruct](https://ai.azure.com/explore/models/Llama-3.3-70B-Instruct/version/4/registry/azureml-meta) |
 
+
+#### Microsoft
+
+| Model | Legacy date (UTC) | Deprecation date (UTC) | Retirement date (UTC) | Suggested replacement model |
+|-------|-------------------|------------------------|-----------------------|-----------------------------|
+| [Phi-3-medium-4k-instruct](https://ai.azure.com/explore/models/Phi-3-medium-4k-instruct/version/6/registry/azureml)     | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4](https://ai.azure.com/explore/models/Phi-4/version/8/registry/azureml)                             |
+| [Phi-3-medium-128k-instruct](https://ai.azure.com/explore/models/Phi-3-medium-128k-instruct/version/7/registry/azureml) | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4](https://ai.azure.com/explore/models/Phi-4/version/8/registry/azureml)                             |
+| [Phi-3-mini-4k-instruct](https://ai.azure.com/explore/models/Phi-3-mini-4k-instruct/version/15/registry/azureml)        | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) |
+| [Phi-3-mini-128k-instruct](https://ai.azure.com/explore/models/Phi-3-mini-128k-instruct/version/13/registry/azureml)    | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) |
+| [Phi-3-small-8k-instruct](https://ai.azure.com/explore/models/Phi-3-small-8k-instruct/version/6/registry/azureml)       | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) |
+| [Phi-3-small-128k-instruct](https://ai.azure.com/explore/models/Phi-3-small-128k-instruct/version/5/registry/azureml)    | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) |
+| [Phi-3.5-mini-instruct](https://ai.azure.com/explore/models/Phi-3.5-mini-instruct/version/6/registry/azureml)           | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) |
+| [Phi-3.5-MoE-instruct](https://ai.azure.com/explore/models/Phi-3.5-MoE-instruct/version/5/registry/azureml)             | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) |
+| [Phi-3.5-vision-instruct](https://ai.azure.com/explore/models/Phi-3.5-vision-instruct/version/2/registry/azureml)       | June 9, 2025          | June 30, 2025              | August 30, 2025           | [Phi-4-mini-instruct](https://ai.azure.com/explore/models/Phi-4-mini-instruct/version/1/registry/azureml) |
+
+
 #### Mistral AI
 
 | Model | Legacy date (UTC) | Deprecation date (UTC) | Retirement date (UTC) | Suggested replacement model |

articles/ai-foundry/concepts/models-featured.md

@@ -340,7 +340,7 @@ See the [Nixtla model collection in Azure AI Foundry portal](https://ai.azure.co
 
 | Model  | Type | Capabilities |
 | ------ | ---- | ------------ |
-| [tsuzumi-7b](https://ai.azure.com/explore/models/Tsuzumi-7b/version/1/registry/azureml-nttdata) | [chat-completion](../model-inference/how-to/use-chat-completions.md?context=/azure/ai-foundry/context/context) | - **Input:** text (8,192 tokens) <br /> - **Output:** text (8,192 tokens) <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
+| [tsuzumi-7b](https://ai.azure.com/explore/models/Tsuzumi-7b/version/2/registry/azureml-nttdata) | [chat-completion](../model-inference/how-to/use-chat-completions.md?context=/azure/ai-foundry/context/context) | - **Input:** text (8,192 tokens) <br /> - **Output:** text (8,192 tokens) <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
 
 ## Stability AI
 

articles/ai-foundry/how-to/benchmark-model-in-catalog.md

@@ -20,8 +20,8 @@ author: lgayhardt
 
 In this article, you learn to streamline your model selection process in the Azure AI Foundry [model catalog](../how-to/model-catalog-overview.md) by comparing models in the model leaderboards (preview) available in Azure AI Foundry portal. This comparison can help you make informed decisions about which models meet the requirements for your particular use case or application. You can compare models by viewing the following leaderboards:
 
-- [Quality, cost, and performance leaderboards](#access-model-leaderboards) to quickly identify the model leaders along a single metric (quality, cost, or throughput);
-- [Trade-off charts](#compare-models-in-the-trade-off-charts) to see how models perform on one metric versus another, such as quality versus cost;
+- [Quality, safety, cost, and performance leaderboards](#access-model-leaderboards) to quickly identify the model leaders along a single criterion (quality, cost, or throughput);
+- [Trade-off charts](#compare-models-in-the-trade-off-charts) to see how models perform on one metric versus another, such as quality versus cost, among different selection criteria;
 - [Leaderboards by scenario](#view-leaderboards-by-scenario) to find the best leaderboards that suite your scenario.
 
 ## Prerequisites
@@ -42,7 +42,7 @@ In this article, you learn to streamline your model selection process in the Azu
 
     :::image type="content" source="../media/how-to/model-benchmarks/leaderboard-entry.png" alt-text="Screenshot showing the entry point from model catalog into model leaderboards." lightbox="../media/how-to/model-benchmarks/leaderboard-entry.png"::: 
 
-    The homepage displays leaderboard highlights for model selection criteria. Quality is the most common criterion for model selection, followed by cost and performance.
+    The homepage displays leaderboard highlights for model selection criteria. Quality is the most common criterion for model selection, followed by safety, cost, and performance.
 
     :::image type="content" source="../media/how-to/model-benchmarks/leaderboard-highlights.png" alt-text="Screenshot showing the highlighted leaderboards in quality, cost, and performance." lightbox="../media/how-to/model-benchmarks/leaderboard-highlights.png":::
 
@@ -52,7 +52,7 @@ In this article, you learn to streamline your model selection process in the Azu
 Trade-off charts allow you to compare models based on the criteria that you care more about. Suppose you care more about cost than quality and you discover that the highest quality model isn't the cheapest model, you might need to make trade-offs among quality, cost, and performance criteria. In the trade-off charts, you can compare how models perform along two metrics at a glance. 
 
 1. Select the **Models selected** dropdown menu to add or remove models from the trade-off chart.
-1. Select the **Quality vs. Throughput** tab and the **Throughput vs Cost** tab to view those charts for your selected models.
+1. Select the **Quality vs. Cost** tab and the **Quality vs Throughput** tab to view those charts for your selected models.
 1. Select **Compare between metrics** to access more detailed results for each model.
 
 :::image type="content" source="../media/how-to/model-benchmarks/leaderboard-trade-off.png" alt-text="Screenshot showing the trade-off charts in quality, cost, and performance." lightbox="../media/how-to/model-benchmarks/leaderboard-trade-off.png":::

articles/ai-foundry/how-to/prompt-flow-troubleshoot.md

@@ -67,16 +67,12 @@ You might encounter a 429 error from Azure OpenAI. This error means that you rea
 
         If you see the message `request canceled` in the logs, it might be because the OpenAI API call is taking too long and exceeding the time-out limit.
 
-        A network issue or a complex request that requires more processing time might cause the OpenAI time out. For more information, see [OpenAI API time out](https://help.openai.com/en/articles/6897186-timeout).
-
+        A network issue or a complex request that requires more processing time might cause the OpenAI time out.
+     
         Wait a few seconds and retry your request. This action usually resolves any network issues.
 
         If retrying doesn't work, check whether you're using a long context model, such as `gpt-4-32k`, and set a large value for `max_tokens`. If so, the behavior is expected because your prompt might generate a long response that takes longer than the interactive mode's upper threshold. In this situation, we recommend that you try `Bulk test` because this mode doesn't have a time-out setting.
 
-1. If you can't find anything in logs to indicate that it's a specific node issue:
-
-    - Contact the prompt flow team ([promptflow-eng](mailto:[email protected])) with the logs. We try to identify the root cause.
-
 ## Compute session failures that use a custom base image: Flow deployment-related issues
 
 ### How do I resolve an upstream request time-out issue?

articles/ai-foundry/model-inference/concepts/models.md

@@ -219,7 +219,7 @@ See [this model collection in Azure AI Foundry portal](https://ai.azure.com/expl
 
 | Model  | Type | Tier | Capabilities |
 | ------ | ---- | --- | ------------ |
-| [tsuzumi-7b](https://ai.azure.com/explore/models/Tsuzumi-7b/version/1/registry/azureml-nttdata) | chat-completion | Global standard | - **Input:** text (8,192 tokens) <br /> - **Output:** text (8,192 tokens) <br /> - **Languages:** `en` and `jp` <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
+| [tsuzumi-7b](https://ai.azure.com/explore/models/Tsuzumi-7b/version/2/registry/azureml-nttdata) | chat-completion | Global standard | - **Input:** text (8,192 tokens) <br /> - **Output:** text (8,192 tokens) <br /> - **Languages:** `en` and `jp` <br />  - **Tool calling:** No <br /> - **Response formats:** Text |
 
 ## Open and protected models
 

articles/ai-services/content-understanding/overview.md

@@ -5,7 +5,7 @@ description: Learn about Azure AI Content Understanding solutions, processes, wo
 author: laujan
 ms.author: lajanuar
 manager: nitinme
-ms.date: 03/06/2025
+ms.date: 06/19/2025
 ms.service: azure-ai-content-understanding
 ms.topic: overview
 ms.custom: