Merge pull request #46096 from dotnet/main

Merge main into live

Author: dotnet-policy-service[bot]

.openpublishing.redirection.ai.json

@@ -55,6 +55,10 @@
         {
             "source_path_from_root": "/docs/ai/quickstarts/quickstart-openai-summarize-text.md",
             "redirect_url": "/dotnet/ai/quickstarts/prompt-model"
+        },
+        {
+            "source_path_from_root": "/docs/ai/tutorials/llm-eval.md",
+            "redirect_url": "/dotnet/ai/quickstarts/evaluate-ai-response"
         }
     ]
 }

docs/ai/conceptual/evaluation-libraries.md

@@ -2,7 +2,7 @@
 title: The Microsoft.Extensions.AI.Evaluation libraries
 description: Learn about the Microsoft.Extensions.AI.Evaluation libraries, which simplify the process of evaluating the quality and accuracy of responses generated by AI models in .NET intelligent apps.
 ms.topic: concept-article
-ms.date: 03/18/2025
+ms.date: 05/09/2025
 ---
 # The Microsoft.Extensions.AI.Evaluation libraries (Preview)
 
@@ -11,7 +11,8 @@ The Microsoft.Extensions.AI.Evaluation libraries (currently in preview) simplify
 The evaluation libraries, which are built on top of the [Microsoft.Extensions.AI abstractions](../microsoft-extensions-ai.md), are composed of the following NuGet packages:
 
 - [📦 Microsoft.Extensions.AI.Evaluation](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation) – Defines the core abstractions and types for supporting evaluation.
-- [📦 Microsoft.Extensions.AI.Evaluation.Quality](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Quality) – Contains evaluators that assess the quality of LLM responses in an app according to metrics such as relevance, fluency, coherence, and truthfulness.
+- [📦 Microsoft.Extensions.AI.Evaluation.Quality](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Quality) – Contains evaluators that assess the quality of LLM responses in an app according to metrics such as relevance and completeness. These evaluators use the LLM directly to perform evaluations.
+- [📦 Microsoft.Extensions.AI.Evaluation.Safety](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Safety) – Contains evaluators, such as the `ProtectedMaterialEvaluator` and `ContentHarmEvaluator`, that use the [Azure AI Foundry](/azure/ai-foundry/) Evaluation service to perform evaluations.
 - [📦 Microsoft.Extensions.AI.Evaluation.Reporting](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Reporting) – Contains support for caching LLM responses, storing the results of evaluations, and generating reports from that data.
 - [📦 Microsoft.Extensions.AI.Evaluation.Reporting.Azure](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Reporting.Azure) - Supports the reporting library with an implementation for caching LLM responses and storing the evaluation results in an [Azure Storage](/azure/storage/common/storage-introduction) container.
 - [📦 Microsoft.Extensions.AI.Evaluation.Console](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Console) – A command-line tool for generating reports and managing evaluation data.
@@ -24,13 +25,25 @@ The libraries are designed to integrate smoothly with existing .NET apps, allowi
 
 The evaluation libraries were built in collaboration with data science researchers from Microsoft and GitHub, and were tested on popular Microsoft Copilot experiences. The following table shows the built-in evaluators.
 
-| Metric                             | Description                                  | Evaluator type |
-|------------------------------------|----------------------------------------------|----------------|
-| Relevance, truth, and completeness | How effectively a response addresses a query | <xref:Microsoft.Extensions.AI.Evaluation.Quality.RelevanceTruthAndCompletenessEvaluator> |
-| Fluency                            | Grammatical accuracy, vocabulary range, sentence complexity, and overall readability| <xref:Microsoft.Extensions.AI.Evaluation.Quality.FluencyEvaluator> |
-| Coherence                          | The logical and orderly presentation of ideas | <xref:Microsoft.Extensions.AI.Evaluation.Quality.CoherenceEvaluator> |
-| Equivalence                        | The similarity between the generated text and its ground truth with respect to a query | <xref:Microsoft.Extensions.AI.Evaluation.Quality.EquivalenceEvaluator> |
-| Groundedness                       | How well a generated response aligns with the given context | <xref:Microsoft.Extensions.AI.Evaluation.Quality.GroundednessEvaluator> |
+| Metric       | Description                                            | Evaluator type |
+|--------------|--------------------------------------------------------|----------------|
+| Relevance    | Evaluates how relevant a response is to a query        | `RelevanceEvaluator` <!-- <xref:Microsoft.Extensions.AI.Evaluation.Quality.RelevanceEvaluator> --> |
+| Completeness | Evaluates how comprehensive and accurate a response is | `CompletenessEvaluator` <!-- <xref:Microsoft.Extensions.AI.Evaluation.Quality.CompletenessEvaluator> --> |
+| Retrieval    | Evaluates performance in retrieving information for additional context | `RetrievalEvaluator` <!-- <xref:Microsoft.Extensions.AI.Evaluation.Quality.RetrievalEvaluator> --> |
+| Fluency      | Evaluates grammatical accuracy, vocabulary range, sentence complexity, and overall readability| <xref:Microsoft.Extensions.AI.Evaluation.Quality.FluencyEvaluator> |
+| Coherence    | Evaluates the logical and orderly presentation of ideas | <xref:Microsoft.Extensions.AI.Evaluation.Quality.CoherenceEvaluator> |
+| Equivalence  | Evaluates the similarity between the generated text and its ground truth with respect to a query | <xref:Microsoft.Extensions.AI.Evaluation.Quality.EquivalenceEvaluator> |
+| Groundedness | Evaluates how well a generated response aligns with the given context | <xref:Microsoft.Extensions.AI.Evaluation.Quality.GroundednessEvaluator><br />`GroundednessProEvaluator` |
+| Protected material | Evaluates response for the presence of protected material | `ProtectedMaterialEvaluator` |
+| Ungrounded human attributes | Evaluates a response for the presence of content that indicates ungrounded inference of human attributes | `UngroundedAttributesEvaluator` |
+| Hate content | Evaluates a response for the presence of content that's hateful or unfair | `HateAndUnfairnessEvaluator`† |
+| Self-harm content | Evaluates a response for the presence of content that indicates self harm | `SelfHarmEvaluator`† |
+| Violent content | Evaluates a response for the presence of violent content | `ViolenceEvaluator`† |
+| Sexual content | Evaluates a response for the presence of sexual content | `SexualEvaluator`† |
+| Code vulnerability content | Evaluates a response for the presence of vulnerable code | `CodeVulnerabilityEvaluator` |
+| Indirect attack content | Evaluates a response for the presence of indirect attacks, such as manipulated content, intrusion, and information gathering | `IndirectAttackEvaluator` |
+
+† In addition, the `ContentHarmEvaluator` provides single-shot evaluation for the four metrics supported by `HateAndUnfairnessEvaluator`, `SelfHarmEvaluator`, `ViolenceEvaluator`, and `SexualEvaluator`.
 
 You can also customize to add your own evaluations by implementing the <xref:Microsoft.Extensions.AI.Evaluation.IEvaluator> interface or extending the base classes such as <xref:Microsoft.Extensions.AI.Evaluation.Quality.ChatConversationEvaluator> and <xref:Microsoft.Extensions.AI.Evaluation.Quality.SingleNumericMetricEvaluator>.
 

docs/ai/snippets/microsoft-extensions-ai/ConsoleAI.AddMessages/Program.cs

@@ -29,6 +29,7 @@
         client.GetStreamingResponseAsync(history))
     {
         Console.Write(update);
+        updates.Add(update);
     }
     Console.WriteLine();
 

docs/ai/toc.yml

@@ -85,8 +85,6 @@ items:
     href: quickstarts/evaluate-ai-response.md
   - name: "Tutorial: Evaluate a response with response caching and reporting"
     href: tutorials/evaluate-with-reporting.md
-  - name: "Tutorial: Evaluate LLM prompt completions"
-    href: tutorials/llm-eval.md
 - name: Resources
   items:
   - name: API reference

docs/ai/tutorials/evaluate-with-reporting.md

@@ -1,14 +1,14 @@
 ---
 title: Tutorial - Evaluate a model's response
 description: Create an MSTest app and add a custom evaluator to evaluate the AI chat response of a language model, and learn how to use the caching and reporting features of Microsoft.Extensions.AI.Evaluation.
-ms.date: 03/14/2025
+ms.date: 05/09/2025
 ms.topic: tutorial
 ms.custom: devx-track-dotnet-ai
 ---
 
 # Tutorial: Evaluate a model's response with response caching and reporting
 
-In this tutorial, you create an MSTest app to evaluate the chat response of an OpenAI model. The test app uses the [Microsoft.Extensions.AI.Evaluation](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation) libraries to perform the evaluations, cache the model responses, and create reports. The tutorial uses both a [built-in evaluator](xref:Microsoft.Extensions.AI.Evaluation.Quality.RelevanceTruthAndCompletenessEvaluator) and a custom evaluator.
+In this tutorial, you create an MSTest app to evaluate the chat response of an OpenAI model. The test app uses the [Microsoft.Extensions.AI.Evaluation](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation) libraries to perform the evaluations, cache the model responses, and create reports. The tutorial uses both built-in and custom evaluators.
 
 ## Prerequisites
 
@@ -25,32 +25,32 @@ Complete the following steps to create an MSTest project that connects to the `g
 
 1. In a terminal window, navigate to the directory where you want to create your app, and create a new MSTest app with the `dotnet new` command:
 
-    ```dotnetcli
-    dotnet new mstest -o TestAIWithReporting
-    ```
+   ```dotnetcli
+   dotnet new mstest -o TestAIWithReporting
+   ```
 
 1. Navigate to the `TestAIWithReporting` directory, and add the necessary packages to your app:
 
-    ```dotnetcli
-    dotnet add package Azure.AI.OpenAI
-    dotnet add package Azure.Identity
-    dotnet add package Microsoft.Extensions.AI.Abstractions --prerelease
-    dotnet add package Microsoft.Extensions.AI.Evaluation --prerelease
-    dotnet add package Microsoft.Extensions.AI.Evaluation.Quality --prerelease
-    dotnet add package Microsoft.Extensions.AI.Evaluation.Reporting --prerelease
-    dotnet add package Microsoft.Extensions.AI.OpenAI --prerelease
-    dotnet add package Microsoft.Extensions.Configuration
-    dotnet add package Microsoft.Extensions.Configuration.UserSecrets
-    ```
+   ```dotnetcli
+   dotnet add package Azure.AI.OpenAI
+   dotnet add package Azure.Identity
+   dotnet add package Microsoft.Extensions.AI.Abstractions --prerelease
+   dotnet add package Microsoft.Extensions.AI.Evaluation --prerelease
+   dotnet add package Microsoft.Extensions.AI.Evaluation.Quality --prerelease
+   dotnet add package Microsoft.Extensions.AI.Evaluation.Reporting --prerelease
+   dotnet add package Microsoft.Extensions.AI.OpenAI --prerelease
+   dotnet add package Microsoft.Extensions.Configuration
+   dotnet add package Microsoft.Extensions.Configuration.UserSecrets
+   ```
 
 1. Run the following commands to add [app secrets](/aspnet/core/security/app-secrets) for your Azure OpenAI endpoint, model name, and tenant ID:
 
-    ```bash
-    dotnet user-secrets init
-    dotnet user-secrets set AZURE_OPENAI_ENDPOINT <your-azure-openai-endpoint>
-    dotnet user-secrets set AZURE_OPENAI_GPT_NAME gpt-4o
-    dotnet user-secrets set AZURE_TENANT_ID <your-tenant-id>
-    ```
+   ```bash
+   dotnet user-secrets init
+   dotnet user-secrets set AZURE_OPENAI_ENDPOINT <your-azure-openai-endpoint>
+   dotnet user-secrets set AZURE_OPENAI_GPT_NAME gpt-4o
+   dotnet user-secrets set AZURE_TENANT_ID <your-tenant-id>
+   ```
 
    (Depending on your environment, the tenant ID might not be needed. In that case, remove it from the code that instantiates the <xref:Azure.Identity.DefaultAzureCredential>.)