Merge pull request #7230 from MicrosoftDocs/main

Auto Publish – main to live - 2025-09-23 17:13 UTC

Author: learn-build-service-prod[bot]

articles/ai-foundry/concepts/evaluation-evaluators/azure-openai-graders.md

@@ -5,23 +5,30 @@ description: Learn about Azure OpenAI Graders for evaluating AI model outputs, i
 author: lgayhardt
 ms.author: lagayhar
 ms.reviewer: mithigpe
-ms.date: 07/16/2025
+ms.date: 09/23/2025
 ms.service: azure-ai-foundry
 ms.topic: reference
 ms.custom:
   - build-aifnd
   - build-2025
 ---
 
-# Azure OpenAI Graders (preview)
+# Azure OpenAI graders (preview)
 
-[!INCLUDE [feature-preview](../../includes/feature-preview.md)]
+Azure OpenAI graders are a new set of evaluation tools in the Azure AI Foundry SDK that evaluate the performance of AI models and their outputs. These graders include:
+
+- [Label grader](#label-grader)
+- [String checker](#string-checker)
+- [Text similarity](#text-similarity)
+- [Python grader](#python-grader) 
 
-The Azure OpenAI Graders are a new set of evaluation graders available in the Azure AI Foundry SDK, aimed at evaluating the performance of AI models and their outputs. These graders including  [Label grader](#label-grader), [String checker](#string-checker), [Text similarity](#text-similarity), and [General grader](#general-grader) can be run locally or remotely. Each grader serves a specific purpose in assessing different aspects of AI model/model outputs.
+You can run graders locally or remotely. Each grader assesses specific aspects of AI models and their outputs.
+
+[!INCLUDE [feature-preview](../../includes/feature-preview.md)]
 
 ## Model configuration for AI-assisted grader
 
-For reference in the following code snippet, the AI-assisted grader uses a model configuration as follows:
+The following code snippet shows the model configuration used by the AI-assisted grader:
 
 ```python
 import os
@@ -33,7 +40,7 @@ model_config = AzureOpenAIModelConfiguration(
     azure_endpoint=os.environ.get("AZURE_ENDPOINT"),
     api_key=os.environ.get("AZURE_API_KEY"),
     azure_deployment=os.environ.get("AZURE_DEPLOYMENT_NAME"),
-    api_version=os.environ.get("AZURE_API_VERSION"),
+    api_version=os.environ.get("AZURE_API_VERSION")
 )
 ```
 
@@ -42,11 +49,11 @@ model_config = AzureOpenAIModelConfiguration(
 `AzureOpenAILabelGrader` uses your custom prompt to instruct a model to classify outputs based on labels you define. It returns structured results with explanations for why each label was chosen.
 
 > [!NOTE]
-> We recommend using Azure OpenAI GPT o3-mini for best results.
+> We recommend using Azure OpenAI o3-mini for the best results.
 
-Here's an example `data.jsonl` that is used in the following code snippets:
+Here's an example of `data.jsonl` used in the following code snippets:
 
-```jsonl
+```json
 [
     {
         "query": "What is the importance of choosing the right provider in getting the most value out of your health insurance plan?",
@@ -83,11 +90,11 @@ from azure.ai.evaluation import AzureOpenAILabelGrader, evaluate
 
 data_file_name="data.jsonl"
 
-#  Evaluation criteria: Determine if the response column contains texts that are "too short", "just right", or "too long" and pass if it is "just right"
+#  Evaluation criteria: Determine if the response column contains text that is "too short," "just right," or "too long," and pass if it is "just right."
 label_grader = AzureOpenAILabelGrader(
     model_config=model_config,
-    input=[{"content": "{{item.response}}", "role": "user"}
-           {"content":"Any text including space that's more than 600 characters are too long, less than 500 characters are too short; 500 to 600 characters are just right.", "role":"user", "type": "message"}],
+    input=[{"content": "{{item.response}}", "role": "user"},
+           {"content": "Any text including space that's more than 600 characters is too long, less than 500 characters is too short; 500 to 600 characters is just right.", "role": "user", "type": "message"}],
     labels=["too short", "just right", "too long"],
     passing_labels=["just right"],
     model="gpt-4o",
@@ -104,7 +111,7 @@ label_grader_evaluation = evaluate(
 
 ### Label grader output
 
-For each of the sets of sample data contained in the data file, an evaluation result of `True` or `False` is returned signifying if the output matches with the passing label defined. The `score` is `1.0` for `True` cases while `score` is `0.0` for `False` cases. The reason for why the model provided the label for the data can be found in `content` under `outputs.label.sample`.
+For each set of sample data in the data file, an evaluation result of `True` or `False` is returned, signifying if the output matches the defined passing label. The `score` is `1.0` for `True` cases, and `0.0` for `False` cases. The reason the model provided the label for the data is in `content` under `outputs.label.sample`.
 
 ```python
 'outputs.label.sample':
@@ -114,12 +121,11 @@ For each of the sets of sample data contained in the data file, an evaluation re
       'content': '{"steps":[{"description":"Calculate the number of characters in the user\'s input including spaces.","conclusion":"The provided text contains 575 characters."},{"description":"Evaluate if the character count falls within the given ranges (greater than 600 too long, less than 500 too short, 500 to 600 just right).","conclusion":"The character count falls between 500 and 600, categorized as \'just right.\'"}],"result":"just right"}'}],
 ...
 ...
-'outputs.label.label_result': 'pass',
 'outputs.label.passed': True,
 'outputs.label.score': 1.0
 ```
 
-Aside from individual data evaluation results, the grader also returns a metric indicating the overall dataset pass rate.
+In addition to individual data evaluation results, the grader returns a metric indicating the overall dataset pass rate.
 
 ```python
 'metrics': {'label.pass_rate': 0.2}, #1/5 in this case
@@ -139,7 +145,7 @@ string_grader = AzureOpenAIStringCheckGrader(
     model_config=model_config,
     input="{{item.query}}",
     name="starts with what is",
-    operation="like", # "eq" for equal, "ne" for not equal, "like" for contain, "ilike" for case insensitive contain
+    operation="like", # "eq" for equal, "ne" for not equal, "like" for contains, "ilike" for case-insensitive contains
     reference="What is",
 )
 
@@ -153,23 +159,22 @@ string_grader_evaluation = evaluate(
 
 ### String checker output
 
-For each of the sets of sample data contained in the data file, an evaluation result of `True` or `False` is returned signifying if the input text matches with pattern matching rules defined. The `score` is `1.0` for `True` cases while `score` is `0.0` for `False` cases.
+For each set of sample data in the data file, an evaluation result of `True` or `False` is returned, indicating whether the input text matches the defined pattern-matching rules. The `score` is `1.0` for `True` cases while `score` is `0.0` for `False` cases.
 
 ```python
-'outputs.string.string_result': 'pass',
 'outputs.string.passed': True,
 'outputs.string.score': 1.0
 ```
 
 The grader also returns a metric indicating the overall dataset pass rate.
 
 ```python
-'metrics': {'string.pass_rate': 0.4}, #2/5 in this case
+'metrics': {'string.pass_rate': 0.4}, # 2/5 in this case
 ```
 
 ## Text similarity
 
-Evaluates how closely input text matches a reference value using similarity metrics like`fuzzy_match`, `BLEU`, `ROUGE`, or `METEOR`. Useful for assessing text quality or semantic closeness.
+Evaluates how closely input text matches a reference value using similarity metrics like `fuzzy_match`, `BLEU`, `ROUGE`, or `METEOR`. This is useful for assessing text quality or semantic closeness.
 
 ### Text similarity example
 
@@ -197,69 +202,83 @@ sim_grader_evaluation
 
 ### Text similarity output
 
-For each set of sample data contained in the data file, a numerical similarity score is generated. This score, ranging from 0 to 1, indicates the degree of similarity, with higher scores representing greater similarity. Additionally, an evaluation result of `True` or `False` is returned, signifying whether the similarity score meets or exceeds the specified threshold based on the evaluation metric defined in the grader.
+For each set of sample data in the data file, a numerical similarity score is generated. This score ranges from 0 to 1 and indicates the degree of similarity, with higher scores representing greater similarity. An evaluation result of `True` or `False` is also returned, signifying whether the similarity score meets or exceeds the specified threshold based on the evaluation metric defined in the grader.
 
 ```python
-'outputs.similarity.similarity_result': 'pass',
 'outputs.similarity.passed': True,
 'outputs.similarity.score': 0.6117136659436009
 ```
 
 The grader also returns a metric indicating the overall dataset pass rate.
 
 ```python
-'metrics': {'similarity.pass_rate': 0.4}, #2/5 in this case
+'metrics': {'similarity.pass_rate': 0.4}, # 2 out of 5 in this case
 ```
 
-## General grader
+## Python Grader
 
-Advanced users have the capability to import or define a custom grader and integrate it into the AOAI general grader. This allows for evaluations to be performed based on specific areas of interest aside from the existing AOAI graders. Following is an example to import the OpenAI `StringCheckGrader` and construct it to be ran as a AOAI general grader on Foundry SDK.
+Advanced users can create or import custom Python grader functions and integrate them into the Azure OpenAI Python grader. This enables evaluations tailored to specific areas of interest beyond the capabilities of the existing Azure OpenAI graders. The following example demonstrates how to import a custom similarity grader function and configure it to run as an Azure OpenAI Python grader using the Azure AI Foundry SDK.
 
 ### Example
 
 ```python
-from openai.types.graders import StringCheckGrader
-from azure.ai.evaluation import AzureOpenAIGrader
+from azure.ai.evaluation import AzureOpenAIPythonGrader
 
-# Define an string check grader config directly using the OAI SDK
-# Evaluation criteria: Pass if query column contains "Northwind"
-oai_string_check_grader = StringCheckGrader(
-    input="{{item.query}}",
-    name="contains hello",
-    operation="like",
-    reference="Northwind",
-    type="string_check"
-)
-# Plug that into the general grader
-general_grader = AzureOpenAIGrader(
-    model_config=model_config,
-    grader_config=oai_string_check_grader
+python_similarity_grader = AzureOpenAIPythonGrader(
+    model_config=model_config_aoai,
+    name="custom_similarity",
+    image_tag="2025-05-08",
+    pass_threshold=0.3,
+    source="""
+    def grade(sample, item) -> float:
+     \"\"\"
+     Custom similarity grader using word overlap.
+     Note: All data is in the 'item' parameter.
+     \"\"\"
+     # Extract from item, not sample!
+     response = item.get("response", "") if isinstance(item, dict) else ""
+     ground_truth = item.get("ground_truth", "") if isinstance(item, dict) else ""
+    
+     # Simple word overlap similarity
+     response_words = set(response.lower().split())
+     truth_words = set(ground_truth.lower().split())
+    
+     if not truth_words:
+     return 0.0
+    
+     overlap = response_words.intersection(truth_words)
+     similarity = len(overlap) / len(truth_words)
+    
+     return min(1.0, similarity)
+""",
 )
+
+file_name = "eval_this.jsonl"
 evaluation = evaluate(
     data=data_file_name,
     evaluators={
-        "general": general_grader,
+        "custom_similarity": python_similarity_grader,
     },
+    #azure_ai_project=azure_ai_project,
 )
 evaluation
 ```
 
 ### Output
 
-For each set of sample data contained in the data file, general grader returns a numerical score that is a 0-1 float and a higher score is better. Given a numerical threshold defined as part of the custom grader, we also output `True` if the score >= threshold, or `False` otherwise.
+For each set of sample data in the data file, the Python grader returns a numerical score based on the defined function. Given a numerical threshold defined as part of the custom grader, we also output `True` if the score >= threshold, or `False` otherwise.
 
 For example:
 
 ```python
-'outputs.general.general_result': 'pass',
-'outputs.general.passed': True,
-'outputs.general.score': 1.0
+"outputs.custom_similarity.passed": false,
+"outputs.custom_similarity.score": 0.0
 ```
 
 Aside from individual data evaluation results, the grader also returns a metric indicating the overall dataset pass rate.
 
 ```python
-'metrics': {'general.pass_rate': 0.4}, #2/5 in this case
+'metrics': {'custom_similarity.pass_rate': 0.0}, #0/5 in this case
 ```
 
 ## Related content

articles/ai-foundry/foundry-local/concepts/foundry-local-architecture.md

@@ -93,6 +93,12 @@ The hardware abstraction layer ensures that Foundry Local can run on various dev
 - **multiple _execution providers_**, such as NVIDIA CUDA, AMD, Qualcomm, Intel.
 - **multiple _device types_**, such as CPU, GPU, NPU.
 
+> [!NOTE]
+> For Intel NPU support on Windows, you need to install the [Intel NPU driver](https://www.intel.com/content/www/us/en/download/794734/intel-npu-driver-windows.html) to enable hardware acceleration.
+
+> [!NOTE]
+> For Qualcomm NPU support, you need to install the [Qualcomm NPU driver](https://softwarecenter.qualcomm.com/catalog/item/QHND). If you encounter the error `Qnn error code 5005: "Failed to load from EpContext model. qnn_backend_manager."`, this typically indicates an outdated driver or NPU resource conflicts. Try rebooting to clear NPU resource conflicts, especially after using Windows Copilot+ features.
+
 ### Developer experiences
 
 The Foundry Local architecture is designed to provide a seamless developer experience, enabling easy integration and interaction with AI models.
@@ -123,19 +129,22 @@ Foundry Local supports integration with various SDKs in most languages, such as
 The AI Toolkit for Visual Studio Code provides a user-friendly interface for developers to interact with Foundry Local. It allows users to run models, manage the local cache, and visualize results directly within the IDE.
 
 **Features**:
-  - Model management: Download, load, and run models from within the IDE.
-  - Interactive console: Send requests and view responses in real-time.
-  - Visualization tools: Graphical representation of model performance and results.
+
+- Model management: Download, load, and run models from within the IDE.
+- Interactive console: Send requests and view responses in real-time.
+- Visualization tools: Graphical representation of model performance and results.
 
 **Prerequisites:**
-  - You have installed [Foundry Local](../get-started.md) and have a model service running.
-  - You have installed the [AI Toolkit for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio) extension.
- 
+
+- You have installed [Foundry Local](../get-started.md) and have a model service running.
+- You have installed the [AI Toolkit for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio) extension.
+
 **Connect Foundry Local model to AI Toolkit:**
-  1. **Add model in AI Toolkit**: Open AI Toolkit from the activity bar of Visual Studio Code. In the 'My Models' panel, click the 'Add model for remote interface' button and then select 'Add a custom model' from the dropdown menu.
-  2. **Enter the chat compatible endpoint URL**: Enter `http://localhost:PORT/v1/chat/completions` where PORT is replaced with the port number of your Foundry Local service endpoint. You can see the port of your locally running service using the CLI command `foundry service status`. Foundry Local dynamically assigns a port, so it might not always the same.
-  3. **Provide model name**: Enter the exact model name you which to use from Foundry Local, for example `phi-3.5-mini`. You can list all previously downloaded and locally cached models using the CLI command `foundry cache list` or use `foundry model list` to see all available models for local use. You’ll also be asked to enter a display name, which is only for your own local use, so to avoid confusion it’s recommended to enter the same name as the exact model name.
-  4. **Authentication**: If your local setup doesn't require authentication *(which is the default for a Foundry Local setup)*, you can leave the authentication headers field blank and press Enter.
+
+1. **Add model in AI Toolkit**: Open AI Toolkit from the activity bar of Visual Studio Code. In the 'My Models' panel, select the 'Add model for remote interface' button and then select 'Add a custom model' from the dropdown menu.
+2. **Enter the chat compatible endpoint URL**: Enter `http://localhost:PORT/v1/chat/completions` where PORT is replaced with the port number of your Foundry Local service endpoint. You can see the port of your locally running service using the CLI command `foundry service status`. Foundry Local dynamically assigns a port, so it might not always be the same.
+3. **Provide model name**: Enter the exact model name you which to use from Foundry Local, for example `phi-3.5-mini`. You can list all previously downloaded and locally cached models using the CLI command `foundry cache list` or use `foundry model list` to see all available models for local use. You’ll also be asked to enter a display name, which is only for your own local use, so to avoid confusion it’s recommended to enter the same name as the exact model name.
+4. **Authentication**: If your local setup doesn't require authentication _(which is the default for a Foundry Local setup)_, you can leave the authentication headers field blank and press Enter.
 
 After completing these steps, your Foundry Local model will appear in the 'My Models' list in AI Toolkit and is ready to be used by right-clicking on your model and select 'Load in Playground'.