Merge branch 'release-build-ai-foundry' into agent-metric-updates

Author: changliu2

articles/ai-foundry/.openpublishing.redirection.ai-studio.json

@@ -1152,6 +1152,21 @@
           "source_path_from_root": "/articles/ai-foundry/concepts/evaluation-metrics-built-in.md",
           "redirect_url": "/azure/ai-foundry/concepts/observability",
           "redirect_document_id": false
+        },
+        {
+          "source_path_from_root": "/articles/ai-foundry/concepts/trace.md",
+          "redirect_url": "/azure/ai-foundry/how-to/develop/trace-application",
+          "redirect_document_id": false
+        },
+        {
+          "source_path_from_root": "/articles/ai-foundry/how-to/develop/trace-local-sdk.md",
+          "redirect_url": "/azure/ai-foundry/how-to/develop/trace-application",
+          "redirect_document_id": true
+        },
+        {
+          "source_path_from_root": "/articles/ai-foundry/how-to/develop/visualize-traces.md",
+          "redirect_url": "/azure/ai-foundry/how-to/develop/trace-application#visualize-your-traces",
+          "redirect_document_id": false
         }
     ]
 }

articles/ai-foundry/at-foundry/ask-at-foundry.md

@@ -20,7 +20,7 @@ author: PatrickFarley
 [Azure AI Foundry](https://ai.azure.com) includes a content filtering system that works alongside core models and image generation models.
 
 > [!IMPORTANT]
-> The content filtering system isn't applied to prompts and completions processed by the Whisper model in Azure OpenAI Service. Learn more about the [Whisper model in Azure OpenAI](../../ai-services/openai/concepts/models.md).
+> The content filtering system isn't applied to prompts and completions processed by the Whisper model in Azure OpenAI in Azure AI Foundry Models. Learn more about the [Whisper model in Azure OpenAI](../../ai-services/openai/concepts/models.md).
 
 ## How it works 
 

articles/ai-foundry/concepts/content-filtering.md

@@ -19,15 +19,15 @@ The model catalog in Azure AI Foundry portal is the hub to discover and use a wi
 
 Deployment options vary depending on the model offering:
 
-* **Azure OpenAI models:** The latest OpenAI models that have enterprise features from Azure with flexible billing options.
-* **Models-as-a-Service models:** These models don't require compute quota from your subscription and are billed per token in a pay-as-you-go fashion. 
+* **Azure OpenAI in Azure AI Foundry Models:** The latest OpenAI models that have enterprise features from Azure with flexible billing options.
+* **Standard deployment:** These models don't require compute quota from your subscription and are billed per token in a pay-as-you-go fashion. 
 * **Open and custom models:** The model catalog offers access to a large variety of models across modalities, including models of open access. You can host open models in your own subscription with a managed infrastructure, virtual machines, and the number of instances for capacity management.
 
 Azure AI Foundry offers four different deployment options:
 
-|Name                           | Azure OpenAI service | Azure AI model inference | Serverless API | Managed compute |
+|Name                           | Azure OpenAI | Azure AI model inference | Standard deployment | Managed compute |
 |-------------------------------|----------------------|-------------------|----------------|-----------------|
-| Which models can be deployed? | [Azure OpenAI models](../../ai-services/openai/concepts/models.md)        | [Azure OpenAI models and Models-as-a-Service](../../ai-foundry/model-inference/concepts/models.md) | [Models-as-a-Service](../how-to/model-catalog-overview.md#content-safety-for-models-deployed-via-serverless-apis) | [Open and custom models](../how-to/model-catalog-overview.md#availability-of-models-for-deployment-as-managed-compute) |
+| Which models can be deployed? | [Azure OpenAI models](../../ai-services/openai/concepts/models.md)        | [Azure OpenAI models and Standard deployment](../../ai-foundry/model-inference/concepts/models.md) | [Standard deployment](../how-to/model-catalog-overview.md#content-safety-for-models-deployed-via-serverless-apis) | [Open and custom models](../how-to/model-catalog-overview.md#availability-of-models-for-deployment-as-managed-compute) |
 | Deployment resource           | Azure OpenAI resource | Azure AI services resource | AI project resource | AI project resource |
 | Requires Hubs/Projects        | No | No | Yes | Yes |
 | Data processing options       | Regional <br /> Data-zone  <br /> Global | Global | Regional | Regional |
@@ -37,7 +37,7 @@ Azure AI Foundry offers four different deployment options:
 | Key-less authentication       | Yes | Yes | No  | No  |
 | Best suited when              | You're planning to use only OpenAI models | You're planning to take advantage of the flagship models in Azure AI catalog, including OpenAI. | You're planning to use a single model from a specific provider (excluding OpenAI). | If you plan to use open models and you have enough compute quota available in your subscription. |
 | Billing bases                 | Token usage & [provisioned throughput units](../../ai-services/openai/concepts/provisioned-throughput.md)        | Token usage       | Token usage<sup>1</sup>      | Compute core hours<sup>2</sup> |
-| Deployment instructions       | [Deploy to Azure OpenAI Service](../how-to/deploy-models-openai.md) | [Deploy to Azure AI model inference](../model-inference/how-to/create-model-deployments.md) | [Deploy to Serverless API](../how-to/deploy-models-serverless.md) | [Deploy to Managed compute](../how-to/deploy-models-managed.md) |
+| Deployment instructions       | [Deploy to Azure OpenAI](../how-to/deploy-models-openai.md) | [Deploy to Azure AI model inference](../model-inference/how-to/create-model-deployments.md) | [Deploy to Standard deployment](../how-to/deploy-models-serverless.md) | [Deploy to Managed compute](../how-to/deploy-models-managed.md) |
 
 <sup>1</sup> A minimal endpoint infrastructure is billed per minute. You aren't billed for the infrastructure that hosts the model in pay-as-you-go. After you delete the endpoint, no further charges accrue.
 
@@ -54,11 +54,11 @@ Azure AI Foundry encourages you to explore various deployment options and choose
 
 * When you're looking to use a specific model:
 
-   * If you're interested in Azure OpenAI models, use the Azure OpenAI Service. This option is designed for Azure OpenAI models and offers a wide range of capabilities for them.
+   * If you're interested in Azure OpenAI models, use Azure OpenAI in Foundry Models. This option is designed for Azure OpenAI models and offers a wide range of capabilities for them.
 
-   * If you're interested in a particular model from Models-as-a-Service, and you don't expect to use any other type of model, use [Serverless API endpoints](../how-to/deploy-models-serverless.md). Serverless endpoints allow deployment of a single model under a unique set of endpoint URL and keys.
+   * If you're interested in a particular model from serverless pay per token offer, and you don't expect to use any other type of model, use [Standard deployment](../how-to/deploy-models-serverless.md). Standard deployments allow deployment of a single model under a unique set of endpoint URL and keys.
 
-* When your model isn't available in Models-as-a-Service and you have compute quota available in your subscription, use [Managed Compute](../how-to/deploy-models-managed.md), which supports deployment of open and custom models. It also allows a high level of customization of the deployment inference server, protocols, and detailed configuration.
+* When your model isn't available in standard deployment and you have compute quota available in your subscription, use [Managed Compute](../how-to/deploy-models-managed.md), which supports deployment of open and custom models. It also allows a high level of customization of the deployment inference server, protocols, and detailed configuration.
 
 
 ## Related content

articles/ai-foundry/concepts/deployments-overview.md

@@ -127,7 +127,7 @@ Customer-managed key encryption is configured via Azure portal in a similar way
 
 * The customer-managed key for encryption can only be updated to keys in the same Azure Key Vault instance.
 * After deployment, your [!INCLUDE [fdp](../includes/fdp-project-name.md)] can't switch from Microsoft-managed keys to customer-managed keys or vice versa.
-* Azure charges will continue to accrue during the soft delete retention period.
+* Azure charges for the AI Foundry resource will continue to accrue during the soft delete retention period. Charges for projects don't continue to accrue during the soft delete retention period.
 
 ::: zone-end
 

articles/ai-foundry/concepts/encryption-keys-portal.md

@@ -11,7 +11,9 @@ ms.author: lagayhar
 author: lgayhardt
 ---
 
-# Azure OpenAI Graders
+# Azure OpenAI Graders (preview)
+
+[!INCLUDE [feature-preview](../../includes/feature-preview.md)]
 
 The Azure OpenAI Graders are a new set of evaluation graders available in the Azure AI Foundry SDK, aimed at evaluating the performance of AI models and their outputs. These graders including  [Label grader](#label-grader), [String checker](#string-checker), [Text similarity](#text-similarity), and [General grader](#general-grader) can be run locally or remotely. Each grader serves a specific purpose in assessing different aspects of AI model/model outputs.
 
@@ -209,17 +211,17 @@ The grader also returns a metric indicating the overall dataset pass rate.
 
 ## General grader
 
-Advanced users have the capability to import or define a custom grader and integrate it into the Azure OpenAI general grader. This allows for evaluations to be performed based on specific areas of interest aside from the existing Azure OpenAI graders. Following is an example to import the OpenAI `EvalStringCheckGrader` and construct it to be ran as an Azure OpenAI general grader on Foundry SDK.
+Advanced users have the capability to import or define a custom grader and integrate it into the AOAI general grader. This allows for evaluations to be performed based on specific areas of interest aside from the existing AOAI graders. Following is an example to import the OpenAI `StringCheckGrader` and construct it to be ran as a AOAI general grader on Foundry SDK.
 
 ### Example
 
 ```python
-from openai.types.eval_string_check_grader import EvalStringCheckGrader
+from openai.types.graders import StringCheckGrader
 from azure.ai.evaluation import AzureOpenAIGrader
-
+ 
 # Define an string check grader config directly using the OAI SDK
 # Evaluation criteria: Pass if query column contains "Northwind"
-oai_string_check_grader = EvalStringCheckGrader(
+oai_string_check_grader = StringCheckGrader(
     input="{{item.query}}",
     name="contains hello",
     operation="like",

articles/ai-foundry/concepts/evaluation-evaluators/azure-openai-graders.md

@@ -0,0 +1,150 @@
+---
+title: Custom evaluators
+titleSuffix: Azure AI Foundry
+description: Learn how to create custom evaluators for your AI applications using code-based or prompt-based approaches.
+manager: scottpolly
+ms.service: azure-ai-foundry
+ms.topic: reference
+ms.date: 05/19/2025
+ms.reviewer: mithigpe
+ms.author: lagayhar
+author: lgayhardt
+---
+
+# Custom evaluators
+
+Built-in evaluators are great out of the box to start evaluating your application's generations. However you might want to build your own code-based or prompt-based evaluator to cater to your specific evaluation needs.
+
+## Code-based evaluators
+
+Sometimes a large language model isn't needed for certain evaluation metrics. This is when code-based evaluators can give you the flexibility to define metrics based on functions or callable class. You can build your own code-based evaluator, for example, by creating a simple Python class that calculates the length of an answer in `answer_length.py` under directory `answer_len/`:
+
+### Code-based evaluator example: Answer length
+
+```python
+class AnswerLengthEvaluator:
+    def __init__(self):
+        pass
+    # A class is made a callable my implementing the special method __call__
+    def __call__(self, *, answer: str, **kwargs):
+        return {"answer_length": len(answer)}
+```
+
+Then run the evaluator on a row of data by importing a callable class:
+
+```python
+from answer_len.answer_length import AnswerLengthEvaluator
+
+answer_length_evaluator = AnswerLengthEvaluator()
+answer_length = answer_length_evaluator(answer="What is the speed of light?")
+```
+
+### Code-based evaluator output: Answer length
+
+```python
+{"answer_length":27}
+```
+
+## Prompt-based evaluators
+
+To build your own prompt-based large language model evaluator or AI-assisted annotator, you can create a custom evaluator based on a **Prompty** file. Prompty is a file with `.prompty` extension for developing prompt template. The Prompty asset is a markdown file with a modified front matter. The front matter is in YAML format that contains many metadata fields that define model configuration and expected inputs of the Prompty. Let's create a custom evaluator `FriendlinessEvaluator` to measure friendliness of a response.
+
+### Prompt-based evaluator example: Friendliness evaluator
+
+First, create a `friendliness.prompty` file that describes the definition of the friendliness metric and its grading rubric:
+
+```markdown
+---
+name: Friendliness Evaluator
+description: Friendliness Evaluator to measure warmth and approachability of answers.
+model:
+  api: chat
+  parameters:
+    temperature: 0.1
+    response_format: { "type": "json" }
+inputs:
+  response:
+    type: string
+outputs:
+  score:
+    type: int
+  explanation:
+    type: string
+---
+
+system:
+Friendliness assesses the warmth and approachability of the answer. Rate the friendliness of the response between one to five stars using the following scale:
+
+One star: the answer is unfriendly or hostile
+
+Two stars: the answer is mostly unfriendly
+
+Three stars: the answer is neutral
+
+Four stars: the answer is mostly friendly
+
+Five stars: the answer is very friendly
+
+Please assign a rating between 1 and 5 based on the tone and demeanor of the response.
+
+**Example 1**
+generated_query: I just dont feel like helping you! Your questions are getting very annoying.
+output:
+{"score": 1, "reason": "The response is not warm and is resisting to be providing helpful information."}
+**Example 2**
+generated_query: I'm sorry this watch is not working for you. Very happy to assist you with a replacement.
+output:
+{"score": 5, "reason": "The response is warm and empathetic, offering a resolution with care."}
+
+
+**Here the actual conversation to be scored:**
+generated_query: {{response}}
+output:
+```
+
+Then create a class `FriendlinessEvaluator` to load the Prompty file and process the outputs with json format:
+
+```python
+import os
+import json
+import sys
+from promptflow.client import load_flow
+
+
+class FriendlinessEvaluator:
+    def __init__(self, model_config):
+        current_dir = os.path.dirname(__file__)
+        prompty_path = os.path.join(current_dir, "friendliness.prompty")
+        self._flow = load_flow(source=prompty_path, model={"configuration": model_config})
+
+    def __call__(self, *, response: str, **kwargs):
+        llm_response = self._flow(response=response)
+        try:
+            response = json.loads(llm_response)
+        except Exception as ex:
+            response = llm_response
+        return response
+```
+
+Now, you can create your own Prompty-based evaluator and run it on a row of data:
+
+```python
+from friendliness.friend import FriendlinessEvaluator
+
+friendliness_eval = FriendlinessEvaluator(model_config)
+
+friendliness_score = friendliness_eval(response="I will not apologize for my behavior!")
+```
+
+### Prompt-based evaluator output: Friendliness evaluator
+
+```python
+{
+    'score': 1, 
+    'reason': 'The response is hostile and unapologetic, lacking warmth or approachability.'
+}
+```
+
+## Related content
+
+- Learn [how to run batch evaluation on a dataset](../../how-to/develop/evaluate-sdk.md#local-evaluation-on-datasets) and [how to run batch evaluation on a target](../../how-to/develop/evaluate-sdk.md#local-evaluation-on-a-target).