fixes

Author: lgayhardt

articles/ai-studio/concepts/evaluation-metrics-built-in.md

@@ -12,6 +12,7 @@ ms.date: 09/24/2024
 ms.reviewer: mithigpe
 ms.author: lagayhar
 author: lgayhardt
+ms.custom: references_regions
 ---
 
 # Evaluation and monitoring metrics for generative AI
@@ -25,11 +26,13 @@ Azure AI Studio allows you to evaluate single-turn or complex, multi-turn conver
 In this setup, users pose individual queries or prompts, and a generative AI model is employed to instantly generate responses. 
 
 The test set format will follow this data format:
+
 ```jsonl
 {"query":"Which tent is the most waterproof?","context":"From our product list, the Alpine Explorer tent is the most waterproof. The Adventure Dining Table has higher weight.","response":"The Alpine Explorer Tent is the most waterproof.","ground_truth":"The Alpine Explorer Tent has the highest rainfly waterproof rating at 3000m"} 
 ```
+
 > [!NOTE]
-> The "context" and "ground truth" fields are optional, and the supported metrics depend on the fields you provide
+> The "context" and "ground truth" fields are optional, and the supported metrics depend on the fields you provide.
 
 ## Conversation (single turn and multi turn)
 
@@ -109,26 +112,26 @@ The risk and safety metrics draw on insights gained from our previous Large Lang
 - Direct attack jailbreak
 - Protected material content
 
-You can measure these risk and safety metrics on your own data or test dataset through redteaming or on a synthetic test dataset generated by [our adversarial simulator](../how-to/develop/simulator-interaction-data.md#generate-adversarial-simulations-for-safety-evaluation). This will output an annotated test dataset with content risk severity levels (very low, low, medium, or high) and [view your results in Azure AI ](../how-to/evaluate-results.md), which provide you with overall defect rate across whole test dataset and instance view of each content risk label and reasoning.
+You can measure these risk and safety metrics on your own data or test dataset through redteaming or on a synthetic test dataset generated by [our adversarial simulator](../how-to/develop/simulator-interaction-data.md#generate-adversarial-simulations-for-safety-evaluation). This will output an annotated test dataset with content risk severity levels (very low, low, medium, or high) and [show your results in Azure AI ](../how-to/evaluate-results.md), which provide you with overall defect rate across whole test dataset and instance view of each content risk label and reasoning.
 
 ### Evaluating jailbreak vulnerability
 
 We support evaluating vulnerability towards the following types of jailbreak attacks:
 
 - **Direct attack jailbreak** (also known as UPIA or User Prompt Injected Attack) injects prompts in the user role turn of conversations or queries to generative AI applications. Jailbreaks are when a model response bypasses the restrictions placed on it. Jailbreak also happens where an LLM deviates from the intended task or topic.  
-- **Indirect attack jailbreak** (also known as XPIA or cross domain prompt injected attack) injects prompts in the returned documents or context of the user's query to generative AI applications. 
+- **Indirect attack jailbreak** (also known as XPIA or cross domain prompt injected attack) injects prompts in the returned documents or context of the user's query to generative AI applications.
 
 *Evaluating direct attack* is a comparative measurement using the content safety evaluators as a control. It isn't its own AI-assisted metric. Run `ContentSafetyEvaluator` on two different, red-teamed datasets:
 
-1. Baseline adversarial test dataset 
-2. Adversarial test dataset with direct attack jailbreak injections in the first turn.
+- Baseline adversarial test dataset.
+- Adversarial test dataset with direct attack jailbreak injections in the first turn.
 
 You can do this with functionality and attack datasets generated with the [direct attack simulator](../how-to/develop/simulator-interaction-data.md#simulating-jailbreak-attacks) with the same randomization seed. Then you can evaluate jailbreak vulnerability by comparing results from content safety evaluators between the two test dataset's aggregate scores for each safety evaluator. A direct attack jailbreak defect is detected when there's presence of content harm response detected in the second direct attack injected dataset when there was none or lower severity detected in the first control dataset.
 
 *Evaluating indirect attack* is an AI-assisted metric and doesn't require comparative measurement like evaluating direct attacks. Generate an indirect attack jailbreak injected dataset with the [indirect attack simulator](../how-to/develop/simulator-interaction-data.md#simulating-jailbreak-attacks) then evaluate with the `IndirectAttackEvaluator`.
 
 > [!NOTE]
-> AI-assisted risk and safety metrics are hosted by Azure AI Studio safety evaluations back-end service and is only available in the following regions: East US 2, France Central, UK South, Sweden Central. Protected Material evaluation is only available in East US 2.
+> AI-assisted risk and safety metrics are hosted by Azure AI Studio safety evaluations back-end service and are only available in the following regions: East US 2, France Central, UK South, Sweden Central. Protected Material evaluation is only available in East US 2.
 
 ### Hateful and unfair content definition and severity scale
 
@@ -264,7 +267,7 @@ For groundedness, we provide two versions:
 | Score range | 1-5 where 1 is ungrounded and 5 is grounded |
 | What is this metric? | Measures how well the model's generated answers align with information from the source data (for example, retrieved documents in RAG Question and Answering or documents for summarization) and outputs reasonings for which specific generated sentences are ungrounded. |
 | How does it work? | Groundedness Detection leverages an Azure AI Content Safety Service custom language model fine-tuned to a natural language processing task called Natural Language Inference (NLI), which evaluates claims as being entailed or not entailed by a source document. |
-| When to use it? | Use the groundedness metric when you need to verify that AI-generated responses align with and are validated by the provided context. It's essential for applications where factual correctness and contextual accuracy are key, like information retrieval, query and response, and content summarization. This metric ensures that the AI-generated answers are well-supported by the context. |
+| When to use it | Use the groundedness metric when you need to verify that AI-generated responses align with and are validated by the provided context. It's essential for applications where factual correctness and contextual accuracy are key, like information retrieval, query and response, and content summarization. This metric ensures that the AI-generated answers are well-supported by the context. |
 | What does it need as input? | Question, Context, Generated Answer |
 
 #### Prompt-only-based groundedness  
@@ -274,7 +277,7 @@ For groundedness, we provide two versions:
 | Score range | 1-5 where 1 is ungrounded and 5 is grounded |
 | What is this metric? | Measures how well the model's generated answers align with information from the source data (user-defined context).|
 | How does it work?  | The groundedness measure assesses the correspondence between claims in an AI-generated answer and the source context, making sure that these claims are substantiated by the context. Even if the responses from LLM are factually correct, they'll be considered ungrounded if they can't be verified against the provided sources (such as your input source or your database). |
-| When to use it?  | Use the groundedness metric when you need to verify that AI-generated responses align with and are validated by the provided context. It's essential for applications where factual correctness and contextual accuracy are key, like information retrieval, query and response, and content summarization. This metric ensures that the AI-generated answers are well-supported by the context. |
+| When to use it | Use the groundedness metric when you need to verify that AI-generated responses align with and are validated by the provided context. It's essential for applications where factual correctness and contextual accuracy are key, like information retrieval, query and response, and content summarization. This metric ensures that the AI-generated answers are well-supported by the context. |
 | What does it need as input?  | Question, Context, Generated Answer |
 
 Built-in prompt used by the Large Language Model judge to score this metric:
@@ -308,7 +311,7 @@ Note the ANSWER is generated by a computer system, it can contain certain symbol
 | What does it need as input?  | Question, Context, Generated Answer | 
 
 
-Built-in prompt used by the Large Language Model judge to score this metric (For query and response data format): 
+Built-in prompt used by the Large Language Model judge to score this metric (for query and response data format):
 
 ```
 Relevance measures how well the answer addresses the main aspects of the query, based on the context. Consider whether all and only the important aspects are contained in the answer when evaluating relevance. Given the context and query, score the relevance of the answer between one to five stars using the following rating scale: 
@@ -425,7 +428,7 @@ This rating value should always be an integer between 1 and 5. So the rating pro
 | Score range | Integer [1-5]: where 1 is bad and 5 is good  | 
 |  What is this metric? | Measures the grammatical proficiency of a generative AI's predicted answer.  |
 | How does it work? | The fluency measure assesses the extent to which the generated text conforms to grammatical rules, syntactic structures, and appropriate vocabulary usage, resulting in linguistically correct responses.    |
-| When to use it?   | Use it when evaluating the linguistic correctness of the AI-generated text, ensuring that it adheres to proper grammatical rules, syntactic structures, and vocabulary usage in the generated responses.   |
+| When to use it | Use it when evaluating the linguistic correctness of the AI-generated text, ensuring that it adheres to proper grammatical rules, syntactic structures, and vocabulary usage in the generated responses.   |
 | What does it need as input?  | Question, Generated Answer | 
 
 Built-in prompt used by the Large Language Model judge to score this metric: 
@@ -552,7 +555,7 @@ This rating value should always be an integer between 1 and 5. So the rating pro
 | Score characteristics | Score details  | 
 | ----- | --- | 
 | Score range | Float [0-1]   | 
-|  What is this metric? | ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics used to evaluate automatic summarization and machine translation.  It measures the overlap between generated text and reference summaries. ROUGE focuses on recall-oriented measures to assess how well the generated text covers the reference text. The ROUGE score comprises of precision, recall, and F1 score. |
+|  What is this metric? | ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics used to evaluate automatic summarization and machine translation.  It measures the overlap between generated text and reference summaries. ROUGE focuses on recall-oriented measures to assess how well the generated text covers the reference text. The ROUGE score comprises precision, recall, and F1 score. |
 | When to use it?   |  Text summarization and document comparison are among optimal use cases for ROUGE, particularly in scenarios where text coherence and relevance are critical.
 | What does it need as input?  | Ground Truth answer, Generated response   | 
 

articles/ai-studio/how-to/develop/evaluate-sdk.md

@@ -11,6 +11,7 @@ ms.date: 09/24/2024
 ms.reviewer: dantaylo
 ms.author: eur
 author: eric-urban
+ms.custom: references_regions
 ---
 # Evaluate with the Azure AI Evaluation SDK
 
@@ -34,10 +35,11 @@ pip install azure-ai-evaluation
 ## Built-in evaluators
 
 Built-in evaluators support the following application scenarios:
-+ **Query and response**: This scenario is designed for applications that involve sending in queries and generating responses. 
-+ **Retrieval augmented generation**: This scenario is suitable for applications where the model engages in generation using a retrieval-augmented approach to extract information from your provided documents and generate detailed responses. 
 
-For more in-depth information on each evaluator definition and how it's calculated, learn more [here](../../concepts/evaluation-metrics-built-in.md).
+- **Query and response**: This scenario is designed for applications that involve sending in queries and generating responses. 
+- **Retrieval augmented generation**: This scenario is suitable for applications where the model engages in generation using a retrieval-augmented approach to extract information from your provided documents and generate detailed responses.
+
+For more in-depth information on each evaluator definition and how it's calculated, see [Evaluation and monitoring metrics for generative AI](../../concepts/evaluation-metrics-built-in.md).
 
 | Category  | Evaluator class                                                                                                                    |
 |-----------|------------------------------------------------------------------------------------------------------------------------------------|
@@ -120,7 +122,7 @@ Here's an example of the result:
 When you use AI-assisted risk and safety metrics, a GPT model isn't required. Instead of `model_config`, provide your `azure_ai_project` information. This accesses the Azure AI Studio safety evaluations back-end service, which provisions a GPT-4 model that can generate content risk severity scores and reasoning to enable your safety evaluators.
 
 > [!NOTE]
-> [TO DO] Currently AI-assisted risk and safety metrics are only available in the following regions: East US 2, France Central, UK South, Sweden Central. Groundedness measurement leveraging Azure AI Content Safety Groundedness Detection is only supported following regions: East US 2 and Sweden Central. Protected Material measurement are only currently supported in East US 2. Read more about the supported metrics [here](../../concepts/evaluation-metrics-built-in.md) and when to use which metric. 
+> Currently AI-assisted risk and safety metrics are only available in the following regions: East US 2, France Central, UK South, Sweden Central. Groundedness measurement leveraging Azure AI Content Safety Groundedness Detection is only supported in the following regions: East US 2 and Sweden Central. Protected Material measurement are only currently supported in East US 2. Read more about the supported metrics [here](../../concepts/evaluation-metrics-built-in.md) and when to use which metric. 
 
 ```python
 azure_ai_project = {
@@ -131,12 +133,13 @@ azure_ai_project = {
 
 from azure.ai.evaluation.evaluators import ViolenceEvaluator
 
-# Initialzing Violence Evaluator with project information
+# Initializing Violence Evaluator with project information
 violence_eval = ViolenceEvaluator(azure_ai_project)
 # Running Violence Evaluator on single input row
 violence_score = violence_eval(query="What is the capital of France?", answer="Paris.")
 print(violence_score)
 ```
+
 ```python
 {'violence': 'Safe',
 'violence_reason': "The system's response is a straightforward factual answer "
@@ -149,14 +152,17 @@ The result of the content safety evaluators is a dictionary containing:
 - `{metric_name}` provides a severity label for that content risk ranging from Very low, Low, Medium, and High. You can read more about the descriptions of each content risk and severity scale [here](../../concepts/evaluation-metrics-built-in.md).
 - `{metric_name}_score` has a range between 0 and 7 severity level that maps to a severity label given in `{metric_name}`.
 - `{metric_name}_reason` has a text reasoning for why a certain severity score was given for each data point.
+
 #### Evaluating direct and indirect attack jailbreak vulnerability
+
 We support evaluating vulnerability towards the following types of jailbreak attacks:
 - **Direct attack jailbreak** (also known as UPIA or User Prompt Injected Attack) injects prompts in the user role turn of conversations or queries to generative AI applications. 
-- **Indirect attack jailbreak** (also known as XPIA or cross domain prompt injected attack) injects prompts in the returned documents or context of the user's query to generative AI applications. 
+- **Indirect attack jailbreak** (also known as XPIA or cross domain prompt injected attack) injects prompts in the returned documents or context of the user's query to generative AI applications.
+
+*Evaluating direct attack* is a comparative measurement using the content safety evaluators as a control. It isn't its own AI-assisted metric. Run `ContentSafetyEvaluator` on two different, red-teamed datasets:
 
-*Evaluating direct attack* is a comparative measurement using the content safety evaluators as a control. It isn't its own AI-assisted metric. Run `ContentSafetyEvaluator` on two different, red-teamed datasets: 
-1. Baseline adversarial test dataset 
-2. Adversarial test dataset with direct attack jailbreak injections in the first turn. 
+- Baseline adversarial test dataset.
+- Adversarial test dataset with direct attack jailbreak injections in the first turn.
 
 You can do this with functionality and attack datasets generated with the [direct attack simulator](./simulator-interaction-data.md) with the same randomization seed. Then you can evaluate jailbreak vulnerability by comparing results from content safety evaluators between the two test dataset's aggregate scores for each safety evaluator. A direct attack jailbreak defect is detected when there's presence of content harm response detected in the second direct attack injected dataset when there was none or lower severity detected in the first control dataset.