Merge pull request #283919 from lgayhardt/aistudiofloweval0824

Update flow-evaluate-sdk.md

Author: prmerger-automator[bot]

articles/ai-studio/how-to/develop/flow-evaluate-sdk.md

@@ -7,12 +7,11 @@ ms.service: azure-ai-studio
 ms.custom:
   - build-2024
 ms.topic: how-to
-ms.date: 5/21/2024
+ms.date: 08/07/2024
 ms.reviewer: dantaylo
 ms.author: eur
 author: eric-urban
 ---
-
 # Evaluate with the prompt flow SDK
 
 [!INCLUDE [Feature preview](~/reusable-content/ce-skilling/azure/includes/ai-studio/includes/feature-preview.md)]
@@ -51,7 +50,10 @@ Built-in composite evaluators are composed of individual evaluators.
 - `ContentSafetyEvaluator` combines all the safety evaluators for a single output of combined metrics for question and answer pairs
 - `ContentSafetyChatEvaluator` combines all the safety evaluators for a single output of combined metrics for chat messages following the OpenAI message protocol that can be found [here](https://platform.openai.com/docs/api-reference/messages/object#messages/object-content).
 
-### Required data input for built-in evaluators
+> [!TIP]
+> For more information about inputs and outputs, see the [Prompt flow Python reference documentation](https://microsoft.github.io/promptflow/reference/python-library-reference/promptflow-evals/promptflow.evals.evaluators.html).
+
+### Data requirements for built-in evaluators
 We require question and answer pairs in `.jsonl` format with the required inputs, and column mapping for evaluating datasets, as follows:
 
 | Evaluator         | `question`      | `answer`      | `context`       | `ground_truth`  |
@@ -160,9 +162,11 @@ chat_evaluator = ChatEvaluator(
 ```
 
 ## Custom evaluators
+
 Built-in evaluators are great out of the box to start evaluating your application's generations. However you might want to build your own code-based or prompt-based evaluator to cater to your specific evaluation needs.
 
 ### Code-based evaluators
+
 Sometimes a large language model isn't needed for certain evaluation metrics. This is when code-based evaluators can give you the flexibility to define metrics based on functions or callable class.  Given a simple Python class in an example `answer_length.py` that calculates the length of an answer:
 ```python
 class AnswerLengthEvaluator:
@@ -186,7 +190,41 @@ The result:
 ```JSON
 {"answer_length":27}
 ```
+#### Log your custom code-based evaluator to your AI Studio project
+```python
+# First we need to save evaluator into separate file in its own directory:
+def answer_len(answer):
+    return len(answer)
+
+# Note, we create temporary directory to store our python file
+target_dir_tmp = "flex_flow_tmp"
+os.makedirs(target_dir_tmp, exist_ok=True)
+lines = inspect.getsource(answer_len)
+with open(os.path.join("flex_flow_tmp", "answer.py"), "w") as fp:
+    fp.write(lines)
+
+from flex_flow_tmp.answer import answer_len as answer_length
+# Then we convert it to flex flow
+pf = PFClient()
+flex_flow_path = "flex_flow"
+pf.flows.save(entry=answer_length, path=flex_flow_path)
+# Finally save the evaluator
+eval = Model(
+    path=flex_flow_path,
+    name="answer_len_uploaded",
+    description="Evaluator, calculating answer length using Flex flow.",
+)
+flex_model = ml_client.evaluators.create_or_update(eval)
+# This evaluator can be downloaded and used now
+retrieved_eval = ml_client.evaluators.get("answer_len_uploaded", version=1)
+ml_client.evaluators.download("answer_len_uploaded", version=1, download_path=".")
+evaluator = load_flow(os.path.join("answer_len_uploaded", flex_flow_path))
+```
+
+After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](../evaluate-generative-ai-app.md#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.
+
 ### Prompt-based evaluators
+
 To build your own prompt-based large language model evaluator, you can create a custom evaluator based on a **Prompty** file. Prompty is a file with `.prompty` extension for developing prompt template. The Prompty asset is a markdown file with a modified front matter. The front matter is in YAML format that contains many metadata fields that define model configuration and expected inputs of the Prompty. Given an example `apology.prompty` file that looks like the following:
 
 ```markdown
@@ -248,13 +286,33 @@ apology_score = apology_eval(
 print(apology_score)
 ```
 
-Here is the result:
+Here's the result:
 ```JSON
 {"apology": 0}
 ```
+#### Log your custom prompt-based evaluator to your AI Studio project
+```python
+# Define the path to prompty file.
+prompty_path = os.path.join("apology-prompty", "apology.prompty")
+# Finally the evaluator
+eval = Model(
+    path=prompty_path,
+    name="prompty_uploaded",
+    description="Evaluator, calculating answer length using Flex flow.",
+)
+flex_model = ml_client.evaluators.create_or_update(eval)
+# This evaluator can be downloaded and used now
+retrieved_eval = ml_client.evaluators.get("prompty_uploaded", version=1)
+ml_client.evaluators.download("prompty_uploaded", version=1, download_path=".")
+evaluator = load_flow(os.path.join("prompty_uploaded", "apology.prompty"))
+```
+
+After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](../evaluate-generative-ai-app.md#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.
 
 ## Evaluate on test dataset using `evaluate()`
-After you spot-check your built-in or custom evaluators on a single row of data, you can combine multiple evaluators with the `evaluate()` API on an entire test dataset. In order to ensure the `evaluate()` can correctly parse the data, you must specify column mapping to map the column from the dataset to key words that are accepted by the evaluators. In this case, we specify the data mapping for `ground_truth`. 
+
+After you spot-check your built-in or custom evaluators on a single row of data, you can combine multiple evaluators with the `evaluate()` API on an entire test dataset. In order to ensure the `evaluate()` can correctly parse the data, you must specify column mapping to map the column from the dataset to key words that are accepted by the evaluators. In this case, we specify the data mapping for `ground_truth`.
+
 ```python
 from promptflow.evals.evaluate import evaluate
 
@@ -276,9 +334,11 @@ result = evaluate(
     output_path="./myevalresults.json"
 )
 ```
+
 > [!TIP]
 > Get the contents of the `result.studio_url` property for a link to view your logged evaluation results in Azure AI Studio.
 The evaluator outputs results in a dictionary which contains aggregate `metrics` and row-level data and metrics. An example of an output:
+
 ```python
 {'metrics': {'answer_length.value': 49.333333333333336,
              'relevance.gpt_relevance': 5.0},
@@ -311,9 +371,17 @@ The evaluator outputs results in a dictionary which contains aggregate `metrics`
            'outputs.answer_length.value': 66,
            'outputs.relevance.gpt_relevance': 5}],
  'traces': {}}
+
 ```
-### Supported data formats for `evaluate()`
-The `evaluate()` API only accepts data in the JSONLines format. For all built-in evaluators, except for `ChatEvaluator` or `ContentSafetyChatEvaluator`, `evaluate()` requires data in the following format with required input fields. See the [previous section on required data input for built-in evaluators](#required-data-input-for-built-in-evaluators).
+
+### Requirements for `evaluate()`
+
+The `evaluate()` API has a few requirements for the data format that it accepts and how it handles evaluator parameter key names so that the charts in your AI Studio evaluation results show up properly.
+
+#### Data format
+
+The `evaluate()` API only accepts data in the JSONLines format. For all built-in evaluators, except for `ChatEvaluator` or `ContentSafetyChatEvaluator`, `evaluate()` requires data in the following format with required input fields. See the [previous section on required data input for built-in evaluators](#data-requirements-for built-in evaluators).
+
 ```json
 {
   "question":"What is the capital of France?",
@@ -322,7 +390,9 @@ The `evaluate()` API only accepts data in the JSONLines format. For all built-in
   "ground_truth": "Paris"
 }
 ```
+
 For the composite evaluator class, `ChatEvaluator` and `ContentSafetyChatEvaluator`, we require an array of messages that adheres to OpenAI's messages protocol that can be found [here](https://platform.openai.com/docs/api-reference/messages/object#messages/object-content). The messages protocol contains a role-based list of messages with the following:
+
 - `content`: The content of that turn of the interaction between user and application or assistant.
 - `role`: Either the user or application/assistant.
 - `"citations"` (within `"context"`): Provides the documents and its ID as key value pairs from the retrieval-augmented generation model. 
@@ -360,7 +430,7 @@ To `evaluate()` with either the `ChatEvaluator` or `ContentSafetyChatEvaluator`,
 result = evaluate(
     data="data.jsonl",
     evaluators={
-        "chatevaluator": chat_evaluator
+        "chat": chat_evaluator
     },
     # column mapping for messages
     evaluator_config={
@@ -371,6 +441,40 @@ result = evaluate(
 )
 ```
 
+#### Evaluator parameter format
+
+When passing in your built-in evaluators, it's important to specify the right keyword mapping in the `evaluators` parameter list. The following is the keyword mapping required for the results from your built-in evaluators to show up in the UI when logged to Azure AI Studio.
+
+| Evaluator                    | keyword param         |
+|------------------------------|-----------------------|
+| `RelevanceEvaluator`         | "relevance"           |
+| `CoherenceEvaluator`         | "coherence"           |
+| `GroundednessEvaluator`      | "groundedness"        |
+| `FluencyEvaluator`           | "fluency"             |
+| `SimilarityEvaluator`        | "similarity"          |
+| `F1ScoreEvaluator`           | "f1_score"            |
+| `ViolenceEvaluator`          | "violence"            |
+| `SexualEvaluator`            | "sexual"              |
+| `SelfHarmEvaluator`          | "self_harm"           |
+| `HateUnfairnessEvaluator`    | "hate_unfairness"     |
+| `QAEvaluator`                | "qa"                  |
+| `ChatEvaluator`              | "chat"                |
+| `ContentSafetyEvaluator`     | "content_safety"      |
+| `ContentSafetyChatEvaluator` | "content_safety_chat" |
+
+Here's an example of setting the `evaluators` parameters:
+```python
+result = evaluate(
+    data="data.jsonl",
+    evaluators={
+        "sexual":sexual_evaluator
+        "self_harm":self_harm_evaluator
+        "hate_unfairness":hate_unfairness_evaluator
+        "violence":violence_evaluator
+    }
+)
+```
+
 ## Evaluate on a target
 
 If you have a list of queries that you'd like to run then evaluate, the `evaluate()` also supports a `target` parameter, which can send queries to an application to collect answers then run your evaluators on the resulting question and answers. 
@@ -399,4 +503,6 @@ result = evaluate(
 ## Related content
 
 - [Get started building a chat app using the prompt flow SDK](../../quickstarts/get-started-code.md)
-- [Work with projects in VS Code](vscode.md)
+- [Prompt flow Python reference documentation](https://microsoft.github.io/promptflow/reference/python-library-reference/promptflow-evals/promptflow.evals.evaluators.html)
+- [Learn more about the evaluation metrics](../../concepts/evaluation-metrics-built-in.md)
+- [View your evaluation results in Azure AI Studio](../../how-to/evaluate-flow-results.md)