You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Built-in evaluators are great out of the box to start evaluating your application's generations. However you might want to build your own code-based or prompt-based evaluator to cater to your specific evaluation needs.
166
167
167
168
### Code-based evaluators
169
+
168
170
Sometimes a large language model isn't needed for certain evaluation metrics. This is when code-based evaluators can give you the flexibility to define metrics based on functions or callable class. Given a simple Python class in an example `answer_length.py` that calculates the length of an answer:
After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/evaluate-generative-ai-app#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.
223
+
224
+
After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](../evaluate-generative-ai-ap.md#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.
225
+
222
226
### Prompt-based evaluators
227
+
223
228
To build your own prompt-based large language model evaluator, you can create a custom evaluator based on a **Prompty** file. Prompty is a file with `.prompty` extension for developing prompt template. The Prompty asset is a markdown file with a modified front matter. The front matter is in YAML format that contains many metadata fields that define model configuration and expected inputs of the Prompty. Given an example `apology.prompty` file that looks like the following:
After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/evaluate-generative-ai-app#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.
309
+
310
+
After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](../evaluate-generative-ai-app.md#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.
311
+
305
312
## Evaluate on test dataset using `evaluate()`
306
-
After you spot-check your built-in or custom evaluators on a single row of data, you can combine multiple evaluators with the `evaluate()` API on an entire test dataset. In order to ensure the `evaluate()` can correctly parse the data, you must specify column mapping to map the column from the dataset to key words that are accepted by the evaluators. In this case, we specify the data mapping for `ground_truth`.
313
+
314
+
After you spot-check your built-in or custom evaluators on a single row of data, you can combine multiple evaluators with the `evaluate()` API on an entire test dataset. In order to ensure the `evaluate()` can correctly parse the data, you must specify column mapping to map the column from the dataset to key words that are accepted by the evaluators. In this case, we specify the data mapping for `ground_truth`.
315
+
307
316
```python
308
317
from promptflow.evals.evaluate import evaluate
309
318
@@ -325,9 +334,11 @@ result = evaluate(
325
334
output_path="./myevalresults.json"
326
335
)
327
336
```
337
+
328
338
> [!TIP]
329
339
> Get the contents of the `result.studio_url` property for a link to view your logged evaluation results in Azure AI Studio.
330
340
The evaluator outputs results in a dictionary which contains aggregate `metrics` and row-level data and metrics. An example of an output:
@@ -360,11 +371,17 @@ The evaluator outputs results in a dictionary which contains aggregate `metrics`
360
371
'outputs.answer_length.value': 66,
361
372
'outputs.relevance.gpt_relevance': 5}],
362
373
'traces': {}}
374
+
363
375
```
376
+
364
377
### Requirements for `evaluate()`
378
+
365
379
The `evaluate()` API has a few requirements for the data format that it accepts and how it handles evaluator parameter key names so that the charts in your AI Studio evaluation results show up properly.
380
+
366
381
#### Data format
367
-
The `evaluate()` API only accepts data in the JSONLines format. For all built-in evaluators, except for `ChatEvaluator` or `ContentSafetyChatEvaluator`, `evaluate()` requires data in the following format with required input fields. See the [previous section on required data input for built-in evaluators](#required-data-input-for-built-in-evaluators).
382
+
383
+
The `evaluate()` API only accepts data in the JSONLines format. For all built-in evaluators, except for `ChatEvaluator` or `ContentSafetyChatEvaluator`, `evaluate()` requires data in the following format with required input fields. See the [previous section on required data input for built-in evaluators](#data-requirements-for built-in evaluators).
384
+
368
385
```json
369
386
{
370
387
"question":"What is the capital of France?",
@@ -373,7 +390,9 @@ The `evaluate()` API only accepts data in the JSONLines format. For all built-in
373
390
"ground_truth": "Paris"
374
391
}
375
392
```
393
+
376
394
For the composite evaluator class, `ChatEvaluator` and `ContentSafetyChatEvaluator`, we require an array of messages that adheres to OpenAI's messages protocol that can be found [here](https://platform.openai.com/docs/api-reference/messages/object#messages/object-content). The messages protocol contains a role-based list of messages with the following:
395
+
377
396
-`content`: The content of that turn of the interaction between user and application or assistant.
378
397
-`role`: Either the user or application/assistant.
379
398
-`"citations"` (within `"context"`): Provides the documents and its ID as key value pairs from the retrieval-augmented generation model.
@@ -421,8 +440,11 @@ result = evaluate(
421
440
}
422
441
)
423
442
```
443
+
424
444
#### Evaluator parameter format
425
-
When passing in your built-in evaluators, it is important to specify the right keyword mapping in the `evaluators` parameter list. The following is the keyword mapping required for the results from your built-in evaluators to show up in the UI when logged to Azure AI Studio.
445
+
446
+
When passing in your built-in evaluators, it's important to specify the right keyword mapping in the `evaluators` parameter list. The following is the keyword mapping required for the results from your built-in evaluators to show up in the UI when logged to Azure AI Studio.
Here's an example of setting the `evaluators` parameters:
443
466
```python
444
467
result = evaluate(
@@ -451,6 +474,7 @@ result = evaluate(
451
474
}
452
475
)
453
476
```
477
+
454
478
## Evaluate on a target
455
479
456
480
If you have a list of queries that you'd like to run then evaluate, the `evaluate()` also supports a `target` parameter, which can send queries to an application to collect answers then run your evaluators on the resulting question and answers.
0 commit comments