Skip to content

Commit d1d082b

Browse files
lgayhardtlgayhardt
committed
Eval Github Action
1 parent a59b0ee commit d1d082b

File tree

5 files changed

+312
-0
lines changed

5 files changed

+312
-0
lines changed

‎articles/ai-foundry/how-to/evaluation-github-action.md

Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
---
2+
title: How to run an evaluation in GitHub Action
3+
titleSuffix: Azure AI Foundry
4+
description: How to evaluation in GitHub Action to streamline the evaluation process, allowing you to assess model performance and make informed decisions before deploying to production.
5+
manager: scottpolly
6+
ms.service: azure-ai-foundry
7+
ms.topic: how-to
8+
ms.date: 05/05/2025
9+
ms.reviewer: hanch
10+
ms.author: lagayhar
11+
author: lgayhardt
12+
---
13+
14+
# How to run an evaluation in GitHub Action
15+
16+
[!INCLUDE [feature-preview](../includes/feature-preview.md)]
17+
18+
This GitHub Action enables offline evaluation of AI models within your CI/CD pipelines. It's designed to streamline the evaluation process, allowing you to assess model performance and make informed decisions before deploying to production.
19+
20+
Offline evaluation involves testing AI models using test datasets to measure their performance on various quality and safety metrics such as fluency, coherence, and content safety. After you select a model in the [Azure AI Model Catalog](https://azure.microsoft.com/products/ai-model-catalog?msockid=1f44c87dd9fa6d1e257fdd6dd8406c42) or [GitHub Model marketplace](https://github.com/marketplace/models), offline pre-production evaluation is crucial for AI application validation during integration testing. This process allows developers to identify potential issues and make improvements before deploying the model or application to production, such as when updating agents.
21+
22+
[!INCLUDE [features](../includes/evaluation-github-action-azure-devops-features.md)]
23+
24+
- **Seamless Integration**: Easily integrate with existing GitHub workflows to run evaluation based on rules that you specify in your workflows (for examples, when changes are committed to agent versions, prompt templates, or feature flag configuration).
25+
- **Statistical Analysis**: Evaluation results include confidence intervals and test for statistical significance to determine if changes are meaningful and not due to random variation.
26+
- **Out-of-box operation metrics**: Automatically generates operational metrics for each evaluation run (client run duration, server run duration, completion tokens, and prompt tokens).
27+
28+
## Prerequisites
29+
30+
Two GitHub Actions are available for evaluating AI applications: **ai-agent-evals** and **genai-evals**.
31+
32+
- If your application is already leveraging AI Foundry agents, **ai-agent-evals** is well-suited as it offers a simplified setup process and direct integration with agent-based workflows. **genai-evals** is intended for evaluating generative AI models outside of the agent framework.
33+
- **genai-evals** is intended for evaluating generative AI models outside of the agent framework.
34+
35+
> [!NOTE]
36+
> The **ai-agent-evals** interface is more straightforward to configure. In contrast, **genai-evals** require customers to prepare structured evaluation input data. Although code samples are provided to facilitate this process, the overall setup might involve additional complexity.
37+
38+
## How to set up
39+
40+
### AI agent evaluations
41+
42+
### AI agent evaluations input
43+
44+
The input of ai-agent-evals includes:
45+
46+
**Required:**
47+
48+
- `azure-aiproject-connection-string`: The connection string for the Azure AI Project. This is used to connect to Azure OpenAI to simulate conversations with each agent, and to connect to the Azure AI evaluation SDK to perform the evaluation.
49+
- `deployment-name`: the deployed model name.
50+
- `data-path`: Path to the input data file containing the conversation starters. Each conversation starter is sent to each agent for a pairwise comparison of evaluation results.
51+
- `evaluators`: built-in evaluator names.
52+
- `data`: a set of conversation starters/queries and ground truth. Ground-truth is optional and only required for a subset of evaluators. (See which [evaluator requires ground-truth](./develop/evaluate-sd.md#data-requirements-for-built-in-evaluators))
53+
- Only single agent turn is supported.
54+
- `agent-ids`: a unique identifier for the agent and comma-separated list of agent IDs to evaluate.
55+
- When only one `agent-id` is specified, the evaluation results include the absolute values for each metric along with the corresponding confidence intervals.
56+
- When multiple a`gent-ids` are specified, the results include absolute values for each agent and a statistical comparison against the designated baseline agent ID.
57+
58+
**Optional:**
59+
60+
- `api-version`: the API version of deployed model.
61+
- `baseline-agent-id`: Agent ID of the baseline agent to compare against. By default, the first agent is used.
62+
- `evaluation-result-view`: Specifies the format of evaluation results. Defaults to "default" (boolean scores such as passing and defect rates) if omitted. Options are "default", "all-scores" (includes all evaluation scores), and "raw-scores-only" (non-boolean scores only).
63+
64+
Here's a sample of the dataset:
65+
66+
```JSON
67+
{
68+
"name": "MyTestData",
69+
"evaluators": [
70+
"GroundednessEvaluator",
71+
"RelevanceEvaluator",
72+
"ViolenceEvaluator",
73+
"HateUnfairnessEvaluator",
74+
"RougeScoreEvaluator"
75+
],
76+
"data": [
77+
{
78+
"query": "Tell me about Tokyo?",
79+
"ground_truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange."
80+
},
81+
{
82+
"query": "Where is Italy?",
83+
"ground_truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo."
84+
}
85+
]
86+
}
87+
88+
```
89+
90+
### AI agent evaluations workflow
91+
92+
To use the GitHub Action, add the GitHub Action to your CI/CD workflows and specify the trigger criteria (for example, on commit) and file paths to trigger your automated workflows.
93+
94+
> [!TIP]
95+
> To minimize costs, you should avoid running evaluation on every commit.
96+
97+
This example illustrates how Azure Agent AI Evaluation can be run when comparing two different agents with agent ID `my-agent-id1` and `my-agent-id2`.
98+
99+
```YAML
100+
name: "AI Agent Evaluation"
101+
102+
on:
103+
workflow_dispatch:
104+
push:
105+
branches:
106+
- main
107+
108+
permissions:
109+
id-token: write
110+
contents: read
111+
112+
jobs:
113+
run-action:
114+
runs-on: ubuntu-latest
115+
steps:
116+
- name: Checkout
117+
uses: actions/checkout@v4
118+
119+
- name: Azure login using Federated Credentials
120+
uses: azure/login@v2
121+
with:
122+
client-id: ${{ vars.AZURE_CLIENT_ID }}
123+
tenant-id: ${{ vars.AZURE_TENANT_ID }}
124+
subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
125+
126+
- name: Run Evaluation
127+
uses: microsoft/ai-agent-evals@v1-beta
128+
with:
129+
# Replace placeholders with values for your Azure AI Project
130+
azure-aiproject-connection-string: "<your-ai-project-conn-str>"
131+
deployment-name: "<your-deployment-name>"
132+
agent-ids: "<your-ai-agent-ids>"
133+
data-path: ${{ github.workspace }}/path/to/your/data-file
134+
135+
```
136+
137+
### AI agent evaluations outputs
138+
139+
Evaluation results are outputted to the summary section for each AI evaluation GitHub Action run under Actions in GitHub.com.
140+
141+
The result includes two main parts:
142+
143+
- The top section summarizes the overview of your AI agent variants. You can select it on the agent ID link, and it directs you to the agent setting page in AI Foundry portal. You can also select the link for Evaluation Results, and it directs you to AI Foundry portal to view individual result in detail.
144+
- The second section includes evaluation scores and comparison between different variants on statistical significance (for multiple agents) and confidence intervals (for single agent).
145+
146+
Multi agent evaluation result:
147+
148+
:::image type="content" source="../media/evaluations/github-action-multi-agent-result.png" alt-text="Screenshot of multi agent evaluation result in GitHub Action." lightbox="../media/evaluations/github-action-multi-agent-result.png":::
149+
150+
Single agent evaluation result:
151+
152+
:::image type="content" source="../media/evaluations/github-action-single-agent-output.png" alt-text="Screenshot of single agent evaluation result in GitHub Action." lightbox="../media/evaluations/github-action-single-agent-output.png":::
153+
154+
## GenAI evaluations
155+
156+
### GenAI evaluations input
157+
158+
The input of genai-evals includes (some of them are optional depending on the evaluator used):
159+
160+
Evaluation configuration file:
161+
162+
- `data`: a set of queries and ground truth. Ground-truth is optional and only required for a subset of evaluators. (See which [evaluator requires ground-truth](./develop/evaluate-sd.md#data-requirements-for-built-in-evaluators)).
163+
164+
Here's a sample of the dataset:
165+
166+
```json
167+
[
168+
{
169+
"query": "Tell me about Tokyo?",
170+
"ground-truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange."
171+
},
172+
{
173+
"query": "Where is Italy?",
174+
"ground-truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo."
175+
},
176+
177+
{
178+
"query": "Where is Papua New Guinea?",
179+
"ground-truth": "Papua New Guinea is an island country that lies in the south-western Pacific. It includes the eastern half of New Guinea and many small offshore islands. Its neighbours include Indonesia to the west, Australia to the south and Solomon Islands to the south-east."
180+
}
181+
]
182+
183+
```
184+
185+
- `evaluators`: built-in evaluator names.
186+
- `ai_model_configuration`: including type, `azure_endpoint`, `azure_deployment` and `api_version`.
187+
188+
### GenAI evaluations workflow
189+
190+
This example illustrates how Azure AI Evaluation can be run when changes are committed to specific files in your repo.
191+
192+
> [!NOTE]
193+
> Update `GENAI_EVALS_DATA_PATH` to point to the correct directory in your repo.
194+
195+
```yml
196+
name: Sample Evaluate Action
197+
on:
198+
workflow_call:
199+
workflow_dispatch:
200+
201+
permissions:
202+
id-token: write
203+
contents: read
204+
205+
jobs:
206+
evaluate:
207+
runs-on: ubuntu-latest
208+
env:
209+
GENAI_EVALS_CONFIG_PATH: ${{ github.workspace }}/evaluate-config.json
210+
GENAI_EVALS_DATA_PATH: ${{ github.workspace }}/.github/.test_files/eval-input.jsonl
211+
steps:
212+
- uses: actions/checkout@v4
213+
- uses: azure/login@v2
214+
with:
215+
client-id: ${{ secrets.OIDC_AZURE_CLIENT_ID }}
216+
tenant-id: ${{ secrets.OIDC_AZURE_TENANT_ID }}
217+
subscription-id: ${{ secrets.OIDC_AZURE_SUBSCRIPTION_ID }}
218+
- name: Write evaluate config
219+
run: |
220+
cat > ${{ env.GENAI_EVALS_CONFIG_PATH }} <<EOF
221+
{
222+
"data": "${{ env.GENAI_EVALS_DATA_PATH }}",
223+
"evaluators": {
224+
"coherence": "CoherenceEvaluator",
225+
"fluency": "FluencyEvaluator"
226+
},
227+
"ai_model_configuration": {
228+
"type": "azure_openai",
229+
"azure_endpoint": "${{ secrets.AZURE_OPENAI_ENDPOINT }}",
230+
"azure_deployment": "${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT }}",
231+
"api_key": "${{ secrets.AZURE_OPENAI_API_KEY }}",
232+
"api_version": "${{ secrets.AZURE_OPENAI_API_VERSION }}"
233+
}
234+
}
235+
EOF
236+
- name: Run AI Evaluation
237+
id: run-ai-evaluation
238+
uses: microsoft/genai-evals@main
239+
with:
240+
evaluate-configuration: ${{ env.GENAI_EVALS_CONFIG_PATH }}
241+
```
242+
243+
### GenAI evaluations outputs
244+
245+
Evaluation results are outputted to the summary section for each AI evaluation GitHub Action run under Actions in GitHub.com.
246+
247+
The results include three parts:
248+
249+
- Test Variants: a summary of variant names and system prompts.
250+
- Average scores: the average score of each evaluator for each variant.
251+
- Individual test scores: detailed result for each individual test case.
252+
253+
:::image type="content" source="../media/evaluations/github-action-output-results.png" alt-text="Screenshot of result output including test variants, average score, and individual test in GitHub Action." lightbox="../media/evaluations/github-action-output-results.png":::
254+
255+
## Related content
256+
257+
- [How to evaluate generative AI models and applications with Azure AI Foundry](./evaluate-generative-ai-app.md)
258+
- [How to view evaluation results in Azure AI Foundry portal](./evaluate-results.md)

‎articles/ai-foundry/includes/evaluation-github-action-azure-devops-features.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
title: Include file
3+
description: Include file
4+
author: lgayhardt
5+
ms.service: azure-ai-foundry
6+
ms.topic: include
7+
ms.date: 4/30/2025
8+
ms.author: lagayhar
9+
ms.custom: include file
10+
---
11+
12+
## Features
13+
14+
- **Automated Evaluation**: Integrate offline evaluation into your CI/CD workflows to automate the pre-production assessment of AI models.
15+
16+
- **Built-in Evaluators**: Leverage existing evaluators provided by the [Azure AI Evaluation SDK](../how-to/develop/evaluate-sdk.md).
17+
18+
The following evaluators are supported:
19+
20+
| Category | Evaluator class/Metrics | AI Agent evals | GenAI evals |
21+
|--|--|--|--|
22+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `GroundednessEvaluator` | Not Supported | Supported |
23+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `GroundednessProEvaluator` | Not Supported | Supported |
24+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `RetrievalEvaluator` | Not Supported | Supported |
25+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `RelevanceEvaluator` | Supported | Supported |
26+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `CoherenceEvaluator` | Supported | Supported |
27+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `FluencyEvaluator` | Supported | Supported |
28+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `SimilarityEvaluator` | Not Supported | Supported |
29+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `IntentResolutionEvaluator` | Supported | Supported |
30+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `TaskAdherenceEvaluator` | Supported | Supported |
31+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `ToolCallAccuracyEvaluator` | Supported | Not Supported |
32+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `ResponseCompletenessEvaluator` | Not Supported | Supported |
33+
| [Performance and quality (AI-assisted)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `DocumentRetrievalEvaluator` | Not Supported | Not Supported |
34+
| [Performance and quality (NLP)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `F1ScoreEvaluator` | Not Supported | Supported |
35+
| [Performance and quality (NLP)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `RougeScoreEvaluator` | Not Supported | Not Supported |
36+
| [Performance and quality (NLP)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `GleuScoreEvaluator` | Not Supported | Supported |
37+
| [Performance and quality (NLP)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `BleuScoreEvaluator ` | Not Supported | Supported |
38+
| [Performance and quality (NLP)](../how-to/develop/evaluate-sdk.md#performance-and-quality-evaluators) | `MeteorScoreEvaluator` | Not Supported | Supported |
39+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `ViolenceEvaluator` | Supported | Supported |
40+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `SexualEvaluator` | Supported | Supported |
41+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `SelfHarmEvaluator` | Supported | Supported |
42+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `HateUnfairnessEvaluator` | Supported | Supported |
43+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `IndirectAttackEvaluator` | Supported | Supported |
44+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `ProtectedMaterialEvaluator` | Supported | Supported |
45+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `CodeVulnerabilityEvaluator` | Supported | Supported |
46+
| [Risk and safety (AI-assisted)](../how-to/develop/evaluate-sdk.md#risk-and-safety-evaluators-preview) | `UngroundedAttributesEvaluator` | Not Supported | Supported |
47+
| [Composite](../how-to/develop/evaluate-sdk.md#composite-evaluators) | `QAEvaluator` | Not Supported | Supported |
48+
| [Composite](../how-to/develop/evaluate-sdk.md#composite-evaluators) | `ContentSafetyEvaluator` | Supported | Supported |
49+
| [Composite](../how-to/develop/evaluate-sdk.md#composite-evaluators) | `AgentOverallEvaluator` | Not Supported | Not Supported |
50+
| Operational metrics | Client run duration | Supported | Not Supported |
51+
| Operational metrics | Server run duration | Supported | Not Supported |
52+
| Operational metrics | Completion tokens | Supported | Not Supported |
53+
| Operational metrics | Prompt tokens | Supported | Not Supported |
54+
| [Custom evaluators](../how-to/develop/evaluate-sdk.md#custom-evaluators) | | Not Supported | Not Supported |

‎articles/ai-foundry/media/evaluations/github-action-multi-agent-result.png

438 KB
Loading

‎articles/ai-foundry/media/evaluations/github-action-output-results.png

474 KB
Loading

‎articles/ai-foundry/media/evaluations/github-action-single-agent-output.png

297 KB
Loading

0 commit comments

Comments
 (0)