@@ -5,7 +5,7 @@ description: How to run evaluation in Azure DevOps which enables offline evaluat
55manager : scottpolly
66ms.service : azure-ai-foundry
77ms.topic : how-to
8- ms.date : 05/19 /2025
8+ ms.date : 07/25 /2025
99ms.reviewer : hanch
1010ms.author : lagayhar
1111author : lgayhardt
@@ -21,8 +21,7 @@ Similar to the [Azure AI evaluation in GitHub Actions](evaluation-github-action.
2121
2222## Prerequisites
2323
24- [ !INCLUDE [ hub-only-prereq] ( ../includes/hub-only-prereq.md )]
25-
24+ - Foundry project or Hubs based project. To learn more, see [ Create a project] ( create-projects.md ) .
2625- Install Azure AI evaluation extension.
2726 - Go to [ Azure DevOps Marketplace] ( https://marketplace.visualstudio.com/azuredevops ) .
2827 - Search for Azure AI evaluation and install the extension into your Azure DevOps organization.
@@ -65,7 +64,51 @@ Similar to the [Azure AI evaluation in GitHub Actions](evaluation-github-action.
6564
6665A sample YAML file:
6766
68- ```yml
67+ # [Foundry project](#tab/foundry-project)
68+
69+ ```yaml
70+
71+ trigger:
72+ - main
73+ pool:
74+
75+ vmImage: 'windows-latest'
76+
77+ steps:
78+
79+ - task: AzureCLI@2
80+ inputs:
81+ addSpnToEnvironment: true
82+ azureSubscription: ${{vars.Service_Connection_Name}}
83+ scriptType: bash
84+ scriptLocation: inlineScript
85+
86+ inlineScript: |
87+ echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$servicePrincipalId"
88+ echo "##vso[task.setvariable variable=ARM_ID_TOEKN]$idToken"
89+ echo "##vso[task.setvariable variable=ARM_TENANT_ID]$tenantId"
90+
91+ - bash: |
92+
93+ az login --service-principal -u $(ARM_CLIENT_ID) --tenant $(ARM_TENANT_ID) --allow-no-subscriptions --federated-token $(ARM_ID_TOEKN)
94+
95+ displayName: 'Login Azure'
96+
97+ - task: UsePythonVersion@0
98+ inputs:
99+ versionSpec: '3.11'
100+ - task: AIAgentEvaluation@0
101+ inputs:
102+ azure-ai-project-endpoint: "<your-ai-project-endpoint>"
103+ deployment-name: "gpt-4o-mini"
104+ data-path: $(Build.SourcesDirectory)\tests\data\golden-dataset-medium.json
105+ agent-ids: "<your-ai-agent-ids>
106+
107+ ```
108+
109+ # [ Hub based project] ( #tab/hub-project )
110+
111+ ``` yaml
69112
70113trigger :
71114- main
@@ -105,6 +148,8 @@ agent-ids: "<your-ai-agent-ids>
105148
106149```
107150
151+ ---
152+
108153## Set up a new pipeline and trigger an evaluation run
109154
110155Commit and run the pipeline in Azure DevOps.
0 commit comments