Skip to content

Conversation

@JennyPng
Copy link
Member

@JennyPng JennyPng commented Dec 17, 2025

closes #13132

  • test that verify setup gets invoked
  • ^ added toolmocks for generate and build code because I wanted to test that verify_setup would be called before those in all repos, but they kept failing so i didn't end up including them because it may be too flaky for now 😸
  • revised pipeline name in the README coz i think it was out of date?

@github-actions github-actions bot added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Dec 17, 2025
@JennyPng JennyPng marked this pull request as ready for review December 17, 2025 22:51
@JennyPng JennyPng requested a review from a team as a code owner December 17, 2025 22:51
Copilot AI review requested due to automatic review settings December 17, 2025 22:51
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds evaluation tests for the VerifySetup functionality and introduces tool mocks for SDK generation and building operations. The changes test that the verify_setup tool is correctly invoked both directly and as a prerequisite step before SDK generation.

  • Added two new tool mocks: PackageGenerateCode and PackageBuildCode to support testing SDK generation workflows
  • Created test to verify that the verify_setup tool is invoked when explicitly requested
  • Created test to verify that verify_setup is called before SDK generation operations

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/ToolMocks/ToolMocks.cs Registered PackageGenerateCode and PackageBuildCode tool mocks in the mock registry
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/ToolMocks/PackageGenerateCode.cs Implemented tool mock for the azsdk_package_generate_code operation with sample responses
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/ToolMocks/PackageBuildCode.cs Implemented tool mock for the azsdk_package_build_code operation with sample responses
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/Scenarios/General/Evaluate_VerifySetup.cs Added test case to verify that verify_setup tool is invoked for DotNet setup verification
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/Scenarios/AzureRestApiSpecs/Evaluate_GenerateSdk.cs Added test case to verify that verify_setup is invoked before SDK generation

@JennyPng JennyPng marked this pull request as draft December 17, 2025 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Evaluation Test for Verify Setup

2 participants