Improve Azure skill routing triggers

[kvenkatrajan]

Summary

• Expand trigger phrases in azure-prepare and azure-deploy descriptions to reduce routing misses.
• Tighten azure-prepare description length to stay under 1024 chars.
• Keep automated tests moving past planning by continuing after plan creation.

Diagnostics

Test	Prompt	azure-prepare	azure-validate	azure-deploy	Overall	Root-cause tag
Python function app with Service Bus trigger	Create an azure python function app that takes input from a service bus trigger and does message processing and deploy to Azure using my current subscription in eastus2 region.	No	No	No	Passed	Routing miss → manual implementation
URL shortener with Terraform	Create a URL shortener service using Azure Functions that creates short links and redirects users to the original URL and deploy to Azure using Terraform infrastructure in my current subscription in eastus2 region.	No	No	No	Passed	Routing miss → manual implementation
HTTP API with Terraform	Create a serverless HTTP API using Azure Functions and deploy to Azure using Terraform infrastructure in my current subscription in eastus2 region.	No	No	No	Failed	Routing miss → manual implementation
Static portfolio (SWA)	Create a static portfolio website and deploy to Azure using my current subscription in eastus2 region.	No	No	No	Passed	Routing miss → manual implementation
Whiteboard (SWA)	Create a static whiteboard web app and deploy to Azure using my current subscription in eastus2 region.	Yes	No	No	Passed	Handoff gap (prepare used, validate/deploy skipped)
Static portfolio (Terraform)	Create a static portfolio website and deploy to Azure using Terraform infrastructure in my current subscription in eastus2 region.	No	No	No	Passed	Routing miss → manual implementation
Whiteboard (Terraform)	Create a static whiteboard web app and deploy to Azure using Terraform infrastructure in my current subscription in eastus2 region.	No	No	No	Passed	Routing miss → manual implementation
brownfield-dotnet_deploys_MvcMovie_90	Please deploy this application to Azure. Use the eastus2 region. Use my current subscription. This is for a small scale production environment. Use standard SKUs.	Yes	No	No	Failed	Planning-only stop (awaited approval)
brownfield-dotnet_deploys_aspire_azure_functions	Same prompt as above	Yes	No	No	Failed	Prepare executed deployment directly (no validate/deploy skills)
brownfield-dotnet_deploys_aspire_client_apps_integration	Same prompt as above	Yes	No	No	Failed	Planning-only stop (awaited approval)
brownfield-dotnet_deploys_aspire_container_build	Same prompt as above	No	No	No	Passed	Routing miss → manual implementation
brownfield-dotnet_deploys_aspire_database_containers	Same prompt as above	Yes	No	No	Failed	Planning-only stop (awaited approval)
brownfield-dotnet_deploys_aspire_health-checks-ui	Same prompt as above	Yes	No	No	Failed	Prepare executed deployment directly (no validate/deploy skills)
brownfield-dotnet_deploys_eShop	Same prompt as above	Yes	No	No	Failed	Prepare executed deployment directly (no validate/deploy skills)
brownfield-dotnet_does_not_deploy_aspire_custom_resources	Same prompt as above	Yes	No	No	Failed	Non-deployable app detected

Report

• https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/22603076174

Files Updated

• plugin/skills/azure-prepare/SKILL.md
• plugin/skills/azure-deploy/SKILL.md
• tests/utils/agent-runner.ts

[github-actions]

🔍 Token Analysis Report

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts compare --base origin/main --head HEAD --markdown

📊 Token Change Report

Comparing origin/main → HEAD

Summary

Metric	Value
📈 Total Change	+221 tokens (+6%)
Before	3,422 tokens
After	3,643 tokens
Files Changed	2

Changed Files

File	Before	After	Change
plugin/skills/azure-prepare/SKILL.md	2,067	2,255	+188 (+9%)
plugin/skills/azure-deploy/SKILL.md	1,355	1,388	+33 (+2%)

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown

📊 Token Limit Check Report

Checked: 448 files
Exceeded: 133 files

⚠️ Files Exceeding Token Limits

File	Tokens	Limit	Over By
.github/skills/file-test-bug/SKILL.md	628	500	+128
.github/skills/sensei/README.md	3530	1000	+2530
.github/skills/sensei/SKILL.md	2382	500	+1882
.github/skills/sensei/references/EXAMPLES.md	3707	1000	+2707
.github/skills/sensei/references/LOOP.md	4181	1000	+3181
.github/skills/sensei/references/SCORING.md	3927	1000	+2927
.github/skills/sensei/references/TOKEN-INTEGRATION.md	1094	1000	+94
.github/skills/skill-authoring/SKILL.md	817	500	+317
plugin/skills/appinsights-instrumentation/SKILL.md	965	500	+465
plugin/skills/azure-ai/SKILL.md	846	500	+346
plugin/skills/azure-ai/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-aigateway/SKILL.md	1294	500	+794
plugin/skills/azure-aigateway/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-aigateway/references/patterns.md	1696	1000	+696
plugin/skills/azure-aigateway/references/policies.md	2342	1000	+1342
plugin/skills/azure-aigateway/references/troubleshooting.md	1971	1000	+971
plugin/skills/azure-cloud-migrate/references/services/functions/assessment.md	1601	1000	+601
plugin/skills/azure-cloud-migrate/references/services/functions/code-migration.md	1515	1000	+515
plugin/skills/azure-cloud-migrate/references/services/functions/lambda-to-functions.md	2600	1000	+1600
plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/csharp.md	1403	1000	+403
plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/java.md	1638	1000	+638
plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/javascript.md	2181	1000	+1181
plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/powershell.md	1261	1000	+261
plugin/skills/azure-cloud-migrate/references/services/functions/runtimes/python.md	1632	1000	+632
plugin/skills/azure-compliance/SKILL.md	1250	500	+750
plugin/skills/azure-compliance/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-compliance/references/azqr-recommendations.md	1447	1000	+447
plugin/skills/azure-compliance/references/azqr-remediation-patterns.md	1987	1000	+987
plugin/skills/azure-compliance/references/azure-keyvault-expiration-audit.md	1286	1000	+286
plugin/skills/azure-compliance/references/azure-quick-review.md	1268	1000	+268
plugin/skills/azure-compute/SKILL.md	2631	500	+2131
plugin/skills/azure-compute/references/retail-prices-api.md	1609	1000	+609
plugin/skills/azure-compute/references/vm-families.md	1234	1000	+234
plugin/skills/azure-compute/references/vmss-guide.md	1621	1000	+621
plugin/skills/azure-cost-optimization/SKILL.md	3468	500	+2968
plugin/skills/azure-cost-optimization/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-deploy/SKILL.md	1388	500	+888
plugin/skills/azure-deploy/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-deploy/references/pre-deploy-checklist.md	1195	1000	+195
plugin/skills/azure-deploy/references/recipes/azd/ef-migrations.md	1318	1000	+318
plugin/skills/azure-deploy/references/recipes/azd/errors.md	1212	1000	+212
plugin/skills/azure-deploy/references/recipes/azd/sql-managed-identity.md	1190	1000	+190
plugin/skills/azure-deploy/references/troubleshooting.md	1527	1000	+527
plugin/skills/azure-diagnostics/SKILL.md	1077	500	+577
plugin/skills/azure-hosted-copilot-sdk/SKILL.md	671	500	+171
plugin/skills/azure-hosted-copilot-sdk/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-hosted-copilot-sdk/references/azure-model-config.md	1151	1000	+151
plugin/skills/azure-kusto/SKILL.md	2175	500	+1675
plugin/skills/azure-messaging/SKILL.md	867	500	+367
plugin/skills/azure-messaging/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-messaging/references/service-troubleshooting.md	1044	1000	+44
plugin/skills/azure-observability/SKILL.md	1048	500	+548
plugin/skills/azure-observability/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-prepare/SKILL.md	2255	500	+1755
plugin/skills/azure-prepare/references/analyze.md	1038	1000	+38
plugin/skills/azure-prepare/references/apim.md	1453	1000	+453
plugin/skills/azure-prepare/references/aspire.md	2735	1000	+1735
plugin/skills/azure-prepare/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-prepare/references/azure-context.md	1019	1000	+19
plugin/skills/azure-prepare/references/plan-template.md	1063	1000	+63
plugin/skills/azure-prepare/references/recipes/azd/aspire.md	1584	1000	+584
plugin/skills/azure-prepare/references/recipes/azd/azure-yaml.md	1803	1000	+803
plugin/skills/azure-prepare/references/recipes/azd/terraform.md	2924	1000	+1924
plugin/skills/azure-prepare/references/research.md	1784	1000	+784
plugin/skills/azure-prepare/references/runtimes/nodejs.md	1508	1000	+508
plugin/skills/azure-prepare/references/security.md	2092	1000	+1092
plugin/skills/azure-prepare/references/services/functions/bicep.md	2184	1000	+1184
plugin/skills/azure-prepare/references/services/functions/templates/SPEC-composable-templates.md	6187	1000	+5187
plugin/skills/azure-prepare/references/services/functions/templates/recipes/README.md	1354	1000	+354
plugin/skills/azure-prepare/references/services/functions/templates/recipes/common/nodejs-entry-point.md	1034	1000	+34
plugin/skills/azure-prepare/references/services/functions/templates/recipes/common/uami-bindings.md	1223	1000	+223
plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md	4564	1000	+3564
plugin/skills/azure-prepare/references/services/functions/templates/recipes/cosmosdb/README.md	1467	1000	+467
plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/README.md	1149	1000	+149
plugin/skills/azure-prepare/references/services/functions/templates/recipes/eventhubs/README.md	1403	1000	+403
plugin/skills/azure-prepare/references/services/functions/templates/recipes/mcp/source/java.md	1312	1000	+312
plugin/skills/azure-prepare/references/services/functions/templates/recipes/mcp/source/python.md	1207	1000	+207
plugin/skills/azure-prepare/references/services/functions/templates/recipes/mcp/source/typescript.md	1138	1000	+138
plugin/skills/azure-prepare/references/services/functions/templates/recipes/servicebus/README.md	1171	1000	+171
plugin/skills/azure-prepare/references/services/functions/templates/recipes/servicebus/source/dotnet.md	1280	1000	+280
plugin/skills/azure-prepare/references/services/functions/templates/recipes/servicebus/source/java.md	1016	1000	+16
plugin/skills/azure-prepare/references/services/functions/templates/recipes/sql/source/java.md	1009	1000	+9
plugin/skills/azure-prepare/references/services/functions/templates/recipes/sql/source/python.md	1080	1000	+80
plugin/skills/azure-prepare/references/services/functions/terraform.md	2545	1000	+1545
plugin/skills/azure-prepare/references/services/service-bus/patterns.md	1122	1000	+122
plugin/skills/azure-resource-lookup/SKILL.md	1389	500	+889
plugin/skills/azure-resource-lookup/references/azure-resource-graph.md	1307	1000	+307
plugin/skills/azure-resource-visualizer/SKILL.md	2105	500	+1605
plugin/skills/azure-storage/SKILL.md	1180	500	+680
plugin/skills/azure-storage/references/auth-best-practices.md	1543	1000	+543
plugin/skills/azure-storage/references/sdk-usage.md	1135	1000	+135
plugin/skills/azure-validate/SKILL.md	761	500	+261
plugin/skills/azure-validate/references/recipes/azd/README.md	1191	1000	+191
plugin/skills/entra-app-registration/SKILL.md	2068	500	+1568
plugin/skills/entra-app-registration/references/api-permissions.md	2545	1000	+1545
plugin/skills/entra-app-registration/references/auth-best-practices.md	1543	1000	+543
plugin/skills/entra-app-registration/references/cli-commands.md	2211	1000	+1211
plugin/skills/entra-app-registration/references/console-app-example.md	2752	1000	+1752
plugin/skills/entra-app-registration/references/first-app-registration.md	1846	1000	+846
plugin/skills/entra-app-registration/references/oauth-flows.md	2375	1000	+1375
plugin/skills/entra-app-registration/references/troubleshooting.md	1896	1000	+896
plugin/skills/microsoft-foundry/SKILL.md	1948	500	+1448
plugin/skills/microsoft-foundry/foundry-agent/create/create.md	3016	1000	+2016
plugin/skills/microsoft-foundry/foundry-agent/create/references/agentframework.md	1300	1000	+300
plugin/skills/microsoft-foundry/foundry-agent/create/references/tool-memory.md	1204	1000	+204
plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md	4013	1000	+3013
plugin/skills/microsoft-foundry/foundry-agent/invoke/invoke.md	1273	1000	+273
plugin/skills/microsoft-foundry/foundry-agent/trace/references/kql-templates.md	1913	1000	+913
plugin/skills/microsoft-foundry/foundry-agent/trace/references/search-traces.md	1366	1000	+366
plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md	1265	1000	+265
plugin/skills/microsoft-foundry/foundry-agent/troubleshoot/troubleshoot.md	1299	1000	+299
plugin/skills/microsoft-foundry/models/deploy-model/SKILL.md	1640	500	+1140
plugin/skills/microsoft-foundry/models/deploy-model/capacity/SKILL.md	1739	500	+1239
plugin/skills/microsoft-foundry/models/deploy-model/customize/EXAMPLES.md	1091	1000	+91
plugin/skills/microsoft-foundry/models/deploy-model/customize/SKILL.md	2235	500	+1735
plugin/skills/microsoft-foundry/models/deploy-model/customize/references/customize-workflow.md	3335	1000	+2335
plugin/skills/microsoft-foundry/models/deploy-model/preset/SKILL.md	1226	500	+726
plugin/skills/microsoft-foundry/models/deploy-model/preset/references/preset-workflow.md	5534	1000	+4534
plugin/skills/microsoft-foundry/models/deploy-model/preset/references/workflow.md	1315	1000	+315
plugin/skills/microsoft-foundry/project/create/create-foundry-project.md	1346	1000	+346
plugin/skills/microsoft-foundry/quota/quota.md	2129	1000	+1129
plugin/skills/microsoft-foundry/quota/references/capacity-planning.md	1968	1000	+968
plugin/skills/microsoft-foundry/quota/references/error-resolution.md	1141	1000	+141
plugin/skills/microsoft-foundry/quota/references/optimization.md	1846	1000	+846
plugin/skills/microsoft-foundry/quota/references/ptu-guide.md	1473	1000	+473
plugin/skills/microsoft-foundry/quota/references/troubleshooting.md	1807	1000	+807
plugin/skills/microsoft-foundry/quota/references/workflows.md	1614	1000	+614
plugin/skills/microsoft-foundry/rbac/rbac.md	1752	1000	+752
plugin/skills/microsoft-foundry/references/auth-best-practices.md	1543	1000	+543
plugin/skills/microsoft-foundry/references/sdk/foundry-sdk-py.md	2060	1000	+1060
plugin/skills/microsoft-foundry/resource/create/create-foundry-resource.md	1489	1000	+489
plugin/skills/microsoft-foundry/resource/create/references/workflows.md	1637	1000	+637
.github/agents/SkillCreator.agent.md	1044	1000	+44

Consider moving content to references/ subdirectories.

---

Automated token analysis. See skill authoring guidelines for best practices.

[Copilot]

Pull request overview

This PR updates Azure skill metadata and the integration-test harness to improve skill routing coverage and reduce test runs that stall at “plan approval”.

Changes:

• Expanded trigger phrases in azure-prepare and azure-deploy skill descriptions and bumped skill versions.
• Strengthened azure-prepare guidance to enforce handoff to azure-validate/azure-deploy and prohibit deploying from azure-prepare.
• Updated the test runner system prompt so integration tests continue execution after plan creation without asking for approval.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
plugin/skills/azure-prepare/SKILL.md	Adds/adjusts routing triggers and makes validate/deploy handoff requirements explicit; bumps version.
plugin/skills/azure-deploy/SKILL.md	Adds more deploy-oriented trigger phrases; bumps version.
tests/utils/agent-runner.ts	Extends default system prompt to prevent planning-only stalls in automated integration tests.

[Copilot]

The default system prompt content is now duplicated between useAgentRunner() and runConversation(). To avoid future drift (e.g., updating one but not the other), consider extracting a shared DEFAULT_SYSTEM_PROMPT (or a small helper that builds it) and reusing it in both places.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

tests/utils/agent-runner.ts:400

• The default system prompt now tells the agent to continue after creating a plan without asking for user approval. This can cause integration tests that are intended to stop before running real Azure commands (e.g., tests that ‘do not expect to deploy’) to proceed into azd / terraform execution, potentially creating resources/cost. Consider making this behavior opt-in (e.g., a dedicated config flag) or gating it to specific tests, rather than changing the runner default globally.

      const systemPrompt = config.systemPrompt ?? {
        mode: "append",
        content: "When a relevant skill is available, prefer using it instead of doing the task manually. This is an automated integration test: if you create a plan, continue execution without asking for user approval."
      };

[Copilot]

Same issue as the runner default above: runConversation also hard-codes the new ‘continue without asking for approval’ instruction as the default system message. If this behavior is only desired for a subset of scenarios, it should be configurable here as well to avoid unintentionally triggering destructive deployments in tests that rely on plan/approval gating.

This issue also appears on line 397 of the same file.

Suggested change

	content: "When a relevant skill is available, prefer using it instead of doing the task manually. This is an automated integration test: if you create a plan, continue execution without asking for user approval."
	content: "When a relevant skill is available, prefer using it instead of doing the task manually. This is an automated integration test."

[Copilot]

The SKILL.md description is used for keyword extraction (words >3 chars) in trigger tests/routing. Adding very generic tokens like main (from infra/main.bicep) and template can increase false positives (e.g., prompts about ‘deploy main’ unrelated to Azure can now match deploy + main). Consider removing/rewriting these phrases to keep description keywords Azure-specific (e.g., refer to ‘Bicep files under infra/’ without introducing main/template).

See below for a potential fix:

description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.yaml, Dockerfiles). Use for create/modernize or create+deploy; not cross-cloud migration (use azure-cloud-migrate). WHEN: \"create app\", \"build web app\", \"create API\", \"create serverless HTTP API\", \"build a service\", \"modernize application\", \"update application\", \"add authentication\", \"add caching\", \"host on Azure\", \"create and deploy\", \"deploy to Azure\", \"deploy to Azure using Terraform\", \"deploy to Azure App Service\", \"deploy to Azure Container Apps\", \"generate Terraform\", \"generate Bicep\", \"function app\", \"timer trigger\", \"service bus trigger\", \"event-driven function\", \"static portfolio website\", \"todo list with frontend and API\", \"Key Vault\", \"managed identity\", \"set up Azure infrastructure\", \"provision Azure resources\", \"scaffold Azure project\", \"azure.yaml\", \"infra Bicep entry file in infra/\", \"Terraform infrastructure\", \"Bicep module\"."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.5"

[Copilot]

Including the phrase az deployment group create in the trigger description introduces the keyword group, which is extremely common in non-deployment Azure prompts (e.g., ‘create an Azure AD group’). Because triggering only requires 2 keyword matches, azure + group can now incorrectly route to azure-deploy. Consider removing or rephrasing this trigger phrase to avoid adding overly-generic keywords to the description.

See below for a potential fix:

description: "Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/plan.md from azure-prepare and validated status from azure-validate. WHEN: \"run azd up\", \"run azd deploy\", \"execute deployment\", \"push to production\", \"push to cloud\", \"go live\", \"ship it\", \"bicep deploy\", \"terraform apply\", \"publish to Azure\", \"launch on Azure\", \"deploy existing app\", \"deploy prepared app\", \"apply Terraform\", \"az deployment create (RG scope)\", \"redeploy\", \"roll out\". DO NOT USE WHEN: \"create and deploy\", \"build and deploy\", \"create a new app\", \"set up infrastructure\", \"create and deploy to Azure using Terraform\" — use azure-prepare for these."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.4"

[JasonYeMSFT]

I suggest remove the part that mentions do not use this skill.

[JasonYeMSFT]

I didn't realize that we added this. If possible, I prefer not touching the system prompt. This will cause our test runs to deviate from what Copilot CLI users get. We have implemented the onPermissionRequest handler which should auto approve all permission requests.

[kvenkatrajan]

cancelling this one - this was before the skill invocation issue was applied