Skip to content

Improve Azure skill routing triggers #1107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kvenkatrajan wants to merge 5 commits into from
Closed

Improve Azure skill routing triggers #1107

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5934bac
test: simplify agent runner default prompt
kvenkatrajan Mar 3, 2026
b350f88
Improve Azure skill routing triggers
kvenkatrajan Mar 3, 2026
0462547
Update trigger snapshots
kvenkatrajan Mar 3, 2026
0e1b5fd
Fix azure-prepare trigger snapshot
kvenkatrajan Mar 3, 2026
d0f0823
Update azure-prepare triggers
kvenkatrajan Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions plugin/skills/azure-deploy/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
name: azure-deploy
description: "Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/plan.md from azure-prepare and validated status from azure-validate. WHEN: \"run azd up\", \"run azd deploy\", \"execute deployment\", \"push to production\", \"push to cloud\", \"go live\", \"ship it\", \"bicep deploy\", \"terraform apply\", \"publish to Azure\", \"launch on Azure\". DO NOT USE WHEN: \"create and deploy\", \"build and deploy\", \"create a new app\", \"set up infrastructure\", \"create and deploy to Azure using Terraform\" — use azure-prepare for these."
description: "Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/plan.md from azure-prepare and validated status from azure-validate. WHEN: \"run azd up\", \"run azd deploy\", \"execute deployment\", \"push to production\", \"push to cloud\", \"go live\", \"ship it\", \"bicep deploy\", \"terraform apply\", \"publish to Azure\", \"launch on Azure\", \"deploy existing app\", \"deploy prepared app\", \"apply Terraform\", \"az deployment group create\", \"redeploy\", \"roll out\". DO NOT USE WHEN: \"create and deploy\", \"build and deploy\", \"create a new app\", \"set up infrastructure\", \"create and deploy to Azure using Terraform\" — use azure-prepare for these."
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including the phrase az deployment group create in the trigger description introduces the keyword group, which is extremely common in non-deployment Azure prompts (e.g., ‘create an Azure AD group’). Because triggering only requires 2 keyword matches, azure + group can now incorrectly route to azure-deploy. Consider removing or rephrasing this trigger phrase to avoid adding overly-generic keywords to the description.

See below for a potential fix:

description: "Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/plan.md from azure-prepare and validated status from azure-validate. WHEN: \"run azd up\", \"run azd deploy\", \"execute deployment\", \"push to production\", \"push to cloud\", \"go live\", \"ship it\", \"bicep deploy\", \"terraform apply\", \"publish to Azure\", \"launch on Azure\", \"deploy existing app\", \"deploy prepared app\", \"apply Terraform\", \"az deployment create (RG scope)\", \"redeploy\", \"roll out\". DO NOT USE WHEN: \"create and deploy\", \"build and deploy\", \"create a new app\", \"set up infrastructure\", \"create and deploy to Azure using Terraform\" — use azure-prepare for these."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.4"

Copilot uses AI. Check for mistakes.
Copy link
Member

@JasonYeMSFT JasonYeMSFT Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest remove the part that mentions do not use this skill.

license: MIT
metadata:
author: Microsoft
version: "1.0.2"
version: "1.0.3"
---

# Azure Deploy
Expand Down
12 changes: 9 additions & 3 deletions plugin/skills/azure-prepare/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
name: azure-prepare
description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.yaml, Dockerfiles). Use for create/modernize or create+deploy; not cross-cloud migration (use azure-cloud-migrate). WHEN: \"create app\", \"build web app\", \"create API\", \"create serverless HTTP API\", \"create frontend\", \"create back end\", \"build a service\", \"modernize application\", \"update application\", \"add authentication\", \"add caching\", \"host on Azure\", \"create and deploy\", \"deploy to Azure\", \"deploy to Azure using Terraform\", \"deploy to Azure App Service\", \"deploy to Azure App Service using Terraform\", \"deploy to Azure Container Apps\", \"deploy to Azure Container Apps using Terraform\", \"generate Terraform\", \"generate Bicep\", \"function app\", \"timer trigger\", \"service bus trigger\", \"event-driven function\", \"containerized Node.js app\", \"social media app\", \"static portfolio website\", \"todo list with frontend and API\", \"prepare my Azure application to use Key Vault\", \"managed identity\"."
description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.yaml, Dockerfiles). Use for create/modernize or create+deploy; not cross-cloud migration (use azure-cloud-migrate). WHEN: \"create app\", \"build web app\", \"create API\", \"create serverless HTTP API\", \"build a service\", \"modernize application\", \"update application\", \"add authentication\", \"add caching\", \"host on Azure\", \"create and deploy\", \"deploy to Azure\", \"deploy to Azure using Terraform\", \"deploy to Azure App Service\", \"deploy to Azure Container Apps\", \"generate Terraform\", \"generate Bicep\", \"function app\", \"timer trigger\", \"service bus trigger\", \"event-driven function\", \"static portfolio website\", \"todo list with frontend and API\", \"Key Vault\", \"managed identity\", \"set up Azure infrastructure\", \"provision Azure resources\", \"scaffold Azure project\", \"azure.yaml\", \"infra/main.bicep\", \"Terraform infrastructure\", \"Bicep template\"."
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SKILL.md description is used for keyword extraction (words >3 chars) in trigger tests/routing. Adding very generic tokens like main (from infra/main.bicep) and template can increase false positives (e.g., prompts about ‘deploy main’ unrelated to Azure can now match deploy + main). Consider removing/rewriting these phrases to keep description keywords Azure-specific (e.g., refer to ‘Bicep files under infra/’ without introducing main/template).

See below for a potential fix:

description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.yaml, Dockerfiles). Use for create/modernize or create+deploy; not cross-cloud migration (use azure-cloud-migrate). WHEN: \"create app\", \"build web app\", \"create API\", \"create serverless HTTP API\", \"build a service\", \"modernize application\", \"update application\", \"add authentication\", \"add caching\", \"host on Azure\", \"create and deploy\", \"deploy to Azure\", \"deploy to Azure using Terraform\", \"deploy to Azure App Service\", \"deploy to Azure Container Apps\", \"generate Terraform\", \"generate Bicep\", \"function app\", \"timer trigger\", \"service bus trigger\", \"event-driven function\", \"static portfolio website\", \"todo list with frontend and API\", \"Key Vault\", \"managed identity\", \"set up Azure infrastructure\", \"provision Azure resources\", \"scaffold Azure project\", \"azure.yaml\", \"infra Bicep entry file in infra/\", \"Terraform infrastructure\", \"Bicep module\"."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.5"

Copilot uses AI. Check for mistakes.
license: MIT
metadata:
author: Microsoft
version: "1.0.1"
version: "1.0.4"
---

# Azure Prepare
Expand Down Expand Up @@ -36,6 +36,8 @@ Activate this skill when user wants to:
6. **Confirm Azure context** — Use `ask_user` for subscription and location per [Azure Context](references/azure-context.md)
7. ❌ **Destructive actions require `ask_user`** — [Global Rules](references/global-rules.md)
8. **Scope: preparation only** — This skill generates infrastructure code and configuration files. Deployment execution (`azd up`, `azd deploy`, `terraform apply`) is handled by the **azure-deploy** skill, which provides built-in error recovery and deployment verification.
9. **Handoff is mandatory** — After plan approval and preparation, you MUST invoke **azure-validate** via skill call. If the user asked to deploy in the same request and validation succeeds, immediately hand off to **azure-deploy**. Do not just mention the next step—invoke it.
10. **Do not deploy in azure-prepare** — Never run deployment or provisioning commands here (e.g., `azd up`, `azd deploy`, `azd provision`, `terraform apply`, `az deployment group create`). Those are exclusively handled by **azure-deploy** after **azure-validate** succeeds.

---

Expand Down Expand Up @@ -103,7 +105,7 @@ Execute the approved plan. Update `.azure/plan.md` status after each step.
| 3 | **Generate Artifacts** — Create infrastructure and configuration files | [generate.md](references/generate.md) |
| 4 | **Harden Security** — Apply security best practices | [security.md](references/security.md) |
| 5 | **Update Plan** — Mark steps complete, set status to `Ready for Validation` | `.azure/plan.md` |
| 6 | **⚠️ Hand Off** — Invoke **azure-validate** skill. Your preparation work is done. Deployment execution is handled by azure-deploy. | — |
| 6 | **⚠️ Hand Off** — Invoke **azure-validate** skill now (skill call). If validation succeeds and the user requested deployment, immediately invoke **azure-deploy**. Do not run azd/terraform deployment commands here. | — |

---

Expand Down Expand Up @@ -136,4 +138,8 @@ Execute the approved plan. Update `.azure/plan.md` status after each step.
>
> Skipping validation leads to deployment failures. Be patient and follow the complete workflow for the highest success outcome.

> **⚠️ REQUIRED ACTION**
>
> Invoke **azure-validate** now. If it succeeds and the user asked to deploy, immediately invoke **azure-deploy**.

**→ Invoke azure-validate now**
12 changes: 7 additions & 5 deletions tests/utils/agent-runner.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,11 @@ export function useAgentRunner() {

const skillDirectory = path.resolve(__dirname, "../../plugin/skills");

const systemPrompt = config.systemPrompt ?? {
mode: "append",
content: "When a relevant skill is available, prefer using it instead of doing the task manually. This is an automated integration test: if you create a plan, continue execution without asking for user approval."
Copy link
Member

@JasonYeMSFT JasonYeMSFT Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize that we added this. If possible, I prefer not touching the system prompt. This will cause our test runs to deviate from what Copilot CLI users get. We have implemented the onPermissionRequest handler which should auto approve all permission requests.

};

const session = await client.createSession({
model: modelOverride || "claude-sonnet-4.5",
onPermissionRequest: approveAll,
Expand All @@ -406,10 +411,7 @@ export function useAgentRunner() {
tools: ["*"]
}
},
systemMessage: config.systemPrompt ?? {
mode: "append",
content: "When a relevant skill is available, prefer using it instead of doing the task manually."
}
systemMessage: systemPrompt
});
entry.session = session;

Expand Down Expand Up @@ -730,7 +732,7 @@ export async function runConversation(config: ConversationConfig): Promise<Conve
},
systemMessage: config.systemPrompt ?? {
mode: "append",
content: "When a relevant skill is available, prefer using it instead of doing the task manually."
content: "When a relevant skill is available, prefer using it instead of doing the task manually. This is an automated integration test: if you create a plan, continue execution without asking for user approval."
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue as the runner default above: runConversation also hard-codes the new ‘continue without asking for approval’ instruction as the default system message. If this behavior is only desired for a subset of scenarios, it should be configurable here as well to avoid unintentionally triggering destructive deployments in tests that rely on plan/approval gating.

This issue also appears on line 397 of the same file.

Suggested change
content: "When a relevant skill is available, prefer using it instead of doing the task manually. This is an automated integration test: if you create a plan, continue execution without asking for user approval."
content: "When a relevant skill is available, prefer using it instead of doing the task manually. This is an automated integration test."

Copilot uses AI. Check for mistakes.
}
Comment on lines 733 to 736
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default system prompt content is now duplicated between useAgentRunner() and runConversation(). To avoid future drift (e.g., updating one but not the other), consider extracting a shared DEFAULT_SYSTEM_PROMPT (or a small helper that builds it) and reusing it in both places.

Copilot uses AI. Check for mistakes.
});

Expand Down
Loading