Skip to content

Refactor agent environment scanners and add standard location support#15022

Merged
JamesNK merged 19 commits intomainfrom
mad-skills
Mar 25, 2026
Merged

Refactor agent environment scanners and add standard location support#15022
JamesNK merged 19 commits intomainfrom
mad-skills

Conversation

@IEvangelist
Copy link
Member

@IEvangelist IEvangelist commented Mar 6, 2026

Description

  • Adds a standard .agents/ based skill.
  • Updates the existing aspire skill to reflect latest changes to CLI
    • Including ./modules/* context for TS polyglot AppHost
  • Adds dotnet inspect skill

In this PR, regardless of whether someone explicitly calls agent init or they accept the template question to include AI, the flow is as follows:

  1. Prompt which locations they want to configure.
  2. Prompt which skills/mcp configs they want installed.
  3. Install selections in selected locations (remove hardcoded install locations from playwright-cli)

Also, fixes a bug where the playwright-cli was never installing on Windows.

Select AI environment locations

image

Select skills/MCP bits

image

Resulting bits on disk

image

Other considerations not in this PR, remove vendor specific skills.

Fixes #14885

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No
  • Does the change require an update in our Aspire docs?

Copilot AI review requested due to automatic review settings March 6, 2026 21:00
@github-actions
Copy link
Contributor

github-actions bot commented Mar 6, 2026

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15022

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15022"

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors agent-environment scanning to centralize skill-file provisioning and adds support for an “industry-standard” .agents/skills/ location, including a new dotnet-inspect skill and updated Aspire skill content.

Changes:

  • Add a new StandardLocationAgentEnvironmentScanner that registers workspace + user (~) .agents/skills/ skill applicators.
  • Extend CommonAgentApplicators.TryAddSkillFileApplicator to support custom skill content and configurable deduplication keys; add dotnet-inspect skill content.
  • Remove per-environment Aspire skill-file creation from existing scanners (Copilot CLI / VS Code / Claude / OpenCode) and add/adjust tests accordingly.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/Aspire.Cli.Tests/Agents/StandardLocationAgentEnvironmentScannerTests.cs Adds coverage for .agents/skills workspace + user locations and update/no-op behavior.
tests/Aspire.Cli.Tests/Agents/OpenCodeAgentEnvironmentScannerTests.cs Updates tests for OpenCode scanner constructor changes.
tests/Aspire.Cli.Tests/Agents/CopilotCliAgentEnvironmentScannerTests.cs Removes assumptions about .github/skills Aspire skill generation in Copilot scanner tests.
tests/Aspire.Cli.Tests/Agents/CommonAgentApplicatorsTests.cs Adds tests for custom content, deduplication keys, and standard location applicators.
src/Aspire.Cli/Program.cs Registers the new StandardLocationAgentEnvironmentScanner.
src/Aspire.Cli/Agents/VsCode/VsCodeAgentEnvironmentScanner.cs Stops adding Aspire skill-file applicators in VS Code scanning paths.
src/Aspire.Cli/Agents/StandardLocationAgentEnvironmentScanner.cs New scanner that adds standard .agents/skills skill applicators.
src/Aspire.Cli/Agents/OpenCode/OpenCodeAgentEnvironmentScanner.cs Removes OpenCode-specific skill file creation; updates constructor signature.
src/Aspire.Cli/Agents/CopilotCli/CopilotCliAgentEnvironmentScanner.cs Stops adding Aspire skill-file applicators.
src/Aspire.Cli/Agents/CommonAgentApplicators.cs Adds dotnet-inspect skill content + standard-location helper; extends skill applicator creation to support custom content/dedup keys; updates Aspire skill guidance.
src/Aspire.Cli/Agents/ClaudeCode/ClaudeCodeAgentEnvironmentScanner.cs Stops adding Aspire skill-file applicators.

You can also share your feedback on Copilot code review. Take the survey.

@davidfowl
Copy link
Contributor

@IEvangelist Did you test this with codex, copilot cli and claude code? Those are the primary 3 that we need to make sure works with this.

@IEvangelist IEvangelist marked this pull request as draft March 7, 2026 02:38
@IEvangelist
Copy link
Member Author

@IEvangelist Did you test this with codex, copilot cli and claude code? Those are the primary 3 that we need to make sure works with this.

I'm going to test it, converted to draft. I was also going to play around with attempting to scope it to Aspire APIs.

@IEvangelist
Copy link
Member Author

IEvangelist commented Mar 7, 2026

I tested all three. Gave each an aspire new app, then called:

dotnet run --project "D:\GitHub\aspire\src\Aspire.Cli\Aspire.Cli.csproj" -- agent init

This resulted in each new ts-starter app getting the skills only in the .agents/skills/* folder. Immediately, all there AI reported that they saw those skills. To double check, I even explicitly asked them what skills they had and again, all three reported them explicitly.

I then tasked each AI, in their respective folders to take the template, and update it with the following prompt:

Great, now help me update this aspire app so that it's TODO app instead.
I want a nice modern UX - (not weather). I want the API to be updated 
to support OTEL with standard values so that aspire can pick it up in 
the dashboard. I want the API to serve a TODO API instead - and 
persist/read from a pg db.

Summary, all three were successful at implementing a rewrite of the UX, API (with OTEL) and PG support.

GitHub Copilot

Used copilot --yolo command to initiate.

  • Model: Claude Opus 4.6
image image image

Code Generated:

.aspire/settings.json

{
  "appHostPath": "../apphost.ts",
  "language": "typescript/nodejs",
  "channel": "staging",
  "sdkVersion": "13.1.0",
  "packages": {
    "Aspire.Hosting.JavaScript": "13.2.0-preview.1.26156.7",
    "Aspire.Hosting.PostgreSQL": "13.1.2"
  }
}

apphost.ts

import { createBuilder } from './.modules/aspire.js';

const builder = await createBuilder();

const postgres = builder.addPostgres("postgres")
    .withLifetime("persistent")
    .withPgAdmin();
const tododb = await postgres.addDatabase("tododb");

const app = await builder
    .addNodeApp("app", "./api", "src/index.ts")
    .withHttpEndpoint({ env: "PORT" })
    .withReference(tododb)
    .waitFor(tododb)
    .withExternalHttpEndpoints();

const frontend = await builder
    .addViteApp("frontend", "./frontend")
    .withServiceReference(app)
    .waitFor(app);

await app.publishWithContainerFiles(frontend, "./static");

await builder.build().run();

I asked the AI:

Please summarize what skills, tools, and CLI commands you used to be successful. Please report it here so that I might analyze how well you did, or if something might need fine-tuning.

Copilot

● Here's a breakdown of everything I used across this session:

Skills Invoked

  • Aspire Skill (.agents/skills/aspire/SKILL.md) — Referenced for CLI command patterns (aspire start, aspire describe, aspire wait, aspire logs, aspire add, aspire stop). Installed via aspire agent init.
  • dotnet-inspect Skill — Installed but not directly invoked (no .NET API queries were needed).

Aspire MCP Tools

  • aspire-search_docs / aspire-get_doc — Looked up PostgreSQL hosting integration docs to understand withReference, connection string env vars, and non-.NET app patterns.
  • aspire-list_resources — Verified all resources were running and healthy after startup.
  • aspire-list_traces — Confirmed OpenTelemetry traces (HTTP spans → pg.query spans → tododb) were flowing to the Aspire dashboard.

CLI Commands (via dotnet run --project)

  • aspire agent init — Interactive setup of skill files, Playwright CLI, and MCP server config.
  • aspire add Aspire.Hosting.PostgreSQL — Added PostgreSQL hosting integration, which regenerated .modules/ with Postgres APIs.
  • aspire start / aspire stop — Background start/stop of the AppHost (used 5+ restarts while debugging).
  • aspire run — Foreground run to capture the CapabilityError stack trace.
  • aspire wait app — Blocked until the API resource was healthy.
  • aspire describe — Listed resource states, endpoints, and env vars (also used --format Json for detailed env var inspection).
  • aspire logs app — Retrieved console logs to diagnose crashes.

Tools

  • powershell — npm install (pg, @opentelemetry/*), builds (npm run build, tsc), API endpoint testing via Invoke-RestMethod/Invoke-WebRequest.
  • view/edit/create — Read and modified apphost.ts, api/src/index.ts, api/src/instrumentation.ts, frontend/src/App.tsx, frontend/src/App.css.
  • grep — Searched .modules/aspire.ts for addPostgres, addDatabase, withReference, ResourceBuilderBase to understand generated API signatures.
  • task (general-purpose agents) — Dispatched two parallel sub-agents: one for API code (OTEL + CRUD), one for frontend (React + CSS).
  • sql — Tracked todos and dependencies for the implementation plan.

Key Debugging Moments

  1. CapabilityError: IResourceBuilder must have AspireDto attribute — Fixed by await-ing postgres.addDatabase() before passing to withReference.
  2. ERR_MODULE_NOT_FOUND: instrumentation.js — Changed import from ./instrumentation.js to ./instrumentation.ts (tsx runtime resolves .ts directly).
  3. getaddrinfo ENOTFOUND base — ConnectionStrings__tododb was in ADO.NET format (Host=...;Port=...) which pg can't parse. Switched to individual env vars (TODODB_HOST, TODODB_PORT, etc.) that Aspire
    provides for non-.NET apps.

Codex App

  • Model: GPT-5.3-Codex

Used codex app command to initiate.

image image image

Code Generated:

.aspire/settings.json

{
  "appHostPath": "../apphost.ts",
  "language": "typescript/nodejs",
  "channel": "staging",
  "sdkVersion": "13.2.0-preview.1.26156.7",
  "packages": {
    "Aspire.Hosting.JavaScript": "13.2.0-preview.1.26156.3",
    "Aspire.Hosting.PostgreSQL": "13.2.0-preview.1.26156.7"
  }
}

apphost.ts

import { createBuilder } from './.modules/aspire.js';

const builder = await createBuilder();

const postgres = await builder
    .addPostgres("postgres")
    .withDataVolume();

const todoDb = await postgres.addDatabase("todos");

const app = await builder
    .addNodeApp("app", "./api", "src/index.ts")
    .withHttpEndpoint({ env: "PORT" })
    .withReference(todoDb, { connectionName: "todos" })
    .waitFor(postgres)
    .withExternalHttpEndpoints();

const frontend = await builder
    .addViteApp("frontend", "./frontend")
    .withServiceReference(app)
    .waitFor(app);

await app.publishWithContainerFiles(frontend, "./static");

await builder.build().run();

I asked the AI:

Please summarize what skills, tools, and CLI commands you used to be successful. Please report it here so that I might analyze how well you did, or if something might need fine-tuning.

Codex App

• Skills Used

  1. aspire skill (C:/Users/dapine/.agents/skills/aspire/SKILL.md): used for Aspire orchestration patterns, command choices (aspire start, aspire describe, aspire logs, aspire agent init, aspire add, aspire
    docs), and TypeScript AppHost integration flow.
  2. Agent skills initialized into repo via aspire agent init: aspire and dotnet-inspect were installed into .agents/skills.

Developer Tools Used

  1. functions.shell_command: primary execution tool for CLI commands and repo inspection.
  2. functions.apply_patch: used for code edits (apphost.ts, API code, frontend code, package files).
  3. multi_tool_use.parallel: used to run independent shell commands in parallel for faster diagnostics and validation.

Aspire / CLI Commands That Drove Success

  1. aspire --version to confirm installed CLI.
  2. aspire start --non-interactive repeatedly to regenerate .modules, restart resources, and validate runtime.
  3. aspire add postgres --non-interactive to add Postgres hosting integration and unlock generated addPostgres/addDatabase APIs.
  4. aspire describe --format json and aspire describe app --format json to inspect resource health, env injection, and endpoints.
  5. aspire logs app to diagnose runtime failures (syntax error, DB connection parsing issue).
  6. aspire docs search "JavaScript integration" and aspire docs get javascript-integration to confirm auto-install behavior and JS integration defaults.
  7. aspire otel traces --apphost apphost.ts --format Json -n 20 to verify OTEL traces were finally flowing.
  8. aspire otel logs --apphost apphost.ts --format Json -n 20 to confirm logs pipeline status.
  9. aspire ps to confirm running AppHost instances.

Other Commands Used

  1. dotnet run --project D:\GitHub\aspire\src\Aspire.Cli\Aspire.Cli.csproj -- ... for local CLI-build testing early on.
  2. rg -n ... for fast source/API surface discovery (.modules/aspire.ts, Aspire CLI source).
  3. curl.exe / Invoke-RestMethod against /api/todos to generate traffic and validate persistence.
  4. Get-Content, Get-ChildItem, git status --short for repo inspection and change tracking.

Key Outcome Enablers

  1. Added Postgres integration first, then used generated TypeScript AppHost APIs.
  2. Parsed Aspire-style connection strings (Host=...;Database=...) for Node pg.
  3. Bootstrapped OTEL before loading Express/pg modules.
  4. Matched Aspire OTLP defaults (gRPC + headers) to make traces visible in dashboard telemetry

Claude Code

  • Model: Sonnet 4.6

Note

I didn't use the CLI, I don't have a subscription. Instead, I used the free version (desktop) with the Fileystem and Desktop Commander extensions to interact with the terminal and perform file system I/O.

After a bunch of attempts, the Desktop app was not able to do this task. It couldn't figure out how to do interactive terminals.

{
  "appHostPath": "../apphost.ts",
  "language": "typescript/nodejs",
  "channel": "staging",
  "sdkVersion": "13.2.0-preview.1.26156.7",
  "packages": {
    "Aspire.Hosting.JavaScript": "13.2.0-preview.1.26156.3"
  }
}
import { createBuilder } from './.modules/aspire.js';

const builder = await createBuilder();

const postgres = await builder.addPostgres('postgres');
const db = await postgres.addDatabase('todos');

const api = await builder
  .addNodeApp('api', './api', 'src/index.ts')
  .withHttpEndpoint({ env: 'PORT' })
  .withExternalHttpEndpoints()
  .withReference(db)
  .waitFor(db);

const frontend = await builder
  .addViteApp('frontend', './frontend')
  .withServiceReference(api)
  .waitFor(api);

await api.publishWithContainerFiles(frontend, './static');

await builder.build().run();

@joperezr
Copy link
Member

@IEvangelist is this ready to review?

@IEvangelist
Copy link
Member Author

@IEvangelist is this ready to review?

It has been ready for review...I can close this as it seems like it might not be something we want.

@davidfowl
Copy link
Contributor

I think we want this.

@IEvangelist
Copy link
Member Author

I think we want this.

Hey @davidfowl - this is ready to go.

@github-actions
Copy link
Contributor

The transient CI rerun workflow requested reruns for the following jobs after analyzing the failed attempt.
GitHub's job rerun API also reruns dependent jobs, so the retry is being tracked in the rerun attempt.
The job links below point to the failed attempt that matched the retry-safe transient failure rules.

  • Tests / Npgsql / Npgsql (windows-latest) - Failed step 'Build test project' will be retried because the job log shows a likely transient infrastructure network failure. Matched pattern: /Unable to load the service index for source https://(?:pkgs.dev.azure.com/dnceng|dnceng.pkgs.visualstudio.com)/public/_packaging//i.

IEvangelist and others added 19 commits March 25, 2026 08:46
* feat: enhance Playwright CLI installation process with improved error handling and cleanup logic

* fix: improve error handling and cleanup in Playwright CLI installation process
…atus enum, improve error handling and messaging (#15356)
…#15386)

* Initial plan

* Fix missing DI registrations for PlaywrightCliInstaller in test setup

Co-authored-by: IEvangelist <7679720+IEvangelist@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: IEvangelist <7679720+IEvangelist@users.noreply.github.com>
Co-authored-by: James Newton-King <james@newtonking.com>
- Remove accidental Debugger.Launch() and unused System.Diagnostics import from AgentInitCommand
- Remove dead InitCommand_NothingSelected and InitCommand_InstallSkillFile resource strings from .resx, .Designer.cs, and all 13 xlf files
- Split platform-agnostic NpmRunnerTests empty-args test into Windows (.cmd) and non-Windows variants
The Sigstore .NET library v0.4.0 ExtractSan() method fails on Windows
because .NET formats URI-type SANs as 'URL=...' but the library
checks for 'URI'. This causes CertificateIdentity.ForGitHubActions()
to always fail with 'Could not extract Subject Alternative Name'.

When the known SAN failure is detected, retry cryptographic
verification without CertificateIdentity, then manually verify the
three identity checks (OIDC issuer, SourceRepositoryUri, SAN pattern)
by reading Fulcio certificate extensions directly from the bundle.
This is safe because the full crypto chain (Fulcio CA, SCT, Rekor,
artifact signature) is still verified, and Fulcio extensions are
parsed from raw DER bytes (not the platform-dependent Format method).

Will be removed when upgrading to Sigstore 0.5.0 which fixes the
upstream bug: mitchdenny/sigstore-dotnet#14

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix extra leading whitespace in VerifyCertificateIdentityFromBundle
- Add comment documenting PayloadDecodeFailed -> AttestationParseFailed change
- Extract ParseUriFromFormattedSan as internal static for testability
- Add unit tests for SAN parsing (URI/URL formats, multi-entry, certs)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@JamesNK JamesNK changed the base branch from release/13.2 to main March 25, 2026 00:48
@JamesNK JamesNK removed the Servicing-consider Issue for next servicing release review label Mar 25, 2026
@JamesNK JamesNK closed this Mar 25, 2026
@JamesNK JamesNK reopened this Mar 25, 2026
@JamesNK JamesNK enabled auto-merge (squash) March 25, 2026 01:49
@JamesNK JamesNK merged commit db249e4 into main Mar 25, 2026
253 checks passed
@JamesNK JamesNK deleted the mad-skills branch March 25, 2026 02:31
@github-actions
Copy link
Contributor

🎬 CLI E2E Test Recordings — 49 recordings uploaded (commit cf7b8cd)

View recordings
Test Recording
AddPackageInteractiveWhileAppHostRunningDetached ▶️ View Recording
AddPackageWhileAppHostRunningDetached ▶️ View Recording
AgentCommands_AllHelpOutputs_AreCorrect ▶️ View Recording
AgentInitCommand_DefaultSelection_InstallsSkillOnly ▶️ View Recording
AgentInitCommand_MigratesDeprecatedConfig ▶️ View Recording
AspireAddPackageVersionToDirectoryPackagesProps ▶️ View Recording
AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps ▶️ View Recording
Banner_DisplayedOnFirstRun ▶️ View Recording
Banner_DisplayedWithExplicitFlag ▶️ View Recording
CertificatesClean_RemovesCertificates ▶️ View Recording
CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate ▶️ View Recording
CertificatesTrust_WithUntrustedCert_TrustsCertificate ▶️ View Recording
ConfigSetGet_CreatesNestedJsonFormat ▶️ View Recording
CreateAndRunAspireStarterProject ▶️ View Recording
CreateAndRunAspireStarterProjectWithBundle ▶️ View Recording
CreateAndRunEmptyAppHostProject ▶️ View Recording
CreateAndRunJsReactProject ▶️ View Recording
CreateAndRunPythonReactProject ▶️ View Recording
CreateAndRunTypeScriptEmptyAppHostProject ▶️ View Recording
CreateAndRunTypeScriptStarterProject ▶️ View Recording
CreateStartAndStopAspireProject ▶️ View Recording
CreateTypeScriptAppHostWithViteApp ▶️ View Recording
DescribeCommandResolvesReplicaNames ▶️ View Recording
DescribeCommandShowsRunningResources ▶️ View Recording
DetachFormatJsonProducesValidJson ▶️ View Recording
DoctorCommand_DetectsDeprecatedAgentConfig ▶️ View Recording
DoctorCommand_WithSslCertDir_ShowsTrusted ▶️ View Recording
DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted ▶️ View Recording
GlobalMigration_HandlesCommentsAndTrailingCommas ▶️ View Recording
GlobalMigration_HandlesMalformedLegacyJson ▶️ View Recording
GlobalMigration_PreservesAllValueTypes ▶️ View Recording
GlobalMigration_SkipsWhenNewConfigExists ▶️ View Recording
GlobalSettings_MigratedFromLegacyFormat ▶️ View Recording
InvalidAppHostPathWithComments_IsHealedOnRun ▶️ View Recording
LogsCommandShowsResourceLogs ▶️ View Recording
PsCommandListsRunningAppHost ▶️ View Recording
PsFormatJsonOutputsOnlyJsonToStdout ▶️ View Recording
PublishWithDockerComposeServiceCallbackSucceeds ▶️ View Recording
RestoreGeneratesSdkFiles ▶️ View Recording
RunWithMissingAwaitShowsHelpfulError ▶️ View Recording
SecretCrudOnDotNetAppHost ▶️ View Recording
SecretCrudOnTypeScriptAppHost ▶️ View Recording
StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels ▶️ View Recording
StopAllAppHostsFromAppHostDirectory ▶️ View Recording
StopAllAppHostsFromUnrelatedDirectory ▶️ View Recording
StopNonInteractiveMultipleAppHostsShowsError ▶️ View Recording
StopNonInteractiveSingleAppHost ▶️ View Recording
StopWithNoRunningAppHostExitsSuccessfully ▶️ View Recording
TypeScriptAppHostWithProjectReferenceIntegration ▶️ View Recording

📹 Recordings uploaded automatically from CI run #23520998239

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add dotnet inspect as an AI skill for Aspire APIs

7 participants