skill-validator: bump per-plugin aggregate description cap from 15K to 20K (and document its informal origin)#655
Open
Evangelink wants to merge 2 commits into
Conversation
…o 20K The dotnet-test plugin (22 skills) is now at ~14K aggregate description characters and bumping into the 15,000 cap on every meaningful description change. This blocks legitimate plugin growth on what is, on inspection, an informal repo guardrail rather than a documented Copilot constraint. Origin of the 15K number Traced via 'git log -S MaxAggregateDescriptionLength' to PR dotnet#238, which references issue dotnet#222. The issue verbatim says: `users may apply a limit to the total size of 'description' across` `all skills in a plugin (...). 15K characters was mentioned, we` `could choose smaller.` So the value is informal — picked because it `was mentioned` and explicitly leaving room to `choose smaller`. Spec status The agentskills.io specification (https://agentskills.io/specification) documents per-skill description (1024), compatibility (500), and name (64) limits. It does NOT define any aggregate per-plugin cap. The `MaxDescriptionLength = 1024` constant in this same file already cites the spec; the aggregate constant carried no citation comment. What this PR does - Bumps `MaxAggregateDescriptionLength` from 15,000 to 20,000. Picks a round value that gives ~30% headroom over the largest current plugin (dotnet-test ~14K) without making the cap meaningless. - Adds a comment documenting the origin (informal, issue dotnet#222), the absence of a spec source, and a TODO to validate this guardrail against literature on skill-routing degradation and to run experiments (skill-validator already has the infrastructure to A/B-measure description noise vs activation accuracy). What this PR does NOT do - It does not remove the cap. The cost concern in dotnet#222 is real (every description is loaded at agent startup for routing); we just don't yet have evidence for the right ceiling. - It does not change per-skill limits. Validation All 545 tests pass; the existing `CheckCommandAggregateDescriptionTests` reference the constant symbolically and continue to pass with the new value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Note This PR is from a fork and modifies infrastructure files ( Changes to infrastructure typically need to be submitted from a branch in Please consider recreating this PR from an upstream branch. If you don't have push access to |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the skill-validator’s per-plugin aggregate description length guardrail to reduce friction as plugins (notably dotnet-test) grow, and documents the historical/unspecified nature of the aggregate cap relative to the agentskills.io spec.
Changes:
- Increase
MaxAggregateDescriptionLengthfrom 15,000 to 20,000 characters. - Add explanatory comments documenting that the aggregate cap is a local repo policy (not a spec limit) and capturing its informal origin + follow-up TODOs.
Show a summary per file
| File | Description |
|---|---|
| eng/skill-validator/src/Check/SkillProfiler.cs | Raises the aggregate description cap and adds context/TODO commentary about the cap’s origin and intent. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 1/1 changed files
- Comments generated: 1
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment on lines
+37
to
+40
| // spec defines per-skill description limits (1024 chars) but no aggregate | ||
| // limit. The original 15,000 was introduced in #238 / discussed in #222 | ||
| // ("15K characters was mentioned, we could choose smaller") as an informal | ||
| // guardrail against bloated metadata costs at startup. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bumps
MaxAggregateDescriptionLengthineng/skill-validator/src/Check/SkillProfiler.csfrom 15,000 → 20,000 characters and adds a comment block documenting the origin of the cap and a follow-up TODO.Why now
The
dotnet-testplugin (22 skills) is now at ~14K aggregate description characters and bumping into the 15,000 ceiling on every meaningful description change. Recent PRs (#647, #652) had to micro-trim wording purely to stay under the limit. The next plugin growth will hit the cap regardless of authoring intent.The cap is not a Copilot constraint
This is a local repo guardrail, not a documented Copilot/agentskills constraint. Evidence:
Spec status: the agentskills.io specification defines per-skill
description(1024),compatibility(500), andname(64) limits. It does not define any aggregate per-plugin cap. TheMaxDescriptionLength = 1024constant in this same file already cites the spec; the aggregate constant carried no citation comment.Origin:
git log -S MaxAggregateDescriptionLengthtraces to PR Enforce description size limits and add spec conformance checks #238, which references issue Measure/limit "description" material #222. Issue Measure/limit "description" material #222 verbatim:Picked because it "was mentioned", with the issue itself acknowledging the value is approximate.
What this PR does
dotnet-test~14K) without making the cap meaningless.What we should investigate next (not this PR)
skill-validatoralready has the infrastructure to A/B measure activation accuracy with and without noise (see the--noise-skills-dirflag). We could deliberately inflate descriptions and measure the activation regression curve, then set the cap at an empirically-derived inflection point.The cost concern in #222 is real (every description is loaded at startup for routing) — we just do not have evidence for the right ceiling. Until that evidence exists, the cap should be loose enough not to obstruct legitimate authoring.
What this PR does NOT do
Validation
dotnet build eng/skill-validator/srcpasses.dotnet test eng/skill-validator/tests545/545 pass, including the fourCheckCommandAggregateDescriptionTestscases (they reference the constant symbolically and continue to verifyUnder/At/Over/MultiplePluginsbehaviour).