diff --git a/openspec/changes/add-verify-skill/design.md b/openspec/changes/add-verify-skill/design.md new file mode 100644 index 00000000..523438e5 --- /dev/null +++ b/openspec/changes/add-verify-skill/design.md @@ -0,0 +1,96 @@ +# Design: Add /opsx:verify Skill + +## Architecture Decision: Dynamic Generation via Setup Command + +### Context + +All existing opsx experimental skills (explore, new, continue, apply, ff, sync, archive) are dynamically generated when users run `openspec artifact-experimental-setup`. They are not manually created files checked into the repository. + +### Decision + +**Integrate verify into the existing artifact-experimental-setup system rather than creating static skill files.** + +### Rationale + +1. **Consistency**: All 7 existing opsx skills follow this pattern. Adding verify as the 8th skill should follow the same architecture. + +2. **Maintainability**: Template functions in `skill-templates.ts` are the single source of truth. Changes to skill definitions automatically propagate to all users when they re-run setup. + +3. **Distribution**: Users get the verify skill automatically when running `openspec artifact-experimental-setup`, just like all other opsx skills. No special installation steps needed. + +4. **Versioning**: Skills are generated from the installed npm package version, ensuring consistency between CLI version and skill behavior. + +### Implementation Approach + +#### 1. Template Functions + +Add two template functions to `src/core/templates/skill-templates.ts`: + +```typescript +export function getVerifyChangeSkillTemplate(): SkillTemplate +export function getOpsxVerifyCommandTemplate(): CommandTemplate +``` + +These return the skill definition (for Agent Skills) and slash command definition (for explicit invocation). + +#### 2. Setup Integration + +Update `artifactExperimentalSetupCommand()` in `src/commands/artifact-workflow.ts`: + +- Import both template functions +- Add verify to the `skills` array (position 8) +- Add verify to the `commands` array (position 8) +- Update help text to list `/opsx:verify` + +#### 3. Generated Artifacts + +When users run `openspec artifact-experimental-setup`, the command creates: + +- `.claude/skills/openspec-verify-change/SKILL.md` - Agent Skills format +- `.claude/commands/opsx/verify.md` - Slash command format + +Both are generated from the template functions, with YAML frontmatter automatically added. + +### Alternatives Considered + +**Alternative 1: Static skill files in repository** + +Create `.claude/skills/openspec-verify-change/SKILL.md` as a static file in the OpenSpec repository. + +**Rejected because:** +- Inconsistent with all other opsx skills +- Requires users to manually copy/update files +- Versioning becomes complicated (repo version vs installed package version) +- Breaks the established pattern + +**Alternative 2: Separate verify setup command** + +Add `openspec setup-verify` as a separate command. + +**Rejected because:** +- Fragments the setup experience +- Users would need to run multiple commands +- Doesn't scale if we add more skills in the future +- Goes against the "setup once, get everything" philosophy + +### Trade-offs + +**Advantages:** +- Consistent with existing architecture +- Zero additional setup burden for users +- Easy to update and maintain +- Automatic version compatibility + +**Disadvantages:** +- Slightly more complex initial implementation (template functions + integration) +- Requires understanding the setup system (but that's already documented) + +### Verification + +The implementation correctly follows this design if: + +1. Both template functions exist in `skill-templates.ts` +2. Verify appears in both skills and commands arrays in `artifact-workflow.ts` +3. Help text mentions `/opsx:verify` +4. Running `openspec artifact-experimental-setup` generates both skill and command files +5. Build succeeds with no TypeScript errors diff --git a/openspec/changes/add-verify-skill/proposal.md b/openspec/changes/add-verify-skill/proposal.md index 576ee30e..00f49198 100644 --- a/openspec/changes/add-verify-skill/proposal.md +++ b/openspec/changes/add-verify-skill/proposal.md @@ -12,9 +12,12 @@ A user requested: "Can we get a :verify that will ensure that the implementation ## What Changes -- Add new `/opsx:verify` slash command skill +- Add `getVerifyChangeSkillTemplate()` function to `skill-templates.ts` +- Add `getOpsxVerifyCommandTemplate()` function to `skill-templates.ts` +- Integrate verify skill into `artifactExperimentalSetupCommand` in `artifact-workflow.ts` +- Add verify to the skills and commands arrays in the setup command +- Update help text to include `/opsx:verify` in the list of available commands - Create `opsx-verify-skill` capability spec -- Create `SKILL.md` file at `.claude/skills/openspec-verify-change/` ## Verification Dimensions @@ -36,5 +39,10 @@ Produces a prioritized report with: ## Impact - Affected specs: New `opsx-verify-skill` spec -- Affected code: New skill file at `.claude/skills/openspec-verify-change/SKILL.md` +- Affected code: + - `src/core/templates/skill-templates.ts` - Added 2 new template functions + - `src/commands/artifact-workflow.ts` - Integrated verify into experimental setup +- Generated artifacts: When users run `openspec artifact-experimental-setup`: + - Creates `.claude/skills/openspec-verify-change/SKILL.md` + - Creates `.claude/commands/opsx/verify.md` - Related skills: Works alongside `/opsx:apply` and before `/opsx:archive` diff --git a/openspec/changes/add-verify-skill/tasks.md b/openspec/changes/add-verify-skill/tasks.md index 1d61abf3..e7b59158 100644 --- a/openspec/changes/add-verify-skill/tasks.md +++ b/openspec/changes/add-verify-skill/tasks.md @@ -1,28 +1,15 @@ # Tasks: Add /opsx:verify Skill -## 1. Skill Implementation -- [ ] 1.1 Create `.claude/skills/openspec-verify-change/SKILL.md` with skill definition - -## 2. Completeness Verification Logic -- [ ] 2.1 Implement task completion parsing (read tasks.md, count checkboxes) -- [ ] 2.2 Implement spec coverage detection (extract requirements, search codebase) - -## 3. Correctness Verification Logic -- [ ] 3.1 Implement requirement-to-code mapping logic -- [ ] 3.2 Implement scenario coverage assessment -- [ ] 3.3 Implement divergence detection heuristics - -## 4. Coherence Verification Logic -- [ ] 4.1 Implement design.md decision extraction -- [ ] 4.2 Implement design adherence checking -- [ ] 4.3 Implement project pattern consistency checking - -## 5. Report Generation -- [ ] 5.1 Implement summary scorecard generation -- [ ] 5.2 Implement issue prioritization (CRITICAL/WARNING/SUGGESTION) -- [ ] 5.3 Implement actionable recommendation generation - -## 6. Integration -- [ ] 6.1 Add opsx:verify alias registration -- [ ] 6.2 Test skill invocation end-to-end -- [ ] 6.3 Update documentation/help text if applicable +## 1. Skill Template Functions +- [x] 1.1 Add `getVerifyChangeSkillTemplate()` to skill-templates.ts +- [x] 1.2 Add `getOpsxVerifyCommandTemplate()` to skill-templates.ts + +## 2. Integration with artifact-experimental-setup +- [x] 2.1 Import verify template functions in artifact-workflow.ts +- [x] 2.2 Add verify to skills array in artifactExperimentalSetupCommand +- [x] 2.3 Add verify to commands array in artifactExperimentalSetupCommand +- [x] 2.4 Add verify to help text output + +## 3. Verification (Build & Test) +- [x] 3.1 Verify TypeScript compilation succeeds +- [x] 3.2 Verify all 8 skills are now included (was 7, now 8) diff --git a/src/commands/artifact-workflow.ts b/src/commands/artifact-workflow.ts index 736073a4..4e902d2f 100644 --- a/src/commands/artifact-workflow.ts +++ b/src/commands/artifact-workflow.ts @@ -28,7 +28,7 @@ import { type SchemaInfo, } from '../core/artifact-graph/index.js'; import { createChange, validateChangeName } from '../utils/change-utils.js'; -import { getExploreSkillTemplate, getNewChangeSkillTemplate, getContinueChangeSkillTemplate, getApplyChangeSkillTemplate, getFfChangeSkillTemplate, getSyncSpecsSkillTemplate, getArchiveChangeSkillTemplate, getOpsxExploreCommandTemplate, getOpsxNewCommandTemplate, getOpsxContinueCommandTemplate, getOpsxApplyCommandTemplate, getOpsxFfCommandTemplate, getOpsxSyncCommandTemplate, getOpsxArchiveCommandTemplate } from '../core/templates/skill-templates.js'; +import { getExploreSkillTemplate, getNewChangeSkillTemplate, getContinueChangeSkillTemplate, getApplyChangeSkillTemplate, getFfChangeSkillTemplate, getSyncSpecsSkillTemplate, getArchiveChangeSkillTemplate, getVerifyChangeSkillTemplate, getOpsxExploreCommandTemplate, getOpsxNewCommandTemplate, getOpsxContinueCommandTemplate, getOpsxApplyCommandTemplate, getOpsxFfCommandTemplate, getOpsxSyncCommandTemplate, getOpsxArchiveCommandTemplate, getOpsxVerifyCommandTemplate } from '../core/templates/skill-templates.js'; import { FileSystemUtils } from '../utils/file-system.js'; // ----------------------------------------------------------------------------- @@ -800,6 +800,7 @@ async function artifactExperimentalSetupCommand(): Promise { const ffChangeSkill = getFfChangeSkillTemplate(); const syncSpecsSkill = getSyncSpecsSkillTemplate(); const archiveChangeSkill = getArchiveChangeSkillTemplate(); + const verifyChangeSkill = getVerifyChangeSkillTemplate(); // Get command templates const exploreCommand = getOpsxExploreCommandTemplate(); @@ -809,6 +810,7 @@ async function artifactExperimentalSetupCommand(): Promise { const ffCommand = getOpsxFfCommandTemplate(); const syncCommand = getOpsxSyncCommandTemplate(); const archiveCommand = getOpsxArchiveCommandTemplate(); + const verifyCommand = getOpsxVerifyCommandTemplate(); // Create skill directories and SKILL.md files const skills = [ @@ -819,6 +821,7 @@ async function artifactExperimentalSetupCommand(): Promise { { template: ffChangeSkill, dirName: 'openspec-ff-change' }, { template: syncSpecsSkill, dirName: 'openspec-sync-specs' }, { template: archiveChangeSkill, dirName: 'openspec-archive-change' }, + { template: verifyChangeSkill, dirName: 'openspec-verify-change' }, ]; const createdSkillFiles: string[] = []; @@ -850,6 +853,7 @@ ${template.instructions} { template: ffCommand, fileName: 'ff.md' }, { template: syncCommand, fileName: 'sync.md' }, { template: archiveCommand, fileName: 'archive.md' }, + { template: verifyCommand, fileName: 'verify.md' }, ]; const createdCommandFiles: string[] = []; @@ -908,6 +912,7 @@ ${template.content} console.log(' • /opsx:apply - Implement tasks'); console.log(' • /opsx:ff - Fast-forward: create all artifacts at once'); console.log(' • /opsx:sync - Sync delta specs to main specs'); + console.log(' • /opsx:verify - Verify implementation matches artifacts'); console.log(' • /opsx:archive - Archive a completed change'); console.log(); console.log(chalk.yellow('💡 This is an experimental feature.')); diff --git a/src/core/templates/skill-templates.ts b/src/core/templates/skill-templates.ts index 9c331a5d..f5f0b488 100644 --- a/src/core/templates/skill-templates.ts +++ b/src/core/templates/skill-templates.ts @@ -1785,6 +1785,174 @@ Main specs are now updated. The change remains active - archive when implementat }; } +/** + * Template for openspec-verify-change skill + * For verifying implementation matches change artifacts before archiving + */ +export function getVerifyChangeSkillTemplate(): SkillTemplate { + return { + name: 'openspec-verify-change', + description: 'Verify implementation matches change artifacts. Use when the user wants to validate that implementation is complete, correct, and coherent before archiving.', + instructions: `Verify that an implementation matches the change artifacts (specs, tasks, design). + +**Input**: Optionally specify a change name. If omitted, MUST prompt for available changes. + +**Steps** + +1. **If no change name provided, prompt for selection** + + Run \`openspec list --json\` to get available changes. Use the **AskUserQuestion tool** to let the user select. + + Show changes that have implementation tasks (tasks artifact exists). + Include the schema used for each change if available. + Mark changes with incomplete tasks as "(In Progress)". + + **IMPORTANT**: Do NOT guess or auto-select a change. Always let the user choose. + +2. **Check status to understand the schema** + \`\`\`bash + openspec status --change "" --json + \`\`\` + Parse the JSON to understand: + - \`schemaName\`: The workflow being used (e.g., "spec-driven", "tdd") + - Which artifacts exist for this change + +3. **Get the change directory and load artifacts** + + \`\`\`bash + openspec instructions apply --change "" --json + \`\`\` + + This returns the change directory and context files. Read all available artifacts from \`contextFiles\`. + +4. **Initialize verification report structure** + + Create a report structure with three dimensions: + - **Completeness**: Track tasks and spec coverage + - **Correctness**: Track requirement implementation and scenario coverage + - **Coherence**: Track design adherence and pattern consistency + + Each dimension can have CRITICAL, WARNING, or SUGGESTION issues. + +5. **Verify Completeness** + + **Task Completion**: + - If tasks.md exists in contextFiles, read it + - Parse checkboxes: \`- [ ]\` (incomplete) vs \`- [x]\` (complete) + - Count complete vs total tasks + - If incomplete tasks exist: + - Add CRITICAL issue for each incomplete task + - Recommendation: "Complete task: " or "Mark as done if already implemented" + + **Spec Coverage**: + - If delta specs exist in \`openspec/changes//specs/\`: + - Extract all requirements (marked with "### Requirement:") + - For each requirement: + - Search codebase for keywords related to the requirement + - Assess if implementation likely exists + - If requirements appear unimplemented: + - Add CRITICAL issue: "Requirement not found: " + - Recommendation: "Implement requirement X: " + +6. **Verify Correctness** + + **Requirement Implementation Mapping**: + - For each requirement from delta specs: + - Search codebase for implementation evidence + - If found, note file paths and line ranges + - Assess if implementation matches requirement intent + - If divergence detected: + - Add WARNING: "Implementation may diverge from spec:
" + - Recommendation: "Review : against requirement X" + + **Scenario Coverage**: + - For each scenario in delta specs (marked with "#### Scenario:"): + - Check if conditions are handled in code + - Check if tests exist covering the scenario + - If scenario appears uncovered: + - Add WARNING: "Scenario not covered: " + - Recommendation: "Add test or implementation for scenario: " + +7. **Verify Coherence** + + **Design Adherence**: + - If design.md exists in contextFiles: + - Extract key decisions (look for sections like "Decision:", "Approach:", "Architecture:") + - Verify implementation follows those decisions + - If contradiction detected: + - Add WARNING: "Design decision not followed: " + - Recommendation: "Update implementation or revise design.md to match reality" + - If no design.md: Skip design adherence check, note "No design.md to verify against" + + **Code Pattern Consistency**: + - Review new code for consistency with project patterns + - Check file naming, directory structure, coding style + - If significant deviations found: + - Add SUGGESTION: "Code pattern deviation:
" + - Recommendation: "Consider following project pattern: " + +8. **Generate Verification Report** + + **Summary Scorecard**: + \`\`\` + ## Verification Report: + + ### Summary + | Dimension | Status | + |--------------|------------------| + | Completeness | X/Y tasks, N reqs| + | Correctness | M/N reqs covered | + | Coherence | Followed/Issues | + \`\`\` + + **Issues by Priority**: + + 1. **CRITICAL** (Must fix before archive): + - Incomplete tasks + - Missing requirement implementations + - Each with specific, actionable recommendation + + 2. **WARNING** (Should fix): + - Spec/design divergences + - Missing scenario coverage + - Each with specific recommendation + + 3. **SUGGESTION** (Nice to fix): + - Pattern inconsistencies + - Minor improvements + - Each with specific recommendation + + **Final Assessment**: + - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." + - If only warnings: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." + - If all clear: "All checks passed. Ready for archive." + +**Verification Heuristics** + +- **Completeness**: Focus on objective checklist items (checkboxes, requirements list) +- **Correctness**: Use keyword search, file path analysis, reasonable inference - don't require perfect certainty +- **Coherence**: Look for glaring inconsistencies, don't nitpick style +- **False Positives**: When uncertain, prefer SUGGESTION over WARNING, WARNING over CRITICAL +- **Actionability**: Every issue must have a specific recommendation with file/line references where applicable + +**Graceful Degradation** + +- If only tasks.md exists: verify task completion only, skip spec/design checks +- If tasks + specs exist: verify completeness and correctness, skip design +- If full artifacts: verify all three dimensions +- Always note which checks were skipped and why + +**Output Format** + +Use clear markdown with: +- Table for summary scorecard +- Grouped lists for issues (CRITICAL/WARNING/SUGGESTION) +- Code references in format: \`file.ts:123\` +- Specific, actionable recommendations +- No vague suggestions like "consider reviewing"` + }; +} + /** * Template for /opsx:archive slash command */ @@ -1964,3 +2132,172 @@ Target archive directory already exists. - If sync is requested, use /opsx:sync approach (agent-driven)` }; } + +/** + * Template for /opsx:verify slash command + */ +export function getOpsxVerifyCommandTemplate(): CommandTemplate { + return { + name: 'OPSX: Verify', + description: 'Verify implementation matches change artifacts before archiving', + category: 'Workflow', + tags: ['workflow', 'verify', 'experimental'], + content: `Verify that an implementation matches the change artifacts (specs, tasks, design). + +**Input**: Optionally specify \`--change \` after \`/opsx:verify\`. If omitted, MUST prompt for available changes. + +**Steps** + +1. **If no change name provided, prompt for selection** + + Run \`openspec list --json\` to get available changes. Use the **AskUserQuestion tool** to let the user select. + + Show changes that have implementation tasks (tasks artifact exists). + Include the schema used for each change if available. + Mark changes with incomplete tasks as "(In Progress)". + + **IMPORTANT**: Do NOT guess or auto-select a change. Always let the user choose. + +2. **Check status to understand the schema** + \`\`\`bash + openspec status --change "" --json + \`\`\` + Parse the JSON to understand: + - \`schemaName\`: The workflow being used (e.g., "spec-driven", "tdd") + - Which artifacts exist for this change + +3. **Get the change directory and load artifacts** + + \`\`\`bash + openspec instructions apply --change "" --json + \`\`\` + + This returns the change directory and context files. Read all available artifacts from \`contextFiles\`. + +4. **Initialize verification report structure** + + Create a report structure with three dimensions: + - **Completeness**: Track tasks and spec coverage + - **Correctness**: Track requirement implementation and scenario coverage + - **Coherence**: Track design adherence and pattern consistency + + Each dimension can have CRITICAL, WARNING, or SUGGESTION issues. + +5. **Verify Completeness** + + **Task Completion**: + - If tasks.md exists in contextFiles, read it + - Parse checkboxes: \`- [ ]\` (incomplete) vs \`- [x]\` (complete) + - Count complete vs total tasks + - If incomplete tasks exist: + - Add CRITICAL issue for each incomplete task + - Recommendation: "Complete task: " or "Mark as done if already implemented" + + **Spec Coverage**: + - If delta specs exist in \`openspec/changes//specs/\`: + - Extract all requirements (marked with "### Requirement:") + - For each requirement: + - Search codebase for keywords related to the requirement + - Assess if implementation likely exists + - If requirements appear unimplemented: + - Add CRITICAL issue: "Requirement not found: " + - Recommendation: "Implement requirement X: " + +6. **Verify Correctness** + + **Requirement Implementation Mapping**: + - For each requirement from delta specs: + - Search codebase for implementation evidence + - If found, note file paths and line ranges + - Assess if implementation matches requirement intent + - If divergence detected: + - Add WARNING: "Implementation may diverge from spec:
" + - Recommendation: "Review : against requirement X" + + **Scenario Coverage**: + - For each scenario in delta specs (marked with "#### Scenario:"): + - Check if conditions are handled in code + - Check if tests exist covering the scenario + - If scenario appears uncovered: + - Add WARNING: "Scenario not covered: " + - Recommendation: "Add test or implementation for scenario: " + +7. **Verify Coherence** + + **Design Adherence**: + - If design.md exists in contextFiles: + - Extract key decisions (look for sections like "Decision:", "Approach:", "Architecture:") + - Verify implementation follows those decisions + - If contradiction detected: + - Add WARNING: "Design decision not followed: " + - Recommendation: "Update implementation or revise design.md to match reality" + - If no design.md: Skip design adherence check, note "No design.md to verify against" + + **Code Pattern Consistency**: + - Review new code for consistency with project patterns + - Check file naming, directory structure, coding style + - If significant deviations found: + - Add SUGGESTION: "Code pattern deviation:
" + - Recommendation: "Consider following project pattern: " + +8. **Generate Verification Report** + + **Summary Scorecard**: + \`\`\` + ## Verification Report: + + ### Summary + | Dimension | Status | + |--------------|------------------| + | Completeness | X/Y tasks, N reqs| + | Correctness | M/N reqs covered | + | Coherence | Followed/Issues | + \`\`\` + + **Issues by Priority**: + + 1. **CRITICAL** (Must fix before archive): + - Incomplete tasks + - Missing requirement implementations + - Each with specific, actionable recommendation + + 2. **WARNING** (Should fix): + - Spec/design divergences + - Missing scenario coverage + - Each with specific recommendation + + 3. **SUGGESTION** (Nice to fix): + - Pattern inconsistencies + - Minor improvements + - Each with specific recommendation + + **Final Assessment**: + - If CRITICAL issues: "X critical issue(s) found. Fix before archiving." + - If only warnings: "No critical issues. Y warning(s) to consider. Ready for archive (with noted improvements)." + - If all clear: "All checks passed. Ready for archive." + +**Verification Heuristics** + +- **Completeness**: Focus on objective checklist items (checkboxes, requirements list) +- **Correctness**: Use keyword search, file path analysis, reasonable inference - don't require perfect certainty +- **Coherence**: Look for glaring inconsistencies, don't nitpick style +- **False Positives**: When uncertain, prefer SUGGESTION over WARNING, WARNING over CRITICAL +- **Actionability**: Every issue must have a specific recommendation with file/line references where applicable + +**Graceful Degradation** + +- If only tasks.md exists: verify task completion only, skip spec/design checks +- If tasks + specs exist: verify completeness and correctness, skip design +- If full artifacts: verify all three dimensions +- Always note which checks were skipped and why + +**Output Format** + +Use clear markdown with: +- Table for summary scorecard +- Grouped lists for issues (CRITICAL/WARNING/SUGGESTION) +- Code references in format: \`file.ts:123\` +- Specific, actionable recommendations +- No vague suggestions like "consider reviewing"` + }; +}