Skip to content

Commit 8fad1a1

Browse files
ashleyshawclaude
andcommitted
feat: Add comprehensive testing documentation and agent permissions system
## Testing Documentation ### PHP Testing Instructions (php-tests.instructions.md) - Complete PHPUnit testing guide for WordPress plugins (891 lines) - Test environment setup and configuration - Testing custom post types, taxonomies, and SCF - WordPress test framework utilities and factories - Mocking strategies and test fixtures - Code coverage configuration - Best practices and CI integration examples ### Playwright Testing Instructions (playwright-tests.instructions.md) - Comprehensive Playwright E2E testing guide (721 lines) - WordPress block editor testing patterns - Test file organization and configuration - Locator strategies and web-first assertions - Block insertion, configuration, and rendering tests - Debugging techniques and troubleshooting - Accessibility testing with aria snapshots - Page object model patterns ## Agent Permissions System ### Frontmatter Schema (frontmatter.schema.json) - Complete JSON schema for agent specification frontmatter - Permissions vocabulary with 12 approved values: - Core: read, write, execute, shell, filesystem, network - GitHub: github:repo, github:issues, github:pulls, github:workflows, github:checks, github:actions - Validation rules for all frontmatter fields - Metadata support for guardrails, rate limits, and dry-run capabilities ### Agent Spec Instructions Updates - Added comprehensive Permissions Vocabulary section (65 lines) - Documented each permission with use cases and examples - Permission-to-tool mapping guidelines - Principle of least privilege enforcement - Updated validation checklist with permissions requirements ### Agent Spec Updates - template.agent.md: Added permissions field, updated to v1.2 - generate-plugin.agent.md: Added permissions [read, write, execute, shell, filesystem] ## Documentation Alignment - All new instructions reference existing docs (jest-tests, testing-e2e, a11y) - Consistent formatting and structure across all testing guides - Cross-referenced with WPCS instructions and custom-instructions.md Phase 6.5: Testing Documentation & Permissions Framework 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent ca5c67f commit 8fad1a1

File tree

6 files changed

+1946
-6
lines changed

6 files changed

+1946
-6
lines changed

.github/agents/generate-plugin.agent.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
name: "Plugin Generator Agent"
33
description: Interactive agent that collects comprehensive requirements and generates a WordPress multi-block plugin with CPT, taxonomies, and SCF fields
44
tools: ["semantic_search", "read_file", "grep_search", "file_search", "run_in_terminal", "create_file", "update_file", "delete_file", "move_file"]
5+
permissions: ["read", "write", "execute", "shell", "filesystem"]
56
---
67

78
# Multi-Block Plugin Scaffold Generator

.github/agents/template.agent.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
---
22
name: "Template: Agent Specification"
33
description: "Standard specification template for LightSpeed multi-block plugin scaffold agents, covering role, tooling, inputs, outputs, and guardrails."
4-
version: "v1.1"
5-
last_updated: "YYYY-MM-DD"
4+
version: "v1.2"
5+
last_updated: "2025-12-16"
66
owners: ["LightSpeedWP Engineering"]
77
status: "draft"
88
apply_to: [".github/agents/*.agent.md"]
99
file_type: "template"
1010
tags: ["agent", "spec", "template", "copilot"]
1111
tools: ["Copilot Agents"]
12+
permissions: ["read"]
1213
metadata:
1314
guardrails: "Agents must never perform destructive or irreversible actions without explicit confirmation and must follow AGENTS.md."
1415
---

.github/instructions/agent-spec.instructions.md

Lines changed: 77 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,85 @@ Use this file when creating or updating `*.agent.md` files inside `.github/agent
2929
## Detailed Guidance
3030

3131
- **Template & metadata:** Copy the latest `.github/agents/template.agent.md`, update frontmatter values, and ensure references include `AGENTS.md` and `.github/instructions/agent-spec.instructions.md`. Use ISO `YYYY-MM-DD` dates.
32-
- **Role & scope first:** Clearly state the agent’s purpose, persona, supported workflows (blocks, generator, release), and explicit boundaries (no deployments, no production writes, no git pushes unless specified).
33-
- **Responsibilities & capabilities:** List only actions the team can support. Make limitations explicit (for example, “read-only for generated plugins”, “no database migrations”).
34-
- **Allowed tools:** Enumerate every tool, script, API, or CLI command the agent may call. Note required environment variables without revealing real values.
32+
- **Role & scope first:** Clearly state the agent's purpose, persona, supported workflows (blocks, generator, release), and explicit boundaries (no deployments, no production writes, no git pushes unless specified).
33+
- **Responsibilities & capabilities:** List only actions the team can support. Make limitations explicit (for example, "read-only for generated plugins", "no database migrations").
34+
- **Allowed tools:** Enumerate every tool, script, API, or CLI command the agent may call. Note required environment variables without revealing real values. Tools are explicitly listed in the `tools` frontmatter array.
35+
- **Permissions:** Define fine-grained permissions using the approved vocabulary (see Permissions Vocabulary section below). Grant permissions conservatively based on agent's actual needs. Include the `permissions` field in frontmatter alongside `tools`.
3536
- **Inputs & outputs:** Define accepted natural language prompts and structured inputs (JSON/YAML). Provide examples and, when useful, JSON Schema. Specify output formats, required fields, and error conventions for deterministic parsing.
3637
- **Safety guardrails:** Include confirmation rules, non-destructive defaults, rate limits, and escalation paths to humans. Align with OWASP practices and repository security expectations.
3738
- **Failure & rollback:** Document how to handle invalid input, tool failures, partial success, and any rollback or manual follow-up steps.
3839
- **Test tasks & observability:** Provide at least three validation tasks (normal, edge, failure). State logging expectations (timestamps, tool calls, external interactions) and privacy considerations.
3940
- **Changelog discipline:** Keep a changelog section in each spec. Update `version`, `last_updated`, and changelog entries whenever behaviour, tools, or guardrails change.
4041

42+
## Permissions Vocabulary
43+
44+
The `permissions` field gates what agents can access and modify. Use the approved enum values from `.github/schemas/frontmatter.schema.json`. Grant permissions conservatively and document why each is needed.
45+
46+
### Core Permissions
47+
48+
- **`read`** - Read files, directories, and repository content. Required for most agents that need to inspect code, configuration, or documentation.
49+
- **`write`** - Create, update, or delete files in the repository. Required for agents that generate code, update configurations, or modify documentation.
50+
- **`execute`** - Execute scripts, commands, or binaries. Required for agents that run build tools, tests, or generation scripts.
51+
- **`shell`** - Access to shell/terminal operations. Required for agents that need to run command-line tools like npm, composer, or git.
52+
- **`filesystem`** - Full filesystem access including directory operations. Required for agents that need to create/manage directory structures.
53+
- **`network`** - Make network requests to external services. Required for agents that fetch data from APIs, check URLs, or integrate with external tools.
54+
55+
### GitHub Permissions
56+
57+
- **`github:repo`** - Access repository information, read branches, tags, and commits. Required for agents working with repository metadata.
58+
- **`github:issues`** - Create, read, update GitHub issues. Required for reporting agents or issue triage automation.
59+
- **`github:pulls`** - Create, read, update pull requests. Required for release agents or PR automation.
60+
- **`github:workflows`** - Trigger or manage GitHub Actions workflows. Required for CI/CD integration agents.
61+
- **`github:checks`** - Read or create status checks. Required for quality gate agents.
62+
- **`github:actions`** - Manage GitHub Actions. Required for workflow management agents.
63+
64+
### Permission Examples
65+
66+
```yaml
67+
# Read-only documentation agent
68+
permissions: ["read"]
69+
70+
# Code quality agent (reads code, writes reports)
71+
permissions: ["read", "write", "execute", "filesystem"]
72+
73+
# Plugin generator agent (full file system access, runs scripts)
74+
permissions: ["read", "write", "execute", "shell", "filesystem"]
75+
76+
# Release agent (GitHub integration, runs commands)
77+
permissions: ["read", "write", "execute", "shell", "filesystem", "github:repo", "github:pulls", "github:workflows"]
78+
79+
# Reporting agent (reads data, creates issues)
80+
permissions: ["read", "github:repo", "github:issues"]
81+
```
82+
83+
### Permission Guidelines
84+
85+
1. **Principle of least privilege** - Only grant permissions actually needed
86+
2. **Document justification** - Explain why each permission is required in agent spec
87+
3. **Review regularly** - Audit permissions when agent capabilities change
88+
4. **Network access** - Requires explicit justification (external API calls, URL validation)
89+
5. **GitHub permissions** - Must document which GitHub features are accessed and why
90+
6. **Shell access** - Document which commands/tools are executed
91+
92+
### Adding Permissions to Specs
93+
94+
When updating or creating agent specifications, add the `permissions` field to frontmatter:
95+
96+
```markdown
97+
---
98+
name: "Example Agent"
99+
description: "Agent description"
100+
tools: ["read_file", "update_file", "run_in_terminal"]
101+
permissions: ["read", "write", "execute", "shell"]
102+
---
103+
```
104+
105+
The permissions array should align with the tools array:
106+
- `read_file` → requires `read` permission
107+
- `update_file`, `create_file` → requires `write` permission
108+
- `run_in_terminal` → requires `execute` and `shell` permissions
109+
- `github_api` → requires appropriate `github:*` permissions
110+
41111
## Examples
42112

43113
- **Frontmatter skeleton:**
@@ -64,10 +134,13 @@ Use this checklist before merging a new or updated agent spec:
64134

65135
- [ ] **Role & scope** – Purpose is unambiguous; boundaries are clear.
66136
- [ ] **Capabilities** – Actions are supportable; no implied powers.
67-
- [ ] **Tools** – All tools/APIs are explicitly listed with required permissions noted.
137+
- [ ] **Tools** – All tools/APIs are explicitly listed in `tools` frontmatter array.
138+
- [ ] **Permissions** – Permissions field is present with appropriate values from approved vocabulary; justification is clear.
139+
- [ ] **Tool-Permission alignment** – Permissions match the capabilities required by listed tools.
68140
- [ ] **Input/Output** – Schemas or formats are clear with examples; error handling is defined.
69141
- [ ] **Safety** – Guardrails reference AGENTS.md/SECURITY.md; confirmation rules exist for risky actions.
70142
- [ ] **Failure/Rollback** – Behaviour for partial failure and recovery is documented.
71143
- [ ] **Testing** – Includes at least one normal task, one edge case, and one failure case.
72144
- [ ] **Observability** – Logging and audit expectations are stated.
73145
- [ ] **Changelog & metadata** – Version, `last_updated`, owners, and status fields are current; changelog updated.
146+
- [ ] **Schema validation** – Run `npm run validate:agents` to ensure frontmatter passes schema validation.

0 commit comments

Comments
 (0)