diff --git a/.github/README.md b/.github/README.md index c9518072..7e016089 100644 --- a/.github/README.md +++ b/.github/README.md @@ -348,6 +348,19 @@ Directory Structure Details │ ├── project-meta-sync.yml │ └── ... (other workflows) │ +├── metrics/ # Metrics collection scripts and configuration +│ ├── README.md +│ ├── metrics.config.json +│ ├── frontmatter-metrics.js +│ ├── branding.json +│ └── branding-log.md +│ +├── reporting/ # Generated reports and metrics artifacts +│ ├── README.md +│ ├── frontmatter/ +│ ├── branding/ +│ └── issue-metrics/ +│ ├── ISSUE_TEMPLATE/ # Issue templates for bugs, features, docs, tasks, etc. │ └── *.md │ diff --git a/.github/agents/README.md b/.github/agents/README.md index a16b8bf7..e9cf835b 100644 --- a/.github/agents/README.md +++ b/.github/agents/README.md @@ -272,6 +272,8 @@ graph TD | **Automation Governance** | Workflow and agent policies | [../AUTOMATION_GOVERNANCE.md](../AUTOMATION_GOVERNANCE.md) | | **Global AI Rules** | Organization-wide AI guidelines | [../../AGENTS.md](../../AGENTS.md) | | **Collections** | Curated agent toolkits | [../collections/](../collections/) | +| **Metrics Directory** | Metrics collection and configuration | [../metrics/](../metrics/) | +| **Reporting Directory** | Generated reports and artifacts | [../reporting/](../reporting/) | --- diff --git a/.github/metrics/README.md b/.github/metrics/README.md new file mode 100644 index 00000000..72e16aa2 --- /dev/null +++ b/.github/metrics/README.md @@ -0,0 +1,377 @@ +--- +title: 'LightSpeed Metrics Directory' +version: 'v1.0' +last_updated: '2025-11-17' +author: 'LightSpeed' +maintainer: 'Ash Shaw' +description: 'Metrics collection scripts, configuration, and automation for tracking repository health, documentation quality, and project activity.' +tags: ['metrics', 'analytics', 'automation', 'monitoring', 'quality'] +file_type: 'documentation' +category: 'infrastructure' +references: + - path: '../reporting/README.md' + description: 'Reporting documentation' + - path: '../agents/metrics.agent.md' + description: 'Metrics agent configuration' + - path: '../../docs/BRANDING.md' + description: 'Branding guidelines' + - path: '../workflows/branding.yml' + description: 'Branding workflow' + - path: '../workflows/issue_metrics.yml' + description: 'Issue metrics workflow' +--- + +# LightSpeed Metrics Directory + +This directory contains metrics collection scripts, configuration files, and automation logic for tracking repository health, documentation quality, and project activity across the LightSpeed organization. + +## Purpose + +- **Metrics Collection**: Scripts and tools for gathering metrics from various sources +- **Configuration Management**: Centralized configuration for all metrics collection +- **Data Validation**: Schema enforcement and quality checks for collected data +- **Automation Integration**: Hooks into workflows for scheduled and event-driven metrics + +## Directory Structure + +```text +.github/metrics/ +├── README.md # This file +├── metrics.config.json # Master configuration for all metrics +├── frontmatter-metrics.js # Frontmatter coverage and quality metrics +├── branding-log.md # Historical branding metrics log +├── branding.json # Latest branding metrics snapshot +└── out/ # Generated output (gitignored) + ├── frontmatter-metrics.json # Latest frontmatter report (JSON) + └── frontmatter-metrics.md # Latest frontmatter report (Markdown) +``` + +**Note**: The `out/` directory is typically gitignored. Generated reports are moved to `.github/reporting/` for version control and distribution. + +## Metrics Types + +### 1. Frontmatter Metrics + +**Script**: `frontmatter-metrics.js` + +**Purpose**: Validate and track frontmatter coverage across all markdown and YAML template files. + +**Collected Metrics**: +- Coverage percentage (valid frontmatter / eligible files) +- Unknown keys (schema violations) +- Broken references (invalid cross-links) +- Version skews (file version > repo version) + +**Configuration**: `metrics.config.json` → `includeGlobs`, `excludeGlobs`, `thresholds` + +**Outputs**: +- JSON artifact: `out/frontmatter-metrics.json` +- Markdown report: `out/frontmatter-metrics.md` + +**Frequency**: Weekly (Monday 03:00 UTC) + +### 2. Branding Metrics + +**Workflow**: `.github/workflows/branding.yml` + +**Purpose**: Track branding automation coverage and effectiveness. + +**Collected Metrics**: +- Coverage: Percentage of docs with branding applied +- Changes: Number of files modified in last run +- Errors: Number of errors encountered +- Opt-outs: Number of files opted out + +**Outputs**: +- Latest snapshot: `branding.json` +- Historical log: `branding-log.md` + +**Frequency**: Weekly (Monday 03:00 UTC) + +### 3. Issue & PR Metrics (Future) + +**Agent Spec**: `.github/agents/metrics.agent.md` + +**Purpose**: Repository health and activity metrics. + +**Planned Metrics**: +- Open/closed issue counts +- PR response times +- Review turnaround +- Project velocity + +**Status**: Planned (see agent spec) + +## Configuration + +### Master Config: `metrics.config.json` + +```json +{ + "includeGlobs": ["**/*.md", ".github/ISSUE_TEMPLATE/*.yml"], + "excludeGlobs": ["**/node_modules/**", "**/.git/**", "**/CHANGELOG.md"], + "frontmatterEligible": { + "md": true, + "issue_template": true, + "pr_template": false, + "discussion_template": true + }, + "thresholds": { + "coveragePctMin": 90, + "unknownKeysMax": 0, + "brokenRefsMax": 0, + "versionSkewMax": 0 + }, + "report": { + "issueTitle": "Weekly Frontmatter Metrics", + "storeJsonArtifact": true, + "artifactPath": "metrics/out/frontmatter-metrics.json", + "reportPath": "metrics/out/frontmatter-metrics.md" + }, + "version": { + "repoVersionFile": "VERSION", + "enforceFileNotAboveRepo": true + } +} +``` + +### Configuration Fields + +| Field | Type | Purpose | +|-------|------|---------| +| `includeGlobs` | array | File patterns to include in metrics | +| `excludeGlobs` | array | File patterns to exclude from metrics | +| `frontmatterEligible` | object | File types expected to have frontmatter | +| `thresholds` | object | Quality gates and failure conditions | +| `report` | object | Output paths and artifact configuration | +| `version` | object | Version enforcement rules | + +## Usage + +### Running Metrics Locally + +**Frontmatter Metrics**: +```bash +# Run from repository root +node .github/metrics/frontmatter-metrics.js + +# Outputs: +# - metrics/out/frontmatter-metrics.json +# - metrics/out/frontmatter-metrics.md +``` + +**Branding Metrics**: +```bash +# Triggered via workflow +gh workflow run branding.yml --ref develop + +# Or manually dispatch in GitHub Actions UI +``` + +### Automated Collection + +Metrics are automatically collected via GitHub Actions workflows: + +1. **Weekly Schedule**: Every Monday at 03:00 UTC +2. **Push Events**: On push to `develop` branch +3. **Manual Dispatch**: Via `workflow_dispatch` trigger + +See `.github/workflows/branding.yml` for automation details. + +### Consuming Metrics + +**From JSON Artifacts**: +```javascript +const fs = require('fs'); +const metrics = JSON.parse( + fs.readFileSync('.github/metrics/out/frontmatter-metrics.json', 'utf8') +); + +console.log(`Coverage: ${metrics.summary.coveragePct}%`); +console.log(`Broken refs: ${metrics.summary.brokenRefs}`); +``` + +**From Markdown Reports**: +- Include in weekly status updates +- Link in project dashboards +- Reference in governance reviews + +## Thresholds and Quality Gates + +Metrics can fail CI/CD builds if thresholds are exceeded: + +| Metric | Threshold | Action | +|--------|-----------|--------| +| Coverage | < 90% | Warning (configurable to fail) | +| Unknown keys | > 0 | Fail build | +| Broken refs | > 0 | Fail build | +| Version skews | > 0 | Fail build | + +Configure thresholds in `metrics.config.json` → `thresholds`. + +Enable build failures with `thresholds.failOnError: true`. + +## Integration + +### Workflow Integration + +Metrics scripts integrate with GitHub Actions workflows: + +```yaml +# Example: .github/workflows/branding.yml +jobs: + metrics-update: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run frontmatter metrics + run: node .github/metrics/frontmatter-metrics.js + - name: Move reports to reporting directory + run: | + mkdir -p .github/reporting/frontmatter + mv metrics/out/frontmatter-metrics.* .github/reporting/frontmatter/ + - name: Commit metrics + run: | + git add .github/reporting + git commit -m "chore: update metrics [skip ci]" +``` + +### Dashboard Integration + +Metrics can feed external dashboards: + +- **PowerBI**: Import JSON artifacts +- **Grafana**: Parse JSON for time-series visualization +- **Custom Dashboards**: Fetch via GitHub API + +### Alert Integration + +Configure alerts based on threshold violations: + +```yaml +# Example: Slack notification on threshold failure +- name: Notify on failure + if: failure() + uses: slackapi/slack-github-action@v1 + with: + payload: | + { + "text": "Metrics thresholds failed in ${{ github.repository }}" + } +``` + +## Development + +### Adding New Metrics + +1. **Create Collection Script**: Add new script in `.github/metrics/` +2. **Update Config**: Add configuration to `metrics.config.json` +3. **Define Output**: Specify artifact and report paths +4. **Integrate Workflow**: Update or create workflow in `.github/workflows/` +5. **Document**: Update this README with metric details +6. **Test**: Run locally and validate outputs + +### Testing Metrics + +```bash +# Test frontmatter metrics locally +node .github/metrics/frontmatter-metrics.js + +# Validate output schema +npx ajv validate -s schemas/metrics-output.schema.json \ + -d metrics/out/frontmatter-metrics.json + +# Run with test fixtures +TEST_MODE=true node .github/metrics/frontmatter-metrics.js +``` + +### Debugging + +```bash +# Enable verbose logging +DEBUG=metrics:* node .github/metrics/frontmatter-metrics.js + +# Dry run (no file writes) +DRY_RUN=true node .github/metrics/frontmatter-metrics.js + +# Test specific file patterns +node .github/metrics/frontmatter-metrics.js --include="docs/**/*.md" +``` + +## Best Practices + +- **Version Control Config**: Always commit `metrics.config.json` changes +- **Document Thresholds**: Explain rationale for threshold values +- **Test Before Deploy**: Run metrics locally before pushing changes +- **Schema Validation**: Validate JSON outputs against schemas +- **Incremental Changes**: Add metrics incrementally, not all at once +- **Monitor Performance**: Track execution time for metrics scripts +- **Archive Old Outputs**: Move historical data to `.github/reporting/archive/` + +## Troubleshooting + +**Script fails with "Cannot find module"**: +```bash +# Install dependencies +npm install +``` + +**Threshold failures causing build issues**: +```bash +# Review thresholds in metrics.config.json +# Adjust or fix underlying issues +# Set failOnError: false for warnings only +``` + +**Output files not generated**: +```bash +# Check output directory exists +mkdir -p metrics/out + +# Verify script permissions +chmod +x .github/metrics/frontmatter-metrics.js + +# Run with debug logging +DEBUG=* node .github/metrics/frontmatter-metrics.js +``` + +**Frontmatter validation errors**: +```bash +# Review schema: schemas/frontmatter.schema.json +# Validate individual file: +npx ajv validate -s schemas/frontmatter.schema.json -d path/to/file.md +``` + +## Related Resources + +| Resource | Purpose | Location | +|----------|---------|----------| +| **Reporting Directory** | Generated report outputs | [.github/reporting/](../reporting/) | +| **Metrics Agent Spec** | Future automated metrics agent | [.github/agents/metrics.agent.md](../agents/metrics.agent.md) | +| **Branding Workflow** | Branding metrics automation | [.github/workflows/branding.yml](../workflows/branding.yml) | +| **Frontmatter Schema** | Validation schema for frontmatter | [schemas/frontmatter.schema.json](../../schemas/frontmatter.schema.json) | +| **Automation Governance** | Metrics and reporting policies | [.github/automation/AUTOMATION_GOVERNANCE.md](../automation/AUTOMATION_GOVERNANCE.md) | + +## Future Enhancements + +See [.github/agents/metrics.agent.md](../agents/metrics.agent.md) for planned metrics automation: + +- Automated issue/PR metrics collection +- Multi-repo aggregation +- Real-time metrics dashboards +- Configurable alert thresholds +- Metrics API endpoint + +## Contributing + +To contribute new metrics or improvements: + +1. Review [CONTRIBUTING.md](../../CONTRIBUTING.md) +2. Follow [coding standards](../instructions/coding-standards.instructions.md) +3. Add tests for new metrics scripts +4. Document configuration changes +5. Submit PR with rationale and examples + +--- + +Made with ❤️ by the LightSpeed team. diff --git a/.github/reporting/README.md b/.github/reporting/README.md new file mode 100644 index 00000000..97a77903 --- /dev/null +++ b/.github/reporting/README.md @@ -0,0 +1,220 @@ +--- +title: 'LightSpeed Reporting Directory' +version: 'v1.0' +last_updated: '2025-11-17' +author: 'LightSpeed' +maintainer: 'Ash Shaw' +description: 'Central location for generated reports, metrics outputs, and analytics artifacts across LightSpeed repositories.' +tags: ['reporting', 'metrics', 'analytics', 'artifacts', 'automation'] +type: 'documentation' +category: 'infrastructure' +references: + - '../metrics/README.md' + - '../metrics/metrics.config.json' + - '../agents/metrics.agent.md' + - '../../docs/BRANDING.md' +--- + +# LightSpeed Reporting Directory + +This directory contains generated reports, metrics outputs, and analytics artifacts produced by automated workflows and metrics collection systems across the LightSpeed organization. + +## Purpose + +- **Centralized Reporting**: All automated reports and metrics artifacts are stored here for easy access and historical tracking +- **Metrics Output**: Generated reports from metrics collection scripts (frontmatter metrics, branding metrics, etc.) +- **Analytics Artifacts**: JSON and markdown outputs for consumption by dashboards, workflows, and CI/CD pipelines +- **Historical Tracking**: Timestamped logs and snapshots for trend analysis and compliance + +## Directory Structure + +```text +.github/reporting/ +├── README.md # This file +├── frontmatter/ # Frontmatter metrics reports +│ ├── frontmatter-metrics.json # Latest frontmatter metrics (JSON) +│ ├── frontmatter-metrics.md # Latest frontmatter metrics (Markdown) +│ └── archive/ # Historical frontmatter reports +├── branding/ # Branding metrics reports +│ ├── branding.json # Latest branding snapshot +│ ├── branding-log.md # Historical branding log +│ └── archive/ # Historical branding reports +├── coverage/ # Test coverage reports +│ └── ... +└── issue-metrics/ # Issue and PR metrics reports + └── ... +``` + +## Report Types + +### Frontmatter Metrics + +Generated by: `.github/metrics/frontmatter-metrics.js` + +**Purpose**: Track frontmatter coverage, unknown keys, broken references, and version skews across all documentation files. + +**Outputs**: +- `frontmatter/frontmatter-metrics.json` - Machine-readable metrics data +- `frontmatter/frontmatter-metrics.md` - Human-readable report + +**Frequency**: Weekly (Monday 03:00 UTC) via branding workflow + +### Branding Metrics + +Generated by: `.github/workflows/branding.yml` + +**Purpose**: Track branding automation coverage, changes, errors, and opt-outs. + +**Outputs**: +- `branding/branding.json` - Latest snapshot +- `branding/branding-log.md` - Historical log with timestamps + +**Frequency**: Weekly (Monday 03:00 UTC) via branding workflow + +### Issue Metrics + +Generated by: `.github/workflows/issue_metrics.yml` + +**Purpose**: Repository health metrics including open/closed issues, PR response times, and project activity. + +**Outputs**: +- `issue-metrics/weekly-report.md` - Weekly metrics report +- `issue-metrics/monthly-report.md` - Monthly rollup + +**Frequency**: Weekly and monthly + +## Usage + +### Accessing Reports + +All reports in this directory are committed to the repository and available via: + +1. **Direct File Access**: Browse this directory in GitHub +2. **Workflow Artifacts**: Download from GitHub Actions runs +3. **API Access**: Use GitHub API to fetch file contents programmatically + +### Consuming Reports + +**JSON Artifacts**: +```javascript +// Example: Fetch frontmatter metrics +const response = await fetch( + 'https://raw.githubusercontent.com/lightspeedwp/.github/develop/.github/reporting/frontmatter/frontmatter-metrics.json' +); +const metrics = await response.json(); +console.log(`Coverage: ${metrics.summary.coveragePct}%`); +``` + +**Markdown Reports**: +- Link directly in issues, PRs, or documentation +- Include in weekly status updates +- Reference in governance reviews + +### Report Retention + +- **Latest Reports**: Always available in root report directories +- **Historical Archives**: Moved to `archive/` subdirectories after new reports are generated +- **Retention Policy**: Keep last 12 months of archived reports + +## Configuration + +Report generation is controlled by: + +- **Metrics Config**: `.github/metrics/metrics.config.json` +- **Workflow Files**: `.github/workflows/branding.yml`, `.github/workflows/issue_metrics.yml` +- **Agent Specs**: `.github/agents/metrics.agent.md` + +See [Metrics Directory](../metrics/README.md) for collection scripts and configuration details. + +## Automation + +Reports are automatically generated by: + +1. **Scheduled Workflows**: Weekly/monthly GitHub Actions +2. **Event-Triggered**: Push to develop, PR merges +3. **Manual Dispatch**: `workflow_dispatch` for on-demand generation + +All automation follows [AUTOMATION_GOVERNANCE.md](../automation/AUTOMATION_GOVERNANCE.md) policies. + +## Integration + +### Dashboards + +Reports can be consumed by external dashboards and monitoring tools: + +- Metrics APIs +- BI tools (PowerBI, Tableau, etc.) +- Custom dashboards + +### Notifications + +Configure alerts based on report thresholds: + +- Slack/Discord webhooks +- Email notifications +- GitHub issue creation + +### CI/CD Pipelines + +Use reports as quality gates: + +```yaml +# Example: Fail build if frontmatter coverage < 90% +- name: Check frontmatter coverage + run: | + COVERAGE=$(jq '.summary.coveragePct' .github/reporting/frontmatter/frontmatter-metrics.json) + if [ "$COVERAGE" -lt 90 ]; then + echo "::error::Frontmatter coverage below threshold: $COVERAGE%" + exit 1 + fi +``` + +## Best Practices + +- **Never manually edit generated reports** - They will be overwritten on next run +- **Reference reports by permalink** - Use specific commit SHAs for stable links +- **Archive old reports** - Move to `archive/` to maintain clean directory structure +- **Document custom reports** - Add README sections for new report types +- **Validate JSON artifacts** - Ensure schema compliance before consumption + +## Troubleshooting + +**Report not generated:** +- Check workflow run status in Actions tab +- Verify metrics configuration in `.github/metrics/metrics.config.json` +- Review error logs in workflow runs + +**Report data looks incorrect:** +- Validate source data collection scripts +- Check for recent config changes +- Review thresholds and filters in metrics.config.json + +**Historical reports missing:** +- Check `archive/` subdirectories +- Review retention policy and cleanup workflows +- Verify git history for deleted files + +## Related Resources + +| Resource | Purpose | Location | +|----------|---------|----------| +| **Metrics Scripts** | Collection logic and configuration | [.github/metrics/](../metrics/) | +| **Metrics Agent Spec** | Future automated reporting agent | [.github/agents/metrics.agent.md](../agents/metrics.agent.md) | +| **Branding Docs** | Branding metrics documentation | [docs/BRANDING.md](../../docs/BRANDING.md) | +| **Automation Governance** | Reporting policies and standards | [.github/automation/AUTOMATION_GOVERNANCE.md](../automation/AUTOMATION_GOVERNANCE.md) | + +## Contributing + +To add new report types: + +1. Create metrics collection script in `.github/metrics/` +2. Add output path configuration to `metrics.config.json` +3. Update this README with report type documentation +4. Create workflow integration in `.github/workflows/` +5. Test end-to-end generation and consumption + +See [CONTRIBUTING.md](../../CONTRIBUTING.md) for general contribution guidelines. + +--- + +Made with ❤️ by the LightSpeed team. diff --git a/README.md b/README.md index d275edc2..a670c45b 100644 --- a/README.md +++ b/README.md @@ -813,6 +813,8 @@ This project is licensed under the GNU General Public License v3.0 - see the [LI - [🤖 Agent Specifications](./.github/agents/README.md) - Automation agents and workflows - [📝 Instructions Library](./.github/instructions/README.md) - Development standards and practices - [📚 Collections Directory](./.github/collections/README.md) - Curated instruction collections +- [📊 Metrics Directory](./.github/metrics/README.md) - Metrics collection and configuration +- [📈 Reporting Directory](./.github/reporting/README.md) - Generated reports and artifacts - [🔧 VS Code Configuration](./.vscode/README.md) - Development environment setup ### �🔄 Workflows & Governance diff --git a/docs/BRANDING.md b/docs/BRANDING.md index bd3715fa..43a067de 100644 --- a/docs/BRANDING.md +++ b/docs/BRANDING.md @@ -166,10 +166,15 @@ Made with ❤️ by the LightSpeed team. ## Metrics -Branding metrics are tracked in two files: +Branding metrics are collected in the [`.github/metrics/`](../.github/metrics/) directory and published to [`.github/reporting/`](../.github/reporting/): -- **[`.github/metrics/branding.json`](../.github/metrics/branding.json)** – Latest snapshot -- **[`.github/metrics/branding-log.md`](../.github/metrics/branding-log.md)** – Historical log +- **[`.github/metrics/branding.json`](../.github/metrics/branding.json)** – Latest snapshot (collection) +- **[`.github/metrics/branding-log.md`](../.github/metrics/branding-log.md)** – Historical log (collection) +- **[`.github/reporting/branding/`](../.github/reporting/branding/)** – Published reports and archives + +For complete metrics and reporting documentation, see: +- [Metrics Directory README](../.github/metrics/README.md) - Collection scripts and configuration +- [Reporting Directory README](../.github/reporting/README.md) - Published reports and consumption ### Metrics Tracked