Usage Metrics Collection (Phase 1 + 2): Track workflow costs per task and stage

[landovsky]

Summary

Implements complete usage metrics collection for the development workflow. Tracks token usage, costs, and duration for each workflow stage (analyst, planner, implementer, reviewer) with git-persisted JSONL storage and human-readable beads comments.

Addresses: Complete blindness to workflow costs (9-10/10 pain identified in validation)

Implementation

Phase 1: Infrastructure

• ✅ SubagentStart/SubagentStop hook scripts
• ✅ JSONL persistence (.claude/workflow-metrics.jsonl)
• ✅ BD comment integration for per-task summaries
• ✅ Portable date handling (macOS/Linux compatibility)
• ✅ Robust error handling (exit 0 always, never breaks workflow)

Phase 2: Real Token Data

• ✅ Agent transcript parsing for token counts
• ✅ Model-specific cost calculation (5 pricing tiers: Opus 4.5, Opus 4, Sonnet 4, Haiku 4.5, Haiku 3.5)
• ✅ Real usage data: input/output tokens, cache read/creation
• ✅ Model name extraction from API responses
• ✅ Backward compatible (same JSONL schema, graceful fallback)

Use Cases Enabled

1. Benchmarking: Compare workflow vs fast-track costs to decide if overhead is worth it
2. Token leak detection: Spot anomalies (e.g., task used 5x normal tokens) → investigate bugs
3. Understanding distribution: Which stages cost most? → inform optimization decisions

Data Format

JSONL Entry:

{
  "event": "stage_end",
  "timestamp": "2026-02-03T14:25:32Z",
  "stage": "planner",
  "task": "Plan OTLP data integration",
  "duration_seconds": 107,
  "tokens": {
    "input": 12500,
    "output": 3200,
    "cache_read": 8000,
    "cache_creation": 15000
  },
  "cost_usd": 0.0523,
  "model": "claude-sonnet-4-5-20250929"
}

Query Examples:

# Total cost by stage
jq -s 'group_by(.stage) | map({stage: .[0].stage, total_cost: (map(.cost_usd // 0) | add)})' .claude/workflow-metrics.jsonl

# Average duration per stage
jq -s 'group_by(.stage) | map({stage: .[0].stage, avg_duration: (map(.duration_seconds) | add / length)})' .claude/workflow-metrics.jsonl

# Most expensive tasks
jq 'select(.event=="stage_end")' .claude/workflow-metrics.jsonl | jq -s 'sort_by(-.cost_usd) | .[0:10]'

Files Changed

New:

• .claude/hooks/metrics-start.sh - Captures stage start events
• .claude/hooks/metrics-end.sh - Captures end events with duration, tokens, cost
• .claude/analysis/.claude-6t5.1-context.md - Phase 2 analyst context
• .claude/specs/.claude-6t5.1-spec.md - Phase 2 specification

Modified:

• .claude/agents/master.md - Added $TASK env var setup instructions
• .claude/commands/develop.md - Added usage metrics documentation
• artifacts/lessons-learned.md - Captured hook patterns, jq usage, transcript parsing

Note: settings.json not included (user-specific). Users must manually add hooks (documented in develop.md).

Testing

Phase 1:

• ✅ Unit tests (malformed JSON, missing TASK, truncation)
• ✅ Integration test (duration calculation: 19 seconds measured)
• ✅ Edge cases (blocked status, session interruption)

Phase 2:

• ✅ Token parsing (250 in / 125 out / 500 cache validated)
• ✅ Cost calculation ($0.0050 verified with manual math)
• ✅ Model pricing patterns (all 5 tiers + fallback)
• ✅ Edge cases (empty transcript, missing file, unknown model, large numbers)

Validation Process

• Validated through /validate workflow
• Challenge Q&A confirmed high ROI (high value, small effort, low risk)
• Scoped to Phase 1+2 only (no dashboards/automation yet)
• Full workflow execution: analyst → planner → implementer → reviewer (both phases)

Known Limitations

• Fast-track tasks (master only) not tracked - only subagent stages
• Concurrent subagents not tested - sequential workflow only
• Requires manual settings.json update per user

Merge Checklist

• All acceptance criteria met
• Tests passed (unit + integration)
• Code reviewed and approved
• Documentation complete
• Backward compatible (Phase 1 → Phase 2 seamless)
• No breaking changes

Related Issues

• Closes .claude-l0a (Phase 1: Usage data collection infrastructure)
• Closes .claude-6t5 (Phase 2: Real OTLP token data integration)

---

🤖 Generated through full /develop workflow with analyst, planner, implementer, and reviewer agents.