Proposal: GEPA prompt optimization and template generation in spec-kit

[adam-paterson]

Summary
Use GEPA to turn our handwritten prompts into managed assets with measurable quality, cost, and latency. On each PR, run evals on changed prompts, compare to baselines, and block merges on significant regressions. Nightly jobs refresh baselines and highlight drift.

Why Now

• Handwritten prompts drift and regress silently.
• Contributors lack feedback loops to know if edits help or harm.
• We can standardize structure, evaluation, and optimization with low friction.

Scope

• Prompts only (this repo). No app code changes required.
• Models: start with one pinned model, expand later.
• Evals: small, representative datasets per prompt.

Deliverables

• GEPA config and prompt pack format (YAML front‑matter + Markdown body):
  • id, intent, owners, inputs schema, expected signals, model pin, version.
• Eval fixtures per prompt (inputs.jsonl) and rubrics (LLM‑as‑judge + checks).
• Optimization sweeps:
  • Variants: wording, structure, examples/few‑shot, system vs. user split.
  • Params: temperature, top_p, presence/frequency penalties.
• Reports and baselines:
  • JSON metrics (quality score, pass rate, cost, latency, variance).
  • Markdown summary for humans; stored under artifacts and baselines/.

CI Integration

• On PRs touching prompts: run evals only for changed prompts, compare to baseline, post summary comment, fail on hard regressions.
• Nightly on main: re-run all prompts, refresh baselines, open issue on drift.

Metrics

• Quality: rubric score (clarity, correctness, completeness, format/schema).
• Stability: variance across k samples (e.g., k=3) and seed sensitivity.
• Cost: input/output tokens and $.
• Latency: p50/p95 end‑to‑end.
• Gate thresholds: e.g., quality −2% warn, −5% fail; cost +20% warn; latency +30% warn.

Repo Structure Changes

• prompts//.md (with YAML front‑matter)
• evals//
  • inputs.jsonl (test cases)
  • rubric.yml (criteria + weights)
  • expectations.md (optional qualitative notes)
• gepa.yaml (global config)
• baselines//.json (frozen results)
• .github/workflows/gepa.yml (CI)

Typical Workflow

1. Contributor edits a prompt or adds a variant file.
2. Local: run spec-kit gepa generate to spot-check outputs.
3. Open PR. CI runs gepa eval on changed IDs; compares to baselines.
4. CI posts a comment with deltas and attaches report.md + raw JSON.
5. If fail thresholds triggered, contributor iterates or adjusts rubric if justified.
6. Merge updates baseline on main via nightly job or explicit “Update baseline” label.

Optimization Loop

• gepa optimize sweeps variants/params and selects “best” under constraints (min cost, target quality).
• Accepted variant replaces canonical prompt; baseline updates accordingly.

Risks & Mitigations

• Model drift: pin model versions; nightly baselines and drift reports.
• Flakiness: low temperature, multiple samples with CI median, stable inputs.
• Secrets/cost: use org secrets; cap max runs; allow “dry-run” on forks.
• Overfitting: keep holdout evals; rotate datasets periodically.

Open Questions

• Initial prompt set to pilot (suggest 3–5 high-traffic prompts).
• Baseline policy: manual approval vs. nightly auto-refresh?
• First model/provider to pin (and token budget per run)?

Rollout

1. Land structure + gepa.yaml + CI workflow scaffolding.
2. Pilot 3–5 prompts with evals and thresholds.
3. Enable PR gating and reports.
4. Expand coverage; add optimization sweeps on demand.