Preliminary QA Checklist

[alethkit]

Severity model

• P0 (Stop-ship): could affect the wrong repo/org, expose secrets, brick first-run for typical users, or silently corrupt tutorial state.
• P1 (Release-blocking): common failure mode without workaround, or major UX confusion likely in workshops.
• P2 (Non-blocking): papercuts, edge cases with workaround, polish, performance.

---

P0 Gates (Stop-ship)

G0. Repo boundary safety (blast radius = quest repo only)

Goal: never touch repos other than the newly created quest repo.
Pass criteria:

• All writes (issues, PRs, branches) happen only under the created quest repo.
• If the user is in an org context, the tool does not “default” to orgs unexpectedly.
• Any ambiguity in target owner/repo results in a hard stop + explicit confirmation.

Test procedure (manual):

• Pre-existing repos in account/org (at least 2) + one repo with similar name to quest.
• Run quest creation; verify only the new quest repo changed (compare recent activity / audit trail).

Evidence to attach: URLs of quest repo + 2 unrelated repos showing no changes, terminal transcript.

---

G1. Secret handling (token safety)

Goal: tokens are never printed, logged, or written insecurely.
Pass criteria:

• Token is never echoed in terminal output, logs, panic traces, or error messages.
• If ~/.rqst-token is used: tool fails with actionable guidance when missing/invalid, and does not “helpfully” print file contents.
• Failure modes (expired token, insufficient scope, org SSO restrictions) yield actionable text.

Test procedure (manual):

• Invalid token, expired token, insufficient-scope token, and token that fails org/SSO.
• Confirm no token leakage in terminal transcript.

Evidence: terminal transcript + screenshots (if needed).

---

G2. First-run workshop success path (typical participant)

Goal: a new user can reach the first meaningful tutorial step in ≤ 10 minutes on supported platforms.
Pass criteria:

• Fresh machine / clean environment: install → auth → run → quest created → first instruction shown.
• No “silent failure” or ambiguous hang states.
• If a prerequisite is missing (git, gh, network), the error tells the user exactly what to do next.

Test procedure (manual):

• “Fresh user” path with GitHub CLI auth.
• “Fresh user” path with token file auth.
• Network on/off toggle mid-run (at least once).

Evidence: screen recording or transcript; created repo URL(s).

---

G3. Idempotency + safe re-run

Goal: re-running does not create confusing duplicate state or corrupt progression.
Pass criteria:

• Re-run after success: either resumes cleanly, or refuses with a clear message and recovery steps.
• Re-run after partial failure: safe retry without duplicating issues/PRs, or explicit recovery instructions.

Test procedure (manual):

• Kill the process mid-way (Ctrl+C) after repo creation, then re-run.
• Interrupt during issue/PR creation, then re-run.

Evidence: before/after repo state, list of issues/PRs, transcript.

---

P1 Gates (Release-blocking)

G4. Install pathways reliability

Goal: installs work the way docs say they work.
Pass criteria:

• Install script path works on supported platforms.
• cargo install ... --locked installs and produces a runnable repo-quest binary.
• repo-quest --help works post-install.

Test matrix:

• macOS ARM64
• macOS x86_64
• Linux x86_64
• Windows x86_64

Evidence: repo-quest --version, install logs, platform info.

---

G5. Auth pathway UX (gh vs token file)

Goal: auth errors are predictable and novice-friendly.
Pass criteria:

• If gh missing: message points to installing gh.
• If gh auth login not done: message instructs to run it.
• If gh auth token fails: message offers fallback to token file.
• Token-file path: message includes exact file path and basic creation guidance.

Evidence: transcripts of each failure mode.

---

G6. Common GitHub API failure handling

Goal: predictable behavior under common API failures.
Pass criteria:

• Rate limit or transient network failures yield retry/backoff or a clear “try again later” with a safe stop.
• Permission errors explain what permission is missing and how to fix it.
• Any partial progress is reported (what was created already).

Evidence: transcripts; repo state.

---

G7. Tutorial integrity (content delivery correctness)

Goal: issues/PRs represent the intended tutorial steps and are consistent.
Pass criteria:

• Issues/PRs appear in correct order and reference the right branches/files.
• Starter code is present and builds/runs when expected (or failure is a deliberate teaching moment with explicit instructions).

Evidence: links to issue/PR sequence; screenshots.

---

P2 Gates (Non-blocking)

G8. Cross-shell / terminal robustness

• Handles common shells (bash/zsh) and typical terminal behaviors.
• Output formatting remains readable on narrow terminals.

G9. Diagnostics for support

• Provide a single “support bundle” artifact: version, platform, config summary (redacted), last-error context.
• Clear error codes or stable error prefixes for searching.

G10. Docs consistency

• README steps match current behavior, including auth options and token expiration notes.
• Troubleshooting section for top 5 workshop failures.

---

Release checklist (operational)

• Run P0 gates on at least one platform.
• Run P1 gates on all supported platforms.
• Dogfood: team member completes one quest end-to-end (or QA does from a fresh account).
• Record evidence links in this issue before tagging a release.

Notes / open questions (for early review)

• Should we add an explicit “target owner” prompt to reduce wrong-org accidents?
• Should we add a --dry-run mode to preview which repo/actions will happen?
• Should we add an official “cleanup” command to delete the quest repo created during workshops?