Add Pandora operator verification console#133
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 41 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (17)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d7151f1a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const latest = rows[0]; | ||
| if (!latest) return { status: "not_run", source: "retrieval_logs", latestRunId: null, latestRunAt: null, resultLabel: "Not run", realResultAvailable: false, warnings: ["No retrieval eval/log rows returned for this operator."] }; | ||
| const label = latest.eval_result ?? latest.result ?? latest.status ?? latest.retrieval_service ?? "Real retrieval log row present; no score column returned"; | ||
| return { status: "pass", source: "retrieval_logs", latestRunId: String(latest.id ?? "unknown"), latestRunAt: String(latest.created_at ?? "No created_at returned"), resultLabel: String(label), realResultAvailable: true, warnings: [] }; |
There was a problem hiding this comment.
In environments where any ordinary retrieval has been logged, this returns status: "pass" and realResultAvailable: true without checking for an eval-specific marker; the existing retrieval service writes generic retrieval_logs rows for scaffold searches, and the core table only has generic retrieval columns. That lets /pandora claim the retrieval eval passed from a normal log row, contrary to the gated/no-fake-eval boundary; keep this not_run or warning unless the row proves an actual eval run/result.
Useful? React with 👍 / 👎.
| const warnings: string[] = []; | ||
| const [packRowsByNamespace, auditRows, retrievalRows] = await Promise.all([ | ||
| Promise.all(namespaces.map((namespace) => readRows(client, "memory_context_packs", input.userId, warnings, namespace, 100))), | ||
| readRows(client, "audit_logs", input.userId, warnings, undefined, 50), |
There was a problem hiding this comment.
When an operator has more than 50 newer audit rows for unrelated actions, this call truncates the audit history before the later filters look for memory_context_pack_distilled and smoke actions. Existing distill or smoke proof just outside the newest 50 rows will be reported as missing/not_run, so a healthy deployment can show false evidence gaps; filter by the target actions in the query or page far enough before deciding evidence is absent.
Useful? React with 👍 / 👎.
1 participant
Motivation
/pandorato prove live dashboard data, namespace invariants, master-pack supersession, audit/smoke evidence, and honest retrieval-eval status without exposing fake claims or unsafe actions.userId, surface missing/unreadable tables as warnings, and preserve existing visual layout and dashboard guards.Description
lib/services/pandora-verification-service.tsthat readsmemory_context_packs,audit_logs, andretrieval_logsscoped to the authenticateduserId, converts read failures to warnings, and computes per-namespace summaries, supersession evidence, retrieval-eval honesty, smoke evidence, and invariant statuses.components/pandora/types.tswith strongly typed verification models includingVerificationStatus,NamespaceVerificationSummary,PackSupersessionSummary,RetrievalEvalSummary,AuditEvidenceItem,SmokeEvidenceSummary, andPandoraVerificationData.components/pandora/:VerificationConsoleCard.tsx,NamespaceInvariantCard.tsx,PackSupersessionCard.tsx,RetrievalEvalCard.tsx, andAuditEvidenceCard.tsx, and wired the verification console intocomponents/pandora/PandoraDashboard.tsxwhile retaining the three-column layout and existing cards.loadPandoraVerificationDataand merge verification warnings into the dashboard warnings without accepting any client-supplieduser_idvalues.mock-data(moved nav constants tonav-items.ts) to satisfy production guard rules.tests/unit/pandora-verification-service.test.ts, updatedtests/unit/pandora-dashboard-ui.test.tsxandtests/unit/pandora-dashboard-guard.test.tsto assert honestnot_runbehavior and that the verification loader is read-only and service-role-free.docs/pandora-verification-console.mddescribing what is live, what remains gated, status semantics, and manual verification steps.Testing
npm run typecheck(TypeScripttsc --noEmit) — passed.npm run lint(ESLint) — passed with pre-existing warnings (not introduced by this change).npm run test(Vitest) — all tests passed: 93 test files, 598 tests (598 passed, 0 failed) after adding and updating verification tests; targeted test files also pass individually.npm run build(Next.js build) — build succeeded (Next.js runtime warnings unrelated to these changes are unchanged).npm run env:policy— Env Broker policy passed for discovered env keys.Commands executed in CI-local verification:
npm run typecheck,npm run lint,npm run test,npm run build,npm run env:policy.Codex Task