Skip to content

test(promptfoo): add oracle export gate#1720

Merged
christso merged 1 commit into
mainfrom
av-rr0r-promptfoo-oracle-gate
Jul 7, 2026
Merged

test(promptfoo): add oracle export gate#1720
christso merged 1 commit into
mainfrom
av-rr0r-promptfoo-oracle-gate

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add Promptfoo defaults export lowering: defaults.provider -> defaultTest.providers, defaults.grader -> defaultTest.options.provider
  • add a deterministic AgentV fixture that exports to Promptfoo and executes with real promptfoo@0.121.15 eval
  • assert Promptfoo JSONL matrix/provider/assertion outcomes for default provider, test options.provider, and assertion-level provider selection

Promptfoo oracle

  • CLI oracle: bunx promptfoo@0.121.15 eval
  • Local reference clone checked: /home/entity/projects/promptfoo/promptfoo at 6bfc5a0c7f16f9c4717ac731d276b578e63d0769
  • The gate proves the exported config is accepted by Promptfoo and that Promptfoo itself executes the expected target-provider matrix and grader-provider precedence.

Validation

  • bun run validate:promptfoo-export
  • bunx biome check .github/workflows/ci.yml scripts/export-promptfoo-config.ts scripts/export-promptfoo-config.test.ts scripts/fixtures/promptfoo-export/oracle-matrix.agentv.yaml scripts/fixtures/promptfoo-export/oracle-target-provider.cjs scripts/fixtures/promptfoo-export/oracle-grader-provider.cjs
  • tmp=$(mktemp -d); bun scripts/export-promptfoo-config.ts --input scripts/fixtures/promptfoo-export/host-environment.agentv.yaml --output "$tmp/promptfooconfig.yaml" && bunx promptfoo@0.121.15 validate config -c "$tmp/promptfooconfig.yaml"

Notes

Bead: av-rr0r

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: b6b6ebf
Status: ✅  Deploy successful!
Preview URL: https://c601ae92.agentv.pages.dev
Branch Preview URL: https://av-rr0r-promptfoo-oracle-gat.agentv.pages.dev

View logs

@christso
christso merged commit 95b763e into main Jul 7, 2026
9 checks passed
@christso
christso deleted the av-rr0r-promptfoo-oracle-gate branch July 7, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant