Skip to content

docs(eval): align llm-rubric quickstart#1614

Merged
christso merged 1 commit into
mainfrom
fix/remove-stale-llm-grader-docs
Jul 3, 2026
Merged

docs(eval): align llm-rubric quickstart#1614
christso merged 1 commit into
mainfrom
fix/remove-stale-llm-grader-docs

Conversation

@christso

@christso christso commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • update README/docs/skills/examples away from authored llm-grader guidance toward promptfoo-style llm-rubric
  • add nested agentv results compare/trend command aliases and update docs examples
  • add a checked-in README quickstart example with default_test.options.rubric_prompt, short-form assertions, structured llm-rubric, and a script grader
  • allow authored llm-rubric as the composite LLM aggregator while preserving existing internal plumbing

Validation

  • bun test packages/core/test/evaluation/loaders/grader-parser.test.ts
  • bun test packages/sdk/test/grader-helpers.test.ts
  • bun run build
  • bun run typecheck
  • bun run validate:examples
  • bun run lint
  • bun apps/cli/dist/cli.js validate examples/features/readme-quickstart/evals/my-eval.eval.yaml
  • live smoke via local OpenAI-compatible endpoint: LOCAL_OPENAI_PROXY_BASE_URL=http://127.0.0.1:10531/v1 LOCAL_OPENAI_PROXY_MODEL=gpt-5.3-codex-spark bun apps/cli/src/cli.ts eval examples/features/readme-quickstart/evals/my-eval.eval.yaml --targets examples/features/readme-quickstart/targets.yaml -> PASS, artifact .agentv/results/2026-07-03T07-13-17-761Z/index.jsonl

Bead: av-ml02

@christso christso force-pushed the fix/remove-stale-llm-grader-docs branch from 86f29ad to bc8677b Compare July 3, 2026 07:29
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: bc8677b
Status: ✅  Deploy successful!
Preview URL: https://287a802a.agentv.pages.dev
Branch Preview URL: https://fix-remove-stale-llm-grader.agentv.pages.dev

View logs

@christso christso merged commit 85acf25 into main Jul 3, 2026
8 checks passed
@christso christso deleted the fix/remove-stale-llm-grader-docs branch July 3, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant