A test runner for agentskills.io-style AI agent skills

Crash-test insurance claim AI agents before production.

Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

Turn feature specs into merged PRs with a self-supervising swarm of coding agents — parallel execution, isolated sandboxes, DAG dependencies. Open-source, self-hostable, model-agnostic (Claude / Gemini / Codex).

Visualize LLM outputs against datasets, manually annotate results, and run automated evaluations to algorithmically optimize prompts.

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

An implementation of the Anthropic's paper and essay on "A statistical approach to model evaluations"

Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.

Three runnable enterprise AI systems showing secure RAG, governed agents, AI release reliability, evals, traces, audit logs, and approval gates.

🧪 A/B test your agent skills — skillcheck measures whether a SKILL.md actually improves an LLM's task performance, with blind grading, bootstrap confidence intervals, and a 0–100 score. CLI for Claude Code, Codex & Cursor skills.

7 Claude Code skills for software architecture review (Python, web, cloud, microservices). Includes A/B benchmarks against unskilled baseline, assertion-graded eval suite, and interactive dashboards.

Replay real agent traces through cheaper models to prove which swaps are safe.

Squeeze your model with pressure prompts to see if its behavior leaks.

AgentDog helps developers inspect, test, score, and monitor AI agent runs locally.

Detecting Relational Boundary Erosion in AI systems. A framework for testing whether models maintain honest, calibrated, and appropriate boundaries.

Because we should all have our own set of LLM evals.

Codex-native autoresearch harness with structured worker/judge turns for optimizing anything you can measure.

Disposable Daytona sandboxes for LLM evals and isolated command execution

A framework for evaluating large language models (LLMs) across a variety of tasks.

End-to-end LLM eval framework for AI products. Provider-agnostic and agent-native, with skills, traces, judge validation, and a human review interface.