Hey everyone,
I built a claude code plugin that generates ade-bench tasks from a user's own dbt project. There are two reasons why I thought this might be valuable to folks out there.
First, everyone who's been in data even for a short time, knows the whole TPC story. Benchmarks are good to give you a broad sense of what a system is capable of but at the end of the day what matters is what the system does against your own workload. I believe the same applies to the AI evals too.
Second, data infrastructure is complex, data is complex, it's hard to assess evals that run against setups you are not familiar with. Being able to run ade-bench against your own stuff fixes this.
A bit more on the plugin itself:
Repo: https://github.com/typedef-ai/ade-bench-plugin
Three slash commands inside Claude Code:
/ade-bench:plan-tasks — interactive pair-planning that reads your models, reasons about meaningful bugs, generates the task scaffolding/ade-bench:create-task — automated pipeline: matches model SQL against a pattern catalog, ranks candidates, generates tasks/ade-bench:setup — installs ade-bench at ~/.ade-bench (clones, uv sync, uv tool install, downloads the databases release)
The plugin follows the ade-bench's existing contract of single-variant tasks with the standard task.yaml + setup/solution scripts + patches shape, plus custom assertion tests in tests/*.sql (non-AUTO_* filenames are preserved by the harness).
Any feedback on the plugin would be highly appreciated.
Proper eval infrastructure for agentic data infra tasks is severely limited today and we definitely need it if we want our industry to move forward and building tooling that will help people understand and use evals will help with that. At least, that's how I think of it.
Hey everyone,
I built a claude code plugin that generates ade-bench tasks from a user's own dbt project. There are two reasons why I thought this might be valuable to folks out there.
First, everyone who's been in data even for a short time, knows the whole TPC story. Benchmarks are good to give you a broad sense of what a system is capable of but at the end of the day what matters is what the system does against your own workload. I believe the same applies to the AI evals too.
Second, data infrastructure is complex, data is complex, it's hard to assess evals that run against setups you are not familiar with. Being able to run ade-bench against your own stuff fixes this.
A bit more on the plugin itself:
Repo: https://github.com/typedef-ai/ade-bench-plugin
Three slash commands inside Claude Code:
/ade-bench:plan-tasks— interactive pair-planning that reads your models, reasons about meaningful bugs, generates the task scaffolding/ade-bench:create-task— automated pipeline: matches model SQL against a pattern catalog, ranks candidates, generates tasks/ade-bench:setup— installs ade-bench at~/.ade-bench(clones,uv sync,uv tool install, downloads thedatabasesrelease)The plugin follows the ade-bench's existing contract of single-variant tasks with the standard
task.yaml+ setup/solution scripts + patches shape, plus custom assertion tests intests/*.sql(non-AUTO_*filenames are preserved by the harness).Any feedback on the plugin would be highly appreciated.
Proper eval infrastructure for agentic data infra tasks is severely limited today and we definitely need it if we want our industry to move forward and building tooling that will help people understand and use evals will help with that. At least, that's how I think of it.