investigate reference files for skills #111
Description
context
we shipped a minimal skill in #110 with no reference files. this is intentional - "least possible tokens until hardened empirically."
anthropic's official skills (pdf, xlsx, etc.) use reference files for things claude genuinely doesn't know (library APIs, file formats). we should revisit this once we have more evals showing where claude struggles.
resources
official docs
- agent skills spec
- claude code skills docs
- skills blog post - skills vs MCP explanation
example implementations
- anthropics/skills - official skills with reference files pattern
- getsentry/sentry-for-claude - sentry code review skill (~100 lines, format-heavy)
- obra/superpowers - process/behavioral skills (systematic-debugging, TDD)
- letta-ai/letta-code - "context window is a public good" philosophy
our approach
- mcpval blog post - eval-driven validation
potential reference file candidates
only add these if evals reveal claude needs help:
| candidate | when to add |
|---|---|
filter-examples.md |
if claude struggles with filter syntax |
event-types.md |
if we add event-based debugging evals |
work-pool-types.md |
if infra type confusion shows up |
key insight from letta
"context window is a public good" - only add what the model doesn't already know
the MCP tool descriptions are already good. the skill teaches when to use tools, not how. reference files should only cover domain knowledge that's not inferrable from tool responses.