Testing

[wlandau]

What are your thoughts on testing when it comes to skills? What kind of testing and how much testing is appropriate before releasing/updating a skill in a marketplace?

Static validation is cheap, but tests of functionality are much harder. Would this be in scope for vitals?

[sims1253]

This could probably work as a kind of eval where you take a number of test-cases, and have an agent use a skill to do work on that case. The hard part is to evaluate how well the agent does. For things like correctness or performance you could use tests and benchmarks but more vague things like style or design decisions, this becomes fuzzy and needs some kind of subjective judging. In theory one could use llm judges (https://eqbench.com/judgemark-v2.html might be a start?) and have them judge the results.

[gadenbuie]

Testing is quite tricky, especially since skills are general-purpose and likely to be used across a variety of coding agents and models. If you have a narrowly scoped skill and target agent harness, it might be worth investing in evals. Whether the skill is oriented toward completing a specific task or toward providing background knowledge in open-ended tasks is also important.

In general, I'm trying to stick with skills whose effectiveness I can judge with my personal expertise (shiny) or by sourcing their material from known and up-to-date materials (the testing-r-packages skills is basically the testthat release posts condensed into a skill) or because they show up in my own workflows.

Like blog posts and other materials, skills are useful long before they're perfect, so I'd lean toward releasing early. Even better if you have at least a few users you know that will give you feedback!

[royfrancis]

Adding a comment here because I would also like to know if these actually work. I am hoping to see some metrics showing performance 'with' and 'without skills'. I understand that Posit is locked into Claude, but for the rest of us, we would like to know how skill usage varies across different models. Would these benefit smaller models more than larger ones? Or does it simply pollute the context if the model is already aware of these things?

[gadenbuie]

I would also like to know if these actually work. I am hoping to see some metrics showing performance 'with' and 'without skills'.

I understand the motivation behind this desire -- as data people, we'd really like to see evidence that skills work and are useful.

That said, I can confidently say that the skills in this repo are useful. How useful they are will vary greatly by model, agent harness and, most importantly, the goals and parameters of your task.

So far, all the skills in the repo were created by people who also use the skills with some regularity, so there's a guarantee that they have improved coding agent outcomes for us and that we'll continue to pay attention to their performance over time.

Beyond that, though, formal and comprehensive evals aren't a priority. If you are interested in running evals yourself, we'd be happy to share the results and to use them to improve the skills. And as always, please let us know if you run into any areas for improvement when using these skills.