Where are the evals?

[oshea00]

Is there any discussion on how these components (prompts, skills, etc.) are evaluated and scored for utility? How are changes evaluated? If skills and prompts are later updated, how do we know the effect one way or another? Are there efforts to study this?

For the moment, it looks like 99% of this is caveat emptor.

[astrokdev]

I also join your feeling, would it be possible to have some acceptance criteria and test results ?

[oshea00]

An example:
"Request clarification on ambiguous schemas, authentication methods, or requirements" found in

awesome-copilot/agents/openapi-to-application.agent.md

Line 31 in d99ba71

- Request clarification on ambiguous schemas, authentication methods, or requirements

How does this prescription in the agent instructions effect to desired behavior of detecting "ambiguous schemas"? Seems a little ambiguous. How would this "clarification" be ranked among the nearly infinite possibilities that it encompasses?

[oshea00]

Hint: how would we evaluate this? How would we determine whether our instruction, after further refinement, results in a positive effect in determining "ambiguoseness"?