Skip to content

Feature Request: Agent Evaluations print detailed results even when all tests pass #3891

@quad2524

Description

@quad2524

Is your feature request related to a problem? Please describe.
In AgentEvaluator, currently print_detailed_results=True only prints detailed results if there are failures. It is difficult to get detailed metrics output for successful runs in CI pipelines where visibility into passing tests is desired.

Describe the solution you'd like
I would like an additional flag, for example print_detailed_results_on_success, that when set to True (along with print_detailed_results=True), forces the printing of detailed results even if the evaluation passes.

Describe alternatives you've considered

  • Changing the default behavior of print_detailed_results to always print, but this would be a breaking change and might clutter logs for users who don't want it.
  • Parsing the return value manually, but evaluate is often used as a high-level entry point where return values are not easily accessible for printing in the same format.

Metadata

Metadata

Assignees

No one assigned

    Labels

    eval[Component] This issue is related to evaluation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions