Github actions to make good looking comments for eval runs

[ryx2]

Description

In my company repo, I wrote a .yml that runs evals on PRs to main.

Then, when those evals are done, it writes a nice github comment with the output results:

I was wondering if this type of .md formatting + commenting could be supported in pydantic since it seems generally useful to all?

References

No response

[DouweM]

@ryx2 Nice work! I definitely want to have an easy way to render an evaluation report as a Markdown table, alongside the existing CLI-table rendering in https://github.com/pydantic/pydantic-ai/blob/main/pydantic_evals/pydantic_evals/reporting/__init__.py. Feel free to submit a PR!

We'd also love to have an official GitHub Action like the one you've built. That likely shouldn't live in this repo though. Would you be interested in open sourcing what you have built so far in its own repo, and then we can look into either recommending it or forking it to our organization and making it official?