Skip to content

Conversation

anmarques
Copy link
Contributor

At the end of the run GuideLLM will print summary metrics that are computed from the raw results, but these are not currently saved anywhere.

This PR adds these metrics as serializable properties of the TextGenerationBenchmark class. Most of the metrics are already declared as properties, but are not serializable. Adding the @computed_field decorator is enough in these cases. Other properties were added to complete the list.

@EricAntidot
Copy link

This is a nice addition we are expecting this to be merged.

@EricAntidot
Copy link

@anmarques In order to calm down the mypy check you should add # type: ignore[misc] to the @computed_field annotation line.
cf: python/mypy#1362
or pydantic doc as well: https://docs.pydantic.dev/2.0/usage/computed_fields/

@anmarques
Copy link
Contributor Author

@anmarques In order to calm down the mypy check you should add # type: ignore[misc] to the @computed_field annotation line. cf: python/mypy#1362 or pydantic doc as well: https://docs.pydantic.dev/2.0/usage/computed_fields/

Thanks for the tip!

@rgreenberg1 rgreenberg1 self-requested a review September 6, 2024 14:37
Copy link
Collaborator

@rgreenberg1 rgreenberg1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Alex!

Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Alex! Could you add a short example to the readme how to get the summary metrics from the json file, hopefully in csv format?

@anmarques
Copy link
Contributor Author

I don't have instructions to convert to csv readily available. Will look into it in a following PR. Will land this one to unblock anyone needing the information saved.

@anmarques anmarques merged commit fd04739 into main Sep 6, 2024
9 checks passed
@anmarques anmarques deleted the output_summary branch September 6, 2024 21:34
@anmarques anmarques restored the output_summary branch December 4, 2024 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants