Skip to content

Conversation

andrewginns
Copy link
Owner

@andrewginns andrewginns commented Jul 6, 2025

Adds cost calculations to the preprocessed output JSON and updates the build backend to support this new functionality.

Migrate from hatchling to uv native build backend

Problem:

Previously, the output JSON generated by the preprocessing step did not include cost information for model runs. This limited the ability to analyse and report on the computational or financial cost associated with benchmark runs.

Solution:

I have updated the preprocessing logic to calculate and embed cost data within the output JSON. Additionally, the build backend has been enhanced to handle these new fields, ensuring that downstream processes can access and utilise this data.

Unlocks:

This update enables more detailed analysis of model usage costs, supports cost-aware benchmarking, and allows for more accurate reporting and optimisation of agent workflows.

Detailed breakdown of changes:

  • Modified the preprocessing script(s) to calculate costs for each relevant operation and include this data in the output JSON.
  • Updated the build backend to correctly process and store the new cost information.
  • Adjusted tests and documentation as required to reflect the new JSON structure.
  • Made minor improvements and refactoring to support the integration of cost tracking within existing workflows.
  • Changed 8 files in total, with significant additions to logic surrounding cost calculation and output formatting.

@andrewginns andrewginns merged commit f54372c into main Jul 6, 2025
1 check passed
@andrewginns andrewginns deleted the add-costs-to-preprocessed-merbench-json branch July 6, 2025 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant