-
Notifications
You must be signed in to change notification settings - Fork 195
[Feature]: Add cost estimation and budget guard for pipeline runs #105
Description
Problem or motivation
PaperBanana's multi-agent pipeline makes 6–9+ API calls per run (retriever, planner, stylist, visualizer × N, critic × N), and with --auto mode allowing up to 30 iterations, costs can silently spiral. There is currently no way to:
- Estimate the cost of a run before it starts
- Set a spending limit to abort if exceeded
- See a cost breakdown after a run completes
All 5 VLM providers already log token usage internally but discard it. Image gen providers have no cost tracking at all.
Proposed solution
1. --cost-only flag (dry-run cost estimation)
Estimate the total cost based on input token count, selected providers/models, and iteration count — without making any API calls. Display a breakdown and exit.
2. --budget <USD> flag (runtime budget guard)
Set a maximum spend in USD. The pipeline checks accumulated cost after each phase and between iterations. If the next step would exceed the budget, abort gracefully and save partial results.
3. Post-run cost report
Include a cost section in run_metadata.json with per-agent token usage, per-call costs, and total spend. Print a summary to the terminal at the end of each run.
Implementation notes
- VLM providers (OpenAI, Gemini, Anthropic, OpenRouter, Bedrock): Token usage is already available via
response.usage— just needs to be returned instead of discarded. - Image gen providers (OpenAI, Google, Bedrock, OpenRouter): Use a per-image pricing lookup table based on model and resolution, since these APIs don't return usage metadata.
- A new
CostTrackerclass would accumulate costs across the pipeline and check against the budget between phases/iterations. - Pricing tables should be externalized (YAML/JSON) so they're easy to update when providers change rates.
Example usage
# Estimate cost without running
paperbanana generate -i method.txt -c "Overview" --cost-only
# Run with a $5 budget cap
paperbanana generate -i method.txt -c "Overview" --auto --budget 5.00Why this matters
--autowith OpenAI can cost $15–25+ per run with no warning- Free-tier Gemini users won't be affected, but mixed-provider setups need visibility
- Cost transparency builds trust and makes the tool safer for students and researchers on limited budgets
Area
CLI
Alternatives considered
Willingness to contribute
- I'd be willing to submit a PR for this feature