Skip to content

[Feature]: Add cost estimation and budget guard for pipeline runs #105

@tmimmanuel

Description

@tmimmanuel

Problem or motivation

PaperBanana's multi-agent pipeline makes 6–9+ API calls per run (retriever, planner, stylist, visualizer × N, critic × N), and with --auto mode allowing up to 30 iterations, costs can silently spiral. There is currently no way to:

  • Estimate the cost of a run before it starts
  • Set a spending limit to abort if exceeded
  • See a cost breakdown after a run completes

All 5 VLM providers already log token usage internally but discard it. Image gen providers have no cost tracking at all.

Proposed solution

1. --cost-only flag (dry-run cost estimation)

Estimate the total cost based on input token count, selected providers/models, and iteration count — without making any API calls. Display a breakdown and exit.

2. --budget <USD> flag (runtime budget guard)

Set a maximum spend in USD. The pipeline checks accumulated cost after each phase and between iterations. If the next step would exceed the budget, abort gracefully and save partial results.

3. Post-run cost report

Include a cost section in run_metadata.json with per-agent token usage, per-call costs, and total spend. Print a summary to the terminal at the end of each run.

Implementation notes

  • VLM providers (OpenAI, Gemini, Anthropic, OpenRouter, Bedrock): Token usage is already available via response.usage — just needs to be returned instead of discarded.
  • Image gen providers (OpenAI, Google, Bedrock, OpenRouter): Use a per-image pricing lookup table based on model and resolution, since these APIs don't return usage metadata.
  • A new CostTracker class would accumulate costs across the pipeline and check against the budget between phases/iterations.
  • Pricing tables should be externalized (YAML/JSON) so they're easy to update when providers change rates.

Example usage

# Estimate cost without running
paperbanana generate -i method.txt -c "Overview" --cost-only

# Run with a $5 budget cap
paperbanana generate -i method.txt -c "Overview" --auto --budget 5.00

Why this matters

  • --auto with OpenAI can cost $15–25+ per run with no warning
  • Free-tier Gemini users won't be affected, but mixed-provider setups need visibility
  • Cost transparency builds trust and makes the tool safer for students and researchers on limited budgets

Area

CLI

Alternatives considered

Willingness to contribute

  • I'd be willing to submit a PR for this feature

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions