Skip to content

[Feature]: Add CLI parameter sweep to compare provider/model and pipeline settings #119

@claytonlin1110

Description

@claytonlin1110

Problem or motivation

Problem

Tuning PaperBanana (VLM/image providers and models, refinement iterations, --optimize, --auto) means many manual runs and informal comparison of outputs/run_* folders. There is no single artifact that lists what was tried and how variants ranked.

Proposal

Add a paperbanana sweep command that:

  • Takes one --input + --caption and sweeps over configurable axes (e.g. providers, models, iterations, optimize/auto).
  • Writes each variant under sweep_<id>/variant_<nnn>/ and produces sweep_report.json with per-variant status, timing, and a ranked summary.
  • Supports --dry-run to preview the variant matrix without API calls.

Proposed solution

  • Add paperbanana sweep: one input + caption; axes as comma-separated lists (providers, models, iterations, optimize/auto); optional --max-variants and --dry-run.
  • Add paperbanana/core/sweep.py: build the variant matrix (Cartesian product), parse values, rank successes, summarize.
  • Per variant: merge into Settings, set output_dir to sweep_/variant_/, run PaperBananaPipeline.generate(), record timing + a simple score from the final critic suggestions; failures get an error field.
  • Output: sweep_report.json with results and ranked list; dry-run writes planned variants only, no API calls.
  • Tests: sweep helpers + CLI dry-run / bad axis input.

Area

Pipeline / agents

Alternatives considered

No response

Willingness to contribute

  • I'd be willing to submit a PR for this feature

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions