Skip to content

Implement convergence speed analysis #35

@evanvolgas

Description

@evanvolgas

Implement convergence speed analysis for publication results section.

Metrics to Implement:

  1. Time-to-X% oracle performance: How many queries to reach 80%, 90%, 95% of oracle quality?
  2. Convergence rate: Quality improvement per query over time
  3. Learning curve comparison: Plot quality over time for all algorithms
  4. Convergence stability: Variance in learning trajectory

Implementation Location: conduit_bench/analysis/convergence_analysis.py

Required Functions:

def compute_time_to_threshold(
    results: BenchmarkResults,
    oracle_results: BenchmarkResults,
    thresholds: List[float] = [0.8, 0.9, 0.95]
) -> Dict[str, Dict[float, int]]:
    """Compute number of queries to reach X% of oracle performance."""
    
def compute_learning_curves(results: BenchmarkResults) -> pd.DataFrame:
    """Compute rolling average quality over time."""
    
def compute_convergence_stability(results: BenchmarkResults, window: int = 100) -> Dict[str, float]:
    """Compute variance in learning trajectory."""

Visualization Requirements:

  • Learning curves (quality over time, all algorithms)
  • Time-to-threshold comparison (bar chart)
  • Convergence stability comparison

Success Criteria:

  • All convergence metrics implemented and tested
  • Integration with existing analyze CLI command
  • Visualization functions ready
  • Statistical significance testing (bootstrap confidence intervals)
  • Documentation and examples added

Timeline: 4 hours

Dependencies: Issue #33 (experiment results)

Publication Impact: Addresses "how fast do bandits learn?" research question

Key Insight: LinUCB may be slower to converge (high-dimensional context) but eventually outperform, or UCB1 may converge faster but plateau lower.

Metadata

Metadata

Assignees

No one assigned

    Labels

    analysisData analysis and metricsdifficulty:intermediateIntermediate difficulty - requires domain knowledgeenhancementNew feature or requestpriority:mediumMedium priority - important but not blockingpublicationPublication preparation and research paper

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions