-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
analysisData analysis and metricsData analysis and metricsdifficulty:intermediateIntermediate difficulty - requires domain knowledgeIntermediate difficulty - requires domain knowledgeenhancementNew feature or requestNew feature or requestpriority:mediumMedium priority - important but not blockingMedium priority - important but not blockingpublicationPublication preparation and research paperPublication preparation and research paper
Description
Implement convergence speed analysis for publication results section.
Metrics to Implement:
- Time-to-X% oracle performance: How many queries to reach 80%, 90%, 95% of oracle quality?
- Convergence rate: Quality improvement per query over time
- Learning curve comparison: Plot quality over time for all algorithms
- Convergence stability: Variance in learning trajectory
Implementation Location: conduit_bench/analysis/convergence_analysis.py
Required Functions:
def compute_time_to_threshold(
results: BenchmarkResults,
oracle_results: BenchmarkResults,
thresholds: List[float] = [0.8, 0.9, 0.95]
) -> Dict[str, Dict[float, int]]:
"""Compute number of queries to reach X% of oracle performance."""
def compute_learning_curves(results: BenchmarkResults) -> pd.DataFrame:
"""Compute rolling average quality over time."""
def compute_convergence_stability(results: BenchmarkResults, window: int = 100) -> Dict[str, float]:
"""Compute variance in learning trajectory."""Visualization Requirements:
- Learning curves (quality over time, all algorithms)
- Time-to-threshold comparison (bar chart)
- Convergence stability comparison
Success Criteria:
- All convergence metrics implemented and tested
- Integration with existing analyze CLI command
- Visualization functions ready
- Statistical significance testing (bootstrap confidence intervals)
- Documentation and examples added
Timeline: 4 hours
Dependencies: Issue #33 (experiment results)
Publication Impact: Addresses "how fast do bandits learn?" research question
Key Insight: LinUCB may be slower to converge (high-dimensional context) but eventually outperform, or UCB1 may converge faster but plateau lower.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
analysisData analysis and metricsData analysis and metricsdifficulty:intermediateIntermediate difficulty - requires domain knowledgeIntermediate difficulty - requires domain knowledgeenhancementNew feature or requestNew feature or requestpriority:mediumMedium priority - important but not blockingMedium priority - important but not blockingpublicationPublication preparation and research paperPublication preparation and research paper