Create O4_MINI_EXPERIMENT_REPORT.md

Author: codelion

examples/algotune/O4_MINI_EXPERIMENT_REPORT.md

@@ -0,0 +1,219 @@
+# O4-Mini AlgoTune Benchmark Report
+
+## Executive Summary
+
+This report documents a comprehensive evaluation of OpenAI's o4-mini model using the AlgoTune benchmark suite. The experiment ran 8 algorithmic optimization tasks through 100 iterations of evolutionary code optimization using the OpenEvolve framework. o4-mini demonstrated strong optimization capabilities, achieving significant performance improvements on 7 out of 8 tasks, with particularly impressive results on convolution and matrix projection problems.
+
+**Key Results:**
+- **Success Rate**: 87.5% (7/8 tasks improved)
+- **Major Breakthroughs**: 182x speedup on convolution, 1.85x on PSD projection
+- **Average Improvement**: 26.4x across successful tasks (excluding the 182x outlier: 1.08x average)
+- **Execution Time**: ~16-17 hours total
+- **Model Performance vs Gemini Flash 2.5**: Competitive optimization quality, 10x slower execution
+
+## Detailed Results
+
+### Task Performance Summary
+
+| Task | o4-mini Speedup | Gemini Flash 2.5 | Status | Improvement |
+|------|-----------------|-------------------|--------|-------------|
+| **convolve2d_full_fill** | **182.114x** | 163.773x | ✅ Complete | +11.2% vs Gemini |
+| **psd_cone_projection** | **1.849x** | 1.068x | ✅ Complete | +73.1% vs Gemini |
+| **polynomial_real** | 1.084x | 1.067x | ✅ Complete | +1.6% vs Gemini |
+| **eigenvectors_complex** | 1.070x | 1.113x | ✅ Complete | -3.9% vs Gemini |
+| **lu_factorization** | 1.062x | 1.055x | ✅ Complete | +0.7% vs Gemini |
+| **affine_transform_2d** | 1.023x | 1.018x | ✅ Complete | +0.5% vs Gemini |
+| **fft_cmplx_scipy_fftpack** | 1.018x | 1.021x | ✅ Complete | -0.3% vs Gemini |
+| **fft_convolution** | 0.951x | 0.962x | ❌ Failed (80 iters) | -1.1% vs Gemini |
+
+## Major Optimizations Found
+
+### 1. FFT-Based Convolution (182x Speedup)
+
+**Task**: convolve2d_full_fill  
+**Original Algorithm**: Direct 2D convolution O(n⁴)  
+**Evolved Algorithm**: FFT-based convolution O(n²log n)
+
+**Before (Initial Code):**
+```python
+def solve(self, problem):
+    a, b = problem
+    result = signal.convolve2d(a, b, mode=self.mode, boundary=self.boundary)
+    return result
+```
+
+**After (Evolved Code):**
+```python
+def solve(self, problem):
+    """Compute full 2D convolution using FFT (zero-padded)."""
+    a, b = problem
+    # ensure contiguous arrays for optimal FFT performance
+    a, b = np.ascontiguousarray(a), np.ascontiguousarray(b)
+    return fftconvolve(a, b, mode=self.mode)
+```
+
+**Key Improvements:**
+- Replaced `signal.convolve2d` with `fftconvolve` (FFT-based algorithm)
+- Added memory layout optimization with `ascontiguousarray`
+- Achieved 182x performance improvement
+- This optimization reduces computational complexity from O(n⁴) to O(n²log n)
+
+### 2. Optimized PSD Cone Projection (1.85x Speedup)
+
+**Task**: psd_cone_projection  
+**Optimization**: Matrix computation efficiency and eigenvalue handling
+
+**Before (Initial Code):**
+```python
+A = np.array(problem["A"])
+eigvals, eigvecs = np.linalg.eig(A)
+eigvals = np.maximum(eigvals, 0)
+X = eigvecs @ np.diag(eigvals) @ eigvecs.T
+return {"X": X}
+```
+
+**After (Evolved Code):**
+```python
+# load matrix and ensure float64
+A = np.array(problem["A"], dtype=np.float64, order='C', copy=False)
+eigvals, eigvecs = np.linalg.eigh(A, UPLO='L')
+if eigvals[0] >= 0:
+    return {"X": A}
+np.maximum(eigvals, 0, out=eigvals)
+# reconstruct via GEMM with scaled eigenvectors
+X = eigvecs @ (eigvecs.T * eigvals)
+return {"X": X}
+```
+
+**Key Improvements:**
+- Used `eigh` instead of `eig` for symmetric matrices (faster, more stable)
+- Added early return for already-positive matrices
+- In-place eigenvalue clamping with `out=eigvals`
+- Optimized matrix reconstruction avoiding `np.diag`
+- Memory layout optimization with `order='C'` and `copy=False`
+- Achieved 1.85x performance improvement
+
+### 3. Minor Optimizations
+
+**Other successful tasks showed modest improvements (1.8% to 8.4%) through:**
+- Memory layout optimizations
+- Algorithm parameter tuning
+- Numerical stability improvements
+- Code structure optimizations
+
+## Execution Time Analysis
+
+### Runtime Comparison
+- **Total Benchmark Time**: ~16-17 hours
+- **Average per Task**: ~2 hours per task
+- **Gemini Flash 2.5**: ~2 hours total (~15 minutes per task)
+- **Speed Ratio**: o4-mini is approximately **10x slower** than Gemini Flash 2.5
+
+### Iteration Timing
+- **Average time per iteration**: ~1.2 minutes
+- **Checkpoint frequency**: Every 10 iterations
+- **Data sizes**: Adjusted to ~60 seconds per single evaluation (×16 for full evaluation)
+
+## Failure Analysis
+
+### fft_convolution Task Failure
+
+**Status**: Stopped at iteration 80/100 with 0.951x speedup (5% degradation)
+
+**Possible Causes:**
+1. **Task Complexity**: This task may have limited optimization potential
+2. **Model Limitations**: o4-mini may struggle with certain algorithmic patterns
+3. **Local Minima**: Evolution may have gotten stuck in suboptimal solutions
+4. **Evaluation Issues**: Possible timeout or stability issues during evaluation
+
+**Comparison**: Gemini Flash 2.5 also struggled with this task (0.962x speedup)
+
+**Analysis**: Both models found this task challenging, suggesting it may be inherently difficult to optimize or have fundamental constraints preventing improvement.
+
+## Model Comparison: o4-mini vs Gemini Flash 2.5
+
+### Optimization Quality
+- **Major Wins for o4-mini**: 2 tasks significantly better
+- **Overall Performance**: Comparable optimization discovery
+- **Success Rate**: o4-mini 87.5% vs Gemini 100%
+
+### Execution Efficiency
+- **Speed**: Gemini 10x faster
+- **Resource Usage**: o4-mini more computationally intensive
+- **Reliability**: Gemini more consistent completion
+
+### Optimization Discovery
+- **FFT Convolution**: Both models found this key optimization
+- **Novel Optimizations**: o4-mini found better PSD projection approach
+- **Consistency**: Gemini more reliable across all tasks
+
+## Technical Configuration
+
+### Model Settings
+```yaml
+model_name: "openai/o4-mini"
+temperature: 0.7
+max_tokens: 4000
+diff_model: true
+num_iterations: 100
+```
+
+### Evolution Parameters
+- **Database Type**: MAP-Elites with island-based evolution
+- **Population**: 16 islands with periodic migration
+- **Evaluation**: 3-stage cascade (validation, performance, comprehensive)
+- **Timeout**: 200 seconds per evaluation stage
+
+### Data Scaling
+Tasks were scaled to run ~60 seconds per single evaluation:
+- **affine_transform_2d**: data_size = 100
+- **convolve2d_full_fill**: data_size = 5  
+- **eigenvectors_complex**: data_size = 25
+- **fft_cmplx_scipy_fftpack**: data_size = 95
+- **fft_convolution**: data_size = 125
+- **lu_factorization**: data_size = 25
+- **polynomial_real**: data_size = 500
+- **psd_cone_projection**: data_size = 35
+
+## Key Insights
+
+### Strengths of o4-mini
+1. **Algorithmic Discovery**: Successfully identified major algorithmic improvements (FFT convolution)
+2. **Optimization Depth**: Found sophisticated optimizations beyond simple parameter tuning
+3. **Mathematical Insight**: Demonstrated understanding of mathematical properties (symmetric matrices)
+4. **Code Quality**: Generated clean, well-commented optimized code
+
+### Limitations
+1. **Execution Speed**: 10x slower than Gemini Flash 2.5
+2. **Reliability**: One task failed to complete
+3. **Consistency**: More variable performance across tasks
+
+### Comparison with Gemini Flash 2.5
+- **Optimization Quality**: Roughly equivalent, with o4-mini having slight edge on major breakthroughs
+- **Speed**: Gemini significantly faster
+- **Reliability**: Gemini more consistent
+- **Cost-Effectiveness**: Gemini better for production use
+
+## Conclusions
+
+o4-mini demonstrated strong algorithmic optimization capabilities in the AlgoTune benchmark, successfully discovering major performance improvements including the critical FFT convolution optimization. While significantly slower than Gemini Flash 2.5, it showed competitive and sometimes superior optimization quality.
+
+**Recommendations:**
+1. **For research/exploration**: o4-mini suitable for deep optimization discovery
+2. **For production**: Gemini Flash 2.5 better balance of speed and quality
+3. **For hybrid approach**: Use o4-mini for initial discovery, Gemini for iteration
+
+**Future Work:**
+- Test with longer iteration counts to see if o4-mini can overcome the fft_convolution failure
+- Experiment with different temperature settings for better exploration
+- Investigate optimization potential beyond 100 iterations
+
+---
+
+**Experiment Details:**
+- **Date**: August 15, 2025
+- **Total Runtime**: ~16-17 hours
+- **Framework**: OpenEvolve v2.0
+- **Tasks**: AlgoTune benchmark suite (8 tasks)
+- **Iterations**: 100 per task
+- **Model**: openai/o4-mini