diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index a209809e..527967a4 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,46 +1,48 @@ # Useful Optimizer - GitHub Copilot Instructions -Useful Optimizer is a Python optimization library containing 58 optimization algorithms for numeric problems. The library uses Poetry for dependency management and provides a comprehensive collection of metaheuristic, gradient-based, and nature-inspired optimization techniques. +Useful Optimizer is a Python optimization library containing 58 optimization algorithms for numeric problems. The project uses **uv** for dependency management, virtual environments, and packaging while providing a comprehensive collection of metaheuristic, gradient-based, and nature-inspired optimization techniques. -**Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.** +**Always reference these instructions first and fall back to search or shell commands only when you encounter unexpected information that does not match the info here.** All commands are written as **single-line Fish-shell commands** so they can be copied directly into the terminal. ## Working Effectively ### Initial Setup and Installation -- Install Poetry package manager: - ```bash - pip3 install poetry - export PATH="$HOME/.local/bin:$PATH" +- Install uv (recommended via installer script): + ```fish + curl -LsSf https://astral.sh/uv/install.sh | sh ``` - **Alternative if PATH export doesn't work:** Use `~/.local/bin/poetry` instead of `poetry` -- Install project dependencies: - ```bash - poetry install +- Ensure `$HOME/.local/bin` is on your Fish PATH (replace with your actual install dir if different): + ```fish + set -Ux PATH $HOME/.local/bin $PATH + ``` +- Install project dependencies and create the managed virtual environment: + ```fish + uv sync ``` **NEVER CANCEL: Takes 2-3 minutes to complete. Set timeout to 5+ minutes.** ### Core Development Commands - Run any Python command in the project environment: - ```bash - poetry run python [your_command] + ```fish + uv run python [your_command] ``` - Test basic functionality: - ```bash - poetry run python -c "import opt; print('Import successful')" + ```fish + uv run python -c "import opt; print('Import successful')" ``` - Run linting (finds code quality issues): - ```bash - poetry run ruff check opt/ + ```fish + uv run ruff check opt/ ``` **Takes 5-10 seconds. Common to find existing issues in codebase.** - Auto-format code: - ```bash - poetry run ruff format opt/ + ```fish + uv run ruff format opt/ ``` **Takes less than 1 second.** - Build the package: - ```bash - poetry build + ```fish + uv build ``` **Takes less than 1 second. Creates wheel and sdist in dist/ directory.** @@ -49,56 +51,26 @@ Useful Optimizer is a Python optimization library containing 58 optimization alg **ALWAYS test your changes with these scenarios after making modifications:** ### Scenario 1: Basic Import and Functionality Test -```bash -poetry run python -c " -from opt.benchmark.functions import shifted_ackley, rosenbrock, sphere -from opt.particle_swarm import ParticleSwarm -print('Testing basic import and optimizer creation...') -pso = ParticleSwarm(func=shifted_ackley, lower_bound=-2.768, upper_bound=2.768, dim=2, max_iter=50) -best_solution, best_fitness = pso.search() -print(f'PSO completed successfully. Fitness: {best_fitness:.6f}') -print('Basic functionality test PASSED') -" +```fish +uv run python -c "from opt.benchmark.functions import shifted_ackley, rosenbrock, sphere; from opt.particle_swarm import ParticleSwarm; print('Testing basic import and optimizer creation...'); pso = ParticleSwarm(func=shifted_ackley, lower_bound=-2.768, upper_bound=2.768, dim=2, max_iter=50); best_solution, best_fitness = pso.search(); print(f'PSO completed successfully. Fitness: {best_fitness:.6f}'); print('Basic functionality test PASSED')" ``` **Expected: Completes in < 1 second, prints fitness value around 0.001-1.0** ### Scenario 2: Multiple Optimizer Test -```bash -poetry run python -c " -from opt.benchmark.functions import shifted_ackley, rosenbrock -from opt.harmony_search import HarmonySearch -from opt.ant_colony import AntColony -print('Testing multiple optimizers...') - -# Test HarmonySearch -hs = HarmonySearch(func=rosenbrock, lower_bound=-5, upper_bound=5, dim=2, max_iter=50) -_, fitness1 = hs.search() - -# Test AntColony -ac = AntColony(func=shifted_ackley, lower_bound=-2.768, upper_bound=2.768, dim=2, max_iter=50) -_, fitness2 = ac.search() - -print(f'HS fitness: {fitness1:.6f}, ACO fitness: {fitness2:.6f}') -print('Multiple optimizer test PASSED') -" +```fish +uv run python -c "from opt.benchmark.functions import shifted_ackley, rosenbrock; from opt.harmony_search import HarmonySearch; from opt.ant_colony import AntColony; print('Testing multiple optimizers...'); hs = HarmonySearch(func=rosenbrock, lower_bound=-5, upper_bound=5, dim=2, max_iter=50); _, fitness1 = hs.search(); ac = AntColony(func=shifted_ackley, lower_bound=-2.768, upper_bound=2.768, dim=2, max_iter=50); _, fitness2 = ac.search(); print(f'HS fitness: {fitness1:.6f}, ACO fitness: {fitness2:.6f}'); print('Multiple optimizer test PASSED')" ``` **Expected: Completes in < 1 second, prints two fitness values** ### Scenario 3: Direct Script Execution Test -```bash -poetry run python opt/harmony_search.py +```fish +uv run python opt/harmony_search.py ``` **Expected: Prints "Best solution found:" and "Best fitness found:" with numerical values** ### Scenario 4: Advanced Import Test -```bash -poetry run python -c " -from opt.abstract_optimizer import AbstractOptimizer -from opt.benchmark.functions import shifted_ackley, sphere, rosenbrock, ackley -from opt.sgd_momentum import SGDMomentum -from opt.adamw import AdamW -print('Advanced imports successful - gradient-based and base classes work') -" +```fish +uv run python -c "from opt.abstract_optimizer import AbstractOptimizer; from opt.benchmark.functions import shifted_ackley, sphere, rosenbrock, ackley; from opt.sgd_momentum import SGDMomentum; from opt.adamw import AdamW; print('Advanced imports successful - gradient-based and base classes work')" ``` ## Project Structure @@ -142,9 +114,15 @@ print('Advanced imports successful - gradient-based and base classes work') ### Code Quality and Linting - **ALWAYS run linting before committing:** - ```bash - poetry run ruff check opt/ - poetry run ruff format opt/ + ```fish + uv run ruff check opt/ && uv run ruff format opt/ + ``` +- Or run them separately: + ```fish + uv run ruff check opt/ + ``` + ```fish + uv run ruff format opt/ ``` - The project uses extensive ruff rules - expect to find existing linting issues - Ruff configuration is in `pyproject.toml` with Google docstring convention @@ -158,16 +136,67 @@ print('Advanced imports successful - gradient-based and base classes work') - Use benchmark functions from `opt.benchmark.functions` for testing ### Testing Changes -- No formal test suite exists - use the validation scenarios above +- Run the test suite: + ```fish + uv run pytest opt/test/ -v + ``` +- Run specific test files: + ```fish + uv run pytest opt/test/test_benchmarks.py -v + ``` - Test with multiple benchmark functions: `shifted_ackley`, `rosenbrock`, `sphere` - Test with different parameter combinations - Ensure optimizers complete within reasonable time (< 1 second for small max_iter) ### Common Issues to Avoid -- Don't modify `poetry.lock` manually - use `poetry add/remove` commands +- Don't modify `uv.lock` manually - use `uv add`, `uv remove`, or `uv sync` to change dependencies - Ruff linting will fail on many existing files - focus on new/modified code - Some algorithms use legacy `np.random.rand` calls - documented in README -- Always use `poetry run` prefix for Python commands to use project environment +- Always use `uv run` (or activate `.venv`) so commands execute inside the project environment + +### CRITICAL: Fish Shell and Terminal Rules +**All commands MUST be single-line.** Multiline commands crash the terminal in Fish shell. + +**Git commit messages:** Always use single-line format with conventional commits: +```fish +git commit -m "feat: add new optimizer algorithm" +``` + +**Conventional commit types:** +- `feat:` - New feature +- `fix:` - Bug fix +- `docs:` - Documentation changes +- `test:` - Adding or updating tests +- `refactor:` - Code refactoring +- `chore:` - Maintenance tasks + +**NEVER use multiline commit messages like this (WILL CRASH):** +```fish +# BAD - DO NOT USE +git commit -m "Title + +Multiline body" +``` + +**For commit messages with title and body**, use multiple `-m` flags on one line: +```fish +git commit -m "Title" -m "Body paragraph 1" -m "Body paragraph 2" +``` + +**For longer commit messages**, use the editor: +```fish +git commit +``` + +**Chaining commands:** Use `&&` to chain multiple commands: +```fish +git add . && git commit -m "message" && git push +``` + +**Python -c commands:** Keep on single line with semicolons: +```fish +uv run python -c "import x; print(x.value); do_something()" +``` ## Project Information - **Version:** 0.1.2 @@ -178,16 +207,16 @@ print('Advanced imports successful - gradient-based and base classes work') - **License:** MIT ## Build and Distribution -- Build package: `poetry build` (< 1 second) +- Build package: `uv build` (< 1 second) - Built artifacts appear in `dist/` directory - CI/CD publishes to PyPI on tag pushes via GitHub Actions - Test PyPI publishing is also configured ## Performance Expectations -- **Poetry install:** 2-3 minutes (NEVER CANCEL) +- **uv sync:** 2-3 minutes (NEVER CANCEL) - **Ruff linting:** 5-10 seconds - **Ruff formatting:** < 1 second -- **Package build:** < 1 second +- **Package build (uv build):** < 1 second - **Small optimization runs:** < 1 second (max_iter=50-100) - **Import operations:** Nearly instantaneous diff --git a/.github/workflows/python-publish.yaml b/.github/workflows/python-publish.yaml index 759b7c6a..a6c8a5f1 100644 --- a/.github/workflows/python-publish.yaml +++ b/.github/workflows/python-publish.yaml @@ -69,7 +69,7 @@ jobs: name: python-package-distributions path: dist/ - name: Sign the dists with Sigstore - uses: sigstore/gh-action-sigstore-python@v3.1.0 + uses: sigstore/gh-action-sigstore-python@v3.2.0 with: inputs: >- ./dist/*.tar.gz diff --git a/opt/test/conftest.py b/opt/test/conftest.py new file mode 100644 index 00000000..b6c2a968 --- /dev/null +++ b/opt/test/conftest.py @@ -0,0 +1,464 @@ +"""Pytest configuration and shared fixtures for optimizer tests. + +This module provides fixtures for benchmark functions with their known optimal +solutions, test configurations, and helper utilities for comprehensive optimizer +testing. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING + +import numpy as np +import pytest + +from opt.benchmark.functions import ackley +from opt.benchmark.functions import beale +from opt.benchmark.functions import booth +from opt.benchmark.functions import easom +from opt.benchmark.functions import goldstein_price +from opt.benchmark.functions import griewank +from opt.benchmark.functions import himmelblau +from opt.benchmark.functions import levi +from opt.benchmark.functions import levi_n13 +from opt.benchmark.functions import matyas +from opt.benchmark.functions import mccormick +from opt.benchmark.functions import rastrigin +from opt.benchmark.functions import rosenbrock +from opt.benchmark.functions import schwefel +from opt.benchmark.functions import shifted_ackley +from opt.benchmark.functions import sphere +from opt.benchmark.functions import three_hump_camel + + +if TYPE_CHECKING: + from collections.abc import Callable + + from numpy import ndarray + + +@dataclass +class BenchmarkFunction: + """Configuration for a benchmark function with known optimal solution. + + Attributes: + name: Human-readable name of the function. + func: The benchmark function callable. + optimal_point: Known optimal point(s) as numpy array. + optimal_value: Known optimal function value. + lower_bound: Lower bound of the search space. + upper_bound: Upper bound of the search space. + dim: Dimensionality of the problem. + tolerance_point: Acceptable tolerance for solution point (distance). + tolerance_value: Acceptable tolerance for fitness value. + difficulty: Estimated difficulty level ('easy', 'medium', 'hard'). + """ + + name: str + func: Callable[[ndarray], float] + optimal_point: ndarray + optimal_value: float + lower_bound: float + upper_bound: float + dim: int + tolerance_point: float = 0.5 # Euclidean distance tolerance + tolerance_value: float = 1.0 # Fitness value tolerance + difficulty: str = "medium" + + +# ============================================================================= +# Benchmark Function Definitions with Known Optima +# ============================================================================= + +BENCHMARK_FUNCTIONS: dict[str, BenchmarkFunction] = { + # Easy functions - unimodal, smooth + "sphere": BenchmarkFunction( + name="Sphere", + func=sphere, + optimal_point=np.array([0.0, 0.0]), + optimal_value=0.0, + lower_bound=-5.12, + upper_bound=5.12, + dim=2, + tolerance_point=0.1, + tolerance_value=0.01, + difficulty="easy", + ), + "shifted_ackley": BenchmarkFunction( + name="Shifted Ackley", + func=shifted_ackley, + optimal_point=np.array([1.0, 0.5]), # shift=(1, 0.5) moves optimum + optimal_value=0.0, + lower_bound=-2.768, + upper_bound=2.768, + dim=2, + tolerance_point=0.2, # Critical: if 1.2 and 0.7 comes out, it's bad + tolerance_value=0.5, + difficulty="medium", + ), + "ackley": BenchmarkFunction( + name="Ackley", + func=ackley, + optimal_point=np.array([0.0, 0.0]), + optimal_value=0.0, + lower_bound=-5.0, + upper_bound=5.0, + dim=2, + tolerance_point=0.3, + tolerance_value=1.0, + difficulty="medium", + ), + "rosenbrock": BenchmarkFunction( + name="Rosenbrock", + func=rosenbrock, + optimal_point=np.array([1.0, 1.0]), + optimal_value=0.0, + lower_bound=-5.0, + upper_bound=10.0, + dim=2, + tolerance_point=0.5, + tolerance_value=1.0, + difficulty="hard", + ), + "rastrigin": BenchmarkFunction( + name="Rastrigin", + func=rastrigin, + optimal_point=np.array([0.0, 0.0]), + optimal_value=0.0, + lower_bound=-5.12, + upper_bound=5.12, + dim=2, + tolerance_point=0.5, + tolerance_value=2.0, + difficulty="hard", + ), + "griewank": BenchmarkFunction( + name="Griewank", + func=griewank, + optimal_point=np.array([0.0, 0.0]), + optimal_value=0.0, + lower_bound=-600.0, + upper_bound=600.0, + dim=2, + tolerance_point=1.0, + tolerance_value=0.5, + difficulty="medium", + ), + "schwefel": BenchmarkFunction( + name="Schwefel", + func=schwefel, + optimal_point=np.array([420.9687, 420.9687]), + optimal_value=0.0, + lower_bound=-500.0, + upper_bound=500.0, + dim=2, + tolerance_point=50.0, + tolerance_value=100.0, + difficulty="hard", + ), + "booth": BenchmarkFunction( + name="Booth", + func=booth, + optimal_point=np.array([1.0, 3.0]), + optimal_value=0.0, + lower_bound=-10.0, + upper_bound=10.0, + dim=2, + tolerance_point=0.3, + tolerance_value=0.5, + difficulty="easy", + ), + "matyas": BenchmarkFunction( + name="Matyas", + func=matyas, + optimal_point=np.array([0.0, 0.0]), + optimal_value=0.0, + lower_bound=-10.0, + upper_bound=10.0, + dim=2, + tolerance_point=0.2, + tolerance_value=0.1, + difficulty="easy", + ), + "himmelblau": BenchmarkFunction( + name="Himmelblau", + func=himmelblau, + # Has 4 identical local minima; testing one of them + optimal_point=np.array([3.0, 2.0]), + optimal_value=0.0, + lower_bound=-5.0, + upper_bound=5.0, + dim=2, + tolerance_point=0.5, + tolerance_value=1.0, + difficulty="medium", + ), + "three_hump_camel": BenchmarkFunction( + name="Three-Hump Camel", + func=three_hump_camel, + optimal_point=np.array([0.0, 0.0]), + optimal_value=0.0, + lower_bound=-5.0, + upper_bound=5.0, + dim=2, + tolerance_point=0.3, + tolerance_value=0.5, + difficulty="easy", + ), + "beale": BenchmarkFunction( + name="Beale", + func=beale, + optimal_point=np.array([3.0, 0.5]), + optimal_value=0.0, + lower_bound=-4.5, + upper_bound=4.5, + dim=2, + tolerance_point=0.5, + tolerance_value=1.0, + difficulty="medium", + ), + "goldstein_price": BenchmarkFunction( + name="Goldstein-Price", + func=goldstein_price, + optimal_point=np.array([0.0, -1.0]), + optimal_value=3.0, + lower_bound=-2.0, + upper_bound=2.0, + dim=2, + tolerance_point=0.3, + tolerance_value=5.0, + difficulty="medium", + ), + "levi": BenchmarkFunction( + name="Levi", + func=levi, + optimal_point=np.array([1.0, 1.0]), + optimal_value=0.0, + lower_bound=-10.0, + upper_bound=10.0, + dim=2, + tolerance_point=0.3, + tolerance_value=0.5, + difficulty="medium", + ), + "levi_n13": BenchmarkFunction( + name="Levi N.13", + func=levi_n13, + optimal_point=np.array([1.0, 1.0]), + optimal_value=0.0, + lower_bound=-10.0, + upper_bound=10.0, + dim=2, + tolerance_point=0.3, + tolerance_value=0.5, + difficulty="medium", + ), + "easom": BenchmarkFunction( + name="Easom", + func=easom, + optimal_point=np.array([np.pi, np.pi]), + optimal_value=-1.0, + lower_bound=-100.0, + upper_bound=100.0, + dim=2, + tolerance_point=0.5, + tolerance_value=0.5, + difficulty="hard", + ), + "mccormick": BenchmarkFunction( + name="McCormick", + func=mccormick, + optimal_point=np.array([-0.54719, -1.54719]), + optimal_value=-1.9133, + lower_bound=-1.5, + upper_bound=4.0, + dim=2, + tolerance_point=0.3, + tolerance_value=0.5, + difficulty="easy", + ), +} + +# Functions suitable for quick tests (easy/medium difficulty) +QUICK_TEST_FUNCTIONS = [ + "sphere", + "shifted_ackley", + "booth", + "matyas", + "three_hump_camel", +] + +# Functions for comprehensive tests (all difficulties) +COMPREHENSIVE_TEST_FUNCTIONS = list(BENCHMARK_FUNCTIONS.keys()) + +# Functions by difficulty +EASY_FUNCTIONS = [k for k, v in BENCHMARK_FUNCTIONS.items() if v.difficulty == "easy"] +MEDIUM_FUNCTIONS = [ + k for k, v in BENCHMARK_FUNCTIONS.items() if v.difficulty == "medium" +] +HARD_FUNCTIONS = [k for k, v in BENCHMARK_FUNCTIONS.items() if v.difficulty == "hard"] + + +# ============================================================================= +# Fixtures +# ============================================================================= + + +@pytest.fixture +def sphere_benchmark() -> BenchmarkFunction: + """Fixture for sphere function benchmark.""" + return BENCHMARK_FUNCTIONS["sphere"] + + +@pytest.fixture +def shifted_ackley_benchmark() -> BenchmarkFunction: + """Fixture for shifted_ackley function benchmark.""" + return BENCHMARK_FUNCTIONS["shifted_ackley"] + + +@pytest.fixture +def rosenbrock_benchmark() -> BenchmarkFunction: + """Fixture for rosenbrock function benchmark.""" + return BENCHMARK_FUNCTIONS["rosenbrock"] + + +@pytest.fixture(params=QUICK_TEST_FUNCTIONS) +def quick_benchmark(request: pytest.FixtureRequest) -> BenchmarkFunction: + """Parametrized fixture for quick benchmark tests.""" + return BENCHMARK_FUNCTIONS[request.param] + + +@pytest.fixture(params=COMPREHENSIVE_TEST_FUNCTIONS) +def all_benchmarks(request: pytest.FixtureRequest) -> BenchmarkFunction: + """Parametrized fixture for all benchmark functions.""" + return BENCHMARK_FUNCTIONS[request.param] + + +@pytest.fixture(params=EASY_FUNCTIONS) +def easy_benchmark(request: pytest.FixtureRequest) -> BenchmarkFunction: + """Parametrized fixture for easy benchmark functions.""" + return BENCHMARK_FUNCTIONS[request.param] + + +@pytest.fixture(params=MEDIUM_FUNCTIONS) +def medium_benchmark(request: pytest.FixtureRequest) -> BenchmarkFunction: + """Parametrized fixture for medium difficulty benchmark functions.""" + return BENCHMARK_FUNCTIONS[request.param] + + +@pytest.fixture(params=HARD_FUNCTIONS) +def hard_benchmark(request: pytest.FixtureRequest) -> BenchmarkFunction: + """Parametrized fixture for hard benchmark functions.""" + return BENCHMARK_FUNCTIONS[request.param] + + +# ============================================================================= +# Optimizer Test Configuration +# ============================================================================= + + +@dataclass +class OptimizerTestConfig: + """Configuration for optimizer testing. + + Attributes: + max_iter_quick: Maximum iterations for quick tests. + max_iter_full: Maximum iterations for full benchmark tests. + population_size: Default population size for population-based methods. + seed: Random seed for reproducibility. + strict_tolerance: Whether to use strict tolerances. + """ + + max_iter_quick: int = 50 + max_iter_full: int = 500 + population_size: int = 30 + seed: int = 42 + strict_tolerance: bool = False + + +# Alias for external use +TestConfig = OptimizerTestConfig + + +@pytest.fixture +def test_config() -> OptimizerTestConfig: + """Fixture providing test configuration.""" + return OptimizerTestConfig() + + +@pytest.fixture +def strict_test_config() -> TestConfig: + """Fixture providing strict test configuration.""" + return TestConfig(max_iter_quick=100, max_iter_full=1000, strict_tolerance=True) + + +# ============================================================================= +# Helper Functions +# ============================================================================= + + +def calculate_solution_quality( + solution: ndarray, fitness: float, benchmark: BenchmarkFunction +) -> dict[str, float | bool]: + """Calculate quality metrics for an optimization solution. + + Args: + solution: The solution found by the optimizer. + fitness: The fitness value of the solution. + benchmark: The benchmark function configuration. + + Returns: + Dictionary with quality metrics: + - point_distance: Euclidean distance from optimal point + - value_error: Absolute error in fitness value + - point_within_tolerance: Whether point is within tolerance + - value_within_tolerance: Whether value is within tolerance + - overall_pass: Whether both tolerances are met + """ + point_distance = float(np.linalg.norm(solution - benchmark.optimal_point)) + value_error = abs(fitness - benchmark.optimal_value) + + point_ok = point_distance <= benchmark.tolerance_point + value_ok = value_error <= benchmark.tolerance_value + + return { + "point_distance": point_distance, + "value_error": value_error, + "point_within_tolerance": point_ok, + "value_within_tolerance": value_ok, + "overall_pass": point_ok and value_ok, + } + + +def is_near_any_optimum( + solution: ndarray, optima: list[ndarray], tolerance: float +) -> bool: + """Check if solution is near any of multiple optimal points. + + Some functions like Himmelblau have multiple global optima. + + Args: + solution: The solution found by the optimizer. + optima: List of known optimal points. + tolerance: Distance tolerance. + + Returns: + True if solution is within tolerance of any optimum. + """ + return any(np.linalg.norm(solution - optimum) <= tolerance for optimum in optima) + + +# Himmelblau has 4 identical minima +HIMMELBLAU_OPTIMA = [ + np.array([3.0, 2.0]), + np.array([-2.805118, 3.131312]), + np.array([-3.779310, -3.283186]), + np.array([3.584428, -1.848126]), +] + + +@pytest.fixture +def himmelblau_optima() -> list[ndarray]: + """Fixture providing all Himmelblau function optima.""" + return HIMMELBLAU_OPTIMA diff --git a/opt/test/test_benchmarks.py b/opt/test/test_benchmarks.py new file mode 100644 index 00000000..c8698b57 --- /dev/null +++ b/opt/test/test_benchmarks.py @@ -0,0 +1,549 @@ +"""Benchmark tests for optimizer quality validation. + +This module tests that optimizers find solutions within acceptable tolerance +of known optimal points. These tests verify the actual optimization quality, +not just that the optimizers run without errors. + +Critical Test Cases: +- shifted_ackley: optimal at (1.0, 0.5) - solutions like (1.2, 0.7) are failures +- sphere: optimal at (0, 0) - should be very accurate +- rosenbrock: optimal at (1, 1) - harder, wider tolerance allowed +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import numpy as np +import pytest + +from opt import BFGS +from opt import LBFGS +from opt import SGD +from opt import ADAGrad +from opt import ADAMOptimization +from opt import AMSGrad +from opt import AdaDelta +from opt import AdaMax +from opt import AdamW +from opt import AntColony +from opt import ArtificialFishSwarm +from opt import AugmentedLagrangian +from opt import BatAlgorithm +from opt import BeeAlgorithm +from opt import CMAESAlgorithm +from opt import CatSwarmOptimization +from opt import CollidingBodiesOptimization +from opt import ConjugateGradient +from opt import CrossEntropyMethod +from opt import CuckooSearch +from opt import CulturalAlgorithm +from opt import DifferentialEvolution +from opt import EagleStrategy +from opt import EstimationOfDistributionAlgorithm +from opt import FireflyAlgorithm +from opt import GeneticAlgorithm +from opt import GlowwormSwarmOptimization +from opt import GreyWolfOptimizer +from opt import HarmonySearch +from opt import HillClimbing +from opt import ImperialistCompetitiveAlgorithm +from opt import LDAnalysis +from opt import Nadam +from opt import NelderMead +from opt import NesterovAcceleratedGradient +from opt import ParticleFilter +from opt import ParticleSwarm +from opt import ParzenTreeEstimator +from opt import Powell +from opt import RMSprop +from opt import SGDMomentum +from opt import ShuffledFrogLeapingAlgorithm +from opt import SimulatedAnnealing +from opt import SineCosineAlgorithm +from opt import SquirrelSearchAlgorithm +from opt import StochasticDiffusionSearch +from opt import StochasticFractalSearch +from opt import SuccessiveLinearProgramming +from opt import TabuSearch +from opt import TrustRegion +from opt import VariableDepthSearch +from opt import VariableNeighborhoodSearch +from opt import VeryLargeScaleNeighborhood +from opt import WhaleOptimizationAlgorithm + +from .conftest import calculate_solution_quality +from .conftest import is_near_any_optimum + + +if TYPE_CHECKING: + from opt import AbstractOptimizer + + from .conftest import BenchmarkFunction + + +# ============================================================================= +# Optimizer Categories for Benchmark Tests +# ============================================================================= + +# High-performing optimizers that should achieve tight tolerances +HIGH_PERFORMANCE_OPTIMIZERS = [ + ParticleSwarm, + DifferentialEvolution, + CMAESAlgorithm, + Powell, + WhaleOptimizationAlgorithm, +] + +# Optimizers that struggle with multimodal functions like shifted_ackley +# They converge to local minima instead of the global optimum +LOCAL_MINIMA_PRONE_OPTIMIZERS = [ + pytest.param( + BFGS, + marks=pytest.mark.xfail( + reason="BFGS converges to local minimum on multimodal shifted_ackley", + strict=False, + ), + ), + pytest.param( + LBFGS, + marks=pytest.mark.xfail( + reason="LBFGS converges to local minimum on multimodal shifted_ackley", + strict=False, + ), + ), + pytest.param( + NelderMead, + marks=pytest.mark.xfail( + reason="NelderMead converges to local minimum on multimodal shifted_ackley", + strict=False, + ), + ), + pytest.param( + GreyWolfOptimizer, + marks=pytest.mark.xfail( + reason="GreyWolfOptimizer has convergence issues on shifted_ackley", + strict=False, + ), + ), +] + +# Medium performance - reasonable results expected +MEDIUM_PERFORMANCE_OPTIMIZERS = [ + GeneticAlgorithm, + AntColony, + FireflyAlgorithm, + CuckooSearch, + HarmonySearch, + SimulatedAnnealing, + BeeAlgorithm, + CrossEntropyMethod, + SineCosineAlgorithm, +] + +# Optimizers that may need more iterations or have variable performance +VARIABLE_PERFORMANCE_OPTIMIZERS = [ + ArtificialFishSwarm, + CatSwarmOptimization, + GlowwormSwarmOptimization, + SquirrelSearchAlgorithm, + CollidingBodiesOptimization, + EagleStrategy, + CulturalAlgorithm, + EstimationOfDistributionAlgorithm, + ImperialistCompetitiveAlgorithm, + ParticleFilter, + ShuffledFrogLeapingAlgorithm, + StochasticDiffusionSearch, + StochasticFractalSearch, + VariableDepthSearch, + VariableNeighborhoodSearch, + VeryLargeScaleNeighborhood, +] + +# Gradient-based optimizers (may converge to local optima) +GRADIENT_OPTIMIZERS = [ + AdaDelta, + ADAGrad, + AdaMax, + AdamW, + ADAMOptimization, + AMSGrad, + Nadam, + NesterovAcceleratedGradient, + RMSprop, + SGD, + SGDMomentum, + ConjugateGradient, + TrustRegion, + HillClimbing, + TabuSearch, +] + +# Constrained/Probabilistic optimizers +SPECIALIZED_OPTIMIZERS = [ + AugmentedLagrangian, + SuccessiveLinearProgramming, + LDAnalysis, + ParzenTreeEstimator, +] + + +# ============================================================================= +# Critical Benchmark Tests - shifted_ackley +# ============================================================================= + + +class TestShiftedAckleyBenchmark: + """Critical tests for shifted_ackley function. + + The shifted_ackley function has its optimum at (1.0, 0.5) due to shift=(1, 0.5). + Solutions like (1.2, 0.7) indicate the optimizer is NOT converging properly + and should be flagged as critical failures. + """ + + OPTIMAL_POINT = np.array([1.0, 0.5]) + CRITICAL_TOLERANCE = 0.2 # Distance > 0.2 is a critical failure + WARNING_TOLERANCE = 0.1 # Distance > 0.1 but <= 0.2 is a warning + + @pytest.mark.parametrize( + "optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS + LOCAL_MINIMA_PRONE_OPTIMIZERS + ) + def test_high_performance_optimizers( + self, + optimizer_class: type[AbstractOptimizer], + shifted_ackley_benchmark: BenchmarkFunction, + ) -> None: + """Test that high-performance optimizers find the shifted_ackley optimum.""" + optimizer = optimizer_class( + func=shifted_ackley_benchmark.func, + lower_bound=shifted_ackley_benchmark.lower_bound, + upper_bound=shifted_ackley_benchmark.upper_bound, + dim=shifted_ackley_benchmark.dim, + max_iter=200, + ) + solution, fitness = optimizer.search() + + # Calculate distance from known optimum + distance = np.linalg.norm(solution - self.OPTIMAL_POINT) + + # Critical assertion - must not be too far from optimum + assert distance <= self.CRITICAL_TOLERANCE, ( + f"{optimizer_class.__name__} CRITICAL FAILURE on shifted_ackley: " + f"Solution {solution} is {distance:.4f} away from optimum {self.OPTIMAL_POINT}. " + f"Fitness: {fitness:.6f}. Maximum allowed distance: {self.CRITICAL_TOLERANCE}" + ) + + @pytest.mark.parametrize("optimizer_class", MEDIUM_PERFORMANCE_OPTIMIZERS) + def test_medium_performance_optimizers( + self, + optimizer_class: type[AbstractOptimizer], + shifted_ackley_benchmark: BenchmarkFunction, + ) -> None: + """Test medium-performance optimizers on shifted_ackley with relaxed tolerance.""" + optimizer = optimizer_class( + func=shifted_ackley_benchmark.func, + lower_bound=shifted_ackley_benchmark.lower_bound, + upper_bound=shifted_ackley_benchmark.upper_bound, + dim=shifted_ackley_benchmark.dim, + max_iter=300, + ) + solution, _fitness = optimizer.search() + + distance = np.linalg.norm(solution - self.OPTIMAL_POINT) + + # More relaxed tolerance for medium performers + relaxed_tolerance = 0.3 + assert distance <= relaxed_tolerance, ( + f"{optimizer_class.__name__} FAILURE on shifted_ackley: " + f"Solution {solution} is {distance:.4f} away from optimum {self.OPTIMAL_POINT}. " + f"Maximum allowed: {relaxed_tolerance}" + ) + + def test_bat_algorithm_shifted_ackley( + self, shifted_ackley_benchmark: BenchmarkFunction + ) -> None: + """Test BatAlgorithm (requires special n_bats parameter).""" + optimizer = BatAlgorithm( + func=shifted_ackley_benchmark.func, + lower_bound=shifted_ackley_benchmark.lower_bound, + upper_bound=shifted_ackley_benchmark.upper_bound, + dim=shifted_ackley_benchmark.dim, + n_bats=30, + max_iter=200, + ) + solution, _fitness = optimizer.search() + + distance = np.linalg.norm(solution - self.OPTIMAL_POINT) + assert distance <= 0.3, ( + f"BatAlgorithm solution {solution} too far from optimum. Distance: {distance:.4f}" + ) + + +# ============================================================================= +# Sphere Function Benchmark Tests +# ============================================================================= + + +class TestSphereBenchmark: + """Tests for sphere function - should be easy for most optimizers.""" + + OPTIMAL_POINT = np.array([0.0, 0.0]) + TIGHT_TOLERANCE = 0.1 + RELAXED_TOLERANCE = 0.5 + + @pytest.mark.parametrize("optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS) + def test_high_performance_on_sphere( + self, + optimizer_class: type[AbstractOptimizer], + sphere_benchmark: BenchmarkFunction, + ) -> None: + """High-performance optimizers should easily solve sphere.""" + optimizer = optimizer_class( + func=sphere_benchmark.func, + lower_bound=sphere_benchmark.lower_bound, + upper_bound=sphere_benchmark.upper_bound, + dim=sphere_benchmark.dim, + max_iter=100, + ) + solution, fitness = optimizer.search() + + distance = np.linalg.norm(solution - self.OPTIMAL_POINT) + assert distance <= self.TIGHT_TOLERANCE, ( + f"{optimizer_class.__name__} failed on simple sphere function. " + f"Distance: {distance:.4f}, Expected < {self.TIGHT_TOLERANCE}" + ) + assert fitness <= 0.01, f"Fitness {fitness} too high for sphere" + + +# ============================================================================= +# Comprehensive Quality Tests +# ============================================================================= + + +class TestOptimizerQuality: + """Comprehensive quality tests across multiple benchmark functions.""" + + @pytest.mark.parametrize( + "optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS + MEDIUM_PERFORMANCE_OPTIMIZERS + ) + def test_optimizer_on_easy_functions( + self, + optimizer_class: type[AbstractOptimizer], + easy_benchmark: BenchmarkFunction, + ) -> None: + """Test optimizers on easy benchmark functions.""" + optimizer = optimizer_class( + func=easy_benchmark.func, + lower_bound=easy_benchmark.lower_bound, + upper_bound=easy_benchmark.upper_bound, + dim=easy_benchmark.dim, + max_iter=200, + ) + solution, fitness = optimizer.search() + + quality = calculate_solution_quality(solution, fitness, easy_benchmark) + + # Easy functions should be solved with reasonable accuracy + assert quality["point_within_tolerance"] or quality["value_within_tolerance"], ( + f"{optimizer_class.__name__} failed on {easy_benchmark.name}: " + f"Point distance: {quality['point_distance']:.4f} " + f"(tolerance: {easy_benchmark.tolerance_point}), " + f"Value error: {quality['value_error']:.4f} " + f"(tolerance: {easy_benchmark.tolerance_value})" + ) + + @pytest.mark.parametrize("optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS) + def test_high_performers_on_medium_functions( + self, + optimizer_class: type[AbstractOptimizer], + medium_benchmark: BenchmarkFunction, + ) -> None: + """Test high-performance optimizers on medium difficulty functions.""" + optimizer = optimizer_class( + func=medium_benchmark.func, + lower_bound=medium_benchmark.lower_bound, + upper_bound=medium_benchmark.upper_bound, + dim=medium_benchmark.dim, + max_iter=500, + ) + solution, fitness = optimizer.search() + + quality = calculate_solution_quality(solution, fitness, medium_benchmark) + + # At least one criterion should be met + assert quality["point_within_tolerance"] or quality["value_within_tolerance"], ( + f"{optimizer_class.__name__} failed on {medium_benchmark.name}: " + f"Solution: {solution}, Fitness: {fitness:.6f}" + ) + + +# ============================================================================= +# Himmelblau Special Case (Multiple Optima) +# ============================================================================= + + +class TestHimmelblauMultipleOptima: + """Test Himmelblau function which has 4 identical global minima.""" + + @pytest.mark.parametrize("optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS) + def test_finds_any_optimum( + self, + optimizer_class: type[AbstractOptimizer], + himmelblau_optima: list[np.ndarray], + ) -> None: + """Test that optimizer finds one of Himmelblau's four minima.""" + from opt.benchmark.functions import himmelblau + + optimizer = optimizer_class( + func=himmelblau, lower_bound=-5.0, upper_bound=5.0, dim=2, max_iter=300 + ) + solution, fitness = optimizer.search() + + # Should be near any of the 4 optima + near_optimum = is_near_any_optimum(solution, himmelblau_optima, tolerance=0.5) + assert near_optimum, ( + f"{optimizer_class.__name__} did not find any Himmelblau optimum. " + f"Solution: {solution}, Fitness: {fitness:.6f}" + ) + assert fitness <= 1.0, f"Fitness {fitness} too high for Himmelblau" + + +# ============================================================================= +# Fitness Value Sanity Tests +# ============================================================================= + + +class TestFitnessSanity: + """Tests ensuring fitness values are reasonable.""" + + @pytest.mark.parametrize( + "optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS + MEDIUM_PERFORMANCE_OPTIMIZERS + ) + def test_fitness_improves_from_random( + self, + optimizer_class: type[AbstractOptimizer], + sphere_benchmark: BenchmarkFunction, + ) -> None: + """Test that optimization actually improves over random initialization.""" + # Generate random fitness for comparison + rng = np.random.default_rng(42) + random_points = rng.uniform( + sphere_benchmark.lower_bound, + sphere_benchmark.upper_bound, + (100, sphere_benchmark.dim), + ) + random_fitnesses = [sphere_benchmark.func(p) for p in random_points] + avg_random_fitness = np.mean(random_fitnesses) + + # Run optimizer + optimizer = optimizer_class( + func=sphere_benchmark.func, + lower_bound=sphere_benchmark.lower_bound, + upper_bound=sphere_benchmark.upper_bound, + dim=sphere_benchmark.dim, + max_iter=100, + ) + _, fitness = optimizer.search() + + # Optimized fitness should be significantly better than random + assert fitness < avg_random_fitness, ( + f"{optimizer_class.__name__} did not improve over random. " + f"Optimized: {fitness:.4f}, Avg random: {avg_random_fitness:.4f}" + ) + + @pytest.mark.parametrize("optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS) + def test_fitness_is_finite( + self, + optimizer_class: type[AbstractOptimizer], + quick_benchmark: BenchmarkFunction, + ) -> None: + """Test that fitness values are finite numbers.""" + optimizer = optimizer_class( + func=quick_benchmark.func, + lower_bound=quick_benchmark.lower_bound, + upper_bound=quick_benchmark.upper_bound, + dim=quick_benchmark.dim, + max_iter=50, + ) + solution, fitness = optimizer.search() + + assert np.isfinite(fitness), f"Non-finite fitness: {fitness}" + assert np.all(np.isfinite(solution)), f"Non-finite solution: {solution}" + + +# ============================================================================= +# Reproducibility Tests +# ============================================================================= + + +class TestReproducibility: + """Tests for optimizer reproducibility with seeds.""" + + @pytest.mark.parametrize( + "optimizer_class", + [ParticleSwarm, DifferentialEvolution, GeneticAlgorithm, FireflyAlgorithm], + ) + def test_seeded_reproducibility( + self, + optimizer_class: type[AbstractOptimizer], + sphere_benchmark: BenchmarkFunction, + ) -> None: + """Test that seeded optimizers produce reproducible results.""" + results = [] + for _ in range(2): + optimizer = optimizer_class( + func=sphere_benchmark.func, + lower_bound=sphere_benchmark.lower_bound, + upper_bound=sphere_benchmark.upper_bound, + dim=sphere_benchmark.dim, + max_iter=50, + seed=42, + ) + solution, fitness = optimizer.search() + results.append((solution.copy(), fitness)) + + # Both runs should produce identical results + np.testing.assert_array_almost_equal( + results[0][0], + results[1][0], + decimal=10, + err_msg=f"{optimizer_class.__name__} not reproducible with same seed", + ) + assert results[0][1] == results[1][1], "Fitness values should be identical" + + +# ============================================================================= +# Solution Bounds Tests +# ============================================================================= + + +class TestSolutionBounds: + """Tests ensuring solutions stay within specified bounds.""" + + @pytest.mark.parametrize( + "optimizer_class", HIGH_PERFORMANCE_OPTIMIZERS + MEDIUM_PERFORMANCE_OPTIMIZERS + ) + def test_solution_within_bounds( + self, + optimizer_class: type[AbstractOptimizer], + quick_benchmark: BenchmarkFunction, + ) -> None: + """Test that solutions respect the search space bounds.""" + optimizer = optimizer_class( + func=quick_benchmark.func, + lower_bound=quick_benchmark.lower_bound, + upper_bound=quick_benchmark.upper_bound, + dim=quick_benchmark.dim, + max_iter=100, + ) + solution, _ = optimizer.search() + + assert np.all(solution >= quick_benchmark.lower_bound - 1e-10), ( + f"{optimizer_class.__name__} returned solution below lower bound: " + f"{solution} < {quick_benchmark.lower_bound}" + ) + assert np.all(solution <= quick_benchmark.upper_bound + 1e-10), ( + f"{optimizer_class.__name__} returned solution above upper bound: " + f"{solution} > {quick_benchmark.upper_bound}" + ) diff --git a/opt/test/test_performance.py b/opt/test/test_performance.py new file mode 100644 index 00000000..342ff33b --- /dev/null +++ b/opt/test/test_performance.py @@ -0,0 +1,349 @@ +"""Performance regression tests for optimizers. + +This module provides tests to detect performance regressions in optimizers. +It establishes baseline expectations for each optimizer on standard benchmarks +and flags deviations that may indicate bugs or improvements. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING + +import numpy as np +import pytest + +from opt import BFGS +from opt import LBFGS +from opt import CMAESAlgorithm +from opt import DifferentialEvolution +from opt import FireflyAlgorithm +from opt import GeneticAlgorithm +from opt import GreyWolfOptimizer +from opt import HarmonySearch +from opt import NelderMead +from opt import ParticleSwarm +from opt import Powell +from opt import SimulatedAnnealing +from opt import WhaleOptimizationAlgorithm +from opt.benchmark.functions import rosenbrock +from opt.benchmark.functions import shifted_ackley +from opt.benchmark.functions import sphere + + +if TYPE_CHECKING: + from opt import AbstractOptimizer + + +@dataclass +class PerformanceBaseline: + """Expected performance baseline for an optimizer. + + Attributes: + optimizer_class: The optimizer class. + function_name: Name of the benchmark function. + expected_fitness_upper: Upper bound on expected fitness (worse case). + expected_fitness_lower: Lower bound on expected fitness (best case). + max_distance_from_optimum: Maximum acceptable distance from known optimum. + max_iter: Number of iterations used for baseline. + """ + + optimizer_class: type[AbstractOptimizer] + function_name: str + expected_fitness_upper: float + expected_fitness_lower: float + max_distance_from_optimum: float + max_iter: int + + +# ============================================================================= +# Performance Baselines +# ============================================================================= + +# Baselines for shifted_ackley (optimum at [1.0, 0.5], optimal value ~0) +SHIFTED_ACKLEY_BASELINES = [ + PerformanceBaseline(ParticleSwarm, "shifted_ackley", 0.5, 0.0, 0.2, 200), + PerformanceBaseline(DifferentialEvolution, "shifted_ackley", 0.5, 0.0, 0.2, 200), + PerformanceBaseline(CMAESAlgorithm, "shifted_ackley", 0.5, 0.0, 0.2, 200), + PerformanceBaseline(GreyWolfOptimizer, "shifted_ackley", 0.5, 0.0, 0.2, 200), + PerformanceBaseline(GeneticAlgorithm, "shifted_ackley", 1.0, 0.0, 0.3, 300), + PerformanceBaseline(FireflyAlgorithm, "shifted_ackley", 1.0, 0.0, 0.3, 300), + PerformanceBaseline(HarmonySearch, "shifted_ackley", 1.5, 0.0, 0.4, 300), + PerformanceBaseline(SimulatedAnnealing, "shifted_ackley", 1.5, 0.0, 0.4, 300), +] + +# Baselines for sphere (optimum at [0, 0], optimal value = 0) +SPHERE_BASELINES = [ + PerformanceBaseline(ParticleSwarm, "sphere", 0.01, 0.0, 0.1, 100), + PerformanceBaseline(BFGS, "sphere", 0.001, 0.0, 0.05, 100), + PerformanceBaseline(LBFGS, "sphere", 0.001, 0.0, 0.05, 100), + PerformanceBaseline(NelderMead, "sphere", 0.01, 0.0, 0.1, 100), + PerformanceBaseline(Powell, "sphere", 0.01, 0.0, 0.1, 100), + PerformanceBaseline(DifferentialEvolution, "sphere", 0.01, 0.0, 0.1, 100), +] + +# Baselines for rosenbrock (optimum at [1, 1], optimal value = 0) +ROSENBROCK_BASELINES = [ + PerformanceBaseline(BFGS, "rosenbrock", 1.0, 0.0, 0.5, 200), + PerformanceBaseline(LBFGS, "rosenbrock", 1.0, 0.0, 0.5, 200), + PerformanceBaseline(NelderMead, "rosenbrock", 1.0, 0.0, 0.5, 300), + PerformanceBaseline(CMAESAlgorithm, "rosenbrock", 5.0, 0.0, 1.0, 500), + PerformanceBaseline(DifferentialEvolution, "rosenbrock", 5.0, 0.0, 1.0, 500), +] + +# Known optimal points for each function +OPTIMAL_POINTS = { + "shifted_ackley": np.array([1.0, 0.5]), + "sphere": np.array([0.0, 0.0]), + "rosenbrock": np.array([1.0, 1.0]), +} + +# Function configurations +FUNCTION_CONFIGS = { + "shifted_ackley": { + "func": shifted_ackley, + "lower_bound": -2.768, + "upper_bound": 2.768, + "dim": 2, + }, + "sphere": {"func": sphere, "lower_bound": -5.12, "upper_bound": 5.12, "dim": 2}, + "rosenbrock": { + "func": rosenbrock, + "lower_bound": -5.0, + "upper_bound": 10.0, + "dim": 2, + }, +} + + +# ============================================================================= +# Regression Tests +# ============================================================================= + + +class TestPerformanceRegression: + """Tests to detect performance regressions.""" + + @pytest.mark.parametrize("baseline", SHIFTED_ACKLEY_BASELINES) + def test_shifted_ackley_regression(self, baseline: PerformanceBaseline) -> None: + """Test optimizer performance on shifted_ackley against baseline.""" + config = FUNCTION_CONFIGS["shifted_ackley"] + optimizer = baseline.optimizer_class( + func=config["func"], + lower_bound=config["lower_bound"], + upper_bound=config["upper_bound"], + dim=config["dim"], + max_iter=baseline.max_iter, + ) + solution, fitness = optimizer.search() + + # Check fitness is within expected range + assert fitness <= baseline.expected_fitness_upper, ( + f"REGRESSION: {baseline.optimizer_class.__name__} fitness {fitness:.4f} " + f"exceeds baseline upper bound {baseline.expected_fitness_upper:.4f}" + ) + + # Check distance from optimum + distance = np.linalg.norm(solution - OPTIMAL_POINTS["shifted_ackley"]) + assert distance <= baseline.max_distance_from_optimum, ( + f"REGRESSION: {baseline.optimizer_class.__name__} solution {solution} " + f"is {distance:.4f} from optimum, exceeds baseline {baseline.max_distance_from_optimum:.4f}" + ) + + @pytest.mark.parametrize("baseline", SPHERE_BASELINES) + def test_sphere_regression(self, baseline: PerformanceBaseline) -> None: + """Test optimizer performance on sphere against baseline.""" + config = FUNCTION_CONFIGS["sphere"] + optimizer = baseline.optimizer_class( + func=config["func"], + lower_bound=config["lower_bound"], + upper_bound=config["upper_bound"], + dim=config["dim"], + max_iter=baseline.max_iter, + ) + solution, fitness = optimizer.search() + + assert fitness <= baseline.expected_fitness_upper, ( + f"REGRESSION: {baseline.optimizer_class.__name__} on sphere: " + f"fitness {fitness:.6f} > {baseline.expected_fitness_upper:.6f}" + ) + + distance = np.linalg.norm(solution - OPTIMAL_POINTS["sphere"]) + assert distance <= baseline.max_distance_from_optimum, ( + f"REGRESSION: {baseline.optimizer_class.__name__} distance {distance:.4f} " + f"exceeds {baseline.max_distance_from_optimum:.4f}" + ) + + @pytest.mark.parametrize("baseline", ROSENBROCK_BASELINES) + def test_rosenbrock_regression(self, baseline: PerformanceBaseline) -> None: + """Test optimizer performance on rosenbrock against baseline.""" + config = FUNCTION_CONFIGS["rosenbrock"] + optimizer = baseline.optimizer_class( + func=config["func"], + lower_bound=config["lower_bound"], + upper_bound=config["upper_bound"], + dim=config["dim"], + max_iter=baseline.max_iter, + ) + _solution, fitness = optimizer.search() + + assert fitness <= baseline.expected_fitness_upper, ( + f"REGRESSION: {baseline.optimizer_class.__name__} on rosenbrock: " + f"fitness {fitness:.4f} > {baseline.expected_fitness_upper:.4f}" + ) + + +# ============================================================================= +# Statistical Performance Tests +# ============================================================================= + + +class TestStatisticalPerformance: + """Statistical tests over multiple runs to assess optimizer reliability.""" + + @pytest.mark.parametrize( + "optimizer_class", [ParticleSwarm, DifferentialEvolution, GeneticAlgorithm] + ) + def test_consistency_over_runs( + self, optimizer_class: type[AbstractOptimizer] + ) -> None: + """Test that optimizer produces consistent results over multiple runs.""" + n_runs = 5 + results = [] + + config = FUNCTION_CONFIGS["shifted_ackley"] + for seed in range(n_runs): + optimizer = optimizer_class( + func=config["func"], + lower_bound=config["lower_bound"], + upper_bound=config["upper_bound"], + dim=config["dim"], + max_iter=100, + seed=seed * 100, # Different seeds for variety + ) + _, fitness = optimizer.search() + results.append(fitness) + + # Calculate statistics + mean_fitness = np.mean(results) + std_fitness = np.std(results) + best_fitness = min(results) + max(results) + + # Basic consistency checks + assert std_fitness < mean_fitness * 2, ( + f"{optimizer_class.__name__} has high variance: " + f"std={std_fitness:.4f}, mean={mean_fitness:.4f}" + ) + + # At least one run should find a good solution + assert best_fitness < 1.0, ( + f"{optimizer_class.__name__} best fitness {best_fitness:.4f} too high" + ) + + @pytest.mark.parametrize( + "optimizer_class", + [ParticleSwarm, GreyWolfOptimizer, WhaleOptimizationAlgorithm], + ) + def test_success_rate(self, optimizer_class: type[AbstractOptimizer]) -> None: + """Test optimizer success rate (finding solution within tolerance).""" + n_runs = 10 + tolerance = 0.3 + successes = 0 + + config = FUNCTION_CONFIGS["shifted_ackley"] + optimal = OPTIMAL_POINTS["shifted_ackley"] + + for seed in range(n_runs): + optimizer = optimizer_class( + func=config["func"], + lower_bound=config["lower_bound"], + upper_bound=config["upper_bound"], + dim=config["dim"], + max_iter=150, + seed=seed * 42, + ) + solution, _ = optimizer.search() + distance = np.linalg.norm(solution - optimal) + if distance <= tolerance: + successes += 1 + + success_rate = successes / n_runs + assert success_rate >= 0.5, ( + f"{optimizer_class.__name__} success rate {success_rate:.1%} < 50%" + ) + + +# ============================================================================= +# Critical Path Tests +# ============================================================================= + + +class TestCriticalShiftedAckley: + """Critical tests specifically for the shifted_ackley convergence issue. + + These tests are marked as critical because solutions like (1.2, 0.7) instead + of the correct (1.0, 0.5) indicate a fundamental problem with the optimizer. + """ + + @pytest.mark.critical + def test_particle_swarm_critical(self) -> None: + """Critical test: PSO must find shifted_ackley optimum accurately.""" + optimizer = ParticleSwarm( + func=shifted_ackley, + lower_bound=-2.768, + upper_bound=2.768, + dim=2, + max_iter=200, + ) + solution, fitness = optimizer.search() + + optimal = np.array([1.0, 0.5]) + distance = np.linalg.norm(solution - optimal) + + # This is a critical assertion + assert distance <= 0.15, ( + f"CRITICAL: ParticleSwarm converged to {solution} " + f"instead of {optimal}. Distance: {distance:.4f}" + ) + assert fitness <= 0.3, f"CRITICAL: Fitness {fitness:.4f} is too high" + + @pytest.mark.critical + def test_differential_evolution_critical(self) -> None: + """Critical test: DE must find shifted_ackley optimum accurately.""" + optimizer = DifferentialEvolution( + func=shifted_ackley, + lower_bound=-2.768, + upper_bound=2.768, + dim=2, + max_iter=200, + ) + solution, _fitness = optimizer.search() + + optimal = np.array([1.0, 0.5]) + distance = np.linalg.norm(solution - optimal) + + assert distance <= 0.15, ( + f"CRITICAL: DifferentialEvolution converged to {solution}" + ) + + @pytest.mark.critical + def test_firefly_critical(self) -> None: + """Critical test: Firefly must not converge far from optimum.""" + optimizer = FireflyAlgorithm( + func=shifted_ackley, + lower_bound=-2.768, + upper_bound=2.768, + dim=2, + max_iter=300, + ) + solution, _fitness = optimizer.search() + + optimal = np.array([1.0, 0.5]) + distance = np.linalg.norm(solution - optimal) + + # Firefly can have more variance but shouldn't be at (1.2, 0.7) + assert distance <= 0.25, ( + f"CRITICAL: FireflyAlgorithm solution {solution} is too far. " + f"Distance: {distance:.4f}. Solutions like (1.2, 0.7) are failures!" + )