Convergence tests for solvers

[GiovanniCanali]

Currently, there are no tests that guarantee solver convergence. Defining a benchmark and running convergence tests on a rolling basis would help ensure that solver behavior remains consistent with expectations.

Ideally, these tests would be managed via a parser file that, given specific inputs (problem, model, solver), automatically generates a Python script to be executed. A test is considered successful if the script runs without errors and the resulting trained model achieves performance above a predefined threshold.

[GiovanniCanali]

Let's set a tentative deadline for the end of the year.

[adendek]

I found this issue interesting. I wiling to contribute, if you need one.

[GiovanniCanali]

Thanks a lot @adendek for your interest! We’re still aligning internally on how we want to approach this issue, but in the meantime feel free to share any ideas or drafts you might have - they’d be very welcome.

[adendek]

Apology for the late response. @GiovanniCanali when do you plan to start working on that? Can this be addressed next week? I'll do my best find some time to work on this.

I have the initial idea:

---

The core idea is to completely separate the test definition (what to test) from the test execution logic (how to run the test and check the results).

1. Test Definition via YAML: All convergence benchmark parameters (problem type, model, solver, data source, and the required performance threshold) are defined in a human-readable convergence_benchmarks.yml file. This centralizes test management.

2. Parametrized Test Runner: A Python parser reads this YAML file and converts each benchmark entry into a list of arguments.

3. Automatic Execution: The @parameterized.expand decorator uses these arguments to automatically call a single test function multiple times.

4. Threshold Assertion: The test function executes the actual solver training and then uses the expected_outcome parameters to assert the final performance metric (e.g., MSE is < 0.1 or Accuracy is > 0.95).

This architecture provides a robust, extensible, and clean way to manage solver convergence tests for our project.

---

1. New Benchmark Definition Format (YAML)

The YAML file will define all necessary inputs for a complete benchmark run, including the problem, model, solver, and the expected performance threshold.

# --- Example Benchmark 1: Simple Linear Regression ---
Benchmark_LR_1:
  name: "Linear Regression - Fast Convergence Test"
  
  inputs:
    problem: "Linear Regression" 
    model: "Simple Linear Model"
    solver: "SGD"
    data_source: "data/toy_linear_dataset.csv"
    solver_params:
      learning_rate: 0.01
      epochs: 50
  
  # Expected Outcome (Performance Metric)
  expected_outcome:
    metric: "MSE"
    threshold: 0.1
    # Condition: Must be less than the threshold (for loss/error metrics)
    condition: "less_than" 

# --- Example Benchmark 2: Classification Accuracy ---
Benchmark_CL_1:
  name: "Binary Classification - Baseline Accuracy"
  inputs:
    problem: "Binary Classification"
    model: "Logistic Regression"
    solver: "Adam"
    data_source: "data/iris_binary_subset.parquet"
    solver_params:
      learning_rate: 0.005
      epochs: 100
  expected_outcome:
    metric: "accuracy"
    threshold: 0.95
    # Condition: Must be greater than the threshold (for success/accuracy metrics)
    condition: "greater_than"

2. Python Parsing Function

This function uses pyyaml to load the structured data and transforms it into the format required by the @parameterized.expand decorator: a list of argument tuples/lists.

import yaml
from typing import List, Dict, Any, Union

# Define the structure for a single test case's arguments
TestExample = List[Union[str, float, Dict[str, Any]]]

def get_convergence_benchmarks_from_yml(filename: str) -> List[TestExample]:
    """
    Reads convergence benchmarks from a YAML file and formats them for 
    parameterized testing: [name, problem, model, solver, data_source, 
    solver_params, metric, threshold, condition].
    """
    all_benchmarks: List[TestExample] = []
    
    with open(filename, "r") as example_yml:
        data = yaml.safe_load(example_yml)
        
        for _, benchmark in data.items():
            inputs = benchmark["inputs"]
            expected = benchmark["expected_outcome"]
            
            # Assemble arguments in the exact order the test function expects
            test_arguments: TestExample = [
                benchmark["name"],
                inputs["problem"],
                inputs["model"],
                inputs["solver"],
                inputs["data_source"],
                inputs.get("solver_params", {}), # Use .get for optional parameters
                expected["metric"],
                expected["threshold"],
                expected["condition"],
            ]
            all_benchmarks.append(test_arguments)

    return all_benchmarks

3. Parametrized Test Implementation

from parameterized import parameterized
# Note: Assume run_solver_benchmark() is the function that executes 
# the training script and returns the final metric.

class TestSolverConvergence(BaseTestCase):
    
    # Loads arguments for every test case defined in the YAML file
    @parameterized.expand(get_convergence_benchmarks_from_yml("convergence_benchmarks.yml"))
    def test_solver_benchmark(
        self, 
        name: str, 
        problem: str, 
        model: str, 
        solver: str, 
        data_source: str,
        solver_params: Dict[str, Any],
        metric: str,
        threshold: float,
        condition: str,
    ) -> None:
        """Runs the solver benchmark and asserts against the expected performance."""
        
        # 1. Execution Step (This is where the actual training/script run happens): 
        # final_performance_metric = run_solver_benchmark(
        #     problem, model, solver, data_source, solver_params
        # )
        
        # --- Placeholder for demonstration ---
        final_performance_metric = 0.05 
        # --- End Placeholder ---

        # 2. Assertion Step: Check if performance meets the threshold
        
        if condition == "less_than":
            # For metrics like 'loss' or 'MSE'
            self.assertLess(
                final_performance_metric, threshold,
                f"Convergence FAIL: {metric} ({final_performance_metric}) did not converge below threshold ({threshold})."
            )
        elif condition == "greater_than":
            # For metrics like 'accuracy' or 'F1-score'
            self.assertGreater(
                final_performance_metric, threshold,
                f"Convergence FAIL: {metric} ({final_performance_metric}) did not converge above threshold ({threshold})."
            )
        else:
            self.fail(f"Unsupported condition defined in YAML: {condition}")

---

Please let me know, if this approach works for you. If yes, next week I'll proceed with some initial PR.

[GiovanniCanali]

Thank you @adendek for the idea — really appreciated.

The general structure looks good to me. Before going broader, I’d suggest we start with a very minimal setup so we can validate the overall flow:

• Problem: use only the Poisson2DSquareProblem problem from the problem/zoo.
• Model: a standard FeedForward network, e.g. 2-64-64-1.
• Metric: the relative error with respect to the analytical solution (included in the problem).
• Optimizer / scheduler: keep the default ones for now.
• Training: run for 5000 epochs.
• Pass condition: relative error < 1%.

For now, the priority is the automation of test creation, not whether this first example passes or not. What matters is that everything is fully driven by YAML, so adding tests in the future requires no manual test coding.

Let’s keep it simple at this stage — we just want to see the structure working end-to-end. If the approach feels clean and maintainable, we can then expand it in a more structured way.

Let’s try this first and then evaluate how to proceed.

[adendek]

Thanks @GiovanniCanali for your comment.

I'll do my best to submit the PR next week. This one is fully packed.

One more question: I'll create a dedicated workflow (GH Action) for these tests to avoid contaminating the Unit Tests, which ideally should not take too much execution time.

My idea to distinguish the convergence tests from the regular UT is to add the test name prefix coverage.

And then the workflow will look like this:

  coverage_tests: #################################################################
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [windows-latest, macos-latest, ubuntu-latest]
        python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']
        
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install Python dependencies
      run: |
        python3 -m pip install --upgrade pip
        python3 -m pip install .[test]

    - name: Test with pytest
      run: |
        python3 -m pytest -k "coverage"  
 

while the existing unit tests workflow will be modified to exclude coverage tests:

        python3 -m pytest -k "not coverage" 

Please let me know if this makes sense.

[GiovanniCanali]

Sorry for the late reply @adendek.

What you suggest makes sense to me. Using a naming prefix and -k filtering seems reasonable. Please go ahead as proposed.