Skip to content

Conversation

ChrisRackauckas-Claude
Copy link
Contributor

Summary

  • Add comprehensive package version tracking to autotune telemetry output
  • Implement correctness checks against LUFactorization baseline
  • Improve reproducibility and debugging of benchmark results

Motivation

Fixes #669 - Improves the autotune results collection by including exact versions of all solver dependencies and ensuring result correctness.

Changes

1. Package Version Tracking

  • Added get_package_versions() function in gpu_detection.jl that collects versions of:
    • LinearSolve.jl and LinearSolveAutotune.jl
    • All solver dependencies (RecursiveFactorization.jl, CUDA.jl, Metal.jl)
    • BLAS libraries (MKL_jll, blis_jll, OpenBLAS_jll, LAPACK_jll)
    • Other relevant packages (Krylov, SparseArrays, etc.)
  • Versions are now displayed in telemetry output under a "Package Versions" section
  • Added Pkg as a dependency to LinearSolveAutotune

2. Correctness Checks

  • Added optional correctness checking in benchmark_algorithms()
  • Each algorithm's solution is compared against LUFactorization baseline
  • Algorithms that fail correctness check (relative error > tolerance) are:
    • Warned with detailed error information
    • Excluded from benchmark results
  • Default tolerance is 1e-2 (very lenient) to catch only major issues
  • Can be configured via correctness_tol parameter

3. Testing

  • Created test script verifying both features work correctly
  • Tests pass with expected output showing:
    • Package versions collected successfully
    • Correctness checks functioning as intended

Example Output

### Package Versions
- **BLAS_lib**: libopenblas64_.so
- **CUDA**: 5.8.3
- **LinearSolve**: 3.27.0
- **Metal**: 1.7.0
- **MKL_jll**: 2025.2.0+0
- **RecursiveFactorization**: 0.2.23
...

Test Plan

  • Run local tests with test_autotune_improvements.jl
  • Verify package versions are collected and displayed
  • Verify correctness checks work and exclude failing algorithms
  • Format code with JuliaFormatter (SciMLStyle)
  • CI tests should pass

This improves the reliability and reproducibility of LinearSolveAutotune benchmark results.

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]

…otune

- Add get_package_versions() function to collect versions of all LinearSolve-related dependencies
- Include versions of LinearSolve, RecursiveFactorization, CUDA, Metal, MKL_jll, blis_jll, etc.
- Display package versions in telemetry output for better reproducibility
- Add correctness check against LUFactorization baseline
- Algorithms that fail correctness check (with lenient tolerance) are warned and excluded
- Default tolerance is 1e-2 (very lenient) to catch major issues
- Helps ensure benchmark results are meaningful and catch any fishy business
Required for get_package_versions() function to use Pkg.dependencies()
@ChrisRackauckas ChrisRackauckas merged commit 2f4b259 into SciML:main Aug 9, 2025
100 of 117 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Autotune Results
2 participants