Skip to content

Conversation

ChrisRackauckas-Claude
Copy link
Contributor

Summary

Previously, the LinearSolveAutotune telemetry system filtered out NaN values from failed benchmark results, making it unclear what algorithms were actually tested but unsuccessful. This PR improves transparency by displaying all attempted algorithms, including those that failed.

Changes

🔧 Telemetry Display Improvements

  • Include all results: Both successful and failed results are now shown in telemetry output
  • Explicit NaN display: Failed results show "NaN" instead of being hidden or showing "N/A"
  • Success/Total counts: Summary statistics now show "2/3" format indicating successful vs total tests
  • Clear status indicators: Both ✅ Success and ❌ Failed statuses are displayed
  • Console reporting: Failed algorithms are reported in console output for immediate visibility

📊 Before vs After

Before: Only successful results were shown, hiding what was actually attempted

| Algorithm | Avg GFLOPs | Std Dev | Tests |
|-----------|------------|---------|-------|
| TestAlg1  | 11.00      | 1.41    | 2     |

After: All attempts are visible with clear success/failure indication

| Algorithm | Avg GFLOPs | Std Dev | Success/Total |
|-----------|------------|---------|---------------|
| TestAlg1  | 11.00      | 1.41    | 2/3           |

| Matrix Size | GFLOPs | Status      |
|-------------|--------|-------------|
| 200         | NaN    | ❌ Failed   |

Files Modified

  • lib/LinearSolveAutotune/src/telemetry.jl: Updated format_results_for_github() and format_detailed_results_markdown() to include failed results
  • lib/LinearSolveAutotune/src/LinearSolveAutotune.jl: Enhanced console output and summary tables to show all tested algorithms

Test Results

Verified that:

  • ✅ NaN values are explicitly displayed as "NaN"
  • ✅ Failed results show ❌ Failed status
  • ✅ Success/Total counts accurately reflect all attempts
  • ✅ Existing successful results continue to display correctly
  • ✅ All algorithm names are preserved in output

Benefits

  • Transparency: Users can see exactly what algorithms were tested
  • Debugging: Failed attempts provide insight into algorithm reliability
  • Coverage: Complete view of benchmark scope, not just successes
  • Clarity: Explicit indication of what didn't work and why

This addresses user feedback requesting visibility into failed benchmark attempts that were previously hidden from telemetry output.

🤖 Generated with Claude Code

Previously, the telemetry system filtered out NaN values from failed
benchmark results, making it unclear what algorithms were actually
tested but unsuccessful. This change improves transparency by:

- Including all tested algorithms in telemetry output, not just successful ones
- Displaying "NaN" explicitly for failed results instead of hiding them
- Showing "Success/Total" counts (e.g., "2/3") in summary statistics
- Adding ❌ Failed status indicators alongside ✅ Success markers
- Reporting algorithms that had failures/timeouts in console output

This allows users to see exactly what was attempted and understand
the complete benchmark coverage, including which algorithms failed
due to timeouts, convergence issues, or other errors.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@ChrisRackauckas ChrisRackauckas merged commit 5b18502 into SciML:main Aug 14, 2025
115 of 118 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants