Skip to content

Conversation

@vkarak
Copy link
Contributor

@vkarak vkarak commented Nov 20, 2025

This PR offers the following features:

  1. Multiple statistics can be presented at once using the special stats aggregator. These includes mean, standard deviation, and 0.01, 0.05, 0.50, 0.95, 0.99 quantiles.
  2. The cmp_spec syntax is updated to support multiple statistics at once, e.g., mean,min,max, and also aggregations on values other than the performance, e.g., min(num_tasks),min(pref) etc.
  3. Two new command-line options are added to control the name of the left and right terms in the comparisons. These are the --term-lhs and --term-rhs and are only meaningful with the --performance-compare and --performance-report options.
  4. The default name suffixes for left/right terms in comparisons are now (lhs) and (rhs) respectively.
  5. The selectors of left and right columns in the cmp spec are now _L and _R instead of _A and _B.

Closes #3436.

Implementation details

The analytics layers is now using Polars through its Python bindings.

Managing the column names was the most challenging part of the implementation, as we wanted to keep a straightforward for selecting test attributes from the command line, specifying the left/right variants quickly and the same time produce tables with meaningful headers (e.g., include the aggregation in the column name, especially now, that multiple aggregations are supported at once). This is handled the the internal _QueryMatch class which replaces the very thin former _Match named tuple.

Todos

  • Update documentation
  • Update tutorial examples

Notes

If the CI does not pass for Python < 3.9, we should simply merge this PR after #3583 is addressed.

@vkarak vkarak added this to the ReFrame 4.10 milestone Nov 20, 2025
@vkarak vkarak self-assigned this Nov 20, 2025
@vkarak vkarak added the reporting Issues related to reporting and processing the test results label Nov 20, 2025
@codecov
Copy link

codecov bot commented Nov 20, 2025

Codecov Report

❌ Patch coverage is 95.33898% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.67%. Comparing base (2546434) to head (68ed859).

Files with missing lines Patch % Lines
reframe/frontend/reporting/utility.py 94.51% 9 Missing ⚠️
reframe/frontend/reporting/__init__.py 96.72% 2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (2546434) and HEAD (68ed859). Click for more details.

HEAD has 14 uploads less than BASE
Flag BASE (2546434) HEAD (68ed859)
23 9
Additional details and impacted files
@@             Coverage Diff              @@
##           develop    #3585       +/-   ##
============================================
- Coverage    91.28%   69.67%   -21.61%     
============================================
  Files           62       60        -2     
  Lines        13583    13519       -64     
============================================
- Hits         12399     9420     -2979     
- Misses        1184     4099     +2915     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vkarak vkarak marked this pull request as draft November 27, 2025 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement prio: normal reporting Issues related to reporting and processing the test results

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

Report important performance statistics all at once with --list-stored-testcases

1 participant