Skip to content

Adds new global reductions#330

Merged
philipwjones merged 3 commits intoE3SM-Project:developfrom
philipwjones:omega/new-reductions
Feb 11, 2026
Merged

Adds new global reductions#330
philipwjones merged 3 commits intoE3SM-Project:developfrom
philipwjones:omega/new-reductions

Conversation

@philipwjones
Copy link

@philipwjones philipwjones commented Jan 22, 2026

This is a rewrite of all the global reductions routines to fix the treatment of restricted address ranges. It includes

  • correct treatment of address ranges for multi-dimensional arrays
  • results returned as return value rather than function parameter
  • reproducibility for both host and device arrays
  • an updated and more robust unit test
  • updated documentation
  • no error return codes, aborts on error instead

This PR does not include multi-field interfaces, except for scalars. Those will be added in a future PR. Also, the reproducible sums for device R8 arrays are actually computed on the host currently - custom kokkos reductions to compute on the device will also be added later. The additional cost of computing on host is negligible compared to MPI overhead.

Fixes #320

Testing:
New unit test driver added for Reduction unit tests and updated to run on 8 MPI ranks
All CTest unit tests pass on Chrysalis (Intel), Frontier CPU and GPU with gnu, amd.

Checklist

  • Documentation:
  • Linting
  • Building
    • CMake build does not produce any new warnings from changes in this PR
  • Testing
    • Add a comment to the PR titled Testing with the following:
      • Which machines CTest unit tests
        have been run on and indicate that are all passing.
      • The Polaris omega_pr test suite
        has passed, using the Polaris e3sm_submodules/Omega baseline
      • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
      • Indicate "All tests passed" or document failing tests
      • Document testing used to verify the changes including any tests that are added/modified/impacted.
      • Performance related PRs: Please include a relevant PACE experiment link documenting performance before and after.
    • New tests:
      • CTest unit tests for new features have been added per the approved design.

  - fixes treatment of index range
  - includes reproducible sums for device arrays
  - adds minval and maxval routines
  - results are return values rather than function parameters
  - adds a more robust unit test for all interfaces
  - updates documentation
  - all routines abort on error rather than returning an error code
Copy link
Member

@amametjanov amametjanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked on chrysalis, pm and aurora and all ctests are passing:

  • chrysalis:
    • intel: 100% tests passed, 0 tests failed out of 35
    • gnu: 100%
  • pm:
    • intel: 100%
    • gnu: 100%
    • gnugpu: 100%
  • aurora:
    • oneapi-ifxgpu: 100%
    • oneapi-ifx: 80% tests passed, 7 tests failed out of 35
      • which is fine b/c the base/fork-point of the branch on develop has the same 7 fails:
           11 - HORZOPERATORS_PLANE_TEST (Failed)                 Omega-0 SERIAL
           12 - HORZOPERATORS_SPHERE_TEST (Failed)                Omega-0 SERIAL
           15 - AUXSTATE_TEST (Failed)                            Omega-0 SERIAL
           20 - TEND_PLANE_TEST (Failed)                          Omega-0 SERIAL
           21 - TEND_PLANE_SINGLE_PRECISION_TEST (Failed)         Omega-0 SERIAL
           22 - TEND_SPHERE_TEST (Failed)                         Omega-0 SERIAL
           34 - EOS_TEST (Failed)                                 Omega-0 SERIAL
        

Copy link

@grnydawn grnydawn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All CTests are passed on PM-GPU as well ass PM-CPU.

@philipwjones philipwjones merged commit e6e7b09 into E3SM-Project:develop Feb 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global reductions do not work correctly for multi-dim arrays and restricted index ranges

3 participants