Commit ac6986d
committed
Merge PR MPAS-Dev#26: Implement OpenACC-friendly halo exchanges
This merge introduces new infrastructure for OpenACC-friendly halo exchanges for
2-d and 3-d real fields, and it updates the halo exchange calls in the MPAS-Atmosphere
dynamics to use new routines for exchanging halos on both CPUs and GPUs.
Existing halo exchange routines that work on MPAS field types require the use of
complicated "multi-halo exchange lists" within the halo exchange routines themselves.
Transferring these data structures onto GPUs to allow direct GPU-to-GPU halo
exchanges is overly complicated with OpenACC, so new routines (collectively using
the interface `mpas_dmpar_exch_halo_acc`) have been developed.
The `mpas_dmpar_exch_halo_acc` routines make use of a simple array of integers to
describe the communication of elements with other MPI tasks, and are thus are more
easily mapped from host to GPU memory.
The example, below, illustrates the use of the new OpenACC-friendly routines as they
are applied in the MPAS-Atmosphere dynamics:
```
integer, dimension(:), pointer :: compactHaloInfo_c_v
type (field2DReal), pointer :: field_2d_ptr
real (kind=RKIND), dimension(:,:), pointer :: theta_m
! Build integer array of halo information for a given field dimensionality
call mpas_pool_get_field(state, 'rho_zz', field_2d_ptr, 1)
call mpas_dmpar_compact_halo_info(field_2d_ptr, compactHaloInfo_c_v)
! Exchange the halo for a field array using the halo information
call mpas_pool_get_array(state, 'theta_m', theta_m, 1)
call mpas_dmpar_exch_halo_acc(theta_m, compactHaloInfo_c_v)
! Halo information can be re-used as many times as needed before being deallocated
deallocate(compactHaloInfo_c_v)
```
Note that the `mpas_pool_get_array_gpu` routine contains an `!ACC ENTER DATA COPYIN`
directive internally to check if data are present on the GPU or should be copied.
This facilitates the GPU direct communication.
Merge branch 'gpu_halos_merge' into atmosphere/gpu
* gpu_halos_merge: (24 commits)
Replacing CPU halo exchange calls with CPU + GPU halo exachage calls.
Initialize commListPtr = 0 when building send/recv lists for OpenACC exchanges
Compute compactHaloInfo arrays needed by scalar transport in mpas_atm_dynamics_init
Move computation of compactHaloInfo_* arrays into mpas_atm_dynamics_init
Add empty mpas_atm_dynamics_{init,finalize} routines
Pre-process out log writes in mpas_dmpar_exch_halo_acc if running on GPUs
Convert all halo exchanges in atm_srk3() to mpas_dmpar_exch_halo_acc interface
Remove a few debugging write statements from mpas_dmpar_exch_halo_acc routines
Only use compactHaloInfo and field arrays in mpas_dmpar_exch_halo_acc routines
Build {send,recv}CommList exclusively from compactHaloInfo in mpas_dmpar_build_comm_lists_acc
Add endpoint IDs to compactHaloInfo array
Use compactHaloInfo to obtain product of inner dims in mpas_dmpar_build_comm_lists_acc
Add sendList and recvList information to compactHaloInfo array
Use constant layout of compactHaloInfo for all field types
Add mpas_dmpar_build_comm_lists_acc routine to create comm lists from compactHaloInfo
In mpas_dmpar_exch_halo_acc routines, get nHaloLayers from compactHaloInfo
Add 'array' argument to mpas_dmpar_exch_halo_acc routines
Get MPI communicator and rank info from compactHaloInfo in mpas_dmpar_exch_halo_acc
mpas_dmpar_exch_halo_acc routines now require compactHaloInfo argument
Add interface mpas_dmpar_compact_halo_info
...4 files changed
Lines changed: 1332 additions & 134 deletions
File tree
- src
- core_atmosphere
- dynamics
- physics
- framework
0 commit comments