Skip to content

Commit ac6986d

Browse files
committed
Merge PR MPAS-Dev#26: Implement OpenACC-friendly halo exchanges
This merge introduces new infrastructure for OpenACC-friendly halo exchanges for 2-d and 3-d real fields, and it updates the halo exchange calls in the MPAS-Atmosphere dynamics to use new routines for exchanging halos on both CPUs and GPUs. Existing halo exchange routines that work on MPAS field types require the use of complicated "multi-halo exchange lists" within the halo exchange routines themselves. Transferring these data structures onto GPUs to allow direct GPU-to-GPU halo exchanges is overly complicated with OpenACC, so new routines (collectively using the interface `mpas_dmpar_exch_halo_acc`) have been developed. The `mpas_dmpar_exch_halo_acc` routines make use of a simple array of integers to describe the communication of elements with other MPI tasks, and are thus are more easily mapped from host to GPU memory. The example, below, illustrates the use of the new OpenACC-friendly routines as they are applied in the MPAS-Atmosphere dynamics: ``` integer, dimension(:), pointer :: compactHaloInfo_c_v type (field2DReal), pointer :: field_2d_ptr real (kind=RKIND), dimension(:,:), pointer :: theta_m ! Build integer array of halo information for a given field dimensionality call mpas_pool_get_field(state, 'rho_zz', field_2d_ptr, 1) call mpas_dmpar_compact_halo_info(field_2d_ptr, compactHaloInfo_c_v) ! Exchange the halo for a field array using the halo information call mpas_pool_get_array(state, 'theta_m', theta_m, 1) call mpas_dmpar_exch_halo_acc(theta_m, compactHaloInfo_c_v) ! Halo information can be re-used as many times as needed before being deallocated deallocate(compactHaloInfo_c_v) ``` Note that the `mpas_pool_get_array_gpu` routine contains an `!ACC ENTER DATA COPYIN` directive internally to check if data are present on the GPU or should be copied. This facilitates the GPU direct communication. Merge branch 'gpu_halos_merge' into atmosphere/gpu * gpu_halos_merge: (24 commits) Replacing CPU halo exchange calls with CPU + GPU halo exachage calls. Initialize commListPtr = 0 when building send/recv lists for OpenACC exchanges Compute compactHaloInfo arrays needed by scalar transport in mpas_atm_dynamics_init Move computation of compactHaloInfo_* arrays into mpas_atm_dynamics_init Add empty mpas_atm_dynamics_{init,finalize} routines Pre-process out log writes in mpas_dmpar_exch_halo_acc if running on GPUs Convert all halo exchanges in atm_srk3() to mpas_dmpar_exch_halo_acc interface Remove a few debugging write statements from mpas_dmpar_exch_halo_acc routines Only use compactHaloInfo and field arrays in mpas_dmpar_exch_halo_acc routines Build {send,recv}CommList exclusively from compactHaloInfo in mpas_dmpar_build_comm_lists_acc Add endpoint IDs to compactHaloInfo array Use compactHaloInfo to obtain product of inner dims in mpas_dmpar_build_comm_lists_acc Add sendList and recvList information to compactHaloInfo array Use constant layout of compactHaloInfo for all field types Add mpas_dmpar_build_comm_lists_acc routine to create comm lists from compactHaloInfo In mpas_dmpar_exch_halo_acc routines, get nHaloLayers from compactHaloInfo Add 'array' argument to mpas_dmpar_exch_halo_acc routines Get MPI communicator and rank info from compactHaloInfo in mpas_dmpar_exch_halo_acc mpas_dmpar_exch_halo_acc routines now require compactHaloInfo argument Add interface mpas_dmpar_compact_halo_info ...
2 parents b5a700e + cd6e603 commit ac6986d

4 files changed

Lines changed: 1332 additions & 134 deletions

File tree

0 commit comments

Comments
 (0)