Open
Conversation
This commit adds support for linking MPAS with the SCOTCH graph partitioning library. This capability is intended as a first step towards MPAS cores being able to generate graph partitions during runtime, by invoking the distributed graph partitioning algorithms provided by PT-SCOTCH. In order for MPAS to be able to link with the SCOTCH library, the SCOTCH environment variable must be set to point to the SCOTCH installation path prior to build MPAS. export SCOTCH=/path/to/scotch/installation make gnu CORE=atmosphere If the SCOTCH env variable has been set, but it does not point to the correct SCOTCH installation directory, then the Scotch C test in the Makefile would fail, yielding the following message: Could not build a simple C program with Scotch. If MPAS has been successfully linked with the SCOTCH library, the build summary will include the message: MPAS has been linked with the Scotch graph partitioning library. Furthermore, the log file at model runtime will also include the following message near the top: SCOTCH support: yes Conversely, if MPAS has not been linked with SCOTCH, then appropriate messages are displayed in the aforementioned locations.
This commit adds a C and Fortran source file to the src/framework directory, and builds on the previous commit to provide a set of routines which enable various MPAS cores to invoke the SCOTCH library to do the following tasks: - Initialize, build and check a distributed SCOTCH graph - Partition and redistribute the distributed graphs - Query distributed graphs - Initialize SCOTCH graph mapping strategies Together these routines enable the MPAS framework to generate online graph partitioning. This commit also includes modifications to the src/framework Makefile to build and C and Fortran interfaces. To avoid unnecessary references to the SCOTCH fortran library, this commit introduces two derived types, scotchm_dgraph and scotchm_strat in the Fortran mpas_ptscotch_interface. These two derived types hold C pointers to memory locations of type SCOTCH_Dgraph and SCOTCH_Strat respectively, which are allocated in the C-interface.
This commit introduces a subroutine mpas_block_decomp_scotch in src/framework/mpas_block_decomp.F, which calls the routines introduced in the previous commit to invoke Scotch libraries. - The routine initializes and builds PT-SCOTCH distributed graphs from the adjacency data available in the partial_global_graph on each MPI task. - After checking the distributed graph for consistency, the SCOTCH graph mapping strategy is initialized, followed by the graph partitioning. Following a successful redistribution, the graph must be redistributed so that each MPI task is aware of all the vertices that it owns. - Finally, the routine writes out the generated graph partitioning data to disk so that it may be reused for the next model run.
… found This commit modifies the existing logic in mpas_block_decomp_cells_for_proc routine inside src/framework/mpas_block_decomp.F to call the mpas_block_decomp_scotch routine, when all of the following are satisfied: - The MPAS core has been built with SCOTCH support - The routine is unable to find a relevant graph partition file on disk As described in the previous commit, the call to mpas_block_decomp_scotch begins the process of online graph partitioning, resulting in the generated partition file being written out to disk. Subsequently, the routine mpas_block_decomp_cells_for_proc attempts to read this newly-generated partition file, and if successful, proceeds with the rest of initialization.
This commit introduces new timers to the mpas_block_decomp_cells_for_proc and the mpas_block_decomp_scotch routines in src/framework/mpas_block_decomp.F. These timers may be useful in tracking the time spent in online graph partitioning or in reading the graph partition files from disk and in collective MPI calls.
abishekg7
force-pushed
the
ptscotch_c_f
branch
from
b9011b0 to
8c8f81e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The title above should be a 1 line short summary of the pull request (i.e. what the project the PR represents is intended to do).
Enter a description of this PR. This should include why this PR was created, and what it does.
Testing and relations to other Pull Requests should be added as subsequent comments.
See the below examples for more information.
MPAS-Dev/MPAS#930
MPAS-Dev/MPAS#931