Enable Parmetis parallel partitioning for large meshes

For larger meshes, this adds a parallel partitioning for reduced memory footprint and faster partitioning. It includes

    - added ParMetisKWay partitioning with a ParMetisKWay config option
    - updated documentation
    - removal of many unnecessary return codes and integration with new error handler

The serial MetisKWay option is retained as the default since it is more efficient for smaller meshes and node counts. To test the new option, the input config must be manually modified to use the ParMetisKWay method in the Decomp config. As noted in the updated documentation, ParMetis gives a somewhat different decomposition than the serial algorithm (and the offline gpmetis tool which matches the serial version).

Author: philipwjones

components/omega/doc/devGuide/Decomp.md

@@ -6,8 +6,8 @@ In order to run across nodes in a parallel computer, Omega subdivides the
 horizontal domain into subdomains that are distributed across the machine
 and communicate using message passing via the Message Passing Interface (MPI).
 To decompose the domain, we utilize the
-[METIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview) library.
-An Omega mesh is fully described by the
+[METIS/ParMETIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview)
+libraries.  An Omega mesh is fully described by the
 [MPAS Mesh Specification](https://mpas-dev.github.io/files/documents/MPAS-MeshSpec.pdf)
 which we will reproduce here eventually. The Decomp class decomposes
 the domain based on the index space and number of MPI tasks (currently
@@ -21,11 +21,14 @@ with the call:
 ```
 This must be called very early in the init process, just after initializing
 the MachEnv, Config and IO.  Mesh information is first read using parallel
-IO into an equally-spaced linear decomposition, then partitioned by METIS
-into a more optimal decomposition. The parallel METIS implementation
-(ParMETIS) will eventually be used to reduce the memory footprint for
-high-resolution configurations, but currently we use the serial METIS library
-for the partitioning.
+IO into an equally-spaced linear decomposition, then partitioned by METIS or
+ParMETIS into a more optimal decomposition. The parallel ParMETIS
+implementation should be used to reduce the memory footprint for
+high-resolution configurations. Although the algorithmic method is similar,
+ParMETIS leads to a somewhat different partitioning than the serial METIS
+version. The serial version should give the same partition as the serial
+offline gpmetis tool used by MPAS when the same version of the METIS library
+is used.
 
 METIS requires information about the connectivity in the mesh. In particular,
 it needs the total number of cells and edges in the mesh and the connectivity

components/omega/doc/userGuide/Decomp.md

@@ -12,25 +12,30 @@ cells, edges and vertices so that a time step can largely be completed
 without the need to communicate.
 
 The partitioning itself is performed using the
-[METIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview) library that
-can partition unstructured domains in a way that also optimizes communication.
-More details on the mesh, connectivity and partitioning can be found in
-the [Developer's Guide](#omega-dev-decomp).
+[METIS/ParMETIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview)
+libraries that can partition unstructured domains in a way that also optimizes
+communication. More details on the mesh, connectivity and partitioning can be
+found in the [Developer's Guide](#omega-dev-decomp).
 
-There are three parameters that are set by the user in the input configuration
+There are two parameters that are set by the user in the input configuration
 file. These are:
 ```yaml
 Decomp:
    HaloWidth: 3
-   MeshFileName: OmegaMesh.nc
    DecompMethod: MetisKWay
 ```
-(until the config module is complete, these are currently hardwired to
-the defaults above). The HaloWidth is set to be able to compute all of the
-baroclinic terms in a timestep without communication and for higher-order
-tracer advection terms, this currently must be at least 3. The MeshFileName
-should include the complete path and filename to a standard Omega mesh file
-that contains at a minimum
+The HaloWidth is set to be able to compute all of the baroclinic terms in a
+timestep without communication and for higher-order tracer advection terms,
+this currently must be at least 3.
+
+METIS and ParMETIS support a number of partitioning schemes, but the MetisKWay
+and its parallel equivalent ParMetisKWay are the only currently supported
+decomposition methods for Omega. The serial form (MetisKWay) is typically
+faster for small meshes and decompositions. The ParMetisKWay option is a
+parallel form that is faster and requires less memory than the serial option.
+Note that the two methods will not result in the same partition.
+
+An input mesh file must be provided that contains at a minimum
   - the total number of cells, edges and vertices (NCells, NEdges, NVertices)
   - the mesh connectivity contained in the arrays CellsOnCell, EdgesOnCell
     VerticesOnCell, CellsOnEdge, EdgesOnEdge, CellsOnVertex, EdgesOnVertex.
@@ -40,12 +45,8 @@ The mesh information is read via parallel IO into an initial linear domain
 decomposition and then is partitioned by METIS and rearranged into the
 final METIS parallel decomposition.
 
-METIS and ParMETIS support a number of partitioning schemes, but MetisKWay
-is currently the only supported decomposition method for Omega and is
-generally the better option.
-
 Once the mesh is decomposed, all of the mesh index arrays are stored in
 a Decomp named Default which can be retrieved as described in the
 Developer guide. In the future, additional decompositions associated
-with processor subsets (as described in MachEnv) but this capability is
-not yet supported.
+with processor subsets (as described in MachEnv) should be possible, but
+this capability is not yet supported.