Merge pull request #566 from JuliaParallel/vc/rewrite_docs

Rewrite configuration docs for MPIPreferences

Author: vchuravy

docs/src/configuration.md

@@ -9,10 +9,6 @@ clusters or multi-GPU machines, you will probably want to configure against a
 system-provided MPI implementation in order to exploit features such as fast network
 interfaces and CUDA-aware MPI interfaces.
 
-MPI.jl will attempt to detect when you are running on a HPC cluster, and warn the user
-about this. To disable this warning, set the environment variable
-`JULIA_MPI_CLUSTER_WARN=n`.
-
 ## Julia wrapper for `mpiexec`
 
 Since you can configure `MPI.jl` to use one of several MPI implementations, you
@@ -52,64 +48,147 @@ with:
 $ mpiexecjl --project=/path/to/project -n 20 julia script.jl
 ```
 
-## Using a system-provided MPI
+## Using MPIPreferences.jl
+
+MPI.jl uses [Preferences.jl](https://github.com/JuliaPackaging/Preferences.jl) to
+allow the user to choose which MPI implementation to use for a project. This provides
+a single source of truth that can be used for JLL packages (Julia packages providing C libraries)
+that link against MPI, localizes the choice of MPI implementation to a project.
+
+Users can use the provided [`use_system_binary`](@ref) or [`use_jll_binary`](@ref)
+to switch MPI implementations. By default, the JLL-provided binaries are used.
+
+### Migration from MPI.jl `v0.19`
 
-### Requirements
+Prior to MPI.jl `v0.20` environment variables were used to configure which MPI
+library to use. These have now been removed and have no effect anymore:
+
+- `JULIA_MPI_BINARY`
+- `JULIA_MPIEXEC`
+- `JULIA_MPI_INCLUDE_PATH`
+- `JULIA_MPI_CFLAGS`
+- `JULIA_MPICC`
+
+### Using a system-provided MPI backend
+
+#### Requirements
 
 MPI.jl requires a shared library installation of a C MPI library, supporting the MPI 3.0
 standard or later.
 
-### Building
+### Configuration
 
-To use the the system MPI, set the environment variable `JULIA_MPI_BINARY=system` and run
-`Pkg.build("MPI")`. This can be done by:
-```
-julia --project -e 'ENV["JULIA_MPI_BINARY"]="system"; using Pkg; Pkg.build("MPI"; verbose=true)'
+To use the system MPI library, run `MPI.use_system_binary()`.
+This will attempt to locate and to identify any available MPI implementation, and create
+a file called `LocalPreferences.toml` adjacent to the current `Project.toml`.
+Use `Base.active_project()` to obtain the location of the currently active project.
+
+```sh
+julia --project -e 'using MPI; MPI.use_system_binary()'
 ```
-This will attempt find and identify any available MPI implementation.
 
-The MPI standard doesn't specify the exact application binary interface (ABI).
-The build script will attempt to build a small C program to
-determine the appropriate type definitions and constants. This requires a compatible C
-compiler (`mpicc` by default).
+!!! note
+    You can copy `LocalPreferences.toml` to a different project folder, but you must list
+    `MPIPreferences` in the `[extras]` section of the `Project.toml` for the settings
+    to take effect. Due to a bug in Julia (until `v1.6.5` and `v1.7.1`), getting preferences
+    from transitive dependencies is broken (https://github.com/JuliaPackaging/Preferences.jl/issues/24).
+    To fix this update your version of Julia, or add `MPIPreferences` as a direct dependency to your project.
 
-The following implementations should work:
+
+The following MPI implementations should work out-of-the-box with MPI.jl:
 
 - [Open MPI](http://www.open-mpi.org/)
 - [MPICH](http://www.mpich.org/) (v3.1 or later)
 - [Intel MPI](https://software.intel.com/en-us/mpi-library)
 - [Microsoft MPI](https://docs.microsoft.com/en-us/message-passing-interface/microsoft-mpi)
 - [IBM Spectrum MPI](https://www.ibm.com/us-en/marketplace/spectrum-mpi)
+- [MVAPICH](http://mvapich.cse.ohio-state.edu/)
+- [Cray MPICH](https://docs.nersc.gov/development/compilers/wrappers/)
+- [Fujitsu MPI](https://www.fujitsu.com/global/about/resources/publications/technicalreview/2020-03/article07.html#cap-03)
+
+If the implementation is changed, you will need to use [`MPI.use_system_binary()`](@ref) or
+[`MPI.use_jll_binary()`](@ref).
 
-If the implementation is changed, you will need to re-run `Pkg.build("MPI")`.
+#### Advanced options
 
-### [Environment variables](@id environment_variables)
+```@doc
+MPI.use_system_binary
+```
+
+You can use the argument `mpiexec` to provide the name (or full path) of the MPI launcher executable. The default is
+`mpiexec`, but some clusters require using the scheduler launcher interface (e.g. `srun`
+on Slurm, `aprun` on PBS). If the MPI library has an uncommon name you can provide it in `library_names`.
+The MPI standard does not specify the exact application binary interface (ABI).
+In case ABI detection fails you can provide a manual choice (either `MPICH`, `MPItrampoline`, `OpenMPI`, or `MicrosoftMPI`),
+but also open an issue such that the automatic detection can be improved.
+`export_prefs=true` can be used to copy the preferences into the `Project.toml` instead of creating a
+`LocalPreferences.toml` file to hold them.
 
-The following optional environment variables can be used to control certain aspects of the
-build script and other library behaviour. The results of these will be cached in a
-configuration file located at `~/.julia/prefs/MPI.toml` and so can be used for subsequent
-MPI builds.
+#### Notes to HPC cluster adminstators
 
-- `JULIA_MPI_BINARY`: can be set to either the empty string (to use the default implementations
-  above) or `system` (to use a system-provided implementation).
-- `JULIA_MPI_PATH`: the top-level installation directory of MPI. i.e. the library should
-  be located in `${JULIA_MPI_PATH}/lib` and `mpiexec` in `${JULIA_MPI_PATH}/bin`
-- `JULIA_MPI_LIBRARY`: the library name or full path of the MPI shared library. By
-  default, it will attempt to look for common MPI library names in the standard library
-  paths (e.g. `libmpi`, `libmpich`, `msmpi`).
-- `JULIA_MPIEXEC`: the name (or full path) of the MPI launcher executable. The default is
-  `mpiexec`, but some clusters require using the scheduler launcher interface (e.g. `srun`
-  on Slurm, `aprun` on PBS).
-- `JULIA_MPIEXEC_ARGS`: Additional arguments to be passed to MPI launcher.
+Preferences are merged across the Julia load path, such that it is feasible to provide a module file that appends a path to
+`JULIA_LOAD_PATH` variable that contains system-wide preferences.
 
-The following variables are also queried:
+As an example you can use [`MPI.use_system_binary()`](@ref) to create a file `LocalPreferences.toml` containing:
+
+```toml
+[MPIPreferences]
+abi = "OpenMPI"
+binary = "system"
+libmpi = "/software/mpi/lib/libmpi.so"
+mpiexec = "/software/mpi/bin/mpiexec"
+```
 
-- `JULIA_MPI_INCLUDE_PATH`: the directory containing the MPI header files.
-- `JULIA_MPI_CFLAGS`: C flags passed to the constant generation build (default: `-lmpi`)
-- `JULIA_MPICC`: MPI C compiler (default: `mpicc`)
+Copying this `LocalPreferences.toml` to a central location such as `/software/mpi/julia` and
+create adjacent to it a `Project.toml` containing:
+
+```toml
+[extras]
+MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"
+```
+
+Now exporting the environment variable `JULIA_LOAD_PATH=":/software/mpi/julia"`
+(note the `:` before the path) in the corresponding
+module file (preferably the module file for the MPI installation or for Julia),
+will cause MPI.jl to default to your cluster MPI installation.
+
+The user can still provide differing MPI configurations for each Julia project that
+will take precedent by modifying the local `Project.toml` or by providing a `LocalPreferences.toml` file.
+
+### Using a different JLL provided MPI library
+
+The following MPI implementations are provided as JLL packages and automatically obtained when installing MPI.jl:
+
+- `MicrosoftMPI_jll`: Default for Windows
+- `MPICH_jll`: Default for all Unix-like systems
+- [`MPItrampoline_jll`](https://github.com/eschnett/MPItrampoline): Binaries built against MPItrampoline can be efficiently retargetted to a system MPI implementation.
+- `OpenMPI_jll`:
+
+```@doc
+MPI.use_jll_binary
+```
+
+## Configuration of the MPI.jl testsuite
+
+### Testing against a different MPI implementation
+
+The `LocalPreferences.toml` must be located within the `test` folder, you can
+either create it in place or copy it into place.
+
+```
+~/MPI> julia --project=test
+julia> using MPIPreferences
+julia> MPIPreferences.use_system_binary()
+~/MPI> rm test/Manifest.toml
+~/MPI> julia --project
+(MPI) pkg> test
+```
 
+### Environment variables
 The test suite can also be modified by the following variables:
 
-- `JULIA_MPIEXEC_TEST_ARGS`: Additional arguments to be passed to the MPI launcher for the tests only.
+- `JULIA_MPI_TEST_NPROCS`: How many ranks to use within the tests
 - `JULIA_MPI_TEST_ARRAYTYPE`: Set to `CuArray` to test the CUDA-aware interface with
   [`CUDA.CuArray](https://github.com/JuliaGPU/CUDA.jl) buffers.
+- `JULIA_MPI_TEST_BINARY`: Check that the specified MPI binary is used for the tests
+- `JULIA_MPI_TEST_ABI`: Check that the specified MPI ABI is used for the tests

src/MPI.jl

@@ -35,6 +35,8 @@ end
 
 
 import MPIPreferences
+const use_jll_binary = MPIPreferences.use_jll_binary
+const use_system_binary = MPIPreferences.use_system_binary
 
 if MPIPreferences.binary == "MPICH_jll"
     import MPICH_jll: libmpi, mpiexec

test/runtests.jl

@@ -11,12 +11,6 @@ else
     ArrayType = Array
 end
 
-if Sys.isunix()
-    # This test doesn't need to be run with mpiexec.  `mpiexecjl` is currently
-    # available only on Unix systems
-    include("mpiexecjl.jl")
-end
-
 nprocs_str = get(ENV, "JULIA_MPI_TEST_NPROCS", "")
 nprocs = nprocs_str == "" ? clamp(Sys.CPU_THREADS, 2, 4) : parse(Int, nprocs_str)
 
@@ -29,6 +23,12 @@ if haskey(ENV,"JULIA_MPI_TEST_ABI")
     @test ENV["JULIA_MPI_TEST_ABI"] == MPIPreferences.abi
 end
 
+if Sys.isunix()
+    # This test doesn't need to be run with mpiexec.  `mpiexecjl` is currently
+    # available only on Unix systems
+    include("mpiexecjl.jl")
+end
+
 testdir = @__DIR__
 istest(f) = endswith(f, ".jl") && startswith(f, "test_")
 testfiles = sort(filter(istest, readdir(testdir)))