revamp MPIPreferences docs (#594)

* revamp MPIPreferences docs

* expand section for cluster admins

* move mpipref docs

* split notes

* Update lib/MPIPreferences/src/MPIPreferences.jl

Co-authored-by: Mosè Giordano <[email protected]>

* Update docs/src/configuration.md

Co-authored-by: Michael Schlottke-Lakemper <[email protected]>

* typos

* mention installation, and indent

* Fix indentation of code blocks in enumerated list

* remove MPIPreferences exports from MPI

* update CUDA known issues

Co-authored-by: Mosè Giordano <[email protected]>
Co-authored-by: Michael Schlottke-Lakemper <[email protected]>
Co-authored-by: Mosè Giordano <[email protected]>

Author: 3 people

docs/make.jl

@@ -59,17 +59,17 @@ makedocs(
         "knownissues.md",
         "Examples" => EXAMPLES,
         "Reference" => [
-            "mpipreferences.md",
-            "library.md",
-            "environment.md",
-            "comm.md",
-            "buffers.md",
-            "pointtopoint.md",
-            "collective.md",
-            "onesided.md",
-            "topology.md",
-            "io.md",
-            "advanced.md",
+            "reference/mpipreferences.md",
+            "reference/library.md",
+            "reference/environment.md",
+            "reference/comm.md",
+            "reference/buffers.md",
+            "reference/pointtopoint.md",
+            "reference/collective.md",
+            "reference/onesided.md",
+            "reference/topology.md",
+            "reference/io.md",
+            "reference/advanced.md",
         ],
         "refindex.md",
     ],

docs/src/configuration.md

@@ -9,94 +9,21 @@ clusters or multi-GPU machines, you will probably want to configure against a
 system-provided MPI implementation in order to exploit features such as fast network
 interfaces and CUDA-aware MPI interfaces.
 
-## Julia wrapper for `mpiexec`
-
-Since you can configure `MPI.jl` to use one of several MPI implementations, you
-may have different Julia projects using different implementation.  Thus, it may
-be cumbersome to find out which `mpiexec` executable is associated to a specific
-project.  To make this easy, on Unix-based systems `MPI.jl` comes with a thin
-project-aware wrapper around `mpiexec`, called `mpiexecjl`.
-
-### Installation
-
-You can install `mpiexecjl` with [`MPI.install_mpiexecjl()`](@ref).  The default
-destination directory is `joinpath(DEPOT_PATH[1], "bin")`, which usually
-translates to `~/.julia/bin`, but check the value on your system.  You can also
-tell `MPI.install_mpiexecjl` to install to a different directory.
-
-```sh
-$ julia
-julia> using MPI
-julia> MPI.install_mpiexecjl()
-```
-
-To quickly call this wrapper we recommend you to add the destination directory
-to your [`PATH`](https://en.wikipedia.org/wiki/PATH_(variable)) environment
-variable.
-
-### Usage
-
-`mpiexecjl` has the same syntax as the `mpiexec` binary that will be called, but
-it takes in addition a `--project` option to call the specific binary associated
-to the `MPI.jl` version in the given project.  If no `--project` flag is used,
-the `MPI.jl` in the global Julia environment will be used instead.
-
-After installing `mpiexecjl` and adding its directory to `PATH`, you can run it
-with:
-
-```sh
-$ mpiexecjl --project=/path/to/project -n 20 julia script.jl
-```
-
-## Using MPIPreferences.jl
-
-MPI.jl uses [Preferences.jl](https://github.com/JuliaPackaging/Preferences.jl) to
-allow the user to choose which MPI implementation to use for a project. This provides
+The MPIPreferences.jl package allows the user to choose which MPI implementation to use in MPI.jl. It uses [Preferences.jl](https://github.com/JuliaPackaging/Preferences.jl) to
+configure the MPI backend for each project separately. This provides
 a single source of truth that can be used for JLL packages (Julia packages providing C libraries)
-that link against MPI, localizes the choice of MPI implementation to a project.
-
-Users can use the provided [`use_system_binary`](@ref MPIPreferences.use_system_binary) or
-[`use_jll_binary`](@ref MPIPreferences.use_jll_binary) to switch MPI implementations. By
-default, the JLL-provided binaries are used.
-
-### Migration from MPI.jl `v0.19`
-
-Prior to MPI.jl `v0.20` environment variables were used to configure which MPI
-library to use. These have now been removed and have no effect anymore:
-
-- `JULIA_MPI_BINARY`
-- `JULIA_MPIEXEC`
-- `JULIA_MPI_INCLUDE_PATH`
-- `JULIA_MPI_CFLAGS`
-- `JULIA_MPICC`
-
-### Using a system-provided MPI backend
-
-#### Requirements
-
-MPI.jl requires a shared library installation of a C MPI library, supporting the MPI 3.0
-standard or later.
-
-### Configuration
-
-To use the system MPI library, run `MPI.use_system_binary()`.
-This will attempt to locate and to identify any available MPI implementation, and create
-a file called `LocalPreferences.toml` adjacent to the current `Project.toml`.
-Use `Base.active_project()` to obtain the location of the currently active project.
+that link against MPI. It can be installed by
 
 ```sh
-julia --project -e 'using MPI; MPI.use_system_binary()'
+julia -e 'using Pkg; Pkg.add("MPIPreferences")'
 ```
 
-!!! note
-    You can copy `LocalPreferences.toml` to a different project folder, but you must list
-    `MPIPreferences` in the `[extras]` section of the `Project.toml` for the settings
-    to take effect. Due to a bug in Julia (until `v1.6.5` and `v1.7.1`), getting preferences
-    from transitive dependencies is broken (https://github.com/JuliaPackaging/Preferences.jl/issues/24).
-    To fix this update your version of Julia, or add `MPIPreferences` as a direct dependency to your project.
+## Using a system-provided MPI backend
 
+### Requirements
 
-The following MPI implementations should work out-of-the-box with MPI.jl:
+MPI.jl requires a shared library installation of a C MPI library, supporting the MPI 3.0
+standard or later. The following MPI implementations should work out-of-the-box with MPI.jl:
 
 - [Open MPI](http://www.open-mpi.org/)
 - [MPICH](http://www.mpich.org/) (v3.1 or later)
@@ -108,67 +35,83 @@ The following MPI implementations should work out-of-the-box with MPI.jl:
 - [Fujitsu MPI](https://www.fujitsu.com/global/about/resources/publications/technicalreview/2020-03/article07.html#cap-03)
 - [HPE MPT/HMPT](https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00105727en_us)
 
-If the implementation is changed, you will need to use [`MPI.use_system_binary()`](@ref MPIPreferences.use_system_binary)
-or [`MPI.use_jll_binary()`](@ref MPIPreferences.use_jll_binary).
+### Configuration
 
-#### Advanced options
+Run `MPIPreferences.use_system_binary()`. This will attempt to locate and to identify any available MPI implementation, and create a file called `LocalPreferences.toml` adjacent to the current `Project.toml`.
 
-```@doc
-MPI.use_system_binary
+```sh
+julia --project -e 'using MPIPreferences; MPIPreferences.use_system_binary()'
 ```
 
-You can use the argument `mpiexec` to provide the name (or full path) of the MPI launcher executable. The default is
-`mpiexec`, but some clusters require using the scheduler launcher interface (e.g. `srun`
-on Slurm, `aprun` on PBS). If the MPI library has an uncommon name you can provide it in `library_names`.
-The MPI standard does not specify the exact application binary interface (ABI).
-In case ABI detection fails you can provide a manual choice (either `MPICH`, `MPItrampoline`, `OpenMPI`, or `MicrosoftMPI`),
-but also open an issue such that the automatic detection can be improved.
-`export_prefs=true` can be used to copy the preferences into the `Project.toml` instead of creating a
-`LocalPreferences.toml` file to hold them.
+If the implementation is changed, you will need to call this function again. See the [`MPIPreferences.use_system_binary`](@ref) documentation for specific options.
+
+!!! note
+    You can copy `LocalPreferences.toml` to a different project folder, but you must list
+    `MPIPreferences` in the `[extras]` or `[deps]` section of the `Project.toml` for the settings
+    to take effect.
+
+!!! note
+    Due to a bug in Julia (until `v1.6.5` and `v1.7.1`), getting preferences
+    from transitive dependencies is broken (https://github.com/JuliaPackaging/Preferences.jl/issues/24).
+    To fix this update your version of Julia, or add `MPIPreferences` as a direct dependency to your project.
+
 
-#### Notes to HPC cluster adminstators
+### Notes to HPC cluster adminstators
 
 Preferences are merged across the Julia load path, such that it is feasible to provide a module file that appends a path to
-`JULIA_LOAD_PATH` variable that contains system-wide preferences.
+`JULIA_LOAD_PATH` variable that contains system-wide preferences. The steps are as follows:
 
-As an example you can use [`MPI.use_system_binary()`](@ref MPIPreferences.use_system_binary)
-to create a file `LocalPreferences.toml` containing:
+1. Run [`MPIPreferences.use_system_binary()`](@ref MPIPreferences.use_system_binary), which will generate a file `LocalPreferences.toml` containing something like the following:
 
-```toml
-[MPIPreferences]
-abi = "OpenMPI"
-binary = "system"
-libmpi = "/software/mpi/lib/libmpi.so"
-mpiexec = "/software/mpi/bin/mpiexec"
-```
+   ```toml
+   [MPIPreferences]
+   abi = "OpenMPI"
+   binary = "system"
+   libmpi = "/software/mpi/lib/libmpi.so"
+   mpiexec = "/software/mpi/bin/mpiexec"
+   ```
 
-Copying this `LocalPreferences.toml` to a central location such as `/software/mpi/julia` and
-create adjacent to it a `Project.toml` containing:
+2. Create a file called `Project.toml` or `JuliaProject.toml` in a central location, for example `/software/mpi/julia` or in the same directory as the MPI library module, and add the following contents:
 
-```toml
-[extras]
-MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"
-```
+   ```toml
+   [extras]
+   MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"
+
+   [preferences.MPIPreferences]
+   abi = "OpenMPI"
+   binary = "system"
+   libmpi = "/software/mpi/lib/libmpi.so"
+   mpiexec = "/software/mpi/bin/mpiexec"
+   ```
+
+   updating the contents of the `[preferences.MPIPreferences]` section match those of the `[MPIPreferences]` in `LocalPreferences.toml`.
 
-Now exporting the environment variable `JULIA_LOAD_PATH=":/software/mpi/julia"`
-(note the `:` before the path) in the corresponding
-module file (preferably the module file for the MPI installation or for Julia),
-will cause MPI.jl to default to your cluster MPI installation.
+3. Append the directory containing the file to the [`JULIA_LOAD_PATH`](https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH) environment variable, with a colon (`:`) separator.
 
-The user can still provide differing MPI configurations for each Julia project that
-will take precedent by modifying the local `Project.toml` or by providing a `LocalPreferences.toml` file.
+   If this variable is _not_ already set, it should be prefixed with a colon to ensure correct
+   behavior of the Julia load path, e.g. `JULIA_LOAD_PATH=":/software/mpi/julia"`.
+   If using environment modules, this can be achieved with
+   ```
+   append-path  -d {} JULIA_LOAD_PATH :/software/mpi/julia
+   ```
+   in the corresponding module file (preferably the module file for the MPI installation or for Julia).
 
-### Using a different JLL provided MPI library
+   The user can still provide differing MPI configurations for each Julia project
+   that will take precedent by modifying the local `Project.toml` or by providing a
+   `LocalPreferences.toml` file.
+
+## Using an alternative JLL-provided MPI library
 
 The following MPI implementations are provided as JLL packages and automatically obtained when installing MPI.jl:
 
-- `MicrosoftMPI_jll`: Default for Windows
-- `MPICH_jll`: Default for all Unix-like systems
-- [`MPItrampoline_jll`](https://github.com/eschnett/MPItrampoline): Binaries built against MPItrampoline can be efficiently retargetted to a system MPI implementation.
-- `OpenMPI_jll`:
+- `MicrosoftMPI_jll`: [Microsoft MPI](https://docs.microsoft.com/en-us/message-passing-interface/microsoft-mpi) Default for Windows
+- `MPICH_jll`: [MPICH](https://www.mpich.org/). Default for all other systems
+- `OpenMPI_jll`: [Open MPI](https://www.open-mpi.org/)
+- `MPItrampoline_jll`: [MPItrampoline](https://github.com/eschnett/MPItrampoline): an MPI forwarding layer.
 
-```@doc
-MPI.use_jll_binary
+Call [`MPIPreferences.use_jll_binary`](@ref), for example
+```sh
+julia --project -e 'using MPIPreferences; MPIPreferences.use_jll_binary("MPItrampoline_jll")'
 ```
 
 ## Configuration of the MPI.jl testsuite
@@ -195,3 +138,16 @@ The test suite can also be modified by the following variables:
   [`CUDA.CuArray](https://github.com/JuliaGPU/CUDA.jl) buffers.
 - `JULIA_MPI_TEST_BINARY`: Check that the specified MPI binary is used for the tests
 - `JULIA_MPI_TEST_ABI`: Check that the specified MPI ABI is used for the tests
+
+
+## Migration from MPI.jl version 0.19 or earlier
+
+Prior to MPI.jl version 0.20, environment variables were used to configure which MPI
+library to use. These have now been removed and no longer have any effect:
+
+- `JULIA_MPI_BINARY`
+- `JULIA_MPIEXEC`
+- `JULIA_MPIEXEC_ARGS`
+- `JULIA_MPI_INCLUDE_PATH`
+- `JULIA_MPI_CFLAGS`
+- `JULIA_MPICC`

docs/src/knownissues.md

@@ -100,17 +100,17 @@ Make sure to:
 - Make sure to have:
     ```
     export JULIA_CUDA_MEMORY_POOL=none
-    export JULIA_MPI_BINARY=system
     export JULIA_CUDA_USE_BINARYBUILDER=false
     ```
-- Add CUDA and MPI packages in Julia. Build MPI.jl in verbose mode to check whether correct versions are built/used: 
+- Add CUDA, MPIPreferences, and MPI packages in Julia. Switch to using the system binary
     ```
-    julia -e 'using Pkg; pkg"add CUDA"; pkg"add MPI"; Pkg.build("MPI"; verbose=true)'
+    julia -e 'using Pkg; pkg"add CUDA, MPIPreferences, MPI"'
+    julia -e 'using MPIPreferences; MPIPreferences.use_system_binary()'
     ```
 - Then in Julia, upon loading MPI and CUDA modules, you can check
   - CUDA version: `CUDA.versioninfo()`
   - If MPI has CUDA: `MPI.has_cuda()`
-  - If you are using correct MPI implementation: `MPI.identify_implementation()`
+  - If you are using correct MPI library: `MPI.libmpi`
 
 After that, it may be preferred to run the Julia MPI script (as suggested [here](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/11)) launching it from a shell script (as suggested [here](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/4)).