Skip to content

Allow one to specify an ABI file via MPIPreferences #600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Allow one to specify an ABI file via MPIPreferences #600

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
3c1670f
Allow to specigy an MPI ABI file via MPIPreferences
sloede May 25, 2022
9b7091e
Amend docstring
sloede May 25, 2022
edaf03b
Add section to docs
sloede May 25, 2022
904d252
Give example in docs
sloede May 25, 2022
ff29239
Add sentence on using standard Julia alises for C types in ABI file
sloede May 29, 2022
13af812
Add `abi_file` to `set_preferences!`
sloede May 30, 2022
573ae38
Merge branch 'master' into msl/allow-specifying-abi-file
sloede May 30, 2022
162653a
Rename `abi_file` --> `abi_source_file`
sloede May 30, 2022
8022f57
Merge branch 'master' into msl/allow-specifying-abi-file
sloede May 30, 2022
9a5d0a9
Use `nothing` instead of "custom" as sentinel for using a custom ABI …
sloede May 31, 2022
b4de3bd
Add comment
sloede May 31, 2022
ff798c3
Indicate custom ABI file using `abi="custom"`
sloede Jun 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docs/src/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ standard or later. The following MPI implementations should work out-of-the-box
- [Fujitsu MPI](https://www.fujitsu.com/global/about/resources/publications/technicalreview/2020-03/article07.html#cap-03)
- [HPE MPT/HMPT](https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00105727en_us)

If you are using an MPI implementation that is not ABI-compatible with any one
of these, please read the section on [Supporting an unknown ABI](@ref) below.

### Configuration

Run `MPIPreferences.use_system_binary()`. This will attempt to locate and to identify any available MPI implementation, and create a file called `LocalPreferences.toml` adjacent to the current `Project.toml`.
Expand Down Expand Up @@ -100,6 +103,50 @@ Preferences are merged across the Julia load path, such that it is feasible to p
that will take precedent by modifying the local `Project.toml` or by providing a
`LocalPreferences.toml` file.


### Supporting an unknown ABI
If you want to use an MPI implementation not officially supported by MPI.jl, you
need to create your own ABI file with all relevant MPI constants. The files for supported
ABIs are stored in the `src/consts/` folder, e.g.,
[`mpich.jl`](https://github.com/JuliaParallel/MPI.jl/blob/master/src/consts/mpich.jl)
for MPICH-compatible implementations. To create your own ABI file, it is
advisable to start with an existing constants file (e.g., for MPICH) and then
adapt each entry to the contents of your MPI implementations's `mpi.h` C header
file.

For example, if your `mpi.h` header file contains something like
```c
typedef unsigned int MPI_Request;
enum {
MPI_REQUEST_NULL = 0
};

#define MPI_ARGV_NULL ((char**) NULL)
```
you need to put the corresponding entries in your ABI file `abi_file.jl`:
```julia
const MPI_Request = Cuint
@const_ref MPI_REQUEST_NULL MPI_Request 0

@const_ref MPI_ARGV_NULL Ptr{Cvoid} C_NULL
```
As you can see, the syntax of such a Julia ABI file is non-trivial, thus the
recommendation to start with an existing ABI file.
It is further advisable to always use the corresponding Julia alias for
standard C types, e.g., `Cuint` for `unsigned int` or `Clonglong` for `long
long`.
Please note that sometimes information is also stored in ancillary header files (e.g.,
`mpi_types.h` or `mpi_ext.h`).

You can then use [`MPIPreferences.use_system_binary`](@ref) to configure MPI.jl
to use your custom file by providing the path via the `abi_file` keyword
argument, e.g.,
```shell
julia --project -e 'using MPIPreferences; MPIPreferences.use_system_binary(; abi_file="path/to/abifile.jl)'
```
You need to restart Julia for the change to take effect.


## Using an alternative JLL-provided MPI library

The following MPI implementations are provided as JLL packages and automatically obtained when installing MPI.jl:
Expand Down
17 changes: 16 additions & 1 deletion lib/MPIPreferences/src/MPIPreferences.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ else
error("Unknown binary: $binary")
end

# Default is empty string, indicating that the information in `abi` should be used to load the
# correct ABI file
const abi_file = @load_preference("abi_file", "")

module System
export libmpi, mpiexec
using Preferences
Expand Down Expand Up @@ -94,6 +98,7 @@ end
library_names = ["libmpi", "libmpi_ibm", "msmpi", "libmpich", "libmpitrampoline"],
mpiexec = "mpiexec",
abi = nothing,
abi_file = nothing,
export_prefs = false,
force = true)

Expand All @@ -115,6 +120,10 @@ Options:
- `abi`: the ABI of the MPI library. By default this is determined automatically
using [`identify_abi`](@ref). See [`abi`](@ref) for currently supported values.

- `abi_file`: the ABI file for the MPI library. By default, for ABIs supported by MPI.jl, the
corresponding ABI file is loaded automatically. This argument allows one to override the
automatic selection, e.g., to provide an ABI file for an MPI ABI unknown to MPI.jl.

- `export_prefs`: if `true`, the preferences into the `Project.toml` instead of `LocalPreferences.toml`.

- `force`: if `true`, the preferences are set even if they are already set.
Expand All @@ -123,6 +132,7 @@ function use_system_binary(;
library_names=["libmpi", "libmpi_ibm", "msmpi", "libmpich", "libmpitrampoline"],
mpiexec="mpiexec",
abi=nothing,
abi_file=nothing,
export_prefs=false,
force=true,
)
Expand All @@ -137,6 +147,11 @@ function use_system_binary(;
if isnothing(abi)
abi = identify_abi(libmpi)
end
if isnothing(abi_file)
abi_file = ""
else
abi_file = abspath(abi_file)
end
Copy link
Member

@simonbyrne simonbyrne May 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd leave it as nothing if it is not set.

maybe a slightly longer name (e.g. abi_source_file?)

also, you need to store it via set_preferences!

Copy link
Member Author

@sloede sloede May 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd leave it as nothing if it is not set.

That won't work, since Nothing is not a valid TOML type:

julia> TOML.print(Dict{String,Any}("abi_file" => nothing))
ERROR: type `Nothing` is not a valid TOML type, pass a conversion function to `TOML.print

This is also the reason why used the empty string as a sentinel. I could also use false if you'd prefer that.

maybe a slightly longer name (e.g. abi_source_file?)

Makes sense, I will do that.

also, you need to store it via set_preferences!

Thanks, I seem to have forgotten to commit that part 🙈

Copy link
Member Author

@sloede sloede May 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a slightly longer name (e.g. abi_source_file?)

Done in 162653a.

Copy link
Member

@simonbyrne simonbyrne May 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That won't work, since Nothing is not a valid TOML type:

Preferences.jl supports it, and it is what is returned when value is not set:

julia> isnothing(load_preference(MPIPreferences, "abi_file"))
true

if mpiexec isa Cmd
mpiexec = collect(mpiexec)
end
Expand All @@ -149,7 +164,7 @@ function use_system_binary(;
force=force
)

@warn "The underlying MPI implementation has changed. You will need to restart Julia for this change to take effect" binary libmpi abi mpiexec
@warn "The underlying MPI implementation has changed. You will need to restart Julia for this change to take effect" binary libmpi abi abi_file mpiexec

if VERSION <= v"1.6.5" || VERSION == v"1.7.0"
@warn """
Expand Down
27 changes: 16 additions & 11 deletions src/consts/consts.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,23 @@ macro const_ref(name, T, expr)
:(const $(esc(name)) = Ref{$T}())
end

@static if MPIPreferences.abi == "MPICH"
include("mpich.jl")
elseif MPIPreferences.abi == "OpenMPI"
include("openmpi.jl")
elseif MPIPreferences.abi == "MicrosoftMPI"
include("microsoftmpi.jl")
elseif MPIPreferences.abi == "MPItrampoline"
include("mpitrampoline.jl")
elseif MPIPreferences.abi == "HPE MPT"
include("mpt.jl")
# If `abi_file` is empty, choose ABI file based on ABI string, otherwise load the specified file
@static if MPIPreferences.abi_file == ""
Copy link
Member

@simonbyrne simonbyrne May 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think i would prefer if this branch was only taken if abi == "custom" or something similar

Copy link
Member Author

@sloede sloede May 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought of the abi_file preference as a "one to rule them all" property that just overrides whatever is set as ABI when it comes to loading the ABI file.

I do see the merits of your suggestion, though. However, what if users set abi_file to something and leave abi unset? Or set it to something other than custom? Wouldn't it be confusing if the abi_file is then ignored? Or should this be handled as an error in the call to use_system_binary?

Copy link
Member

@simonbyrne simonbyrne May 29, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this option will be rarely used (and is untested), I would be extremely reluctant to make it the "one to rule them all".

The configuration is complicated in any case (which is part of the reason I am reluctant to add this functionality), but I would prefer that any requirements be very explicit. The logic in use_system_binary should handle these cases.

@static if MPIPreferences.abi == "MPICH"
include("mpich.jl")
elseif MPIPreferences.abi == "OpenMPI"
include("openmpi.jl")
elseif MPIPreferences.abi == "MicrosoftMPI"
include("microsoftmpi.jl")
elseif MPIPreferences.abi == "MPItrampoline"
include("mpitrampoline.jl")
elseif MPIPreferences.abi == "HPE MPT"
include("mpt.jl")
else
error("Unknown MPI ABI $(MPIPreferences.abi)")
end
else
error("Unknown MPI ABI $(MPIPreferences.abi)")
include(MPIPreferences.abi_file)
end

# Initialize the ref constants from the library.
Expand Down