OpenFOAM-v1806-foss-2018b.eb -- using mpi 3.1.1 creates segmentation faults

[flowgeek]

hi all,

i have a question - not sure if this is the right spot to ask though - but i see it as an issue of the easybuild of openfoam with foss-2018b.

when i build openfoam with OpenFOAM-v1806-foss-2018b.eb it will be loaded with openmpi 3.1.1. This version is known to not work with openfoam and creating regular segmentation faults:
https://www.cfd-online.com/Forums/openfoam-solving/212574-segmentation-fault-mpi.html

it is therefore recommended to use an older version of openmpi (e.g. 2.1.1)
https://bugs.openfoam.org/view.php?id=3071

how can i do this?!? (i also tried to build openfoam with a toolchain that uses openmpi 2.1.1, this however fails when building ncurses...)

[boegel]

@flowgeek To build OpenFOAM with a different OpenMPI version, the easiest way is indeed to use a different toolchain...

Which issue are you seeing with ncurses exactly? That sounds fishy...

Looking at the release notes of OpenMPI 3.1.4 (https://www.mail-archive.com/announce@lists.open-mpi.org/msg00123.html), it looks like the bug you are hitting may be fixed in that version:

Fix possible floating point rounding and division issues in OMPIO
which led to crashes and/or data corruption with very large data.

So, a relatively easy way to fix this could be to update foss/2018b to use OpenMPI 3.1.4 rather than 3.1.3, and then build OpenFOAM on top of that modified toolchain?

Since the actual fix is a fairly modest change (see open-mpi/ompi#6344), so we could consider including that as a patch in the OpenMPI 3.1.1 easyconfig file...

Are you up for trying that yourself?

[flowgeek]

hi,
thats quick :-)

so the ncurses issue is very similar to this one:
#4387
i just can't get it go away...

i'm farly new to using easybuild, and open to try stuff - but and i'm not really sure if i have the necessary access to modify the toolchain to use open mpi 3.1.4 as I work on an hpc cluster as a simple user and not an administrator...

so if it is possible to modify the toolchain as a normal user we can try that - but you would have to give me a really good explanation on how to do that...
patching openmpi 3.1.1 and rebuilding it with an existing toolchain would be ok too, if this is simpler?

thanks a lot!

[boegel]

@flowgeek You can basically just define a custom foss/2018b in which you bump the OpenMPI version to 3.1.4, by taking the foss-2018b.eb easyconfig file, tweaking it, and doing a eb foss-2018b.eb --force. That way you'll just install your own foss/2018b module which will shadow the system one.
It's a bit more involved than that though, since you'll need to provide the easyconfig file for OpenMPI 3.1.4 too, and probably also customize the gompi/2018b subtoolchain, and build a custom FFTW and ScaLAPACK as well.

So, yes, just installing a custom patched version of OpenMPI 3.1.1 is easier.
Short instructions:

• copy the OpenMPI-3.1.1-GCC-7.3.0-2.30.eb easyconfig file (see eb --search OpenMPI-3.1.1-GCC-7.3.0-2.30.eb)
• create a patch file for the fix in Pr/v3.1.x floating point division problem open-mpi/ompi#6344 (this may work: https://patch-diff.githubusercontent.com/raw/open-mpi/ompi/pull/6344.patch, but it may need tweaking to make it apply to OpenMPI 3.1.1, unsure about that)
• add the patch file to your OpenMPI-3.1.1-GCC-7.3.0-2.30.eb copy
• install your custom OpenMPI using eb OpenMPI-3.1.1-GCC-7.3.0-2.30.eb --force (which will shadow the central OpenMPI installation)

And then try rebuilding OpenFOAM-v1806-foss-2018b.eb on top of that.

If that worked out, please consider contributing back the patched OpenMPI 3.1.1 easyconfig file (see https://easybuild.readthedocs.io/en/latest/Contributing.html#contributing-easyconfig-files).

Do let us know if something isn't clear or working out as expected (our Slack channel is a good place to get quick help: https://easybuild.readthedocs.io/en/latest/index.html#getting-help).

For the ncurses issue, please comment in #4387 with more information (OS, processor info, what type of filesystem you're building and installing on (NFS?), etc); we can re-open that issue and try to sort it out...