Examples and all user code no longer compiles with 2.1.0 using mxm

[lancewilson-monash]

When trying to compile the examples after configuring, compiling and installing 2.1.0 no code will compile and returns errors similar to the following.

mpicc -g -I/usr/local/openmpi/2.1.0-gcc4/include -I/usr/local/gcc/4.9.3/include/c++/4.9.3/x86_64-unknown-linux-gnu/ -I/usr/local/gcc/4.9.3/include -L/usr/local/openmpi/2.1.0-gcc4/lib -L/usr/local/gcc/4.9.3/lib64  hello_c.c   -o hello_c
gcc: error: 25517_LDFLAGS: No such file or directory
gcc: error: 25517_LDFLAGS: No such file or directory
make: *** [hello_c] Error 1 

If I recompile openmpi with the same parameters the error message changes with a new random number.

< #define WRAPPER_EXTRA_LDFLAGS "-L/opt/slurm-16.05.4/lib -L/usr/lib64 25517_LDFLAGS -L/opt/mellanox/mxm/lib 25517_LDFLAGS    -Wl,-rpath -Wl,/opt/slurm-16.05.4/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/opt/mellanox/mxm/lib -Wl,-rpath -Wl,@{libdir} -Wl,--enable-new-dtags"
---
> #define WRAPPER_EXTRA_LDFLAGS "-L/opt/slurm-16.05.4/lib -L/usr/lib64 5362_LDFLAGS -L/opt/mellanox/mxm/lib 5362_LDFLAGS    -Wl,-rpath -Wl,/opt/slurm-16.05.4/lib -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/opt/mellanox/mxm/lib -Wl,-rpath -Wl,@{libdir} -Wl,--enable-new-dtags"

It looks like there is a bug in the configure script that is collecting the number from somewhere and putting it into the LDFLAGS

mtl_mxm_LDFLAGS="$$_LDFLAGS $ompi_check_mxm_LDFLAGS"

Changing the configure script to remove the number causes software to fail on run even though it compiles.

Device name: mlx5_0 with OFED 3.2

[rhc54]

Why are you adding all those flags to mpicc? You shouldn't add anything to it - all the required options are already there.

[lancewilson-monash]

That is what the makefile is doing by default. Removing those options results in exactly the same error. Also all the directories in that list exist, are readable, and have the necessary files.

On our cluster we run multiple version of openmpi and compilers so often these are added by the software when it is compiled.

[lancewilson-monash]

It is a regression as the examples for 2.0.2 compile and run without the compile fail.

[jsquyres]

Can you please send all the information listed here: https://www.open-mpi.org/community/help/

[ggouaillardet]

it looks like 8c20869 was not backported

[jsquyres]

@jladd-mlnx Is Mellanox not testing MXM with v2.1.x? Can you please file PRs to v2.x and/or v2.0.x if necessary. Thanks.

[ggouaillardet]

Fwiw, v2.0.x uses an older logic, and v3.x is up-to-date
so PR is only required for v2.x

[lancewilson-monash]

@jsquyres Please find attached the requested output. Github required a zip file so it has been put into a zip file.
ompi-output.tar.bz2.zip

[ggouaillardet]

@lancewilson-monash i think the issue is known and was already fixed in master
can you please manually download and apply the patch available at https://github.com/open-mpi/ompi/pull/3240.patch and try again ?
note you need recent autotools, and you will have to run ./autogen.pl --force

[jladd-mlnx]

@jsquyres Very strange. We test MXM with 2.x nightly and have no issues.

[rhc54]

Ah, but do you build the examples in those tests? I think that is the only issue being raised here