Libraries are not being added to wrapper compiler when building with external hwloc

[rhc54]

When building with an external hwloc and devel headers, I'm finding that the wrapper compilers are not getting the required flags to build the ORTE tests:

$ mpicc --showme
gcc -I/home/common/openmpi/build/foobar/include/openmpi -I/home/common/local/include
-I/home/common/openmpi/build/foobar/include -pthread -L/home/common/local/lib -Wl,-rpath
-Wl,/home/common/local/lib -Wl,-rpath -Wl,/home/common/openmpi/build/foobar/lib
-Wl,--enable-new-dtags -L/home/common/openmpi/build/foobar/lib -lmpi

I have to manually add -lhwloc -lopen-rte -lopen-pal to get it to build. Will check to see if this impacts 2.x as well

[ggouaillardet]

@rhc54 i do not understand what the issue is ...

as far as i am concerned, mpicc should only use -lmpi
if you are building ORTE tests, then shouldn't you use ortecc instead ?

in my (outdated) build with all (pmix, libevent and hwloc) external libs, mpicc uses -lmpi
and ortecc uses -lopen-rte -lopen-pal

also, if ORTE tests do not invoke OPAL directly, then -lopen-pal should not even be needed.

of course, feel free to disagree if you consider mpicc should work to build apps that directly invoke ORTE and/or OPAL.

[rhc54]

It's an interesting point, but it has worked just fine for years, which indicates that something changed recently. Note that it continues to work today unless I build with external (pmix, libevent, and hwloc), which would appear to indicate a problem with how we are handling that scenario.

[ggouaillardet]

how do you evidence the issue ?
make check ?
other ?

[ggouaillardet]

can you also please post your configure command line, and your LD_LIBRARY_PATH ?

[rhc54]

Sure - the issue can be evidenced by just changing to the orte/test/mpi directory and doing "make clean hello"

Configure line:

./configure --enable-orterun-prefix-by-default --prefix=foo --with-hwloc=bar --with-libevent=bar --with-pmix=bar --with-devel-headers

[ggouaillardet]

ok, that is something different than what i initially expected.

the orte/test/mpi Makefile invokes mpirun --openmpi:linkall and it links with -lmpi -lopen-rte -lopen-pal -levent -levent_pthreads -lutil -lm

the issue here is hwloc_bitmap_alloc is an undefined reference in hello.o
with internal hwloc, it is defined in libopen-pal.so so we are good, but with external hwloc, it is defined in libhwloc.so so we do need -lhwloc
bottom line, mpirun --openmpi:linkall should link with -lpmix -lhwloc to be on the safe side when openmpi was built with external components, i will have a look at it from now

[ggouaillardet]

@jsquyres can you please take care of this ?
i did a bit of research, and event, pmix and hwloc have all different logics
(i guess the rationale is hwloc needs to special tweaking to handle OPAL_HWLOC_WANT_VERBS_HELPER), so some variable is set but too late for the wrapper.

pmix is not a STOP_AT_FIRST framework, (i guess because we always try to build s1,s2,singleton and only one of pmix114 or external, not sure where cray sits here), so -lpmix is not appended to the wrapper flags

bottom line, i am lost and cannot find a way to get it work

[jsquyres]

@rhc54 Are you sure that building with externals worked properly recently? I did a git bisect back to last November, and I still don't see the externals' flags being pulled into the wrappers.

[jsquyres]

There are two problems:

1. The hwloc WRAPPER_EXTRA flags are set in POST_CONFIG, not in CONFIG. This is easy enough to fix.
2. The PMIX external/configure.m4 is somewhat of a mess.
   • I'm not sure why the bulk of its test is AC_REQUIRE'ed (which makes it run out of sequence), and is defined in opal_pmi.m4 -- but is not used by any other component.
   • It sets external_WRAPPER_EXTRA_CPPFLAGS and missed the pmix_ prefix
   • It doesn't set pmix_external_WRAPPER_EXTRA_LIBS and _LDFLAGS

I fixed the hwloc stuff, but do not have the cycles to work on the pmix stuff this week; sorry. The above should be enough to prod someone in the right direction to fix the pmix configure.m4.

[jsquyres]

Fixed via #1663.