[patch] Linker error -- confusion with -L option

[eschnett]

I see this build error with OpenMPI on Shelob at LSU:

  FCLD     libmpi_usempi_ignore_tkr.la
/usr/bin/ld: cannot find -l-L/project/eschnett/shelob/software/gcc-5.1.0/lib/gcc/x86_64-unknown-linux-gnu/5.1.0
collect2: error: ld returned 1 exit status

I configured with

/project/eschnett/shelob/software/src/openmpi-1.8.5/src/openmpi-1.8.5/configure --prefix=/project/eschnett/shelob/software/openmpi-1.8.5 --with-hwloc=/project/eschnett/shelob/software/hwloc-1.10.1 --disable-vt CC=/project/eschnett/shelob/software/llvm-3.6.0/bin/wrap-clang CXX=/project/eschnett/shelob/software/llvm-3.6.0/bin/wrap-clang++ FC=/project/eschnett/shelob/software/gcc-5.1.0/bin/wrap-gfortran CFLAGS=-I/opt/ofed/include -I/project/eschnett/shelob/software/hwloc-1.10.1/include CXXFLAGS=-I/opt/ofed/include -I/project/eschnett/shelob/software/hwloc-1.10.1/include LDFLAGS=-L/opt/ofed/lib64 -L/opt/rocks/lib -L/project/eschnett/shelob/software/hwloc-1.10.1/lib -Wl,-rpath,/project/eschnett/shelob/software/hwloc-1.10.1/lib LIBS=-lhwloc -lpthread -lpthread

I am using a self-built clang 3.6.0 that is called via the "wrap-clang" script, which calls clang, adding a few command line options to set include and library paths pointing to the self-build gcc 5.1 installation that provides libstdc++. (This is necessary since not all software packages honours CFLAGS, CXXFLAGS, etc.; I don't think this wrapper would be necessary for OpenMPI.) I am describing this wrapper here for completeness, I do not think it causes the problem I see.

In particular, the -L option mentioned in the error message is added by wrap-clang. I believe this is due to a problem in OpenMPI's configure script. The following patch remedies the problem:

--- configure
+++ configure
@@ -298951,6 +298951,7 @@
        # Remove the space.
        if test $p = "-L" ||
-          test $p = "-R"; then
+          test $p = "-R" ||
+          test $p = "-l"; then
     prev=$p
     continue
        fi
@@ -300857,6 +300858,7 @@
        # Remove the space.
        if test $p = "-L" ||
-          test $p = "-R"; then
+          test $p = "-R" ||
+          test $p = "-l"; then
     prev=$p
     continue
        fi

By the way, a more complete list of compiler command line options that takes paths as arguments are the letters "FILRYlouz"; you may want to check for all of these instead.

[jsquyres]

@eschnett Gah -- somehow I totally missed that you filed this issue until now. :-(

Unfortunately, this code comes directly from Libtool -- not Open MPI (i.e., it's part of the code that Libtool injects into Open MPI's configure script).

Can you post the wrap-clang script, and post the output of make V=1 for this failed CCLD command?

[eschnett]

This is the wrap-clang script:

#! /bin/bash

# This file was generated by Simfactory
# <https://bitbucket.org/ianhinder/simfactory3>. It is a convenience
# wrapper for clang that automatically adds certain compiler flags and
# libraries.

# Determine what the compiler driver will do
for arg in "$@"; do
    if [[ -n $skip ]]; then
        skip=""
    else
        case "$arg" in
            (-[FILRYlouz]) skip=1;;
            (-E) preprocess=1;;
            (-S) assemble=1;;
            (-c) compile=1;;
            ([^-]*.[cS]) preprocess_files=1;;
            ([^-]*.i) compile_files=1;;
            ([^-]*.s) assemble_files=1;;
            ([^-]*.[ao]) link_files=1;;
        esac
    fi
done
if [[ -z $preprocess && -z $assemble && -z $compile ]]; then link=1; fi

# Choose which flags to pass to the compiler driver
if [[ (-n $preprocess || -n $assemble || -n $compile || -n $link) &&
      (-n $preprocess_files || -n $compile_files) ]]; then
    cflags="$LLVM_CFLAGS"
fi
if [[ -n $link &&
      (-n $preprocess_files || -n $compile_files || -n $assemble_files ||
       -n $link_files) ]];
then
    ldflags="$LLVM_C_LDFLAGS $LLVM_LDFLAGS"
    libs="$LLVM_LIBS"
fi

exec $LLVM_CC "$@" $cflags $ldflags $libs

The environment variables use in this script (with the LLVM_ prefix) are set via these commands:

        gcc_version=${SIMFACTORY_GCC_VERSION#gcc-}
        export LLVM_CC="$LLVM_DIR/bin/clang"
        export LLVM_CXX="$LLVM_DIR/bin/clang++"
        export LLVM_CFLAGS="-I$LLVM_DIR/include -I$(echo $GCC_DIR/lib/gcc/*-*-*/$gcc_version/include-fixed) $GCC_CFLAGS"
        # Clang does not seem to support the abi_tag attribute
        export LLVM_CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0 -I$LLVM_DIR/include -I$GCC_DIR/include/c++/$gcc_version -I$(echo $GCC_DIR/include/c++/$gcc_version/*-*-*) -I$(echo $GCC_DIR/lib/gcc/*-*-*/$gcc_version/include-fixed) $GCC_CXXFLAGS"
        export LLVM_C_LDFLAGS="$GCC_C_LDFLAGS"
        export LLVM_CXX_LDFLAGS="-stdlib=libstdc++ $GCC_CXX_LDFLAGS"
        libdirs=""
        for dir in lib lib64 $(dirname $(cd $GCC_DIR && find . -name libgcc.a)); do
            if [[ -d $GCC_DIR/$dir ]]; then
                libdirs="$GCC_DIR/$dir $libdirs"
            fi
        done
        export LLVM_LDFLAGS="-L$LLVM_DIR/lib $GCC_LDFLAGS"
        export LD_LIBRARY_PATH="$LLVM_DIR/lib:$LD_LIBRARY_PATH"
        export DYLD_LIBRARY_PATH="$LLVM_DIR/lib:$DYLD_LIBRARY_PATH"
        for libdir in $libdirs; do
            LLVM_LDFLAGS="-L$libdir -Wl,-rpath,$libdir $LLVM_LDFLAGS"
            LD_LIBRARY_PATH="$libdir:$LD_LIBRARY_PATH"
            DYLD_LIBRARY_PATH="$libdir:$DYLD_LIBRARY_PATH"
        done
        export LLVM_LIBS="$GCC_LIBS"

and have these values:

LLVM_CC=clang-mp-3.6
LLVM_CFLAGS=
LLVM_CXX=clang++-mp-3.6
LLVM_CXXFLAGS=
LLVM_CXX_LDFLAGS=
LLVM_C_LDFLAGS=
LLVM_DIR=/Users/eschnett/software/llvm-3.6.1
LLVM_LDFLAGS=-L/Users/eschnett/software/gcc-5.1.0/lib -Wl,-rpath,/Users/eschnett/software/gcc-5.1.0/lib 
LLVM_LIBS=-lgfortran

I will post the make V=1 output later.

[jsquyres]

@eschnett Did this issue get solved by 67fdfdd?

[eschnett]

@jsquyres: No, this issue did not get resolved. I still need to apply that part of the patch that checks for the -l option.

[jsquyres]

Oops!

@ggouaillardet it looks like your fix to autogen.pl didn't add in the processing for the -l option. Can you amend the fix to autogen.pl?

[jsquyres]

@rhc54 Yes, we'd like to include this in v1.10 if possible.

[ggouaillardet]

@jsquyres i made PR #808 and i will do the PRs for the release branches

@eschnett per the output you posted, the error occurs when linking libmpi_usempi_ignore_tkr.la
in this case, wrap-gfortran is invoked instead of wrap-clang
could you please

cd ompi/mpi/fortran/use-mpi-ignore-tkr
make -n libmpi_usempi_ignore_tkr.la

then manually run the command returned by make -n libmpi_usempi_ignore_tkr.la after having manually removed the --silent flag, and post the outputs

[eschnett]

For the record:

Original error message:

  FCLD     libmpi_usempi_ignore_tkr.la
/usr/bin/ld: cannot find -l-L/home/eschnett/software/gcc-5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0
collect2: error: ld returned 1 exit status

Then:

$ make -n libmpi_usempi_ignore_tkr.la
echo "  FCLD    " libmpi_usempi_ignore_tkr.la;/bin/sh ../../../../libtool --silent --tag=FC   --mode=link /home/eschnett/software/gcc-5.2.0/bin/wrap-gfortran -I../../../../ompi/include -I/home/eschnett/software/src/openmpi-v2.x-dev-187-g8516895/src/openmpi-v2.x-dev-187-g8516895/ompi/include -I../../../.. -I/home/eschnett/software/src/openmpi-v2.x-dev-187-g8516895/src/openmpi-v2.x-dev-187-g8516895   -version-info 0:0:0  -L/opt/ofed/lib64 -L/home/eschnett/software/hwloc-1.11.0/lib -Wl,-rpath,/home/eschnett/software/hwloc-1.11.0/lib   -L/home/eschnett/software/hwloc-1.11.0/lib -o libmpi_usempi_ignore_tkr.la -rpath /home/eschnett/software/openmpi-v2.x-dev-187-g8516895/lib mpi-ignore-tkr.lo mpi-ignore-tkr-sizeof.lo  -lrt -lutil -lhwloc -lpthread -lpthread   -lrt -lutil -lhwloc -lpthread -lpthread  

Output:

  FCLD     libmpi_usempi_ignore_tkr.la
libtool: link: /home/eschnett/software/gcc-5.2.0/bin/wrap-gfortran -shared  -fPIC  .libs/mpi-ignore-tkr.o .libs/mpi-ignore-tkr-sizeof.o   -Wl,-rpath -Wl,/home/eschnett/software/hwloc-1.11.0/lib -Wl,-rpath -Wl,/home/eschnett/software/gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,/home/eschnett/software/hwloc-1.11.0/lib -Wl,-rpath -Wl,/home/eschnett/software/gcc-5.2.0/lib/../lib64 -L/opt/ofed/lib64 -L/home/eschnett/software/hwloc-1.11.0/lib -lrt -lutil /home/eschnett/software/hwloc-1.11.0/lib/libhwloc.so -L/home/eschnett/software/gcc-5.2.0/./lib/gcc/x86_64-unknown-linux-gnu/5.2.0 -L/home/eschnett/software/gcc-5.2.0/lib64 -L/home/eschnett/software/gcc-5.2.0/lib -l -L/home/eschnett/software/gcc-5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0 -L/home/eschnett/software/gcc-5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L. -L/home/eschnett/software/gcc-5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/../../.. /home/eschnett/software/gcc-5.2.0/lib/../lib64/libstdc++.so /home/eschnett/software/gcc-5.2.0/lib/../lib64/libsupc++.a /home/eschnett/software/gcc-5.2.0/lib/../lib64/libatomic.so -lpthread /home/eschnett/software/gcc-5.2.0/lib/../lib64/libgfortran.so -L/home/eschnett/software/gcc-5.2.0/lib/../lib64 /home/eschnett/software/gcc-5.2.0/lib/../lib64/libquadmath.so -lm -lc -lgcc_s  -Wl,-rpath -Wl,/home/eschnett/software/hwloc-1.11.0/lib   -pthread -Wl,-soname -Wl,libmpi_usempi_ignore_tkr.so.0 -o .libs/libmpi_usempi_ignore_tkr.so.0.0.0
/usr/bin/ld: cannot find -l-L/home/eschnett/software/gcc-5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0
collect2: error: ld returned 1 exit status

Note the empty -l in the command. This is already present in config.status; see https://gist.github.com/eschnett/a48f63e9b4da66aaee61. See also https://gist.github.com/eschnett/608ad2b9f7694b2054f6 for config.log.

[ggouaillardet]

thanks
i noted the empty -l from config.status is in the postdeps_FC definition only.
could you please post your wrap-gfortran and all the environment variables it uses ?

[jsquyres]

@eschnett We just merged #808 into master -- the nightly tarball tonight should (finally) contain the fix. Can you verify that it works for you? If so, we'll merge it over to v2.x.

[eschnett]

Yes, I can build dev-2312-g5ef0632 without any of my current patches (the one discussed here, and another one having to do with Lustre).