Odd flang OpenMP Linking Behavior

[mathomp4]

All,

Last month, I built flang-new and I was able to build OpenMP code:

> flang-new --version
flang-new version 20.0.0git (git@github.com:GEOS-ESM/build-llvm-flang.git a62667f983fc27f97ed4658cc2beb288b00450c9)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /ford1/share/gmao_SIteam/llvm-flang/2024-09-06/bin
> flang-new -fopenmp test.F90
> OMP_NUM_THREADS=2 ./a.out
 Hello from process:  0
 Hello from process:  1

However, I built flang-new as of yesterday and:

> flang-new --version
flang version 20.0.0git (git@github.com:GEOS-ESM/build-llvm-flang.git 697744d14fec6505d07f95249388ac2452ee5151)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /ford1/share/gmao_SIteam/llvm-flang/2024-10-22/bin
> flang-new -fopenmp test.F90
flang-20: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
/usr/bin/ld: cannot find /usr/lib64/libatomic.so.1.2.0
flang-20: error: linker command failed with exit code 1 (use -v to see invocation)

Now, I do have libatomic.so.1.2.0 and it was around when I built flang-new with GCC 12.1.0 but it's the GCC 12 one:

> ls -1 /ford1/local/gcc/gcc-12.1.0/lib64/libatomic.*
/ford1/local/gcc/gcc-12.1.0/lib64/libatomic.a
/ford1/local/gcc/gcc-12.1.0/lib64/libatomic.la*
/ford1/local/gcc/gcc-12.1.0/lib64/libatomic.so@
/ford1/local/gcc/gcc-12.1.0/lib64/libatomic.so.1@
/ford1/local/gcc/gcc-12.1.0/lib64/libatomic.so.1.2.0*

and I have that in my LD_LIBRARY_PATH:

> echo $LD_LIBRARY_PATH
/ford1/share/gmao_SIteam/llvm-flang/2024-10-22/lib:/ford1/local/gcc/gcc-12.1.0/lib64:/home/mathomp4/lib

but it is not in /usr/lib64:

> ls -1 /usr/lib64/libatomic*
/usr/lib64/libatomic_ops_gpl.so.1@
/usr/lib64/libatomic_ops_gpl.so.1.1.2*
/usr/lib64/libatomic_ops.so.1@
/usr/lib64/libatomic_ops.so.1.1.1*

So if I build pointing to that:

> flang-new -fopenmp test.F90 -L/ford1/local/gcc/gcc-12.1.0/lib64 -latomic
flang-20: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
> OMP_NUM_THREADS=2 ./a.out
 Hello from process:  0
 Hello from process:  1

My question is: why does flang-new seem to think libatomic is in /usr/lib64? The build couldn't have seen it there since it didn't exist there during the build. And it's not like ldd wants it there:

> ldd /ford1/share/gmao_SIteam/llvm-flang/2024-10-22/bin/flang-new
	linux-vdso.so.1 (0x00007fff7bb0c000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000151cb7967000)
	libquadmath.so.0 => /ford1/local/gcc/gcc-12.1.0/lib64/libquadmath.so.0 (0x0000151cb7722000)
	librt.so.1 => /lib64/librt.so.1 (0x0000151cb751a000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000151cb7316000)
	libz.so.1 => /lib64/libz.so.1 (0x0000151cb70fe000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x0000151cb6e5a000)
	libstdc++.so.6 => /ford1/local/gcc/gcc-12.1.0/lib64/libstdc++.so.6 (0x0000151cb6a44000)
	libm.so.6 => /lib64/libm.so.6 (0x0000151cb66c2000)
	libgcc_s.so.1 => /ford1/local/gcc/gcc-12.1.0/lib64/libgcc_s.so.1 (0x0000151cb64a4000)
	libc.so.6 => /lib64/libc.so.6 (0x0000151cb60ce000)
	/lib64/ld-linux-x86-64.so.2 (0x0000151cb7b87000)

[mathomp4]

Note: C doesn't seem to be affected:

> clang -fopenmp test.c
> OMP_NUM_THREADS=2 ./a.out
Hello, world.
Hello, world.

[mjklemm]

I think you're a victim of this: #112202.

We need libatomic for some atomic operations. With the PR, Flang now recognizes this and automatically adds -latomic to the link line, when OpenMP is active and if it was configured to use the GCC runtime. If you setup Flang to work with the Clang C runtime, then the atomic operations should be included there and libatomic should not be needed.

Flang uses -latomic and then relies on the default search path in the environment for libraries.

BTW, before the PR, you would have seen a linker error barfing at missing symbols, when you have used certain atomic operations, e.g. CAS atomics.

[mjklemm]

PS: Please try setting LIBRARY_PATH to the location of libatomic. That should resolve the issue.

If not, please post the full output of flang -v -fopenmp ....

[mathomp4]

PS: Please try setting LIBRARY_PATH to the location of libatomic. That should resolve the issue.

Sadly, doesn't seem to help:

> echo $LIBRARY_PATH
/ford1/share/gmao_SIteam/llvm-flang/2024-10-22/lib:/ford1/local/gcc/gcc-12.1.0/lib64

> flang-new -fopenmp  test.F90
flang-20: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
/usr/bin/ld: cannot find /usr/lib64/libatomic.so.1.2.0
flang-20: error: linker command failed with exit code 1 (use -v to see invocation)

If not, please post the full output of flang -v -fopenmp ....

> flang -v -fopenmp test.F90
flang version 20.0.0git (git@github.com:GEOS-ESM/build-llvm-flang.git 697744d14fec6505d07f95249388ac2452ee5151)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /ford1/share/gmao_SIteam/llvm-flang/2024-10-22/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
flang-20: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
 "/ford1/share/gmao_SIteam/llvm-flang/2024-10-22/bin/flang" -fc1 -triple x86_64-unknown-linux-gnu -emit-obj -fcolor-diagnostics -mrelocation-model pic -pic-level 2 -pic-is-pie -target-cpu x86-64 -fopenmp -resource-dir /ford1/share/gmao_SIteam/llvm-flang/2024-10-22/lib/clang/20 -mframe-pointer=all -o /home/mathomp4/tmpdir/test-68da60.o -x f95-cpp-input test.F90
 "/usr/bin/ld" --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -pie -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/Scrt1.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/8/crtbeginS.o -L/ford1/share/gmao_SIteam/llvm-flang/2024-10-22/bin/../lib/x86_64-unknown-linux-gnu -L/usr/lib/gcc/x86_64-redhat-linux/8 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/lib -L/usr/lib -L/ford1/share/gmao_SIteam/llvm-flang/2024-10-22/lib -L/ford1/local/gcc/gcc-12.1.0/lib64 /home/mathomp4/tmpdir/test-68da60.o -L/ford1/share/gmao_SIteam/llvm-flang/2024-10-22/lib -lFortranFloat128Math --as-needed -lquadmath --no-as-needed -lFortranRuntime -lFortranDecimal -latomic -lm -lomp -L/ford1/share/gmao_SIteam/llvm-flang/2024-10-22/lib -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/8/crtendS.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o
/usr/bin/ld: cannot find /usr/lib64/libatomic.so.1.2.0
flang-20: error: linker command failed with exit code 1 (use -v to see invocation)

This is interesting. I mean, I see in there:

-L/ford1/local/gcc/gcc-12.1.0/lib64 ... -latomic

It just seems hellbent on finding it in /usr/lib64. Here is the modulefile I'm using in case anything pops out. It's pretty boring.

Also, learned that apparently there is a flang alias. Huh.

[mjklemm]

Also, learned that apparently there is a flang alias. Huh.

Just quickly between meetings: It's the other way round. It's flang now and flang-new is the alias. We have switched about two weeks ago, so that some point flang-new will cease to exist.

I'll have a look at the other things when I have a free minute today. Thanks for sharing the output!

[mathomp4]

Also, learned that apparently there is a flang alias. Huh.

Just quickly between meetings: It's the other way round. It's flang now and flang-new is the alias. We have switched about two weeks ago, so that some point flang-new will cease to exist.

Well then. Time to scour my system for things (Makefiles, etc.) that have been looking for flang-new!

And thanks for looking. For now I can sort of "shove" CMake to be happy with:

-DCMAKE_EXE_LINKER_FLAGS='-L/ford1/local/gcc/gcc-12.1.0/lib64 -latomic'

Not portable, but I can keep going for now.