ctest failure when compiled with aocc

[DJDavies2]

What happened?

Compiling with aocc results in ctest failures for some linear algebra tests. For example eckit_test_linalg_dense_backend fails like this:

Test "test backend (section: dot)" failed: Condition failed: linalg.dot(a, a) == 21.

Similar for other sections. Printing the result of linalg.dot(a,a) indicates that it is returning 0 for aocc.

What are the steps to reproduce the bug?

Build and run with aocc.

Version

develop

Platform (OS and architecture)

Linux

Relevant log output

Accompanying data

No response

Organisation

No response

[wdeconinck]

It would be good if we could reproduce this as well on our system.
Do you have any more information on the aocc version? Did you try with more versions?

[DJDavies2]

AOCC version we have is:

david.davies@exa-login05:~ $ clang-14 -v
AMD clang version 14.0.6 (CLANG: AOCC_4.0.0-Build#434 2022_10_28) (based on LLVM Mirror.Version.14.0.6)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/AMD/aocc-compiler-4.0.0/bin
Found candidate GCC installation: /usr/lib64/gcc/x86_64-suse-linux/7
Selected GCC installation: /usr/lib64/gcc/x86_64-suse-linux/7
Candidate multilib: .;@m64
Selected multilib: .;@m64

Unfortunately we only have one version.

[DJDavies2]

job.out.gz
job.err.gz

[DJDavies2]

I've attached cmake output. It is from a bundle, but hopefully the eckit stuff is findable.

One difference is that for AOCC it seems to claim the OpenMP version is 5.0. We also have a gnu compiler which claims to use OpenMP version 4.5; this code works fine with gnu. I don't know if that is relevant.

[wdeconinck]

I could not reproduce this with our version of aocc 4.0.0 just trying to compile eckit.
On our ATOS:

module load prgenv/amd
module load aocc/4.0.0

> clang-14 -v
AMD clang version 14.0.6 (CLANG: AOCC_4.0.0-Build#434 2022_10_28) (based on LLVM Mirror.Version.14.0.6)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/apps/aocc/4.0.0/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda, version

The compile command for src/eckit/linalg/dense/LinearAlgebraGeneric.cc contains -fopenmp=libomp

/usr/local/apps/aocc/4.0.0/bin/clang++ -Deckit_linalg_EXPORTS -I/perm/nawd/eckit-issue-233/build-aocc-4.0.0/src -I/perm/nawd/eckit-issue-233/source/eckit/src -pipe -Wall -Wextra -Wno-unused-parameter -Wno-unused-variable -Wno-sign-compare -O2 -g -DNDEBUG -std=gnu++17 -fPIC -fopenmp=libomp -MD -MT src/eckit/linalg/CMakeFiles/eckit_linalg.dir/dense/LinearAlgebraGeneric.cc.o -MF CMakeFiles/eckit_linalg.dir/dense/LinearAlgebraGeneric.cc.o.d -o CMakeFiles/eckit_linalg.dir/dense/LinearAlgebraGeneric.cc.o -c /perm/nawd/eckit-issue-233/source/eckit/src/eckit/linalg/dense/LinearAlgebraGeneric.cc

And the CMake detection of OpenMP states:

-- Found OpenMP_CXX: -fopenmp=libomp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0") found components: CXX
-- eckit FOUND OpenMP
-- Found package OpenMP required for feature OMP
-- Feature OMP enabled
...
-- system : [ac6-102.bullx] [Linux-4.18.0-553.74.1.el8_10.x86_64] [linux.64]
-- processor        : [x86_64]
-- endiness         : Little Endian -- IEEE []
-- build type       : [RelWithDebInfo]
-- timestamp        : [20260122102545]
-- install prefix   : [/usr/local]
--   bin dir        : [/usr/local/bin]
--   lib dir        : [/usr/local/lib64]
--   include dir    : [/usr/local/include]
--   data dir       : [/usr/local/share/eckit]
--   cmake dir      : [/usr/local/lib64/cmake/eckit]
-- ---------------------------------------------------------
-- C -- Clang 14.0.6
--     compiler   : /usr/local/apps/aocc/4.0.0/bin/clang
--     flags      :  -pipe -O2 -g -DNDEBUG
--     link flags :
-- CXX -- Clang 14.0.6
--     compiler   : /usr/local/apps/aocc/4.0.0/bin/clang++
--     flags      :  -pipe -Wall -Wextra -Wno-unused-parameter -Wno-unused-variable -Wno-sign-compare -O2 -g -DNDEBUG
--     link flags :
-- linker : /usr/local/apps/aocc/4.0.0/bin/ld.lld
-- ar     : /usr/local/apps/aocc/4.0.0/bin/llvm-ar
-- ranlib : /usr/local/apps/aocc/4.0.0/bin/llvm-ranlib
-- link flags
--     executable [    -Wl,--disable-new-dtags ]
--     shared lib [ -Wl,--disable-new-dtags ]
--     static lib [ -Wl,--disable-new-dtags ]
-- install rpath  : $ORIGIN/../lib64

So I am not certain that this is a problem with aocc itself.

Since with your suggested PR #244 you are essentially disabling OMP for eckit for this one particular configuration, you may as well just turn off the feature in your bundle for eckit: -DECKIT_ENABLE_OMP=OFF

[DJDavies2]

-DECKIT_ENABLE_OMP=OFF seems to work for me and I am happy to go with that for now. I don't know what is the difference between our installation of aocc and yours. I notice you are compiling with clang++ directly whereas we are using CC (Cray compiler wrapper); maybe that is something I can try if I can get it to work.

[wdeconinck]

We don't have this in a Cray programming environment, rather this is in our Atos HPC.
With Cray PE it is always a bit trickier.
Perhaps you can find a reproducer with standard CMake and a simple C++ program and request assistance from your HPE contractors?
It is possible to customise the libraries OpenMP is using manually. You can do this with a toolchain file or on the command line. On LUMI, but with the Cray compiler itself (not aocc), we had to this as well:

-DOpenMP_C_LIB_NAMES=craymp -DOpenMP_CXX_LIB_NAMES=craymp -DOpenMP_Fortran_LIB_NAMES=craymp \
-DOpenMP_craymp_LIBRARY=craymp

And you can also set the OpenMP flags of course. For instance

-DOpenMP_Fortran_FLAGS=-homp

[DJDavies2]

Using clang++ direclty instead of CC didn't make any different for me. I will try and create a bug report for our HPC. In the meantime I am happy to close this issue and PR as switching off OpenMP for eckit for AOCC works for me. Thanks for the help.