BLIS works on methane, fails on histamine & dopamine

[mgates3]

blas.cc.txt

I'm wondering if BLIS is compiled for the wrong architecture, because it has an illegal instruction when run on histamine & dopamine (AMD CPUs), but works on methane (Intel CPUs).

sh methane blaspp> module purge
sh methane blaspp> module load amd-aocl
Loading amd-aocl/4.2/gcc-11.4.1-l5myig
  Loading requirement: glibc/2.34/gcc-11.4.1-dtaqd4
  gcc-runtime/11.4.1/gcc-11.4.1-no4eyi bzip2/1.0.8/gcc-11.4.1-n7kvt3
  libmd/1.0.4/gcc-11.4.1-crgsfc libbsd/0.12.2/gcc-11.4.1-njr4pw
  expat/2.6.2/gcc-11.4.1-65qru7 ncurses/6.5/gcc-11.4.1-soxhau
  readline/8.2/gcc-11.4.1-uebdnx gdbm/1.23/gcc-11.4.1-6v2jvd
  libiconv/1.17/gcc-11.4.1-ynxtpm xz/5.4.6/gcc-11.4.1-vuvxyo
  zlib-ng/2.2.1/gcc-11.4.1-wwmm37 libxml2/2.10.3/gcc-11.4.1-ewigrh
  pigz/2.8/gcc-11.4.1-ywpabn zstd/1.5.6/gcc-11.4.1-qnrc4s
  tar/1.34/gcc-11.4.1-bvbsgm gettext/0.22.5/gcc-11.4.1-x7sso7
  libffi/3.4.6/gcc-11.4.1-jtnh3b libxcrypt/4.4.35/gcc-11.4.1-wvymff
  openssl/3.3.1/gcc-11.4.1-4iiodh sqlite/3.46.0/gcc-11.4.1-kefle4
  util-linux-uuid/2.40.2/gcc-11.4.1-csmedo
  python/3.11.9/gcc-11.4.1-hxxchl amdblis/4.2/gcc-11.4.1-rii3hi
  cuda/11.8.0/gcc-11.4.1-q6jpi3 libpciaccess/0.17/gcc-11.4.1-k5nwmf
  hwloc/2.9.3/gcc-11.4.1-w2wap2 libevent/2.1.12/gcc-11.4.1-eitmnr
  numactl/2.0.14/gcc-11.4.1-3tmyb5 slurm/22.05.9/gcc-11.4.1-5mlc5h
  check/0.15.2/gcc-11.4.1-ctf5dj gdrcopy/2.4.1/gcc-11.4.1-lx4vdv
  libnl/3.3.0/gcc-11.4.1-a5eaep rdma-core/52.0/gcc-11.4.1-5kuqln
  ucx/1.15.0/gcc-11.4.1-sr34kx openmpi/5.0.2/gcc-11.4.1-tzydwy
  berkeley-db/18.1.40/gcc-11.4.1-fxuu75 perl/5.38.2/gcc-11.4.1-iucw2s
  texinfo/7.1/gcc-11.4.1-6hpk6i amdfftw/4.2/gcc-11.4.1-irus4b
  aocl-utils/4.2/gcc-11.4.1-dakldb amdlibflame/4.2/gcc-11.4.1-o3vhta
  gmp/6.3.0/gcc-11.4.1-vhhzgh mpfr/4.2.1/gcc-11.4.1-55ywxq
  amdlibm/4.2/gcc-11.4.1-i7u5td amdscalapack/4.2/gcc-11.4.1-6smtud
  aocl-compression/4.2/gcc-11.4.1-wpqg2b
  aocl-crypto/4.2/gcc-11.4.1-h33cx7 aocl-libmem/4.2/gcc-11.4.1-wykjyz
  aocl-sparse/4.2/gcc-11.4.1-jvxwdm

# This gcc module doesn't conflict with the AOCL module.
sh methane blaspp> module load gcc/11.3
Loading gcc/11.3.0/gcc-11.4.1-3u5a22
  Loading requirement: mpc/1.3.1/gcc-11.4.1-6rsmso

sh methane blaspp> export CPATH=$ICL_AMDBLIS_ROOT/include/blis
sh methane blaspp> export LIBRARY_PATH=$ICL_AMDBLIS_ROOT/lib
sh methane blaspp> export LD_LIBRARY_PATH=$ICL_AMDBLIS_ROOT/lib

sh methane blaspp> g++ -Wall -o config/blas config/blas.cc -lblis
sh methane blaspp> ./config/blas
x[ 0 ] = 1.0; y[ 0 ] = 5.0
x[ 1 ] = 2.0; y[ 1 ] = 4.0
x[ 2 ] = 3.0; y[ 2 ] = 3.0
x[ 3 ] = 4.0; y[ 3 ] = 2.0
x[ 4 ] = 5.0; y[ 4 ] = 1.0
result = 35.0; should be 35.0
ok

#-------------------------------------------------------------------------------

sh histamine0 blaspp> module purge
sh histamine0 blaspp> module load amd-aocl
Loading amd-aocl/4.2/gcc-11.4.1-l5myig
  Loading requirement: glibc/2.34/gcc-11.4.1-dtaqd4
  gcc-runtime/11.4.1/gcc-11.4.1-no4eyi bzip2/1.0.8/gcc-11.4.1-n7kvt3
  libmd/1.0.4/gcc-11.4.1-crgsfc libbsd/0.12.2/gcc-11.4.1-njr4pw
  expat/2.6.2/gcc-11.4.1-65qru7 ncurses/6.5/gcc-11.4.1-soxhau
  readline/8.2/gcc-11.4.1-uebdnx gdbm/1.23/gcc-11.4.1-6v2jvd
  libiconv/1.17/gcc-11.4.1-ynxtpm xz/5.4.6/gcc-11.4.1-vuvxyo
  zlib-ng/2.2.1/gcc-11.4.1-wwmm37 libxml2/2.10.3/gcc-11.4.1-ewigrh
  pigz/2.8/gcc-11.4.1-ywpabn zstd/1.5.6/gcc-11.4.1-qnrc4s
  tar/1.34/gcc-11.4.1-bvbsgm gettext/0.22.5/gcc-11.4.1-x7sso7
  libffi/3.4.6/gcc-11.4.1-jtnh3b libxcrypt/4.4.35/gcc-11.4.1-wvymff
  openssl/3.3.1/gcc-11.4.1-4iiodh sqlite/3.46.0/gcc-11.4.1-kefle4
  util-linux-uuid/2.40.2/gcc-11.4.1-csmedo
  python/3.11.9/gcc-11.4.1-hxxchl amdblis/4.2/gcc-11.4.1-rii3hi
  cuda/11.8.0/gcc-11.4.1-q6jpi3 libpciaccess/0.17/gcc-11.4.1-k5nwmf
  hwloc/2.9.3/gcc-11.4.1-w2wap2 libevent/2.1.12/gcc-11.4.1-eitmnr
  numactl/2.0.14/gcc-11.4.1-3tmyb5 slurm/22.05.9/gcc-11.4.1-5mlc5h
  check/0.15.2/gcc-11.4.1-ctf5dj gdrcopy/2.4.1/gcc-11.4.1-lx4vdv
  libnl/3.3.0/gcc-11.4.1-a5eaep rdma-core/52.0/gcc-11.4.1-5kuqln
  ucx/1.15.0/gcc-11.4.1-sr34kx openmpi/5.0.2/gcc-11.4.1-tzydwy
  berkeley-db/18.1.40/gcc-11.4.1-fxuu75 perl/5.38.2/gcc-11.4.1-iucw2s
  texinfo/7.1/gcc-11.4.1-6hpk6i amdfftw/4.2/gcc-11.4.1-irus4b
  aocl-utils/4.2/gcc-11.4.1-dakldb amdlibflame/4.2/gcc-11.4.1-o3vhta
  gmp/6.3.0/gcc-11.4.1-vhhzgh mpfr/4.2.1/gcc-11.4.1-55ywxq
  amdlibm/4.2/gcc-11.4.1-i7u5td amdscalapack/4.2/gcc-11.4.1-6smtud
  aocl-compression/4.2/gcc-11.4.1-wpqg2b
  aocl-crypto/4.2/gcc-11.4.1-h33cx7 aocl-libmem/4.2/gcc-11.4.1-wykjyz
  aocl-sparse/4.2/gcc-11.4.1-jvxwdm

# This gcc module doesn't conflict with the AOCL module.
sh histamine0 blaspp> module load gcc/11.3
Loading gcc/11.3.0/gcc-11.4.1-3u5a22
  Loading requirement: mpc/1.3.1/gcc-11.4.1-6rsmso

sh histamine0 blaspp> export CPATH=$ICL_AMDBLIS_ROOT/include/blis
sh histamine0 blaspp> export LIBRARY_PATH=$ICL_AMDBLIS_ROOT/lib
sh histamine0 blaspp> export LD_LIBRARY_PATH=$ICL_AMDBLIS_ROOT/lib

sh histamine0 blaspp> g++ -Wall -o config/blas config/blas.cc -lblis
sh histamine0 blaspp> ./config/blas
x[ 0 ] = 1.0; y[ 0 ] = 5.0
x[ 1 ] = 2.0; y[ 1 ] = 4.0
x[ 2 ] = 3.0; y[ 2 ] = 3.0
x[ 3 ] = 4.0; y[ 3 ] = 2.0
x[ 4 ] = 5.0; y[ 4 ] = 1.0
Illegal instruction (core dumped)

This test is calling BLAS ddot when it has an illegal instruction. See attached.

[mgates3]

Tagging Max. @TeachRaccooon

[G-Ragghianti]

The illegal instruction appears to be a missing avx512 on the AMD CPUs on dopamine, histamine, and leconte. On serotonin (with a newer AMD CPU), the CPU supports avx512 and the test passes. The amdblis package is built with arch=x86_64 (in spack), which shouldn't include this instruction. This is probably a bug in the amdblbis package that is causing it to not compile for the correct requested architecture. I will do some test builds of blis to see where this is occurring.

[G-Ragghianti]

I have filed an issue with spack and the AMD maintainers: spack/spack#49376