AMD-AOCC-AOCL update and minor fixed

Author: QuantumMisaka

toolchain/README.md

@@ -27,7 +27,7 @@ and give setup files that you can use to compile ABACUS.
 - [x] Support for [LibRI](https://github.com/abacusmodeling/LibRI) by submodule or automatic installation from github.com (but installed LibRI via `wget` seems to have some problem, please be cautious)
 - [x] A mirror station by Bohrium database, which can download CEREAL, LibNPY, LibRI and LibComm by `wget` in China Internet. 
 - [x] Support for GPU compilation, users can add `-DUSE_CUDA=1` in builder scripts.
-- [ ] Support for AMD compiler and math lib like `AOCL` and `AOCC` (doing)
+- [x] Support for AMD compiler and math lib  `AOCL` and `AOCC` (not fully complete due to flang and AOCC-ABACUS compliation error)
 - [ ] Change the downloading url from cp2k mirror to other mirror or directly downloading from official website. (doing)
 - [ ] Support a JSON or YAML configuration file for toolchain, which can be easily modified by users.
 - [ ] A better README and Detail markdown file.
@@ -50,6 +50,8 @@ There are also well-modified script to run *install_abacus_toolchain.sh* for `gn
 > ./toolchain_gnu.sh
 # for intel-mkl
 > ./toolchain_intel.sh
+# for amd aocc-aocl
+> ./toolchain_amd.sh
 # for intel-mkl-mpich
 > ./toolchain_intel-mpich.sh
 ```
@@ -121,8 +123,12 @@ The needed dependencies version default:
 - `ELPA` 2025.01.001
 - `CEREAL` 1.3.2
 - `RapidJSON` 1.1.0
-And Intel-oneAPI need user or server manager to manually install from Intel.
-[Intel-oneAPI](https://www.intel.cn/content/www/cn/zh/developer/tools/oneapi/toolkits.html)
+And:
+- Intel-oneAPI need user or server manager to manually install from Intel.
+- - [Intel-oneAPI](https://www.intel.cn/content/www/cn/zh/developer/tools/oneapi/toolkits.html)
+- AMD AOCC-AOCL need user or server manager to manually install from AMD.
+- - [AOCC](https://www.amd.com/zh-cn/developer/aocc.html)
+- - [AOCL](https://www.amd.com/zh-cn/developer/aocl.html)
 
 Dependencies below are optional， which is NOT installed by default:
 
@@ -152,6 +158,8 @@ If compliation is successful, a message will be shown like this:
 >     ./build_abacus_gnu.sh
 > To build ABACUS by intel-toolchain, just use:
 >     ./build_abacus_intel.sh
+> To build ABACUS by amd-toolchain in gcc-aocl, just use:
+>     ./build_abacus_amd.sh
 > or you can modify the builder scripts to suit your needs.
 ```
 
@@ -183,6 +191,14 @@ or you can also do it in a more completely way:
 
 ### Intel-oneAPI problem
 
+#### OneAPI 2025.0 problem
+
+Generally, OneAPI 2025.0 can be useful to compile basic function of ABACUS, but one will encounter compatible problem related to something. Here is the treatment
+- related to rapidjson: 
+- - Not to use rapidjson in your toolchain
+- - or use the master branch of [RapidJSON](https://github.com/Tencent/rapidjson)
+- related to LibRI: not to use LibRI or downgrade your OneAPI.
+
 #### ELPA problem via Intel-oneAPI toolchain in AMD server
 
 The default compiler for Intel-oneAPI is `icpx` and `icx`, which will cause problem when compling ELPA in AMD server. (Which is a problem and needed to have more check-out)
@@ -210,12 +226,27 @@ And will not occur in Intel-MPI before 2021.10.0 (Intel-oneAPI before 2023.2.0)
 
 More problem and possible solution can be accessed via [#2928](https://github.com/deepmodeling/abacus-develop/issues/2928)
 
+### AMD AOCC-AOCL problem
+
+You cannot use AOCC to complie abacus now, see [#5982](https://github.com/deepmodeling/abacus-develop/issues/5982) .
+
+However, use AOCC-AOCL to compile dependencies is permitted and usually get boosting in ABACUS effciency. But you need to get rid of `flang` while compling ELPA. Toolchain itself help you make this `flang` shade in default, and you can manully use `flang` by setting `--with-flang=yes` in `toolchain_amd.sh` to have a try. 
+
+Notice: ABACUS via GCC-AOCL in AOCC-AOCL toolchain have no application with DeePKS, DeePMD and LibRI. 
 
 ### OpenMPI problem
 
 #### in EXX and LibRI
 
-- GCC toolchain with OpenMPI cannot compile LibComm v0.1.1 due to the different MPI variable type from MPICH and IntelMPI, see discussion here [#5033](https://github.com/deepmodeling/abacus-develop/issues/5033), you can switch to GCC-MPICH or Intel toolchain
+- GCC toolchain with OpenMPI cannot compile LibComm v0.1.1 due to the different MPI variable type from MPICH and IntelMPI, see discussion here [#5033](https://github.com/deepmodeling/abacus-develop/issues/5033), you can try use a newest branch of LibComm by 
+```
+git clone https://gitee.com/abacus_dft/LibComm -b MPI_Type_Contiguous_Pool
+``` 
+or pull the newest master branch of LibComm
+```
+git clone https://github.com/abacusmodeling/LibComm
+```
+. yet another is switching to GCC-MPICH or Intel toolchain
 - It is recommended to use Intel toolchain if one wants to include EXX feature in ABACUS, which can have much better performance and can use more than 16 threads in OpenMP parallelization to accelerate the EXX process.
 
 #### OpenMPI-v5 

toolchain/build_abacus_gnu-aocl.sh

@@ -0,0 +1,82 @@
+#!/bin/bash
+#SBATCH -J build
+#SBATCH -N 1
+#SBATCH -n 16
+#SBATCH -o install.log
+#SBATCH -e install.err
+# JamesMisaka in 2025.03.09
+
+# Build ABACUS by amd-openmpi toolchain
+
+# module load openmpi aocc aocl
+
+ABACUS_DIR=..
+TOOL=$(pwd)
+INSTALL_DIR=$TOOL/install
+source $INSTALL_DIR/setup
+cd $ABACUS_DIR
+ABACUS_DIR=$(pwd)
+#AOCLhome=/opt/aocl  # user can specify this parameter
+
+BUILD_DIR=build_abacus_gnu
+rm -rf $BUILD_DIR
+
+PREFIX=$ABACUS_DIR
+ELPA=$INSTALL_DIR/elpa-2025.01.001/cpu
+CEREAL=$INSTALL_DIR/cereal-1.3.2/include/cereal
+LIBXC=$INSTALL_DIR/libxc-7.0.0
+RAPIDJSON=$INSTALL_DIR/rapidjson-1.1.0/
+# LAPACK=$AOCLhome/lib
+# SCALAPACK=$AOCLhome/lib
+# FFTW3=$AOCLhome
+# LIBRI=$INSTALL_DIR/LibRI-0.2.1.0
+# LIBCOMM=$INSTALL_DIR/LibComm-0.1.1
+# LIBTORCH=$INSTALL_DIR/libtorch-2.1.2/share/cmake/Torch
+# LIBNPY=$INSTALL_DIR/libnpy-1.0.1/include
+# DEEPMD=$HOME/apps/anaconda3/envs/deepmd # v3.0 might have problem
+
+# if clang++ have problem, switch back to g++
+
+cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \
+        -DCMAKE_CXX_COMPILER=clang++ \
+        -DMPI_CXX_COMPILER=mpicxx \
+        -DELPA_DIR=$ELPA \
+        -DCEREAL_INCLUDE_DIR=$CEREAL \
+        -DLibxc_DIR=$LIBXC \
+        -DENABLE_LCAO=ON \
+        -DENABLE_LIBXC=ON \
+        -DUSE_OPENMP=ON \
+        -DUSE_ELPA=ON \
+        -DENABLE_RAPIDJSON=ON \
+        -DRapidJSON_DIR=$RAPIDJSON \
+#         -DLAPACK_DIR=$LAPACK \
+#         -DSCALAPACK_DIR=$SCALAPACK \
+#         -DFFTW3_DIR=$FFTW3 \
+#         -DENABLE_DEEPKS=1 \
+#         -DTorch_DIR=$LIBTORCH \
+#         -Dlibnpy_INCLUDE_DIR=$LIBNPY \
+#         -DENABLE_LIBRI=ON \
+#         -DLIBRI_DIR=$LIBRI \
+#         -DLIBCOMM_DIR=$LIBCOMM \
+# 	      -DDeePMD_DIR=$DEEPMD \
+
+# if one want's to include deepmd, your system gcc version should be >= 11.3.0 for glibc requirements
+
+cmake --build $BUILD_DIR -j `nproc` 
+cmake --install $BUILD_DIR 2>/dev/null
+
+# generate abacus_env.sh
+cat << EOF > "${TOOL}/abacus_env.sh"
+#!/bin/bash
+source $INSTALL_DIR/setup
+export PATH="${PREFIX}/bin":\${PATH}
+EOF
+
+# generate information
+cat << EOF
+========================== usage =========================
+Done!
+To use the installed ABACUS version
+You need to source ${TOOL}/abacus_env.sh first !
+"""
+EOF

toolchain/build_abacus_gnu.sh

@@ -4,7 +4,6 @@
 #SBATCH -n 16
 #SBATCH -o install.log
 #SBATCH -e install.err
-# install ABACUS with libxc and deepks
 # JamesMisaka in 2025.03.09
 
 # Build ABACUS by gnu-toolchain

toolchain/build_abacus_intel-mpich.sh

@@ -4,13 +4,12 @@
 #SBATCH -n 16
 #SBATCH -o install.log
 #SBATCH -e install.err
-# build and install ABACUS with libxc, also can with deepks and deepmd
 # JamesMisaka in 2025.03.09
 
 # Build ABACUS by intel-toolchain with mpich
 
 # module load mkl compiler
-# source path/to/vars.sh
+# source path/to/setvars.sh
 
 ABACUS_DIR=..
 TOOL=$(pwd)

toolchain/build_abacus_intel.sh

@@ -4,13 +4,12 @@
 #SBATCH -n 16
 #SBATCH -o install.log
 #SBATCH -e install.err
-# install ABACUS with libxc and deepks
 # JamesMisaka in 2025.03.09
 
 # Build ABACUS by intel-toolchain
 
 # module load mkl compiler mpi
-# source path/to/vars.sh
+# source path/to/setvars.sh
 
 ABACUS_DIR=..
 TOOL=$(pwd)

toolchain/install_abacus_toolchain.sh

@@ -159,6 +159,8 @@ The --with-PKG options follow the rules:
                           Default = yes
   --with-amd              Use the AMD compiler to build CP2K.
                           Default = system
+  --with-flang            Use flang in AMD compiler, which may lead to problem and efficiency loss in ELPA
+                          Default = no
   --with-cmake            Cmake utilities
                           Default = install
   --with-openmpi          OpenMPI, important if you want a parallel version of ABACUS.
@@ -183,6 +185,10 @@ The --with-PKG options follow the rules:
                           it replaces the FFTW library. If the ScaLAPACK component is
                           found, it replaces the one specified by --with-scalapack.
                           Default = system
+  --with-aocl             AMD Optimizing CPU Libraries, which provides LAPACK, BLAS, FFTW, ScaLAPACK
+                          the ScaLAPACK and FFTW can directly use which in AOCL by setting --with-scalapack=system and --with-fftw=system if AOCL in system environment.
+                          related scripts are in development to incorporate scalapack and fftw once for all.
+                          Default = system
   --with-openblas         OpenBLAS is a free high performance LAPACK and BLAS library,
                           the successor to GotoBLAS.
                           Default = install
@@ -238,7 +244,7 @@ EOF
 # ------------------------------------------------------------------------
 tool_list="gcc intel amd cmake"
 mpi_list="mpich openmpi intelmpi"
-math_list="mkl openblas"
+math_list="mkl aocl openblas"
 lib_list="fftw libxc scalapack elpa cereal rapidjson libtorch libnpy libri libcomm"
 package_list="${tool_list} ${mpi_list} ${math_list} ${lib_list}"
 # ------------------------------------------------------------------------
@@ -266,6 +272,9 @@ with_scalapack="__INSTALL__"
 if [ "${MKLROOT}" ]; then
   export MATH_MODE="mkl"
   with_mkl="__SYSTEM__"
+elif [ "${AOCLhome}" ]; then
+  export MATH_MODE="aocl"
+  with_aocl="__SYSTEM__"
 else
   export MATH_MODE="openblas"
 fi
@@ -292,6 +301,7 @@ if (command -v mpiexec > /dev/null 2>&1); then
     echo "MPI is detected and it appears to be Intel MPI"
     with_gcc="__DONTUSE__"
     with_amd="__DONTUSE__"
+    with_aocl="__DONTUSE__"
     with_intel="__SYSTEM__"
     with_intelmpi="__SYSTEM__"
     export MPI_MODE="intelmpi"
@@ -318,10 +328,11 @@ export intel_classic="no"
 # and will lead to problem in force calculation
 # but icx is recommended by intel compiler
 # option: --with-intel-classic can change it to yes/no
-# zhaoqing by 2023.08
+# JamesMisaka by 2023.08
 export intelmpi_classic="no"
-export with_ifx="yes"
-export openmpi_4th="no"
+export with_ifx="yes" # whether ifx is used in oneapi
+export with_flang="no" # whether flang is used in aocc
+export openmpi_4th="no" # whether openmpi downgrade
 export GPUVER="no"
 export MPICH_DEVICE="ch4"
 export TARGET_CPU="native"
@@ -342,6 +353,7 @@ if [ "${CRAY_LD_LIBRARY_PATH}" ]; then
   # set default value for some installers appropriate for CLE
   with_gcc="__DONTUSE__"
   with_amd="__DONTUSE__"
+  with_aocl="__DONTUSE__"
   with_intel="__DONTUSE__"
   with_fftw="__SYSTEM__"
   with_scalapack="__DONTUSE__"
@@ -416,6 +428,12 @@ while [ $# -ge 1 ]; do
         cray)
           export MATH_MODE="cray"
           ;;
+        aocl)
+          export MATH_MODE="aocl"
+          with_aocl="__SYSTEM__"
+          with_fftw="__SYSTEM__"
+          with_scalapack="__SYSTEM__"
+          ;;
         mkl)
           export MATH_MODE="mkl"
           ;;
@@ -424,7 +442,7 @@ while [ $# -ge 1 ]; do
           ;;
         *)
           report_error ${LINENO} \
-            "--math-mode currently only supports mkl, and openblas as options"
+            "--math-mode currently only supports mkl, aocl, openblas and cray as options"
           ;;
       esac
       ;;
@@ -519,9 +537,6 @@ while [ $# -ge 1 ]; do
         export MPI_MODE=intelmpi
       fi
       ;;
-    --with-amd*)
-      with_amd=$(read_with "${1}" "__SYSTEM__")
-      ;;
     --with-intel-classic*)
       intel_classic=$(read_with "${1}" "no") # default new intel compiler
       ;;
@@ -534,6 +549,15 @@ while [ $# -ge 1 ]; do
     --with-ifx*)
       with_ifx=$(read_with "${1}" "yes") # default yes
       ;;
+    --with-amd*)
+      with_amd=$(read_with "${1}" "__SYSTEM__")
+      ;;
+    --with-flang*)
+      with_flang=$(read_with "${1}" "no")
+      ;;
+    --with-aocl*)
+      with_aocl=$(read_with "${1}" "__SYSTEM__")
+      ;;
     --with-libxc*)
       with_libxc=$(read_with "${1}")
       ;;
@@ -864,6 +888,8 @@ To build ABACUS by gnu-toolchain, just use:
     ./build_abacus_gnu.sh
 To build ABACUS by intel-toolchain, just use:
     ./build_abacus_intel.sh
+To build ABACUS by amd-toolchain in gcc-aocl, just use:
+    ./build_abacus_gnu-aocl.sh
 or you can modify the builder scripts to suit your needs.
 """
 EOF

toolchain/scripts/stage0/install_amd.sh

@@ -33,9 +33,12 @@ case "${with_amd}" in
     echo "==================== Finding AMD compiler from system paths ===================="
     check_command clang "amd" && CC="$(realpath $(command -v clang))" || exit 1
     check_command clang++ "amd" && CXX="$(realpath $(command -v clang++))" || exit 1
-    check_command gfortran "gcc" && FC="gfortran" || exit 1
-    add_lib_from_paths GCC_LDFLAGS "libgfortran.*" ${LIB_PATHS}
-    #check_command flang "amd" && FC="$(realpath $(command -v flang))" || exit 1
+    if [ "${with_flang}" = "yes" ]; then
+        check_command flang "amd" && FC="$(realpath $(command -v flang))" || exit 1
+    else
+        check_command gfortran "gcc" && FC="gfortran" || exit 1
+        add_lib_from_paths GCC_LDFLAGS "libgfortran.*" ${LIB_PATHS}
+    fi
     F90="${FC}"
     F77="${FC}"
     ;;
@@ -50,8 +53,11 @@ case "${with_amd}" in
     check_dir "${pkg_install_dir}/include"
     check_command ${pkg_install_dir}/bin/clang "amd" && CC="${pkg_install_dir}/bin/clang" || exit 1
     check_command ${pkg_install_dir}/bin/clang++ "amd" && CXX="${pkg_install_dir}/bin/clang++" || exit 1
-    check_command gfortran "gcc" && FC="$(command -v gfortran)" || exit 1
-    #check_command ${pkg_install_dir}/bin/flang "amd" && FC="${pkg_install_dir}/bin/flang" || exit 1
+    if [ "${with_flang}" = "yes" ]; then
+        check_command ${pkg_install_dir}/bin/flang "amd" && FC="${pkg_install_dir}/bin/flang" || exit 1
+    else
+        check_command gfortran "gcc" && FC="$(command -v gfortran)" || exit 1
+    fi
     F90="${FC}"
     F77="${FC}"
     AMD_CFLAGS="-I'${pkg_install_dir}/include'"
@@ -64,8 +70,11 @@ if [ "${with_amd}" != "__DONTUSE__" ]; then
   echo "CXX is ${CXX}"
   [ $(realpath $(command -v clang++) | grep -e aocc-compiler) ] || echo "Check the AMD C++ compiler path"
   echo "FC  is ${FC}"
-  #[ $(realpath $(command -v flang) | grep -e aocc-compiler) ] || echo "Check the AMD Fortran compiler path"
-  [ $(realpath $(command -v gfortran) | grep -e aocc-compiler) ] || echo "Check the AMD Fortran compiler path"
+  if [ "${with_flang}" = "yes" ]; then
+    [ $(realpath $(command -v flang) | grep -e aocc-compiler) ] || echo "Check the AMD Fortran compiler path"
+  else
+    [ $(realpath $(command -v gfortran) | grep -e aocc-compiler) ] || echo "Check the GNU Fortran compiler path"
+  fi
   cat << EOF > "${BUILDDIR}/setup_amd"
 export CC="${CC}"
 export CXX="${CXX}"