diff --git a/toolchain/Details.md b/toolchain/Details.md deleted file mode 100644 index 202a492350..0000000000 --- a/toolchain/Details.md +++ /dev/null @@ -1,315 +0,0 @@ -# The ABACUS Toolchain - -## Author -QuantumMisaka (Zhaoqing Liu) @PKU @AISI - -Inspired by cp2k-toolchain, still in improvement. - -## Options - -To use the ABACUS toolchain installer, you may want to first follow -the instructions given in installer help message: - -```shell -> ./install_abacus_toolchain.sh --help -``` - -## Basic usage - -If you are new to ABACUS, and want a basic ABACUS binary, then just calling - -```shell -> ./toolchain_gnu.sh -``` -to install ABACUS requirements by gcc-openmpi-openblas toolchain or -```shell -> ./toolchain_intel.sh -``` -to install ABACUS requirements by intel-mpi-mkl toolchain (intel-environments need to be loaded first) -may be enough. -The gcc-toolchain will use your system gcc, and mpi library (if -existing) and build scalapack, fftw, openblas, elpa and libxc from scratch, -and give you compiled libraries that allow you to compile ABACUS. -The intel-toolchain will use your system intel-mpi and mkl library environments -to build elpa and libxc from scratch, and give you compiled libraries that allow you to compile ABACUS. - -After finish the toolchain requirements installation, you can use the following command to compile ABACUS: - -```shell -> ./build_abacus_gnu.sh -``` -or -```shell -> ./build_abacus_intel.sh -``` -to get ABACUS binary file. - - -## Complete toolchain build - -For a complete toolchain build, with everything installed from -scratch, use: - -```shell -> ./install_abacus_toolchain.sh --install-all -``` - -### Package settings - -One can then change settings for some packages, by setting -`--with-PKG` options after the `--install-all` option. e.g.: - -```shell -> ./install_abacus_toolchain.sh --install-all --with-mkl=system -``` - -will set the script to look for a system MKL library to link, while -compile other packages from scratch. - - -### MPI implementation choice - -If you do not have a MPI installation, by default the `--install-all` -option will install MPICH for you. You can change this default -behavior by setting `--mpi-mode` after the `--install-all` option. - -## Trouble Shooting - -Below are solutions to some of the common problems you may encounter when -running this script. - -### The script terminated with an error message - -Look at the error message. If it does not indicate the reason for -failure then it is likely that some error occurred during -compilation of the package. You can look at the compiler log in -the file make.log in the source directory of the package in -`./build`. - -One of the causes on some systems may be the fact that too many -parallel make processes were initiated. By default the script -tries to use all of the processors on you node. You can override -this behavior using `-j` option. - -### The script failed at a tarball downloading stage - -Try run again with `--no-check-certificate` option. See the help -section for this option for details. - -### I've used --with-XYZ=system cannot find the XYZ library - -The installation script in "system" mode will try to find a library -in the following system PATHS: `LD_LIBRARY_PATH`, `LD_RUN_PATH`, -`LIBRARY_PATH`, `/usr/local/lib64`, `/usr/local/lib`, `/usr/lib64`, -`/usr/lib`. - -For MKL libraries, the installation script will try to look for -MKLROOT environment variable. - -You can use: - -```shell -> module show XYZ -``` - -to see exactly what happens when the module XYZ is loaded into your -system. Sometimes a module will define its own PATHS and -environment variables that is not in the default installation -script search path. And as a result the given library will likely -not be found. - -The simplest solution is perhaps to find where the root -installation directory of the library or package is, and then use -`--with-XYZ=/some/location/to/XYZ` to tell the script exactly where -to look for the library. - -## Licenses - -The toolchain only downloads and installs packages that are -[compatible with the GPL](https://www.gnu.org/licenses/gpl-faq.html#WhatDoesCompatMean). -The following table list the licenses of all those packages. While the toolchain -does support linking proprietary software packages, like e.g. MKL, these have to -be installed separately by the user. - -| Package | License | GPL Compatible | -| --------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -| cmake | [BSD 3-Clause](https://gitlab.kitware.com/cmake/cmake/raw/master/Copyright.txt) | Yes | -| elpa | [LGPL](https://gitlab.mpcdf.mpg.de/elpa/elpa/blob/master/LICENSE) | Yes | -| fftw | [GPL](http://www.fftw.org/doc/License-and-Copyright.html) | Yes | -| gcc | [GPL](https://gcc.gnu.org/git/?p=gcc.git;a=blob_plain;f=COPYING;hb=HEAD) | Yes | -| libxc | [MPL](https://gitlab.com/libxc/libxc/blob/master/COPYING) | Yes | -| mpich | [MPICH](https://github.com/pmodels/mpich/blob/master/COPYRIGHT) | [Yes](https://enterprise.dejacode.com/licenses/public/mpich/#license-conditions) | -| openblas | [BSD 3-Clause](https://github.com/xianyi/OpenBLAS/blob/develop/LICENSE) | Yes | -| openmpi | [BSD 3-Clause](https://github.com/open-mpi/ompi/blob/master/LICENSE) | Yes | -| scalapack | [BSD 3-Clause](http://www.netlib.org/scalapack/LICENSE) | Yes | - -## For Developers - -### Structure of the toolchain scripts - -- `install_abacus_toolchain.sh` is the main script that will call all - other scripts. It contains default flag settings, user input - parser, calls to each package installation scripts and the - generator of the ABACUS arch files. - -- `script/install_*.sh` are the installation scripts for individual - packages. They are relatively independent, in the sense that by - running `script/install_PKG.sh` it should install the package on its - own. However, in practice due to dependencies to other libraries, - sometimes for a package to be installed this way, it will depend - on other libraries being already installed and the correct - environment variables set. At the end of each script, it should - write to __two__ files: `build/setup_PKG` and `install/setup`. - - - The `build/setup_PKG` file contains all the instructions to set - the variables used by the `install_abacus_toolchain.sh` and other - `script/install_PKG.sh` scripts in order for them to correctly - compile the toolchain and set the correct library flags for the - arch files. - - The `install/setup` file contains all the instructions for setting - up the correct environment before the user can compile and/or - run ABACUS. - -- `script/toolkit.sh` contains all the macros that may be used by all - of the scripts, and provides functionalities such as prepending a - path, checking if a library exists etc. - -- `script/common_var.sh` contains all of the common variables used by - each installation scripts. All of the variables in the file should - have a default value, but allow the environment to set the values, - using: `VAR=${VAR:-default_value}`. - -- `script/parse_if.py` is a python code for parsing the `IF_XYZ(A|B)` - constructs in the script. Nested structures will be parsed - correctly. See - [`IF_XYZ` constructs](./README_FOR_DEVELOPERS.md#the-if_xyz-constructs) below. - -- `checksums.sha256` contains the pre-calculated SHA256 checksums for - the tar balls of all of the packages. This is used by the - `download_pkg` macro in `script/toolkit.sh`. - -- `arch_base.tmpl` contains the template skeleton structure for the - arch files. The `install_ABACUS_toolchain` script will set all the - variables used in the template file, and then do an eval to expand - all of `${VARIABLE}` items in `arch_base.tmpl` to give the ABACUS arch - files. - -### `enable-FEATURE` options - -The `enable-FEATURE` options control whether a FEATURE is enabled or disabled. -Possible values are: - -- `yes` (equivalent to using the option-keyword alone) -- `no` - -### `with_PKG` and `PKG_MODE` variables - -The `with_PKG` options controls how a package is going to be installed: - -- either compiled and installed from source downloaded - (`install`, or the option-keyword alone), -- or linked to locations provided by system search paths (`system`), -- or linked to locations provided by the user (``, path to some directory), -- or that the installer won't be used (`no`). - -For most packages the `with_pkg` variables will act like a switch for -turning on or off the support for this package. However, for -packages serving the same purpose, with the installer needing only -one, an extra variable `PKG_MODE` (e.g. `MPI_MODE`) are used as a -selector. In this case, while `with_PKG` controls the installation -method, the `PKG_MODE` variable picks which package to actually use. -This provides more flexibility. - -### The IF_XYZ constructs - -Due to the fact that `install_ABACUS_toolchain.sh` needs to produce -several different versions of the arch files: `psmp`, `pdbg`, -`ssmp`, `sdbg`, etc, it will have to resolve different flags for -different arch file versions. - -The solution used by this script is to use a syntax construct: - -```shell -IF_XYZ(A | B) -``` - -A parser will then parse this expression to *A* if *XYZ* is passed -to the parser (python `parse_if.py` filename XYZ); and to *B* if *XYZ* -is not passed as command line option (python `parse_if.py` filename). - -The `IF_XYZ(A|B)` construct can be nested, so things like: - -```shell -IF_XYZ(IF_ABC(flag1|flag2) | flag3) -``` - -will parse to *flag1* if both *XYZ* and *ABC* are present in the command -line arguments of `parser_if.py`, to *flag2* if only *XYZ* is present, -and *flag3* if nothing is present. - -### To ensure portability - -- one should always pass compiler flags through the - `allowed_gcc_flags` and `allowed_gfortran_flags` filters in - `scripts/toolkit.sh` to omit any flags that are not supported by - the gcc version used (or installed by this script). - -- note that `allowed_gcc_flags` and `allowed_gfortran_flags` do not work - with `IF_XYZ` constructs. So if you have something like: - -```shell -FCFLAGS="IF_XYZ(flag1 flag2 | flag3 flag4)" -``` - -Then you should break this into: - -```shell -XYZ_TRUE_FLAGS="flags1 flags2" -XYZ_FALSE_FLAGS="flags3 flags4" -# do filtering -XYZ_TRUE_FLAGS="$(allowed_gcc_flags $XYZ_TRUE_FLAGS)" -XYZ_FALSE_FLAGS="$(allowed_gcc_flags $XYZ_FALSE_FLAGS)" -``` - -So that: - -```shell -FCFLAGS="IF_XYZ($XYZ_TRUE_FLAGS | $XYZ_FALSE_FLAGS)" -``` - -- For any intrinsic fortran modules that may be used, it is best to - check with `check_gfortran_module` macro defined in - `script/tool_kit.sh`. Depending on the gcc version, some intrinsic - modules may not exist. - -- Try to avoid as much hard coding as possible: - e.g. instead of setting: - -```shell -./configure --prefix=some_dir CC=mpicc FC=mpif90 -``` - -use the common variables: - -```shell -./configure --prefix=some_dir CC=${MPICC} FC=${MPIFC} -``` - -## To keep maintainability it is recommended that we follow these practices - -- Reuse as much functionality from the macros defined in the - `script/toolkit.sh` as possible - -- When the existing macros in `script/toolkit.sh` do not provide the - functionalities you want, it is better to write the new - functionality as a macro in `script/toolkit.sh`, and then use the - macro (repeatedly if required) in the actual installation - script. This keeps the installation scripts uncluttered and more - readable. - -- All packages should install into their own directories, and with a - lock file created in their respective directory to indicate - installation has been successful. This allows the script to skip - over the compilation stages of already installed packages if the - user terminated the toolchain script at the middle of a run and - then restarted the script. diff --git a/toolchain/README.md b/toolchain/README.md index feae610a78..48cfe7d1be 100644 --- a/toolchain/README.md +++ b/toolchain/README.md @@ -1,310 +1,330 @@ -# The ABACUS Toolchain +# ABACUS Toolchain -Version 2025.2 +[![Version](https://img.shields.io/badge/version-2025.3-blue.svg)](https://github.com/deepmodeling/abacus-develop/tree/develop/toolchain) +[![License](https://img.shields.io/badge/license-GPL--compatible-green.svg)](#license) +[![Platform](https://img.shields.io/badge/platform-Linux-lightgrey.svg)]() -## Main Developer +> **Automated dependency management and compilation toolchain for ABACUS** -[QuantumMisaka](https://github.com/QuantumMisaka) -(Zhaoqing Liu) @PKU @AISI +## Table of Contents -Inspired by cp2k-toolchain, still in improvement. +- [Overview](#overview) +- [Features](#features) +- [Quick Start](#quick-start) +- [Installation Methods](#installation-methods) +- [Supported Toolchains](#supported-toolchains) +- [Dependencies](#dependencies) +- [GPU Support](#gpu-support) +- [Troubleshooting](#troubleshooting) +- [Advanced Usage](#advanced-usage) +- [Developer Guide](#developer-guide) +- [License](#license) +- [Contributing](#contributing) -You should have read this README before using this toolchain. +## Overview -## Introduction +The ABACUS Toolchain is an automated build system inspired by the cp2k-toolchain that simplifies the compilation and installation of ABACUS and its dependencies. It supports both online and offline installation modes, multiple compiler toolchains, and provides a streamlined path from dependency installation to ABACUS compilation. -This toolchain will help you easily compile and install, -or link libraries ABACUS depends on -in ONLINE or OFFLINE way, -and give setup files that you can use to compile ABACUS. +### Main Developer -## Todo +**[QuantumMisaka](https://github.com/QuantumMisaka)** (Zhaoqing Liu) +*Peking University, CCME* -- [x] `gnu-openblas` toolchain support for `openmpi` and `mpich`. -- [x] `intel-mkl-mpi` toolchain support using `icc`/`icpc`/`ifort` or `icx`/`icpx`/`ifort`. (`icx` as default, but will have problem for ELPA in AMD machine, one can specify `--with-intel-classic=yes` to use `icc`), -- [x] `intel-mkl-mpich` toolchain support. -- [x] Automatic installation of [CEREAL](https://github.com/USCiLab/cereal) and [LIBNPY](https://github.com/llohse/libnpy) (by github.com) -- [x] Support for [LibRI](https://github.com/abacusmodeling/LibRI) by submodule or automatic installation from github.com (but installed LibRI via `wget` seems to have some problem, please be cautious) -- [x] A mirror station by Bohrium database, which can download CEREAL, LibNPY, LibRI and LibComm by `wget` in China Internet. -- [x] Support for GPU-PW and GPU-LCAO compilation (elpa, cusolvermp is developing), and `-DUSE_CUDA=1` is needed builder scripts. -- [x] Support for AMD compiler and math lib `AOCL` and `AOCC` (not fully complete due to flang and AOCC-ABACUS compliation error) -- [ ] Support for more GPU device out of Nvidia. -- [ ] Change the downloading url from cp2k mirror to other mirror or directly downloading from official website. (doing) -- [ ] Support a JSON or YAML configuration file for toolchain, which can be easily modified by users. -- [ ] A better README and Detail markdown file. -- [ ] Automatic installation of [DEEPMD](https://github.com/deepmodeling/deepmd-kit). -- [ ] Modulefile generation scripts. +## Features +- ✅ **Multiple Toolchain Support**: GNU, Intel OneAPI, AMD AOCC/AOCL +- ✅ **Flexible Installation**: Online/offline modes with automatic dependency resolution +- ✅ **GPU Acceleration**: CUDA support for NVIDIA GPUs with ELPA and cuSolverMP +- ✅ **MPI Implementations**: OpenMPI, MPICH, Intel MPI support +- ✅ **Math Libraries**: OpenBLAS, Intel MKL, AMD AOCL integration +- ✅ **Advanced Features**: LibRI, LibComm and MLALGO support +- ✅ **Resumable Installation**: Interrupt and resume capability +- ✅ **Environment Management**: Automatic setup file generation -## Usage Online & Offline +## Quick Start -Main script is *install_abacus_toolchain.sh*, -which will use scripts in *scripts* directory -to compile install dependencies of ABACUS. -It can be directly used, but not recommended. +### Prerequisites -There are also well-modified script to run *install_abacus_toolchain.sh* for `gnu` (gcc-openblas), `intel` (intel-mkl-mpi-compiler), `gcc-aocl` and `aocc-aocl` toolchains dependencies. +- **GCC**: Version ≥ 5.0 (recommended ≥ 7.3.0) +- **Internet Connection**: For online installation mode +- **System Libraries**: Basic development tools (see [System Requirements](#system-requirements)) -```shell -# for gnu-openblas -> ./toolchain_gnu.sh -# for intel-mkl -> ./toolchain_intel.sh -# for AMD gcc-aocl -> ./toolchain_gcc-aocl.sh -# for AMD aocc-aocl -> ./toolchain_aocc-aocl.sh -``` +### Basic Installation -It is recommended to run one of them first to get a fast installation of ABACUS under certain environments. +For new users, start with one of these pre-configured toolchains: -If you are using Intel environments via Intel-OneAPI: please note: -1. After version 2024.0, Intel classic compilers `icc` and `icpc` are not present, so as `ifort` after version 2025.0. Intel MPI compiler will also be updated to `mpiicx`, `mpiicpx` and `mpiifx`. -2. toolchain will detect `icx`, `icpx`, `ifx`, `mpiicx`, `mpiicpx` and `mpiifx` as default compiler. -3. Users can manually specify `--with-intel-classic=yes` to use Intel classic compiler in `toolchain*.sh`, or specify `--with-intel-mpi-clas=yes` to use Intel MPI classic compiler in `toolchain*.sh` while keep the CC, CXX and F90 compiler to new version. -4. Users can manually specify `--with-ifx=no` in `toolchain*.sh` to use `ifort` while keep other compiler to new version. -5. More information is in the later part of this README. +```bash +# GNU toolchain (GCC + OpenMPI + OpenBLAS) +./toolchain_gnu.sh -If you are using AMD AOCL and AOCC, please note: +# Intel toolchain (Intel compilers + Intel MPI + MKL) +./toolchain_intel.sh +# AMD toolchain options +./toolchain_gcc-aocl.sh # GCC + AMD AOCL +./toolchain_aocc-aocl.sh # AMD AOCC + AOCL +``` -**Notice: You GCC version should be no lower than 5 !!!. The toolchain will check it, and gcc with version larger than 7.3.0 is recommended.** +### Build ABACUS -**Notice: You SHOULD `source` or `module load` related environments before use toolchain method for installation, especially for `intel`, `gcc-aocl` or `aocc-aocl` toolchain! For example, `module load mkl mpi icc compiler` for loading oneapi envs.** +After successful toolchain installation: -**Notice: You SHOULD keep your environments systematic, for example, you CANNOT load `intel-OneAPI` environments while use gcc toolchain !!!** +```bash +# For GNU toolchain +./build_abacus_gnu.sh -**Notice: If your server system already have libraries like `cmake`, `openmpi`, please change related setting in `toolchain*.sh` like `--with-cmake=system`, note that the environments of these system package will not be added into install/setup file** +# For Intel toolchain +./build_abacus_intel.sh +# For AMD toolchains +./build_abacus_gcc-aocl.sh +./build_abacus_aocc-aocl.sh +``` -All packages will be downloaded from [cp2k-static/download](https://www.cp2k.org/static/downloads). by `wget` , and will be detailedly compiled and installed in `install` directory by toolchain scripts, despite of: +### Environment Setup -- `CEREAL` which will be downloaded from [CEREAL](https://github.com/USCiLab/cereal) -- `Libnpy` which will be downloaded from [LIBNPY](https://github.com/llohse/libnpy) -- `LibRI` which will be downloaded from [LibRI](https://github.com/abacusmodeling/LibRI) -- `LibCOMM` which will be downloaded from [LibComm](https://github.com/abacusmodeling/LibComm) -- `RapidJSON` which will be downloaded from [RapidJSON](https://github.com/Tencent/rapidjson) -Notice: These packages will be downloaded by `wget` from `codeload.github.com`, which bypass the difficulty of Chinese Internet in some extent. If any downloading problem occurs, you may need to use offline installation method. +```bash +# Source the generated environment +source install/setup -Instead of github.com, we offer other package station, you can use it by: -```shell -wget https://bohrium-api.dp.tech/ds-dl/abacus-deps-93wi-v3 -O abacus-deps-v3.zip +# Or use the generated ABACUS environment +source abacus_env.sh ``` -`unzip` it ,and you can do offline installation of these packages above after rename. -```shell -# packages downloaded from github.com -mv v1.3.2.tar.gz build/cereal-1.3.2.tar.gz -``` -The above station will be updated handly but one should notice that the version will always lower than github repo. -If one want to install ABACUS by toolchain OFFLINE, -one can manually download all the packages from [cp2k-static/download](https://www.cp2k.org/static/downloads) or official website -and put them in *build* directory by formatted name -like *fftw-3.3.10.tar.gz*, or *openmpi-5.0.7.tar.bz2*, -then run this toolchain. -All package will be detected and installed automatically. -Also, one can install parts of packages OFFLINE and parts of packages ONLINE -just by using this toolchain +## Installation Methods -```shell -# for OFFLINE installation -# in toolchain directory -> mkdir build -> cp ***.tar.gz build/ -``` - -The needed dependencies version default: - -- `cmake` 3.31.7 -- `gcc` 13.2.0 (which will always NOT be installed, But use system) -- `OpenMPI` 5.0.7 (Version 5 OpenMPI is good but will have compability problem, user can manually downarade to Version 4 in toolchain scripts by specify `--with-openmpi4`) -- `MPICH` 4.3.0 -- `OpenBLAS` 0.3.29 (Intel toolchain need `get_vars.sh` tool from it) -- `ScaLAPACK` 2.2.2 -- `FFTW` 3.3.10 -- `LibXC` 7.0.0 -- `ELPA` 2025.01.001 (may not be conpatiable for gpu-ver) -- `CEREAL` master (for oneapi compatibility) -- `RapidJSON` master (for oneapi compatibility) -And: -- Intel-oneAPI need user or server manager to manually install from Intel. -- - [Intel-oneAPI](https://www.intel.cn/content/www/cn/zh/developer/tools/oneapi/toolkits.html) -- AMD AOCC-AOCL need user or server manager to manually install from AMD. -- - [AOCC](https://www.amd.com/zh-cn/developer/aocc.html) -- - [AOCL](https://www.amd.com/zh-cn/developer/aocl.html) - -Dependencies below are optional， which is NOT installed by default: -- `LibTorch` 2.1.2 -- `Libnpy` 1.0.1 -- `LibRI` 0.2.1.0 -- `LibComm` master (for openmpi compatibility) - -Users can install them by using `--with-*=install` in toolchain*.sh, which is `no` in default. Also, user can specify the absolute path of the package by `--with-*=path/to/package` in toolchain*.sh to allow toolchain to use the package. -> Notice: LibTorch always suffer from GLIBC_VERSION problem, if you encounter this, please downgrade LibTorch version to 1.12.1 in scripts/stage4/install_torch.sh -> -> Notice: LibRI, LibComm, Rapidjson and Libnpy is on actively development, you should check-out the package version when using this toolchain. - -Users can easily compile and install dependencies of ABACUS -by running these scripts after loading related environment. - -The toolchain installation process can be interrupted at anytime. -just re-run *toolchain_\*.sh*, toolchain itself may fix it. If you encouter some problem like file corrupted, you can always remove some package in the interrupted points and re-run the toolchain. - -Some useful options: -- `--dry-run`: just run the main install scripts for environment setting, without any package downloading or installation. -- `--pack-run`: just run the install scripts without any package building, which helps user to download and check the packages, paticularly for offline installation to a server. - -If compliation is successful, a message will be shown like this: +### Online Installation -```shell -========================== usage ========================= -Done! -To use the installed tools and libraries and ABACUS version -compiled with it you will first need to execute at the prompt: - source ${SETUPFILE} -To build ABACUS by gnu-toolchain, just use: - ./build_abacus_gnu.sh -To build ABACUS by intel-toolchain, just use: - ./build_abacus_intel.sh -To build ABACUS by amd-toolchain in gcc-aocl, just use: - ./build_abacus_gnu-aocl.sh -To build ABACUS by amd-toolchain in aocc-aocl, just use: - ./build_abacus_aocc-aocl.sh -or you can modify the builder scripts to suit your needs. -``` - -You can run *build_abacus_gnu.sh* or *build_abacus_intel.sh* to build ABACUS -by gnu-toolchain or intel-toolchain respectively, same for the `gcc-aocl` and `aocc-aocl` toolchain. -Then, the builder scripts will automatically locate the environment and compile ABACUS. -You can manually change the builder scripts to suit your needs. -The builder scripts will generate `abacus_env.sh` for source - -Then, after `source abacus_env.sh`, one can easily -run builder scripts to build ABACUS binary software. - -If users want to use toolchain but lack of some system library -dependencies, *install_requirements.sh* scripts will help. - -If users want to re-install all the package, just do: +Downloads packages automatically from official sources: -```shell -> rm -rf install +```bash +./toolchain_gnu.sh # Uses system package managers and official repositories ``` -or you can also do it in a more completely way: +Before running the toolchain, please make sure you have loaded the related environments and set the environment variables. +- You SHOULD source or module load related environments before use toolchain method for installation, especially for *intel*, *gcc-aocl* or *aocc-aocl* toolchain! For example, `module load mkl mpi icc compiler` for loading intel-oneapi envs. +- You SHOULD keep your environments systematic, for example, you CANNOT load intel-OneAPI environments while use gcc toolchain !!! + +**Package Sources:** +- **Build Tools:** + - [GCC](https://mirrors.tuna.tsinghua.edu.cn/gnu/gcc/) - GNU Compiler Collection + - [CMake](https://cmake.org/download/) - Cross-platform build system +- **MPI Libraries:** + - [OpenMPI](https://download.open-mpi.org/release/open-mpi/) - Open source MPI implementation + - [MPICH](https://www.mpich.org/downloads/) - High-performance MPI implementation +- **Math Libraries:** + - [OpenBLAS](https://github.com/xianyi/OpenBLAS/releases) - Optimized BLAS library + - [ScaLAPACK](http://www.netlib.org/scalapack/) - Scalable Linear Algebra PACKage +- **Scientific Libraries:** + - [FFTW](http://www.fftw.org/) - Fast Fourier Transform library + - [LibXC](https://www.tddft.org/programs/libxc/) - Exchange-correlation functionals + - [ELPA](https://elpa.mpcdf.mpg.de/) - Eigenvalue solver +- **Advanced Features:** + - [LibTorch](https://download.pytorch.org/libtorch/cpu/) - PyTorch C++ API + - [LibNPY](https://github.com/llohse/libnpy) - NumPy I/O for C++ + - [LibRI](https://github.com/abacusmodeling/LibRI) - Resolution of Identity library + - [LibComm](https://github.com/abacusmodeling/LibComm) - Communication library + - [NEP](https://github.com/brucefan1983/NEP_CPU) - Neuroevolution Potential + - [Cereal](https://github.com/USCiLab/cereal) - C++ serialization library + - [RapidJSON](https://github.com/Tencent/rapidjson) - Fast JSON parser/generator +- **Reference mirror:** [CP2K static downloads](https://www.cp2k.org/static/downloads) +- All package from GitHub will be downloaded by `wget` from `codeload.github.com`, which bypass the difficulty of CN Internet in some extent. + +### Offline Installation + +For air-gapped systems or unreliable internet: -```shell -> rm -rf install build/*/* build/OpenBLAS*/ build/setup_* +```bash +# 1. Create build directory and download packages +mkdir build +# Download required packages to build/ directory with proper naming +# e.g., fftw-3.3.10.tar.gz, openmpi-5.0.8.tar.bz2 + +# 2. Run toolchain (will detect local packages) +./toolchain_gnu.sh ``` -## GPU version of ABACUS +### Hybrid Installation -Toolchain supports compiling GPU version of ABACUS with Nvidia-GPU and CUDA. For usage, adding following options in build*.sh: +Mix online and offline packages as needed - the toolchain automatically detects locally available packages and downloads missing ones. -```shell -# in build_abacus_gnu.sh -cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ - -DCMAKE_CXX_COMPILER=g++ \ - -DMPI_CXX_COMPILER=mpicxx \ - ...... - -DUSE_CUDA=ON \ - # -DCMAKE_CUDA_COMPILER=${path to cuda toolkit}/bin/nvcc \ # add if needed - ...... -# in build_abacus_intel.sh -cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ - -DCMAKE_CXX_COMPILER=icpc \ - -DMPI_CXX_COMPILER=mpiicpc \ - ...... - -DUSE_CUDA=ON \ - # -DCMAKE_CUDA_COMPILER=${path to cuda toolkit}/bin/nvcc \ # add if needed - ...... -``` -which will enable GPU version of ABACUS, and the `ks_solver cusolver` method can be directly used for PW and LCAO calculation. - -Notice: You CANNOT use `icpx` compiler for GPU version of ABACUS for now, see discussion here [#2906](https://github.com/deepmodeling/abacus-develop/issues/2906) and [#4976](https://github.com/deepmodeling/abacus-develop/issues/4976) - -If you wants to use ABACUS GPU-LCAO by `cusolvermp` or `elpa` for multiple-GPU calculation, please compile according to the following usage: - -1. For the elpa method, add -```shell -export CUDA_PATH=/path/to/CUDA -# install_abacus_toolchain.sh part options ---enable-cuda \ ---gpu-ver=(GPU-compatibility-number) \ +## Supported Toolchains + +### GNU Toolchain +- **Compilers**: System GCC (≥5.0) +- **MPI**: OpenMPI or MPICH +- **Math**: OpenBLAS + ScaLAPACK +- **Features**: Most stable, widely compatible + +### Intel Toolchain +- **Compilers**: Intel OneAPI (icx/icpx/ifx or classic icc/icpc/ifort) +- **MPI**: Intel MPI +- **Math**: Intel MKL +- **Features**: Optimized performance, EXX support + +### AMD Toolchain +- **Compilers**: AMD AOCC or GCC +- **Math**: AMD AOCL (Optimized math libraries) +- **Features**: AMD processor optimization + +## Dependencies + +### Supported Packages + +| Package | Version (main/alt) | Purpose | License | Default | +|---------|-------------------|---------|---------|---------| +| **Build Tools** ||||| +| CMake | 3.31.7 / 3.30.5 | Build system | BSD-3-Clause | Install | +| GCC | 13.2.0 / 11.4.0 | C/C++ compiler | GPL-3.0-or-later WITH GCC-exception-3.1 | Install | +| **MPI Libraries** ||||| +| OpenMPI | 5.0.8 / 4.1.6 | MPI implementation | BSD-3-Clause-Open-MPI | Install | +| MPICH | 4.3.1 / 4.1.0 | Alternative MPI | mpich2 (BSD-like) | Alternative | +| **Math Libraries** ||||| +| OpenBLAS | 0.3.30 / 0.3.27 | Linear algebra | BSD-3-Clause | Install | +| ScaLAPACK | 2.2.2 / 2.2.1 | Parallel linear algebra | BSD-3-Clause | Install | +| **Scientific Libraries** ||||| +| FFTW | 3.3.10 / 3.3.10 | Fast Fourier Transform | GPL-2.0-or-later | Install | +| LibXC | 7.0.0 / 6.2.2 | Exchange-correlation | MPL-2.0 | Install | +| ELPA | 2025.06.001 / 2024.05.001 | Eigenvalue solver | LGPL-3.0-only | Install | +| **Advanced Features** ||||| +| Cereal | master | C++ Serialization | BSD | Install | +| RapidJSON | master | JSON parsing | MIT | Install | +| LibRI | master | EXX calculations | GPL-3.0 | Install | +| LibComm | master | EXX calculations | GPL-3.0 | Install | +| LibTorch | 2.1.2 / 1.12.1 | MLALGO support | BSD-3-Clause | Optional | +| LibNPY | 1.0.1 / 1.0.1 | NumPy I/O | MIT | Optional | +| NEP | main | Neural network potential | MIT | Optional | + +Also, [Intel-oneAPI](https://www.intel.cn/content/www/cn/zh/developer/tools/oneapi/toolkits.html) and AMD [AOCC](https://www.amd.com/zh-cn/developer/aocc.html) and [AOCL](https://www.amd.com/zh-cn/developer/aocl.html) are supported in toolchain by setting them to system option, but one should install them manually by server administrator. + +### Package Version Switching + +The toolchain supports a dual-version system for most packages, providing both **main** (latest stable) and **alt** (alternative/legacy) versions. This allows users to choose between cutting-edge features and proven stability based on their specific requirements. + +#### Version Selection Methods + +The `--package-version` parameter supports two flexible usage patterns: + +**Method 1: Multiple Independent Parameters** (Original support) +```bash +./toolchain_gnu.sh --package-version libtorch:alt --package-version elpa:alt ``` -to the `toolchain_*.sh`, and then follow the normal step to install the dependencies using `./toolchain_*.sh`. For checking the GPU compatibility number, you can refer to the [CUDA compatibility](https://developer.nvidia.com/cuda-gpus). -Afterwards, make sure these option are enable in your `build_abacus_*.sh` script -```shell --DUSE_ELPA=ON \ --DUSE_CUDA=ON \ +**Method 2: Single Parameter with Multiple Key-Value Pairs** (Enhanced functionality) +```bash +./toolchain_gnu.sh --package-version "libtorch:alt elpa:alt" ``` -then just build the abacus executable program by compiling it with `./build_abacus_*.sh`. -The ELPA method need more parameter setting, but it doesn't seem to be affected by the CUDA toolkits version, and it is no need to manually install and package. +One can also manually edit the `toolchain_gnu.sh` for selecting specific version of packages. -Note: ELPA-2025.01.001 may have problem in nvidia-GPU compilation on some V100-GPU with AMD-CPU machine, error message: ```bash - 1872 | static __forceinline void CONCAT_8ARGS(hh_trafo_complex_kernel_,ROW_LENGTH,_,SIMD_SET,_,BLOCK,hv_,WORD_LENGTH) (DATA_TYPE_PTR q, DATA_TYPE_PTR hh, int nb, int ldq - | ^~~~~~~~~~~~~~~~~~~~~~~~ -../src/elpa2/kernels/complex_128bit_256bit_512bit_BLOCK_template.c:51:47: note: in definition of macro 'CONCAT2_8ARGS' - 51 | #define CONCAT2_8ARGS(a, b, c, d, e, f, g, h) a ## b ## c ## d ## e ## f ## g ## h - | ^ -../src/elpa2/kernels/complex_128bit_256bit_512bit_BLOCK_template.c:1872:27: note: in expansion of macro 'CONCAT_8ARGS' - 1872 | static __forceinline void CONCAT_8ARGS(hh_trafo_complex_kernel_,ROW_LENGTH,_,SIMD_SET,_,BLOCK,hv_,WORD_LENGTH) (DATA_TYPE_PTR q, DATA_TYPE_PTR hh, int nb, int ldq - | ^~~~~~~~~~~~ - PPFC src/GPU/libelpa_openmp_private_la-mod_vendor_agnostic_general_layer.lo - PPFC test/shared/GPU/libelpatest_openmp_la-test_gpu_vendor_agnostic_layer.lo -../src/GPU/CUDA/./cudaFunctions_template.h(942): error: identifier "creal" is undefined - double alpha_real = creal(alpha); - ^ - -../src/GPU/CUDA/./cudaFunctions_template.h(960): error: identifier "creal" is undefined - float alpha_real = creal(alpha); -``` - -And you may need to change ELPA version to 2024.05.001, edit `toolchain/scripts/stage3/install_elpa.sh` to do it. - -1. For the cusolvermp method, toolchain_*.sh does not need to be changed, just follow it directly install dependencies using `./toolchain_*.sh`, and then add -```shell --DUSE_CUDA=ON \ --DENABLE_CUSOLVERMP=ON \ --D CAL_CUSOLVERMP_PATH=/path/to/math.libs/1x.x/target/x86_64-linux/lib \ +# ============================================================================ +# Package Version Selection (main/alt versions) +# ============================================================================ +# Choose between main (latest stable) and alt (alternative/legacy) versions +# Refer to scripts/package_versions.sh for specific version numbers + +CMAKE_VERSION="main" # main=3.31.7, alt=3.30.5 +OPENMPI_VERSION="main" # main=5.0.8, alt=4.1.6 +OPENBLAS_VERSION="main" # main=0.3.30, alt=0.3.27 +ELPA_VERSION="main" # main=2025.06.001, alt=2024.05.001 +LIBXC_VERSION="main" # main=7.0.0, alt=6.2.2 +SCALAPACK_VERSION="main" # main=2.2.2, alt=2.2.1 +# Optional Libraries +LIBTORCH_VERSION="main" # main=2.1.2, alt=1.12.1 (use alt for older GLIBC) ``` -to the `build.abacus_*.sh` file. add the following three items to the environment (assuming you are using hpcsdk): -```shell -export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/comm_libs/1x.x/hpcx/hpcx-x.xx/ucc/lib -export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/comm_libs/1x.x/hpcx/hpcx-x.xx/ucx/lib -export CPATH=$CPATH:/path/to/math_libs/1x.x/targets/x86_64-linux/include + +and other `toolchain_*.sh` scripts share the same version selection. + +#### Global Version Strategy + +- **Default Behavior**: All packages use their **main** versions unless explicitly overridden +- **Selective Override**: Use `--package-version` to specify alternative versions for specific packages +- **Consistency**: Version selections are validated against available options in `scripts/package_versions.sh` +- **Backward Compatibility**: Both usage methods are fully supported to ensure existing scripts continue to work + + +### System Requirements + +Install system dependencies using provided scripts: + +```bash +# Ubuntu/Debian +sudo ./root_requirements/install_requirements_ubuntu.sh + +# Fedora/RHEL/CentOS +sudo ./root_requirements/install_requirements_fedora.sh + +# Generic +sudo ./root_requirements/install_requirements.sh ``` -Just enough to build the abacus executable program by compiling it with `./build_abacus_*.sh`. -You can refer to the linking video for auxiliary compilation and installation. [Bilibili](https://www.bilibili.com/video/BV1eqr5YuETN/). +## GPU Support -The cusolverMP requires installation from sources such as apt or yum, which is suitable for containers or local computers. -The second choice is using [NVIDIA HPC_SDK](https://developer.nvidia.com/hpc-sdk-downloads) for installation, which is relatively simple, but the package from NVIDIA HPC_SDK may not be suitable, especially for muitiple-GPU parallel running. To better use cusolvermp and its dependency (libcal, ucx, ucc) in multi-GPU running, please contact your server manager. +### CUDA Support for NVIDIA GPUs -After compiling, you can specify `device GPU` in INPUT file to use GPU version of ABACUS. +#### Basic GPU Support +Add to your build script: -## Common Problems and Solutions +```bash +cmake -B $BUILD_DIR \ + -DUSE_CUDA=ON \ + -DCMAKE_CUDA_COMPILER=/path/to/cuda/bin/nvcc \ + # ... other options +``` -### Intel-oneAPI problem +#### Multi-GPU with ELPA -#### OneAPI 2025.0 problem +1. **Configure toolchain with CUDA:** +```bash +export CUDA_PATH=/path/to/CUDA +./toolchain_gnu.sh --enable-cuda --gpu-ver=70 # For V100 (compute capability 7.0) +``` -Generally, OneAPI 2025.0 can be useful to compile basic function of ABACUS, but one will encounter compatible problem related to something. -- related to LibRI: refer to [#6190](https://github.com/deepmodeling/abacus-develop/issues/6190), it is recommended not to use LibRI or downgrade your OneAPI now. +2. **Build with ELPA GPU support:** +```bash +cmake -B $BUILD_DIR \ + -DUSE_CUDA=ON \ + -DUSE_ELPA=ON \ + # ... other options +``` + +#### Multi-GPU with cuSolverMP -#### ELPA problem via Intel-oneAPI toolchain in AMD server +1. **Check or install cuSolverMP manually:** +One may use NVIDIA HPC_SDK as an easy way to install cuSolverMP. + +2. **Install dependencies normally:** +```bash +./toolchain_gnu.sh +``` -The default compiler for Intel-oneAPI is `icpx` and `icx`, which will cause problem when compling ELPA in AMD server. (Which is a problem and needed to have more check-out) +3. **Build with cuSolverMP:** +```bash +cmake -B $BUILD_DIR \ + -DUSE_CUDA=ON \ + -DENABLE_CUSOLVERMP=ON \ + -DCAL_CUSOLVERMP_PATH=/path/to/math_libs/lib \ + # ... other options +``` -The best way is to change `icpx` to `icpc`, `icx` to `icc`. user can manually change it in *toolchain_intel.sh* via `--with-intel-classic=yes` +3. **Set environment variables:** +```bash +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/hpcx/ucc/lib +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/hpcx/ucx/lib +export CPATH=$CPATH:/path/to/math_libs/include +``` -Notice: `icc` and `icpc` from Intel Classic Compiler of Intel-oneAPI is not supported for 2024.0 and newer version. And Intel-OneAPI 2023.2.0 can be found in QE website. You need to download Base-toolkit for MKL and HPC-toolkit for MPi and compiler for Intel-OneAPI 2023.2.0, while in Intel-OneAPI 2024.x, only the HPC-toolkit is needed. +**Note**: cuSolverMP requires NVIDIA HPC SDK or system installation via package manager. -You can get Intel-OneAPI in [QE-managed website](https://pranabdas.github.io/espresso/setup/hpc/#installing-intel-oneapi-libraries), and use this code to get Intel oneAPI Base Toolkit and HPC Toolkit: +## Troubleshooting + +### Common Issues + +#### Intel OneAPI Problems + +Most of the OneAPI problem arise from the newer version of Intel-OneAPI. For users who encounter OneAPI problem, one can get Intel-OneAPI in [QE-managed website](https://pranabdas.github.io/espresso/setup/hpc/#installing-intel-oneapi-libraries), and use this code to get Intel oneAPI Base Toolkit and HPC Toolkit: ```shell wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397_offline.sh wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/0722521a-34b5-4c41-af3f-d5d14e88248d/l_HPCKit_p_2023.2.0.49440_offline.sh @@ -312,100 +332,388 @@ wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/0722521a-34b5-4 Related discussion here [#4976](https://github.com/deepmodeling/abacus-develop/issues/4976) -#### linking problem in early 2023 version oneAPI +**OneAPI 2025.0 Compatibility:** +- LibRI compatibility issues ([#6190](https://github.com/deepmodeling/abacus-develop/issues/6190)) +- Solution: Using the patch from Cereal and the master version of Cereal to fix the compatibility issue (included in toolchain). -Sometimes Intel-oneAPI have problem to link `mpirun`, -which will always show in 2023.2.0 version of MPI in Intel-oneAPI. -Try `source /path/to/setvars.sh` or install another version of IntelMPI may help. - -which is fixed in 2024.0.0 version of Intel-oneAPI, -And will not occur in Intel-MPI before 2021.10.0 (Intel-oneAPI before 2023.2.0) +**ELPA on AMD servers with Intel compilers:** +```bash +# Use Intel classic compilers instead +./toolchain_intel.sh --with-intel-classic=yes +``` -More problem and possible solution can be accessed via [#2928](https://github.com/deepmodeling/abacus-develop/issues/2928) +Notice: `icc` and `icpc` from Intel Classic Compiler of Intel-oneAPI are not supported for 2024.0 and newer version. And Intel-OneAPI 2023.2.0 can be found in QE website. You need to download Base-toolkit for MKL and HPC-toolkit for MPi and compiler for Intel-OneAPI 2023.2.0, while in Intel-OneAPI 2024.x, only the HPC-toolkit is needed. -#### gcc-MKL problem +#### Gcc-MKL Issues You cannot use gcc as compiler while using MKL as math library for compile ABACUS, there will be lots of error in the lask linking step. See [#3198](https://github.com/deepmodeling/abacus-develop/issues/3198) -### AMD AOCC-AOCL problem +#### AMD AOCC-AOCL problem -Use AOCC-AOCL to compile dependencies is permitted and usually get boosting in ABACUS efficiency. But you need to get rid of `flang` while compiling ELPA. Toolchain itself helps you make this `flang` shade in default of `aocc-aocl` toolchain, and you can manually use `flang` by setting `--with-flang=yes` in `toolchain_aocc-aocl.sh` to have a try, while toolchain helps you to bypass the possible errors in compiling ELPA with AOCC-AOCL, but the computing efficiency will be relatively lower compared to `gnu` or `gcc-aocl` toolchain. +Use AOCC-AOCL to compile dependencies is permitted and usually get boosting in ABACUS efficiency. But you need to get rid of `flang` while compiling ELPA. Toolchain itself helps you make this `flang` shade in default of `aocc-aocl` toolchain, and you can manually use `flang` by setting `--with-flang=yes` in `toolchain_aocc-aocl.sh` to have a try, while toolchain helps you to bypass the possible errors in compiling ELPA with AOCC-AOCL, but the computing efficiency will be relatively lower compared to `gnu` or `gcc-aocl` toolchain. There are some issues related to the numeric instability of ABACUS compiled by AOCC-AOCL toolchain, see [#6420](https://github.com/deepmodeling/abacus-develop/issues/6420) The `gcc-aocl` toolchain will have no problem above for aocc-dependent aocl. However, the gcc-dependent aocl will have some package linking problem related to OpenMPI. Take it with caution. +#### OpenMPI Issues -### OpenMPI problem +**Version 5 compatibility problems:** +```bash +# Use OpenMPI v4 instead +./toolchain_gnu.sh --package-version openmpi:alt +# an deprecated option, but still works +./toolchain_gnu.sh --with-openmpi-4th=yes +``` -#### in EXX and LibRI +**LibComm compilation with OpenMPI:** +- Fixed in toolchain 2025.2 which downlo the master branch of LibComm +- Alternative: Use MPICH or Intel MPI -- [Fixed in Toolchain 2025-02] GCC toolchain with OpenMPI cannot compile LibComm v0.1.1 due to the different MPI variable type from MPICH and IntelMPI, see discussion here [#5033](https://github.com/deepmodeling/abacus-develop/issues/5033), you can try use a newest branch of LibComm by +#### Shell and Permission Issues + +**Line ending problems:** +```bash +./pre_set.sh # Fixes line endings and permissions +# Or manually: +dos2unix *.sh +chmod +x *.sh ``` -git clone https://gitee.com/abacus_dft/LibComm -b MPI_Type_Contiguous_Pool -``` -or pull the newest master branch of LibComm + +#### Library Version Issues + +**LibTorch GLIBC errors:** +- Requires GLIBCXX_3.4.26 +- Change version from 2.1.2 to 1.12.1 +- Use `--package-version libtorch:alt` when calling toolchain +- Can combine with other packages using two writing styles: + - Multiple independent parameters: `--package-version libtorch:alt --package-version elpa:alt` + - Single parameter with multiple key-value pairs: `--package-version libtorch:alt elpa:alt` + +**DeepMD GLIBC errors:** +- Requires GCC ≥ 11.3.1 for GLIBCXX_3.4.29 +- Upgrade system GCC or use newer toolchain + +### Getting Help + +1. **Check logs**: Look in `build/PKG_NAME/make.log` for compilation errors +2. **Reduce parallelism**: Use `NPROCS_OVERWRITE=N` environment variable to limit parallel processes +3. **System libraries**: Use `--with-PKG=system` for system-installed packages +4. **Clean installation**: Remove `install/` and `build/` directories to restart +5. **Certificate issues**: Use `DOWNLOAD_CERT_POLICY=skip` for download problems + +## Advanced Usage + +### Package-Specific Options + +```bash +# Use Intel MKL instead of installing OpenBLAS +./toolchain_gnu.sh --with-mkl=system + +# Use system FFTW instead of installing +./toolchain_gnu.sh --with-fftw=system + +# Specify custom package installation path +./toolchain_gnu.sh --with-fftw=/path/to/custom/fftw ``` -git clone https://github.com/abacusmodeling/LibComm + +### Execution Mode Control + +```bash +# Test configuration without actual installation (recommended for first run) +./toolchain_gnu.sh --dry-run + +# Only download packages without building (useful for offline preparation) +./toolchain_gnu.sh --pack-run ``` -. yet another is switching to GCC-MPICH or Intel toolchain -- It is recommended to use Intel toolchain if one wants to include EXX feature in ABACUS, which can have much better performance and can use more than 16 threads in OpenMP parallelization to accelerate the EXX process. -#### OpenMPI-v5 +### Environment Variable Configuration + +The toolchain supports several environment variables for advanced configuration: + +#### Download Certificate Verification + +Control SSL/TLS certificate verification during package downloads: + +```bash +# Strict mode: Always verify certificates (secure) +export DOWNLOAD_CERT_POLICY=strict +./toolchain_gnu.sh + +# Smart mode: Try secure first, fallback if needed (default) +export DOWNLOAD_CERT_POLICY=smart # or leave unset +./toolchain_gnu.sh + +# Skip mode: Skip certificate verification (legacy compatibility) +export DOWNLOAD_CERT_POLICY=skip +./toolchain_gnu.sh +``` + +**Smart Mode Behavior**: The default `smart` mode first attempts secure downloads with certificate verification. If this fails (e.g., due to corporate firewalls or outdated certificates), it automatically falls back to skipping certificate verification while providing clear user feedback. + +#### Parallel Compilation Control + +Override the automatic CPU core detection for compilation: + +```bash +# Use 8 cores for compilation (useful for resource-limited systems) +export NPROCS_OVERWRITE=8 +./toolchain_gnu.sh + +# Use single core for debugging compilation issues +export NPROCS_OVERWRITE=1 +./toolchain_gnu.sh + +# Or specify inline +NPROCS_OVERWRITE=4 ./toolchain_gnu.sh --with-gcc --with-openmpi +``` + +**Use Cases**: +- **Resource-limited systems**: Reduce parallelism to avoid memory exhaustion +- **Shared servers**: Limit resource usage to be considerate of other users +- **CI/CD environments**: Match container resource limits +- **Debugging**: Use single-core compilation for clearer error messages + +### Legacy Script Options + +The deprecated `install_abacus_toolchain.sh` supports additional options: + +| Option | Description | Availability | +|--------|-------------|--------------| +| `--dry-run` | Test configuration without installation | ✅ New & Legacy | +| `--pack-run` | Download packages without building | ✅ New & Legacy | +| `--no-check-certificate` | Skip SSL certificate verification | ⚠️ Legacy only (use `DOWNLOAD_CERT_POLICY=skip`) | +| `-j N` | Limit parallel compilation processes | ⚠️ Legacy only (use `NPROCS_OVERWRITE=N`) | + +> **Migration Note**: The new toolchain system (`toolchain_*.sh` scripts) is recommended over the legacy `install_abacus_toolchain.sh`. Legacy options like `--no-check-certificate` and `-j N` are replaced by environment variables `DOWNLOAD_CERT_POLICY` and `NPROCS_OVERWRITE` respectively. + +### Environment Management + +The toolchain generates several setup files: + +- `install/setup`: Main environment setup +- `build/setup_PKG`: Individual package environments +- `abacus_env.sh`: ABACUS-specific environment (generated by build scripts) + +## Developer Guide + +### Toolchain Architecture + +The toolchain follows a modular design with staged dependency installation: + +``` +scripts/ +├── stage0/ # Compilers and build tools +├── stage1/ # MPI implementations +├── stage2/ # Math libraries (BLAS, LAPACK) +├── stage3/ # Scientific libraries (FFTW, LibXC, ELPA) +├── stage4/ # Advanced features (LibTorch, LibRI) +└── lib/ # Core toolchain libraries +``` + +### Key Components + +| File | Purpose | +|------|---------| +| `install_abacus_toolchain_new.sh` | Main orchestration script (new version) | +| `install_abacus_toolchain.sh` | Legacy main script (deprecated) | +| `toolchain_*.sh` | Frontend scripts for specific toolchains | +| `scripts/lib/config_manager.sh` | Configuration management | +| `scripts/lib/package_manager.sh` | Package installation logic | +| `scripts/lib/user_interface.sh` | User interaction and output | +| `scripts/common_vars.sh` | Shared variables and defaults | +| `scripts/tool_kit.sh` | Utility functions and macros | +| `scripts/parse_if.py` | Parser for IF_XYZ constructs | +| `checksums.sha256` | Pre-calculated SHA256 checksums for packages | + +### Script Structure Details + +**Individual Package Scripts**: Each `scripts/stage*/install_PKG.sh` script is relatively independent and should: + +1. **Generate setup files**: Write to both `build/setup_PKG` and `install/setup` + - `build/setup_PKG`: Variables for toolchain compilation and arch file flags + - `install/setup`: Environment setup for compiling/running ABACUS + +2. **Handle dependencies**: May depend on other libraries being installed with correct environment variables -OpenMPI in version 5 has huge update, lead to compatibility problem. If one wants to use the OpenMPI in version 4 (4.1.6), one can specify `--with-openmpi-4th=yes` in *toolchain_gnu.sh* +3. **Use toolkit macros**: Leverage functionality from `scripts/tool_kit.sh` for common operations +### Package Installation Scripts -### Shell problem +Each `scripts/stage*/install_PKG.sh` script: -If you encounter problem like: +1. **Downloads** the package (if not available locally) +2. **Configures** build with appropriate flags +3. **Compiles** with error handling and logging +4. **Installs** to the toolchain directory +5. **Generates** setup files for environment configuration + +### Configuration System + +#### Package Control Options (`--with-PKG`) + +The `--with-PKG` options control how a package is going to be installed: + +- `--with-PKG=install` (or `--with-PKG` alone): Compile and install from source downloaded (default) +- `--with-PKG=system`: Link to locations provided by system search paths +- `--with-PKG=/path/to/pkg`: Link to locations provided by the user (custom path) +- `--with-PKG=no`: Skip package installation entirely + +**System Search Paths**: When using `system` mode, the installation script searches in: +- `LD_LIBRARY_PATH`, `LD_RUN_PATH`, `LIBRARY_PATH` +- `/usr/local/lib64`, `/usr/local/lib`, `/usr/lib64`, `/usr/lib` +- For MKL libraries: `MKLROOT` environment variable + +**Troubleshooting System Libraries**: If `--with-PKG=system` cannot find the library: +1. Use `module show PKG` to see module-defined paths +2. Find the root installation directory manually +3. Use `--with-PKG=/path/to/pkg` to specify exact location + +#### Feature Control Options (`--enable-FEATURE`) + +The `--enable-FEATURE` options control whether optional features are enabled: + +- `--enable-FEATURE=yes` (or `--enable-FEATURE` alone): Enable the feature +- `--enable-FEATURE=no`: Disable the feature + +#### Mode Selection (`PKG_MODE` Variables) + +For packages serving the same purpose, mode variables act as selectors: + +- `--mpi-mode=openmpi|mpich|intelmpi`: Choose MPI implementation +- `--math-mode=openblas|mkl|aocl`: Choose math library + +**Note**: While `--with-PKG` controls the installation method, the `PKG_MODE` variable picks which package to actually use, providing maximum flexibility. + +### Adding New Packages + +1. **Create installation script**: `scripts/stageN/install_newpkg.sh` +2. **Add to stage script**: Include in `scripts/stageN/install_stageN.sh` +3. **Update configuration**: Add options to `config_manager.sh` +4. **Add version info**: Update `scripts/package_versions.sh` +5. **Test thoroughly**: Verify with different toolchain combinations + +### Advanced Developer Features + +#### The IF_XYZ Constructs + +The toolchain uses a special syntax construct for conditional compilation flags: + +```shell +IF_XYZ(A | B) +``` + +This construct is parsed by `scripts/parse_if.py`: +- Evaluates to *A* if *XYZ* is passed as command line option +- Evaluates to *B* if *XYZ* is not passed + +**Nested Constructs**: The `IF_XYZ(A|B)` construct can be nested: + +```shell +IF_XYZ(IF_ABC(flag1|flag2) | flag3) +``` + +This parses to: +- *flag1* if both *XYZ* and *ABC* are present +- *flag2* if only *XYZ* is present +- *flag3* if neither is present + +#### Portability Requirements + +**Compiler Flag Filtering**: Always pass compiler flags through compatibility filters: + +```shell +# Filter flags for GCC compatibility +CFLAGS="$(allowed_gcc_flags $CFLAGS)" +FCFLAGS="$(allowed_gfortran_flags $FCFLAGS)" +``` + +**IF_XYZ with Flag Filtering**: Since filters don't work with IF_XYZ constructs, break them down: ```shell -/bin/bash^M: bad interpreter: No such file or directory +# Instead of: FCFLAGS="IF_XYZ(flag1 flag2 | flag3 flag4)" +XYZ_TRUE_FLAGS="flag1 flag2" +XYZ_FALSE_FLAGS="flag3 flag4" +# Apply filtering +XYZ_TRUE_FLAGS="$(allowed_gcc_flags $XYZ_TRUE_FLAGS)" +XYZ_FALSE_FLAGS="$(allowed_gcc_flags $XYZ_FALSE_FLAGS)" +# Reconstruct +FCFLAGS="IF_XYZ($XYZ_TRUE_FLAGS | $XYZ_FALSE_FLAGS)" ``` -or `permission denied` problem, you can simply run: +**Fortran Module Checking**: Check intrinsic Fortran modules with: ```shell -./pre_set.sh +check_gfortran_module module_name ``` -And also, you can fix `permission denied` problem via `chmod +x` -if *pre_set.sh* have no execution permission; -if the *pre_set.sh* also have `/bin/bash^M` problem, you can run: +**Avoid Hard Coding**: Use common variables instead of hard-coded paths: ```shell -> dos2unix pre_set.sh +# Good practice +./configure --prefix=some_dir CC=${MPICC} FC=${MPIFC} +# Avoid +./configure --prefix=some_dir CC=mpicc FC=mpif90 ``` -to fix it +### Best Practices + +- **Reuse toolkit functions**: Use macros from `scripts/tool_kit.sh` +- **Modular functionality**: Add new functionality as macros in `scripts/tool_kit.sh` rather than inline code +- **Portable compiler flags**: Filter through `allowed_gcc_flags` and `allowed_gfortran_flags` +- **Environment variables**: Use `${VAR:-default}` pattern for configurable defaults +- **Lock files**: Create completion markers for resumable installation +- **Separate directories**: Install each package in its own directory +- **Error handling**: Provide clear error messages and recovery suggestions + +## License -### Libtorch and DeePKS problem +The ABACUS Toolchain downloads and installs only [GPL-compatible](https://www.gnu.org/licenses/gpl-faq.html#WhatDoesCompatMean) packages. All included packages maintain their original licenses as listed in the Dependencies section above. -If deepks feature have problem, you can manually change libtorch version -from 2.1.2 to 2.0.1 or 1.12.0 in `toolchain/scripts/stage4/install_libtorch.sh`. +**License Compatibility**: All packages use GPL-compatible licenses including BSD, MIT, LGPL, MPL-2.0, and GPL variants, ensuring seamless integration with GPL-licensed software. -Also, you can install ABACUS without deepks by removing all the deepks and related options. +**Note**: Proprietary packages like Intel OneAPI (MKL/Compiler/MPI) and AMD AOCC/AOCL are supported but must be installed separately by the user. -NOTICE: if you want deepks feature, your intel-mkl environment should be accessible in building process. you can check it in `build_abacus_gnu.sh` +## Contributing -### DeePMD feature problem +We welcome contributions to improve the ABACUS Toolchain! Here's how you can help: -When you encounter problem like `GLIBCXX_3.4.29 not found`, it is sure that your `gcc` version is lower than the requirement of `libdeepmd`. +### Reporting Issues -After my test, you need `gcc`>11.3.1 to enable deepmd feature in ABACUS. +1. **Search existing issues** before creating new ones +2. **Provide detailed information**: + - Operating system and version + - Compiler versions + - Complete error messages and logs + - Steps to reproduce +### Contributing Code -## Advanced Installation Usage +1. **Fork the repository** and create a feature branch +2. **Follow coding standards**: + - Use consistent shell scripting style + - Add comments for complex logic + - Test with multiple toolchain combinations +3. **Update documentation** for new features +4. **Submit pull request** with clear description + +### Development Setup + +```bash +# Clone the repository +git clone https://github.com/deepmodeling/abacus-develop.git +cd abacus-develop/toolchain + +# Test your changes +./toolchain_gnu.sh --dry-run +``` -1. Users can move toolchain directory to anywhere you like, -and complete installation by change the setting in -`toolchain_*.sh` and `build_*.sh` by your own setting. -By moving toolchain out or rename it ,one can make toolchain independent -from ABACUS repo, make dependencies package more independent and flexible. -2. Users can manually change `pkg_install_dir` variable -in `scripts/stage*/install*` to change the installation directory -of each packages, which may let the installation more fiexible. +### Areas for Contribution +- 🔧 **New package support**: Add support for additional scientific libraries +- 🐛 **Bug fixes**: Resolve compatibility issues and installation problems +- 📚 **Documentation**: Improve guides and troubleshooting information +- 🧪 **Testing**: Expand test coverage for different systems and configurations +- 🚀 **Performance**: Optimize installation speed and resource usage -## More +--- -More infomation can be read from `Details.md`. +**For questions, issues, or contributions, please visit the [ABACUS GitHub repository](https://github.com/deepmodeling/abacus-develop).** diff --git a/toolchain/build_abacus_aocc-aocl.sh b/toolchain/build_abacus_aocc-aocl.sh index c9801b6887..a586442421 100755 --- a/toolchain/build_abacus_aocc-aocl.sh +++ b/toolchain/build_abacus_aocc-aocl.sh @@ -1,13 +1,13 @@ -#!/bin/bash -#SBATCH -J build +#!/bin/bash -e +#SBATCH -J build_abacus_aocc-aocl #SBATCH -N 1 #SBATCH -n 16 #SBATCH -o install.log #SBATCH -e install.err -# JamesMisaka in 2025.03.09 -# Build ABACUS by amd-openmpi toolchain +# Build ABACUS by aocc-aocl toolchain +# load aocc-aocl env at first # module load openmpi aocc aocl ABACUS_DIR=.. @@ -16,26 +16,27 @@ INSTALL_DIR=$TOOL/install source $INSTALL_DIR/setup cd $ABACUS_DIR ABACUS_DIR=$(pwd) -#AOCLhome=/opt/aocl-linux-aocc-5.0.0/5.0.0/aocl/ # user should specify this parameter +#AOCLhome=/opt/aocl-linux-aocc-5.0.0/5.0.0/aocl/ # user should specify this parameter to the aocl installation path -BUILD_DIR=build_abacus_aocl +BUILD_DIR=build_abacus_aocc_aocl rm -rf $BUILD_DIR PREFIX=$ABACUS_DIR -ELPA=$INSTALL_DIR/elpa-2025.01.001/cpu -# ELPA=$INSTALL_DIR/elpa-2025.01.001/nvidia # for gpu-lcao +ELPA=$INSTALL_DIR/elpa-2025.06.001/cpu +# ELPA=$INSTALL_DIR/elpa-2025.06.001/nvidia # for elpa-gpu CEREAL=$INSTALL_DIR/cereal-master/include/cereal LIBXC=$INSTALL_DIR/libxc-7.0.0 -RAPIDJSON=$INSTALL_DIR/rapidjson-master/ +RAPIDJSON=$INSTALL_DIR/rapidjson-master LAPACK=$AOCLhome/lib SCALAPACK=$AOCLhome/lib FFTW3=$AOCLhome +LIBRI=$INSTALL_DIR/LibRI-master +LIBCOMM=$INSTALL_DIR/LibComm-master +USE_CUDA=OFF # set ON to enable gpu-abacus # NEP_DIR=$INSTALL_DIR/NEP_CPU-main -# LIBRI=$INSTALL_DIR/LibRI-0.2.1.0 -# LIBCOMM=$INSTALL_DIR/LibComm-master # LIBTORCH=$INSTALL_DIR/libtorch-2.1.2/share/cmake/Torch # LIBNPY=$INSTALL_DIR/libnpy-1.0.1/include -# DEEPMD=$HOME/apps/anaconda3/envs/deepmd # v3.0 might have problem +# DEEPMD=$HOME/apps/anaconda3/envs/deepmd # if clang++ have problem, switch back to g++ @@ -54,20 +55,19 @@ cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DUSE_ELPA=ON \ -DENABLE_RAPIDJSON=ON \ -DRapidJSON_DIR=$RAPIDJSON \ -# -DENABLE_DEEPKS=1 \ + -DENABLE_LIBRI=ON \ + -DLIBRI_DIR=$LIBRI \ + -DLIBCOMM_DIR=$LIBCOMM \ + -DUSE_CUDA=$USE_CUDA \ +# -DCMAKE_CUDA_COMPILER=/path/to/cuda/bin/nvcc \ +# -DNEP_DIR=$NEP_DIR \ +# -DENABLE_MLALGO=1 \ # -DTorch_DIR=$LIBTORCH \ # -Dlibnpy_INCLUDE_DIR=$LIBNPY \ -# -DENABLE_LIBRI=ON \ -# -DLIBRI_DIR=$LIBRI \ -# -DLIBCOMM_DIR=$LIBCOMM \ # -DDeePMD_DIR=$DEEPMD \ -# -DNEP_DIR=$NEP_DIR \ -# -DUSE_CUDA=ON \ # -DENABLE_CUSOLVERMP=ON \ # -D CAL_CUSOLVERMP_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/2x.xx/math_libs/1x.x/targets/x86_64-linux/lib -# if one want's to include deepmd, your system gcc version should be >= 11.3.0 for glibc requirements - cmake --build $BUILD_DIR -j `nproc` cmake --install $BUILD_DIR 2>/dev/null @@ -84,5 +84,4 @@ cat << EOF Done! To use the installed ABACUS version You need to source ${TOOL}/abacus_env.sh first ! -""" EOF \ No newline at end of file diff --git a/toolchain/build_abacus_gcc-aocl.sh b/toolchain/build_abacus_gcc-aocl.sh index 1cd50a1f50..65e2491ee9 100755 --- a/toolchain/build_abacus_gcc-aocl.sh +++ b/toolchain/build_abacus_gcc-aocl.sh @@ -1,13 +1,13 @@ -#!/bin/bash -#SBATCH -J build +#!/bin/bash -e +#SBATCH -J build_abacus_aocl #SBATCH -N 1 #SBATCH -n 16 #SBATCH -o install.log #SBATCH -e install.err -# JamesMisaka in 2025.03.09 -# Build ABACUS by amd-openmpi toolchain +# Build ABACUS by gcc-aocl toolchain +# load system env modules at first # module load openmpi aocc aocl ABACUS_DIR=.. @@ -16,26 +16,27 @@ INSTALL_DIR=$TOOL/install source $INSTALL_DIR/setup cd $ABACUS_DIR ABACUS_DIR=$(pwd) -#AOCLhome=/opt/aocl-linux-aocc-5.0.0/5.0.0/aocl/ # user should specify this parameter +#AOCLhome=/opt/aocl-linux-aocc-5.0.0/5.0.0/aocl/ # user should specify this parameter to the aocl installation path -BUILD_DIR=build_abacus_aocl +BUILD_DIR=build_abacus_gcc_aocl rm -rf $BUILD_DIR PREFIX=$ABACUS_DIR -ELPA=$INSTALL_DIR/elpa-2025.01.001/cpu -# ELPA=$INSTALL_DIR/elpa-2025.01.001/nvidia # for gpu-lcao +ELPA=$INSTALL_DIR/elpa-2025.06.001/cpu +# ELPA=$INSTALL_DIR/elpa-2025.06.001/nvidia # for elpa-gpu CEREAL=$INSTALL_DIR/cereal-master/include/cereal LIBXC=$INSTALL_DIR/libxc-7.0.0 -RAPIDJSON=$INSTALL_DIR/rapidjson-master/ +RAPIDJSON=$INSTALL_DIR/rapidjson-master LAPACK=$AOCLhome/lib SCALAPACK=$AOCLhome/lib FFTW3=$AOCLhome +LIBRI=$INSTALL_DIR/LibRI-master +LIBCOMM=$INSTALL_DIR/LibComm-master +USE_CUDA=OFF # set ON to enable gpu-abacus # NEP_DIR=$INSTALL_DIR/NEP_CPU-main -# LIBRI=$INSTALL_DIR/LibRI-0.2.1.0 -# LIBCOMM=$INSTALL_DIR/LibComm-master # LIBTORCH=$INSTALL_DIR/libtorch-2.1.2/share/cmake/Torch # LIBNPY=$INSTALL_DIR/libnpy-1.0.1/include -# DEEPMD=$HOME/apps/anaconda3/envs/deepmd # v3.0 might have problem +# DEEPMD=$HOME/apps/anaconda3/envs/deepmd cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCMAKE_CXX_COMPILER=g++ \ @@ -52,20 +53,19 @@ cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DUSE_ELPA=ON \ -DENABLE_RAPIDJSON=ON \ -DRapidJSON_DIR=$RAPIDJSON \ -# -DENABLE_DEEPKS=1 \ + -DENABLE_LIBRI=ON \ + -DLIBRI_DIR=$LIBRI \ + -DLIBCOMM_DIR=$LIBCOMM \ + -DUSE_CUDA=$USE_CUDA \ +# -DCMAKE_CUDA_COMPILER=/path/to/cuda/bin/nvcc \ +# -DNEP_DIR=$NEP_DIR \ +# -DENABLE_MLALGO=1 \ # -DTorch_DIR=$LIBTORCH \ # -Dlibnpy_INCLUDE_DIR=$LIBNPY \ -# -DENABLE_LIBRI=ON \ -# -DLIBRI_DIR=$LIBRI \ -# -DLIBCOMM_DIR=$LIBCOMM \ # -DDeePMD_DIR=$DEEPMD \ -# -DNEP_DIR=$NEP_DIR \ -# -DUSE_CUDA=ON \ # -DENABLE_CUSOLVERMP=ON \ # -D CAL_CUSOLVERMP_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/2x.xx/math_libs/1x.x/targets/x86_64-linux/lib -# if one want's to include deepmd, your system gcc version should be >= 11.3.0 for glibc requirements - cmake --build $BUILD_DIR -j `nproc` cmake --install $BUILD_DIR 2>/dev/null @@ -82,5 +82,4 @@ cat << EOF Done! To use the installed ABACUS version You need to source ${TOOL}/abacus_env.sh first ! -""" EOF \ No newline at end of file diff --git a/toolchain/build_abacus_gnu.sh b/toolchain/build_abacus_gnu.sh index f231855102..b373ca1a10 100755 --- a/toolchain/build_abacus_gnu.sh +++ b/toolchain/build_abacus_gnu.sh @@ -1,14 +1,13 @@ #!/bin/bash -#SBATCH -J build +#SBATCH -J build_abacus_gnu #SBATCH -N 1 #SBATCH -n 16 #SBATCH -o install.log #SBATCH -e install.err -# JamesMisaka in 2025.03.09 # Build ABACUS by gnu-toolchain -# module load openmpi +# load system env modules at first ABACUS_DIR=.. TOOL=$(pwd) @@ -21,20 +20,21 @@ BUILD_DIR=build_abacus_gnu rm -rf $BUILD_DIR PREFIX=$ABACUS_DIR -LAPACK=$INSTALL_DIR/openblas-0.3.29/lib +LAPACK=$INSTALL_DIR/openblas-0.3.30/lib SCALAPACK=$INSTALL_DIR/scalapack-2.2.2/lib -ELPA=$INSTALL_DIR/elpa-2025.01.001/cpu -# ELPA=$INSTALL_DIR/elpa-2025.01.001/nvidia # for gpu-lcao +ELPA=$INSTALL_DIR/elpa-2025.06.001/cpu +# ELPA=$INSTALL_DIR/elpa-2025.06.001/nvidia # for elpa-gpu FFTW3=$INSTALL_DIR/fftw-3.3.10 CEREAL=$INSTALL_DIR/cereal-master/include/cereal LIBXC=$INSTALL_DIR/libxc-7.0.0 RAPIDJSON=$INSTALL_DIR/rapidjson-master/ +LIBRI=$INSTALL_DIR/LibRI-master +LIBCOMM=$INSTALL_DIR/LibComm-master +USE_CUDA=OFF # set ON to enable gpu-abacus # NEP_DIR=$INSTALL_DIR/NEP_CPU-main -# LIBRI=$INSTALL_DIR/LibRI-0.2.1.0 -# LIBCOMM=$INSTALL_DIR/LibComm-master # LIBTORCH=$INSTALL_DIR/libtorch-2.1.2/share/cmake/Torch # LIBNPY=$INSTALL_DIR/libnpy-1.0.1/include -# DEEPMD=$HOME/apps/anaconda3/envs/deepmd # v3.0 might have problem +# DEEPMD=$HOME/apps/anaconda3/envs/deepmd # cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCMAKE_CXX_COMPILER=g++ \ @@ -51,25 +51,19 @@ cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DUSE_ELPA=ON \ -DENABLE_RAPIDJSON=ON \ -DRapidJSON_DIR=$RAPIDJSON \ -# -DENABLE_DEEPKS=1 \ + -DENABLE_LIBRI=ON \ + -DLIBRI_DIR=$LIBRI \ + -DLIBCOMM_DIR=$LIBCOMM \ + -DUSE_CUDA=$USE_CUDA \ +# -DCMAKE_CUDA_COMPILER=/path/to/cuda/bin/nvcc \ +# -DNEP_DIR=$NEP_DIR \ +# -DENABLE_MLALGO=1 \ # -DTorch_DIR=$LIBTORCH \ # -Dlibnpy_INCLUDE_DIR=$LIBNPY \ -# -DENABLE_LIBRI=ON \ -# -DLIBRI_DIR=$LIBRI \ -# -DLIBCOMM_DIR=$LIBCOMM \ # -DDeePMD_DIR=$DEEPMD \ -# -DNEP_DIR=$NEP_DIR \ -# -DUSE_CUDA=ON \ # -DENABLE_CUSOLVERMP=ON \ # -D CAL_CUSOLVERMP_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/2x.xx/math_libs/1x.x/targets/x86_64-linux/lib -# # add mkl env for libtorch to link -# if one want to install libtorch, mkl should be load in build process -# for -lmkl when load libtorch -# module load mkl - -# if one want's to include deepmd, your system gcc version should be >= 11.3.0 for glibc requirements - cmake --build $BUILD_DIR -j `nproc` cmake --install $BUILD_DIR 2>/dev/null @@ -86,5 +80,4 @@ cat << EOF Done! To use the installed ABACUS version You need to source ${TOOL}/abacus_env.sh first ! -""" EOF diff --git a/toolchain/build_abacus_intel.sh b/toolchain/build_abacus_intel.sh index 9e69f37b52..29575bf4e5 100755 --- a/toolchain/build_abacus_intel.sh +++ b/toolchain/build_abacus_intel.sh @@ -1,13 +1,13 @@ #!/bin/bash -#SBATCH -J build +#SBATCH -J build_abacus_intel #SBATCH -N 1 #SBATCH -n 16 #SBATCH -o install.log #SBATCH -e install.err -# JamesMisaka in 2025.03.09 # Build ABACUS by intel-toolchain +# load intel-oneapi env at first # module load mkl compiler mpi # source path/to/setvars.sh @@ -22,19 +22,20 @@ BUILD_DIR=build_abacus_intel rm -rf $BUILD_DIR PREFIX=$ABACUS_DIR -ELPA=$INSTALL_DIR/elpa-2025.01.001/cpu -# ELPA=$INSTALL_DIR/elpa-2025.01.001/nvidia # for gpu-lcao -CEREAL=$INSTALL_DIR/cereal-master/include/cereal +ELPA=$INSTALL_DIR/elpa-2024.05.001/cpu +# ELPA=$INSTALL_DIR/elpa-2024.05.001/nvidia # for gpu-lcao +CEREAL=$INSTALL_DIR/cereal-master/include LIBXC=$INSTALL_DIR/libxc-7.0.0 -RAPIDJSON=$INSTALL_DIR/rapidjson-master/ -# LIBTORCH=$INSTALL_DIR/libtorch-2.1.2/share/cmake/Torch -# LIBNPY=$INSTALL_DIR/libnpy-1.0.1/include -LIBRI=$INSTALL_DIR/LibRI-0.2.1.1 +RAPIDJSON=$INSTALL_DIR/rapidjson-master +LIBRI=$INSTALL_DIR/LibRI-master LIBCOMM=$INSTALL_DIR/LibComm-master +USE_CUDA=OFF # set ON to enable gpu-abacus # NEP_DIR=$INSTALL_DIR/NEP_CPU-main +# LIBTORCH=$INSTALL_DIR/libtorch-2.1.2/share/cmake/Torch +# LIBNPY=$INSTALL_DIR/libnpy-1.0.1/include # DEEPMD=$HOME/apps/anaconda3/envs/deepmd # v3.0 might have problem -# Notice: if you are compiling with AMD-CPU or GPU-version ABACUS, then `icpc` and `mpiicpc` compilers are recommended +# Notice: if you are compiling with AMD-CPU or GPU-version ABACUS, then `icpc` and `mpiicpc` compilers are needed cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCMAKE_CXX_COMPILER=icpx \ -DMPI_CXX_COMPILER=mpiicpx \ @@ -48,23 +49,22 @@ cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DUSE_ELPA=ON \ -DENABLE_RAPIDJSON=ON \ -DRapidJSON_DIR=$RAPIDJSON \ + -DENABLE_LIBRI=ON \ -DLIBRI_DIR=$LIBRI \ - -DLIBCOMM_DIR=$LIBCOMM + -DLIBCOMM_DIR=$LIBCOMM \ + -DUSE_CUDA=$USE_CUDA \ +# -DCMAKE_CUDA_COMPILER=/path/to/cuda/bin/nvcc \ +# -DNEP_DIR=$NEP_DIR \ # -DENABLE_MLALGO=1 \ # -DTorch_DIR=$LIBTORCH \ # -Dlibnpy_INCLUDE_DIR=$LIBNPY \ -# -DENABLE_LIBRI=ON \ # -DDeePMD_DIR=$DEEPMD \ -# -DNEP_DIR=$NEP_DIR \ -# -DUSE_CUDA=ON \ # -DENABLE_CUSOLVERMP=ON \ # -D CAL_CUSOLVERMP_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/2x.xx/math_libs/1x.x/targets/x86_64-linux/lib cmake --build $BUILD_DIR -j `nproc` cmake --install $BUILD_DIR 2>/dev/null -# if one want's to include deepmd, your system gcc version should be >= 11.3.0 for glibc requirements - # generate abacus_env.sh cat << EOF > "${TOOL}/abacus_env.sh" #!/bin/bash @@ -78,5 +78,4 @@ cat << EOF Done! To use the installed ABACUS version You need to source ${TOOL}/abacus_env.sh first ! -""" EOF diff --git a/toolchain/install_abacus_toolchain.sh b/toolchain/install_abacus_toolchain.sh index 4e70843e2c..910927ded6 100755 --- a/toolchain/install_abacus_toolchain.sh +++ b/toolchain/install_abacus_toolchain.sh @@ -40,6 +40,29 @@ export SHA256_CHECKSUM="${SCRIPTDIR}/checksums.sha256" # ------------------------------------------------------------------------ TOOLCHAIN_OPTIONS="$@" +# ------------------------------------------------------------------------ +# DEPRECATED WARNING +# ------------------------------------------------------------------------ +echo "" +echo -e "\033[1;31m╔══════════════════════════════════════════════════════════════════════════════╗\033[0m" +echo -e "\033[1;31m║ [DEPRECATED] ║\033[0m" +echo -e "\033[1;31m║ ║\033[0m" +echo -e "\033[1;33m║ This script (install_abacus_toolchain.sh) will be deprecated soon. ║\033[0m" +echo -e "\033[1;33m║ ║\033[0m" +echo -e "\033[1;32m║ Please migrate to the refactored version: ║\033[0m" +echo -e "\033[1;32m║ → install_abacus_toolchain_new.sh ║\033[0m" +echo -e "\033[1;32m║ ║\033[0m" +echo -e "\033[1;36m║ Migration Guide: ║\033[0m" +echo -e "\033[1;36m║ • Use toolchain_*.sh frontend scripts for easier configuration ║\033[0m" +echo -e "\033[1;36m║ • New version supports main/alt package version switch ║\033[0m" +echo -e "\033[1;36m║ • Improved parameter handling and error reporting ║\033[0m" +echo -e "\033[1;31m║ ║\033[0m" +echo -e "\033[1;31m╚══════════════════════════════════════════════════════════════════════════════╝\033[0m" +echo "" +echo -e "\033[1;33mContinuing with legacy script in 3 seconds...\033[0m" +sleep 3 +echo "" + # ------------------------------------------------------------------------ # Load common variables and tools # ------------------------------------------------------------------------ @@ -106,8 +129,7 @@ OPTIONS: or --with-openblas options will switch --math-mode to the respective modes. --gpu-ver Selects the GPU architecture for which to compile. Available - options are: K20X, K40, K80, P100, V100, Mi50, Mi100, Mi250, - and no. + options: CUDA architecture number (7.5 / 75, 8.0 / 80, etc) or no This setting determines the value of nvcc's '-arch' flag. Default = no. --log-lines Number of log file lines dumped in case of a non-zero exit code. @@ -268,19 +290,8 @@ with_gcc="__SYSTEM__" with_fftw="__INSTALL__" with_libxc="__INSTALL__" with_scalapack="__INSTALL__" -# default math library settings, MATH_MODE picks the math library -# to use, and with_* defines the default method of installation if it -# is picked. For non-CRAY systems defaults to mkl if $MKLROOT is -# available, otherwise defaults to openblas -if [ "${MKLROOT}" ]; then - export MATH_MODE="mkl" - with_mkl="__SYSTEM__" -elif [ "${AOCLhome}" ]; then - export MATH_MODE="aocl" - with_aocl="__SYSTEM__" -else - export MATH_MODE="openblas" -fi +# default math library setting: openblas +export MATH_MODE="openblas" with_openblas="__INSTALL__" with_elpa="__INSTALL__" with_cereal="__INSTALL__" @@ -873,7 +884,7 @@ for ii in ${package_list}; do done # ------------------------------------------------------------------------ -# Build packages unless dry-run mode is enabled. +# Build packages unless dry-run or pack-run mode is enabled. # ------------------------------------------------------------------------ if [ "${dry_run}" = "__TRUE__" ]; then echo "Wrote only configuration files (--dry-run)." diff --git a/toolchain/install_abacus_toolchain_new.sh b/toolchain/install_abacus_toolchain_new.sh new file mode 100755 index 0000000000..6669c87dc3 --- /dev/null +++ b/toolchain/install_abacus_toolchain_new.sh @@ -0,0 +1,224 @@ +#!/bin/bash -e + +# ============================================================================ +# ABACUS Toolchain Installation Script (New Version - Enhanced UI) +# ============================================================================ +# This is the new refactored version of the ABACUS toolchain installation script. +# It provides the same functionality as the original script but with improved +# modularity, maintainability, extensibility, and beautiful terminal output. +# +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# Enhanced: Beautiful terminal UI with harmonious color scheme +# ============================================================================ + +# Set script name for error reporting +[ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 + +# Set directory variables +export ROOTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +export SCRIPTDIR="${ROOTDIR}/scripts" +export BUILDDIR="${ROOTDIR}/build" +export INSTALLDIR="${ROOTDIR}/install" +export SETUPFILE="${INSTALLDIR}/setup" +export SHA256_CHECKSUM="${SCRIPTDIR}/checksums.sha256" + +# Make a copy of all options for $SETUPFILE +TOOLCHAIN_OPTIONS="$@" + +# Source required modules +source "${SCRIPTDIR}/tool_kit.sh" +source "${SCRIPTDIR}/lib/error_handler.sh" +source "${SCRIPTDIR}/lib/config_manager.sh" +source "${SCRIPTDIR}/lib/version_helper.sh" +source "${SCRIPTDIR}/lib/package_manager.sh" +source "${SCRIPTDIR}/lib/user_interface.sh" +source "${SCRIPTDIR}/lib/config_validator.sh" + +# Initialize modules +version_helper_init +ui_init +ui_setup_signals + +# Show help function +show_help() { + ui_show_help +} + +# Main function +main() { + local args=("$@") + + # Initialize configuration with command line arguments + if ! config_init "${args[@]}"; then + show_help + exit 0 + fi + + # Handle special version-related requests + if [[ "$(config_get show_version)" == "__TRUE__" ]]; then + version_show_available + exit 0 + fi + + local version_info_request="$(config_get show_version_info)" + if [[ -n "$version_info_request" ]]; then + if [[ "$version_info_request" == "all" ]]; then + version_show_available + else + version_show_available "$version_info_request" + fi + exit 0 + fi + + if [[ "$(config_get show_help)" == "__TRUE__" ]]; then + show_help + version_show_help + exit 0 + fi + + # Show beautiful welcome banner (only when actually running installation) + ui_welcome_banner + + # Show configuration summary with enhanced styling + if [[ "$(config_get dry_run)" == "__TRUE__" ]]; then + ui_section "Dry Run Mode" + ui_warning "Configuration will be written but no packages will be installed" + echo "" + fi + + if [[ "$(config_get PACK_RUN)" == "__TRUE__" ]]; then + ui_section "Pack Run Mode" + ui_info "Only downloading packages, skipping installation" + echo "" + fi + + # Show version configuration + version_show_current + + # Show enhanced summary + ui_show_summary + + # Validate version configuration + if ! version_validate_config; then + echo "" + ui_warning "Some version configuration issues were detected" + ui_info "Please review the warnings above" + echo "" + fi + + # Run configuration validation unless skipped + if ! should_skip_validation; then + echo "" + ui_section "System Validation" + if ! validate_configuration; then + echo "" + report_error $LINENO "Configuration validation failed with errors" "CONFIG_ERROR" + ui_error "Please fix the configuration issues above and try again" + ui_info "You can skip validation with --skip-system-checks if needed" + exit 1 + fi + ui_success "System validation completed successfully" + fi + + # Skip user confirmation - proceed directly with installation + + # Export configuration to environment variables + config_export_to_env + + # Export version configuration for stage scripts + package_export_version_config + + # Preliminaries + ui_section "Preparing Installation Environment" + ui_info "Creating installation directories..." + mkdir -p ${INSTALLDIR} + + # Start writing setup file + ui_info "Generating setup configuration..." + cat << EOF > "$SETUPFILE" +#!/bin/bash +source "${SCRIPTDIR}/tool_kit.sh" +export ABACUS_TOOLCHAIN_OPTIONS="${TOOLCHAIN_OPTIONS}" +EOF + + ui_info "Compiling with $(get_nprocs) processes for target ${TARGET_CPU}" + + write_toolchain_env ${INSTALLDIR} + + # write toolchain config + ui_info "Writing toolchain configuration..." + echo "tool_list=\"${tool_list}\"" > ${INSTALLDIR}/toolchain.conf + for ii in ${package_list}; do + install_mode="$(eval echo \${with_${ii}})" + echo "with_${ii}=\"${install_mode}\"" >> ${INSTALLDIR}/toolchain.conf + done + + # Exit if dry run + if [[ "$(config_get dry_run)" == "__TRUE__" ]]; then + ui_success "Configuration files generated successfully (dry-run mode)" + ui_info "To proceed with actual installation, run without --dry-run" + exit 0 + fi + + # Build packages (following original toolchain logic) + ui_section "Starting Package Installation" + ui_info "Initializing build environment..." + echo "# Leak suppressions" > ${INSTALLDIR}/lsan.supp + + # Install packages stage by stage with enhanced progress display + ui_stage_progress "0" "Build Tools & Compilers" + ./scripts/stage0/install_stage0.sh + + ui_stage_progress "1" "MPI Implementations" + ./scripts/stage1/install_stage1.sh + + ui_stage_progress "2" "Mathematical Libraries" + ./scripts/stage2/install_stage2.sh + + ui_stage_progress "3" "Scientific Computing Libraries" + ./scripts/stage3/install_stage3.sh + + ui_stage_progress "4" "Advanced Feature Libraries" + ./scripts/stage4/install_stage4.sh + + # Show beautiful completion message + ui_section "Installation Complete" + ui_success "ABACUS Toolchain installation completed successfully!" + echo "" + + # Enhanced usage instructions with beautiful formatting + ui_print_color "${UI_BLUE}${UI_BOLD}" "🎉 Congratulations! Your ABACUS toolchain installation finished successfully." + echo "" + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '═%.0s' $(seq 1 60))" + ui_print_color "${UI_BLUE}${UI_BOLD}" " USAGE INSTRUCTIONS" + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '═%.0s' $(seq 1 60))" + echo "" + + ui_print_color "${UI_GREEN}${UI_BOLD}" "🔧 To activate the toolchain environment:" + ui_print_color "${UI_WHITE}" " source ${SETUPFILE}" + echo "" + + ui_print_color "${UI_GREEN}${UI_BOLD}" "🚀 Build ABACUS with your preferred toolchain:" + ui_print_color "${UI_WHITE}" " ./build_abacus_gnu.sh # GNU toolchain (GCC + OpenBLAS)" + ui_print_color "${UI_WHITE}" " ./build_abacus_intel.sh # Intel toolchain (ICC + MKL)" + ui_print_color "${UI_WHITE}" " ./build_abacus_gcc-aocl.sh # AMD GCC + AOCL" + ui_print_color "${UI_WHITE}" " ./build_abacus_aocc-aocl.sh # AMD AOCC + AOCL" + echo "" + + ui_print_color "${UI_ORANGE}${UI_BOLD}" "💡 Pro Tips:" + ui_print_color "${UI_GRAY}" " • Modify the builder scripts to suit your specific needs" + ui_print_color "${UI_GRAY}" " • The environment setup is required for each new shell session" + ui_print_color "${UI_GRAY}" " • Check the generated setup file for all available environment variables" + echo "" + + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '═%.0s' $(seq 1 60))" + ui_print_color "${UI_GREEN}${UI_BOLD}" "✨ Happy DFT calculation with ABACUS! ✨" + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '═%.0s' $(seq 1 60))" + echo "" + + return 0 +} + +# Run main function with all arguments +main "$@" \ No newline at end of file diff --git a/toolchain/scripts/VERSION b/toolchain/scripts/VERSION index e94549a66e..6f4471799a 100644 --- a/toolchain/scripts/VERSION +++ b/toolchain/scripts/VERSION @@ -1,2 +1,2 @@ # version file to force a rebuild of the entire toolchain -VERSION="2025.2" \ No newline at end of file +VERSION="2025.3" \ No newline at end of file diff --git a/toolchain/scripts/common_vars.sh b/toolchain/scripts/common_vars.sh index 51f941f6fe..fd003555c3 100755 --- a/toolchain/scripts/common_vars.sh +++ b/toolchain/scripts/common_vars.sh @@ -14,7 +14,7 @@ ARCH_FILE_TEMPLATE=${ARCH_FILE_TEMPLATE:-"${SCRIPTDIR}/arch_base.tmpl"} VERSION_FILE=${VERSION_FILE:-"${SCRIPTDIR}/VERSION"} # system arch gotten from OpenBLAS prebuild -OPENBLAS_ARCH=${OPENBLAS_ARCH:-"x86_64"} +OPENBLAS_ARCH=${OPENBLAS_ARCH:-''} OPENBLAS_LIBCORE=${OPENBLAS_LIBCORE:-''} # search paths @@ -31,4 +31,10 @@ ENABLE_CRAY=${ENABLE_CRAY:-"__FALSE__"} MPI_MODE=${MPI_MODE:-openmpi} MATH_MODE=${MATH_MODE:-openblas} +# default for log file dump size +export LOG_LINES="200" + +# download configuration +DOWNLOAD_CERT_POLICY=${DOWNLOAD_CERT_POLICY:-"smart"} # strict|smart|skip + export NVCC=${NVCC:-nvcc} diff --git a/toolchain/scripts/get_openblas_arch.sh b/toolchain/scripts/get_openblas_arch.sh index f0fb8b1aa8..6f61db4612 100755 --- a/toolchain/scripts/get_openblas_arch.sh +++ b/toolchain/scripts/get_openblas_arch.sh @@ -3,57 +3,72 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2024-0811 - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")" && pwd -P)" -openblas_ver="0.3.29" # Keep in sync with install_openblas.sh -openblas_sha256="38240eee1b29e2bde47ebb5d61160207dc68668a54cac62c076bb5032013b1eb" +# Load centralized version management +source "${SCRIPT_DIR}"/package_versions.sh + +# Load OpenBLAS package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "openblas:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "openblas:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "openblas" "$version_suffix" openblas_pkg="OpenBLAS-${openblas_ver}.tar.gz" +find_openblas_dir() { + local __dir='' + for __dir in *OpenBLAS*; do + if [ -d "$__dir" ]; then + echo "$__dir" + return + fi + done + echo '' +} + source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env -find_openblas_dir() { - local __dir='' - for __dir in *OpenBLAS*; do - if [ -d "$__dir" ]; then - echo "$__dir" - return 0 - fi - done - echo '' -} - ! [ -d "${BUILDDIR}" ] && mkdir -p "${BUILDDIR}" cd "${BUILDDIR}" echo "==================== Getting proc arch info using OpenBLAS tools ====================" -# find existing openblas source dir -openblas_dir="$(find_openblas_dir)" -# if cannot find openblas source dir, try download one -if ! [ "$openblas_dir" ]; then - if [ -f ${openblas_pkg} ]; then +if [ -f ${openblas_pkg} ]; then echo "${openblas_pkg} is found" - else - #download_pkg_from_ABACUS_org "${openblas_sha256}" "${openblas_pkg}" - # using codeload.github +else +# using codeload.github url="https://codeload.github.com/OpenMathLib/OpenBLAS/tar.gz/v${openblas_ver}" download_pkg_from_url "${openblas_sha256}" "${openblas_pkg}" "${url}" - fi - tar -xzf ${openblas_pkg} - openblas_dir="$(find_openblas_dir)" fi +# if toolchain run in pack-run mode, do exit +if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip arch detection" + exit 0 +fi +tar -xzf ${openblas_pkg} +openblas_dir="$(find_openblas_dir)" openblas_conf="${openblas_dir}/Makefile.conf" # try find Makefile.config, if not then generate one with make lapack_prebuild if ! [ -f "$openblas_conf" ]; then - cd "$openblas_dir" - make lapack_prebuild - cd .. + cd "$openblas_dir" + make lapack_prebuild + cd .. fi OPENBLAS_LIBCORE="$(grep 'LIBCORE=' $openblas_conf | cut -f2 -d=)" OPENBLAS_ARCH="$(grep 'ARCH=' $openblas_conf | cut -f2 -d=)" diff --git a/toolchain/scripts/lib/config_manager.sh b/toolchain/scripts/lib/config_manager.sh new file mode 100644 index 0000000000..3ba967c5e4 --- /dev/null +++ b/toolchain/scripts/lib/config_manager.sh @@ -0,0 +1,1236 @@ +#!/bin/bash + +# ============================================================================ +# ABACUS Toolchain Configuration Manager +# ============================================================================ +# Handles configuration parsing, validation, and management +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# ============================================================================ + +# Global configuration variables +declare -A CONFIG_CACHE +declare -A USER_EXPLICIT_MPI +declare -A USER_EXPLICIT_MATH +CONFIG_INITIALIZED=false +CONFIG_FILE_LOADED=false + +# Package lists (from original script) +tool_list="gcc intel amd cmake" +mpi_list="mpich openmpi intelmpi" +math_list="mkl aocl openblas" +lib_list="fftw libxc scalapack elpa cereal rapidjson libtorch libnpy libri libcomm nep" +package_list="${tool_list} ${mpi_list} ${math_list} ${lib_list}" + +# Configuration file paths for loading in advance +# Not in use now +CONFIG_FILE_PATHS=( + "${ROOTDIR}/toolchain_config.conf" +) + +# Load configuration from file +# Usage: config_load_from_file [config_file_path] +config_load_from_file() { + local config_file="$1" + + # If no file specified, search for default locations + if [[ -z "$config_file" ]]; then + for path in "${CONFIG_FILE_PATHS[@]}"; do + if [[ -f "$path" && -r "$path" ]]; then + config_file="$path" + break + fi + done + fi + + # If still no config file found, return silently (not an error) + if [[ -z "$config_file" || ! -f "$config_file" ]]; then + return 0 + fi + + echo "Loading configuration from: $config_file" + + # Read configuration file line by line + local line_num=0 + while IFS= read -r line || [[ -n "$line" ]]; do + line_num=$((line_num + 1)) + + # Skip empty lines and comments + [[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue + + # Remove leading/trailing whitespace + line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + + # Parse variable assignments + if [[ "$line" =~ ^([A-Z_][A-Z0-9_]*)=(.*)$ ]]; then + local var_name="${BASH_REMATCH[1]}" + local var_value="${BASH_REMATCH[2]}" + + # Remove quotes if present + var_value=$(echo "$var_value" | sed 's/^"//;s/"$//') + + # Validate and store configuration + case "$var_name" in + # Basic configuration + INSTALL_PREFIX|BUILD_PREFIX) + CONFIG_CACHE["$var_name"]="$var_value" + ;; + NPROCS) + if [[ "$var_value" =~ ^[0-9]+$ ]]; then + CONFIG_CACHE["NPROCS_OVERWRITE"]="$var_value" + else + echo "Warning: Invalid NPROCS value '$var_value' in config file line $line_num" + fi + ;; + + # Mode selections + MPI_MODE) + case "$var_value" in + mpich|openmpi|intelmpi|none) + CONFIG_CACHE["MPI_MODE"]="$var_value" + ;; + *) + echo "Warning: Invalid MPI_MODE value '$var_value' in config file line $line_num" + ;; + esac + ;; + MATH_MODE) + case "$var_value" in + mkl|aocl|openblas|none) + CONFIG_CACHE["MATH_MODE"]="$var_value" + ;; + *) + echo "Warning: Invalid MATH_MODE value '$var_value' in config file line $line_num" + ;; + esac + ;; + + # Package configurations + WITH_*) + local package_name=$(echo "${var_name#WITH_}" | tr '[:upper:]' '[:lower:]') + # Handle special case for 4th-openmpi (convert to OPENMPI_4TH) + if [[ "$package_name" == "4th-openmpi" ]]; then + case "$var_value" in + yes|no|__DONTUSE__) + # Map __DONTUSE__ to "no" for OPENMPI_4TH + if [[ "$var_value" == "__DONTUSE__" ]]; then + CONFIG_CACHE["OPENMPI_4TH"]="no" + else + CONFIG_CACHE["OPENMPI_4TH"]="$var_value" + fi + ;; + *) + echo "Warning: Invalid value '$var_value' for WITH_4TH_OPENMPI in config file line $line_num" + ;; + esac + else + case "$var_value" in + __SYSTEM__|__INSTALL__|__DONTUSE__) + CONFIG_CACHE["with_${package_name}"]="$var_value" + ;; + *) + # Allow custom paths for some packages + CONFIG_CACHE["with_${package_name}"]="$var_value" + ;; + esac + fi + ;; + + # Version strategy options (NEW) + VERSION_STRATEGY) + case "$var_value" in + main|alt) + CONFIG_CACHE["VERSION_STRATEGY"]="$var_value" + ;; + *) + echo "Warning: Invalid VERSION_STRATEGY value '$var_value' in config file line $line_num. Valid options: main, alt" + ;; + esac + ;; + PACKAGE_VERSION_*) + # Extract package name from PACKAGE_VERSION_PACKAGENAME + local pkg_name="${var_name#PACKAGE_VERSION_}" + case "$var_value" in + main|alt) + CONFIG_CACHE["PACKAGE_VERSION_${pkg_name}"]="$var_value" + ;; + *) + echo "Warning: Invalid package version value '$var_value' for $var_name in config file line $line_num. Valid options: main, alt" + ;; + esac + ;; + + # Advanced options + # Not in use, leave it for future + DEBUG_MODE|VERBOSE_MODE|SKIP_SYSTEM_CHECKS|FORCE_REINSTALL) + case "$var_value" in + true|false) + CONFIG_CACHE["$var_name"]="$var_value" + ;; + *) + echo "Warning: Invalid boolean value '$var_value' for $var_name in config file line $line_num" + ;; + esac + ;; + + *) + echo "Warning: Unknown configuration option '$var_name' in config file line $line_num" + ;; + esac + elif [[ -n "$line" ]]; then + echo "Warning: Invalid configuration syntax in config file line $line_num: $line" + fi + done < "$config_file" + + CONFIG_FILE_LOADED=true + echo "Configuration loaded successfully from: $config_file" + return 0 +} + +# Apply mode-based configuration after loading from file +# Usage: config_apply_modes_from_file +config_apply_modes_from_file() { + # Apply MPI mode if set in config file (silent mode to avoid duplicate output) + if [[ -n "${CONFIG_CACHE[MPI_MODE]}" ]]; then + case "${CONFIG_CACHE[MPI_MODE]}" in + mpich) + CONFIG_CACHE["with_mpich"]="__INSTALL__" + CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + openmpi) + CONFIG_CACHE["with_mpich"]="__DONTUSE__" + CONFIG_CACHE["with_openmpi"]="__INSTALL__" + CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + intelmpi) + CONFIG_CACHE["with_mpich"]="__DONTUSE__" + CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + CONFIG_CACHE["with_intelmpi"]="__INSTALL__" + ;; + none) + CONFIG_CACHE["with_mpich"]="__DONTUSE__" + CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + esac + fi + + # Apply Math mode if set in config file (silent mode to avoid duplicate output) + if [[ -n "${CONFIG_CACHE[MATH_MODE]}" ]]; then + case "${CONFIG_CACHE[MATH_MODE]}" in + mkl) + CONFIG_CACHE["with_mkl"]="__SYSTEM__" + CONFIG_CACHE["with_aocl"]="__DONTUSE__" + CONFIG_CACHE["with_openblas"]="__DONTUSE__" + # MKL provides FFTW and ScaLAPACK, so disable them (original logic) + CONFIG_CACHE["with_fftw"]="__DONTUSE__" + CONFIG_CACHE["with_scalapack"]="__DONTUSE__" + ;; + aocl) + CONFIG_CACHE["with_mkl"]="__DONTUSE__" + CONFIG_CACHE["with_aocl"]="__INSTALL__" + CONFIG_CACHE["with_openblas"]="__DONTUSE__" + ;; + openblas) + CONFIG_CACHE["with_mkl"]="__DONTUSE__" + CONFIG_CACHE["with_aocl"]="__DONTUSE__" + CONFIG_CACHE["with_openblas"]="__INSTALL__" + ;; + none) + CONFIG_CACHE["with_mkl"]="__DONTUSE__" + CONFIG_CACHE["with_aocl"]="__DONTUSE__" + CONFIG_CACHE["with_openblas"]="__DONTUSE__" + ;; + esac + fi +} + +# Initialize configuration manager +# Usage: config_manager_init +config_manager_init() { + if [[ "$CONFIG_INITIALIZED" == "true" ]]; then + return 0 + fi + + # Version management is now handled directly in individual scripts via package_versions.sh + + # Set default configuration first + config_set_defaults + + # Load configuration from file (if available) - this will override defaults + config_load_from_file + + # Apply mode-based configurations from file - this will override defaults + config_apply_modes_from_file + + CONFIG_INITIALIZED=true + + return 0 +} + +# Set default configuration values +# Usage: config_set_defaults +config_set_defaults() { + # First set everything to __DONTUSE__ + for ii in ${package_list}; do + CONFIG_CACHE["with_${ii}"]="__DONTUSE__" + done + + # Tools to turn on by default + CONFIG_CACHE["with_gcc"]="__SYSTEM__" + CONFIG_CACHE["with_cmake"]="__INSTALL__" + + # Set default parallel jobs (NEW: Fix for parallel jobs parameter) + if [[ -z "${NPROCS_OVERWRITE}" ]]; then + if command -v nproc > /dev/null 2>&1; then + CONFIG_CACHE["NPROCS_OVERWRITE"]="$(nproc --all)" + elif command -v sysctl > /dev/null 2>&1; then + CONFIG_CACHE["NPROCS_OVERWRITE"]="$(sysctl -n hw.ncpu)" + else + CONFIG_CACHE["NPROCS_OVERWRITE"]="1" + fi + else + CONFIG_CACHE["NPROCS_OVERWRITE"]="${NPROCS_OVERWRITE}" + fi + + # Default MPI and Math modes (following original script logic) + # Default math library settings to openblas + CONFIG_CACHE["MATH_MODE"]="openblas" + + # Set math library defaults based on MATH_MODE (before user input processing) + case "${CONFIG_CACHE[MATH_MODE]}" in + mkl) + CONFIG_CACHE["with_mkl"]="__SYSTEM__" + ;; + aocl) + CONFIG_CACHE["with_aocl"]="__SYSTEM__" + ;; + openblas) + CONFIG_CACHE["with_openblas"]="__INSTALL__" + ;; + esac + + # For MPI, we try to detect system MPI variant (following original script logic) + # Only detect if MPI_MODE is not already set (to avoid duplicate detection) + if [[ -z "${CONFIG_CACHE[MPI_MODE]}" ]] && command -v mpiexec > /dev/null 2>&1; then + # check if we are dealing with openmpi, mpich or intelmpi + if mpiexec --version 2>&1 | grep -s -q "HYDRA"; then + if command -v ui_info &> /dev/null; then + ui_info "🔍 Detected system MPI: MPICH" + else + echo "MPI is detected and it appears to be MPICH" + fi + CONFIG_CACHE["MPI_MODE"]="mpich" + CONFIG_CACHE["with_mpich"]="__SYSTEM__" + elif mpiexec --version 2>&1 | grep -s -q "OpenRTE"; then + if command -v ui_info &> /dev/null; then + ui_info "🔍 Detected system MPI: OpenMPI" + else + echo "MPI is detected and it appears to be OpenMPI" + fi + CONFIG_CACHE["MPI_MODE"]="openmpi" + CONFIG_CACHE["with_openmpi"]="__SYSTEM__" + elif mpiexec --version 2>&1 | grep -s -q "Intel"; then + if command -v ui_info &> /dev/null; then + ui_info "🔍 Detected system MPI: Intel MPI" + else + echo "MPI is detected and it appears to be Intel MPI" + fi + CONFIG_CACHE["with_gcc"]="__DONTUSE__" + CONFIG_CACHE["with_amd"]="__DONTUSE__" + CONFIG_CACHE["with_aocl"]="__DONTUSE__" + CONFIG_CACHE["with_intel"]="__SYSTEM__" + CONFIG_CACHE["with_intelmpi"]="__SYSTEM__" + CONFIG_CACHE["MPI_MODE"]="intelmpi" + else # default to mpich + if command -v ui_info &> /dev/null; then + ui_info "🔍 MPI detected, defaulting to MPICH configuration" + else + echo "MPI is detected and defaults to MPICH" + fi + CONFIG_CACHE["MPI_MODE"]="mpich" + CONFIG_CACHE["with_mpich"]="__SYSTEM__" + fi + else + if command -v report_warning &> /dev/null; then + report_warning ${LINENO} "No MPI installation detected (ignore this message in Cray Linux Environment or when MPI installation was requested)." + else + echo "Warning: No MPI installation detected (ignore this message in Cray Linux Environment or when MPI installation was requested)." + fi + CONFIG_CACHE["MPI_MODE"]="no" + fi + + # Default libraries to install + CONFIG_CACHE["with_fftw"]="__INSTALL__" + CONFIG_CACHE["with_libxc"]="__INSTALL__" + CONFIG_CACHE["with_scalapack"]="__INSTALL__" + CONFIG_CACHE["with_elpa"]="__INSTALL__" + CONFIG_CACHE["with_cereal"]="__INSTALL__" + CONFIG_CACHE["with_rapidjson"]="__INSTALL__" + CONFIG_CACHE["with_libri"]="__INSTALL__" + CONFIG_CACHE["with_libcomm"]="__INSTALL__" + CONFIG_CACHE["with_nep"]="__DONTUSE__" + + # Default enable options (following original script logic) + CONFIG_CACHE["dry_run"]="__FALSE__" + CONFIG_CACHE["enable_tsan"]="__FALSE__" + CONFIG_CACHE["enable_opencl"]="__FALSE__" + CONFIG_CACHE["enable_cuda"]="__FALSE__" + CONFIG_CACHE["enable_hip"]="__FALSE__" + CONFIG_CACHE["intel_classic"]="no" + CONFIG_CACHE["PACK_RUN"]="__FALSE__" + CONFIG_CACHE["INTELMPI_CLASSIC"]="no" + CONFIG_CACHE["WITH_IFX"]="yes" + CONFIG_CACHE["WITH_FLANG"]="no" + CONFIG_CACHE["OPENMPI_4TH"]="no" + CONFIG_CACHE["GPUVER"]="no" + CONFIG_CACHE["MPICH_DEVICE"]="ch4" + CONFIG_CACHE["TARGET_CPU"]="native" + CONFIG_CACHE["LOG_LINES"]="200" + CONFIG_CACHE["show_help"]="false" + CONFIG_CACHE["DOWNLOADER_FLAGS"]="" + + # Version strategy defaults (NEW) + CONFIG_CACHE["VERSION_STRATEGY"]="main" + + # Defaults for CRAY Linux Environment (following original script logic) + if [[ -n "${CRAY_LD_LIBRARY_PATH}" ]]; then + CONFIG_CACHE["enable_cray"]="__TRUE__" + CONFIG_CACHE["MATH_MODE"]="cray" + # Default MPI used by CLE is assumed to be MPICH, in any case + # do not use the installers for the MPI libraries + CONFIG_CACHE["with_mpich"]="__DONTUSE__" + CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + CONFIG_CACHE["MPI_MODE"]="mpich" + # set default value for some installers appropriate for CLE + CONFIG_CACHE["with_gcc"]="__DONTUSE__" + # Reset math library defaults for CRAY environment + CONFIG_CACHE["with_mkl"]="__DONTUSE__" + CONFIG_CACHE["with_aocl"]="__DONTUSE__" + CONFIG_CACHE["with_openblas"]="__DONTUSE__" + fi + + return 0 +} + +# Helper function to read --enable-* options +# Usage: read_enable_option "--enable-feature=value" +read_enable_option() { + local arg="$1" + if [[ "$arg" == *"="* ]]; then + local value="${arg#*=}" + case "$value" in + yes|true|1) echo "__TRUE__" ;; + no|false|0) echo "__FALSE__" ;; + *) echo "__INVALID__" ;; + esac + else + echo "__TRUE__" + fi +} + +# Helper function to read --with-* options +# Usage: read_with_option "--with-package=value" +read_with_option() { + local arg="$1" + if [[ "$arg" == *"="* ]]; then + local value="${arg#*=}" + case "$value" in + install) echo "__INSTALL__" ;; + system) echo "__SYSTEM__" ;; + no) echo "__DONTUSE__" ;; + *) echo "$value" ;; # Allow custom paths + esac + else + echo "__INSTALL__" + fi +} + +# Validate configuration +# Usage: config_validate +config_validate() { + # Validate numeric values + if [[ -n "${CONFIG_CACHE[NPROCS_OVERWRITE]}" ]]; then + if ! [[ "${CONFIG_CACHE[NPROCS_OVERWRITE]}" =~ ^[0-9]+$ ]]; then + report_error ${LINENO} "Invalid number of processes: ${CONFIG_CACHE[NPROCS_OVERWRITE]}" + exit 1 + fi + fi + + if ! [[ "${CONFIG_CACHE[LOG_LINES]}" =~ ^[0-9]+$ ]]; then + report_error ${LINENO} "Invalid log lines value: ${CONFIG_CACHE[LOG_LINES]}" + exit 1 + fi + + # Validate GPU version - support only numeric formats + local gpu_ver="${CONFIG_CACHE[GPUVER]}" + if [[ "$gpu_ver" != "no" ]]; then + # Set ARCH_NUM based on GPUVER (remove decimal point for numeric versions) + local arch_num="${gpu_ver//.}" + + # Check if it's a valid numeric format (like 8.0, 70, 80, etc.) + if [[ "$arch_num" =~ ^[1-9][0-9]*$ ]]; then + CONFIG_CACHE["ARCH_NUM"]="$arch_num" + else + report_error ${LINENO} "Invalid GPU version: $gpu_ver. Supported formats: numeric with decimal (6.0, 7.0, 8.0, 8.9, etc.) or numeric without decimal (60, 70, 80, 89, etc.)" + exit 1 + fi + else + CONFIG_CACHE["ARCH_NUM"]="no" + fi + + return 0 +} + +# Apply environment variable processing logic (from original script lines 668-744) +# Usage: config_apply_env_logic +config_apply_env_logic() { + # Compiler conflicts (from original script L668-677) + if [ "${CONFIG_CACHE[with_intel]}" != "__DONTUSE__" ] && [ "${CONFIG_CACHE[with_gcc]}" = "__INSTALL__" ]; then + echo "You have chosen to use the Intel compiler, therefore the installation of the GNU compiler will be skipped." + CONFIG_CACHE["with_gcc"]="__SYSTEM__" + fi + if [ "${CONFIG_CACHE[with_amd]}" != "__DONTUSE__" ] && [ "${CONFIG_CACHE[with_gcc]}" = "__INSTALL__" ]; then + echo "You have chosen to use the AMD compiler, therefore the installation of the GNU compiler will be skipped." + CONFIG_CACHE["with_gcc"]="__SYSTEM__" + fi + if [ "${CONFIG_CACHE[with_amd]}" != "__DONTUSE__" ] && [ "${CONFIG_CACHE[with_intel]}" != "__DONTUSE__" ]; then + report_error "You have chosen to use the AMD and the Intel compiler to compile dependent packages. Select only one compiler." + exit 1 + fi + + # MPI library conflicts (from original script L679-710) + if [ "${CONFIG_CACHE[MPI_MODE]}" = "no" ]; then + if [ "${CONFIG_CACHE[with_scalapack]}" != "__DONTUSE__" ]; then + echo "Not using MPI, so scalapack is disabled." + CONFIG_CACHE["with_scalapack"]="__DONTUSE__" + fi + if [ "${CONFIG_CACHE[with_elpa]}" != "__DONTUSE__" ]; then + echo "Not using MPI, so ELPA is disabled." + CONFIG_CACHE["with_elpa"]="__DONTUSE__" + fi + else + # if gcc is installed, then mpi needs to be installed too + if [ "${CONFIG_CACHE[with_gcc]}" = "__INSTALL__" ]; then + echo "You have chosen to install the GNU compiler, therefore MPI libraries have to be installed too" + case ${CONFIG_CACHE[MPI_MODE]} in + mpich) + CONFIG_CACHE["with_mpich"]="__INSTALL__" + CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + ;; + openmpi) + CONFIG_CACHE["with_mpich"]="__DONTUSE__" + CONFIG_CACHE["with_openmpi"]="__INSTALL__" + ;; + esac + echo "and the use of the Intel compiler and Intel MPI will be disabled." + CONFIG_CACHE["with_intel"]="__DONTUSE__" + CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + fi + # Enable only one MPI implementation + case ${CONFIG_CACHE[MPI_MODE]} in + mpich) + CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + openmpi) + CONFIG_CACHE["with_mpich"]="__DONTUSE__" + CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + intelmpi) + CONFIG_CACHE["with_mpich"]="__DONTUSE__" + CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + ;; + esac + fi + + # If MATH_MODE is mkl, then openblas, scalapack and fftw is not needed + # QuantumMisaka in 2023-09-17 + if [ "${CONFIG_CACHE[MATH_MODE]}" = "mkl" ]; then + if [ "${CONFIG_CACHE[with_openblas]}" != "__DONTUSE__" ]; then + echo "Using MKL, so openblas is disabled." + CONFIG_CACHE["with_openblas"]="__DONTUSE__" + fi + if [ "${CONFIG_CACHE[with_scalapack]}" != "__DONTUSE__" ]; then + echo "Using MKL, so scalapack is disabled." + CONFIG_CACHE["with_scalapack"]="__DONTUSE__" + fi + if [ "${CONFIG_CACHE[with_fftw]}" != "__DONTUSE__" ]; then + echo "Using MKL, so fftw is disabled." + CONFIG_CACHE["with_fftw"]="__DONTUSE__" + fi + fi + + # Select the correct compute number based on the GPU architecture + # QuantumMisaka in 2025-03-19 + local gpuver="${CONFIG_CACHE[GPUVER]}" + if [ "$gpuver" != "no" ]; then + export ARCH_NUM="${gpuver//.}" + CONFIG_CACHE["ARCH_NUM"]="$ARCH_NUM" + else + export ARCH_NUM="no" + CONFIG_CACHE["ARCH_NUM"]="no" + fi + + # If CUDA or HIP are enabled, make sure the GPU version has been defined. + if [ "${CONFIG_CACHE[enable_cuda]}" = "__TRUE__" ] || [ "${CONFIG_CACHE[enable_hip]}" = "__TRUE__" ]; then + if [ "${CONFIG_CACHE[GPUVER]}" = "no" ]; then + report_error "Please choose GPU architecture to compile for with --gpu-ver" + exit 1 + fi + if [[ "$ARCH_NUM" =~ ^[1-9][0-9]*$ ]] || [ "$ARCH_NUM" = "no" ]; then + echo "Notice: GPU compilation is enabled, and GPU compatibility is set via --gpu-ver to sm_${ARCH_NUM}." + else + report_error ${LINENO} \ + "When GPU compilation is enabled, the --gpu-ver variable should be properly set regarding to GPU compatibility. For check your GPU compatibility, visit https://developer.nvidia.com/cuda-gpus. For example: A100 -> 8.0 (or 80), V100 -> 7.0 (or 70), 4090 -> 8.9 (or 89)" + exit 1 + fi + fi + + # ABACUS itself and some dependencies require cmake. + if [ "${CONFIG_CACHE[with_cmake]}" = "__DONTUSE__" ]; then + report_error "CMake is required for ABACUS and some dependencies. Please enable it." + exit 1 + fi +} + +# Export configuration to environment variables +# Usage: config_export_to_env +config_export_to_env() { + # Apply environment variable processing logic first + config_apply_env_logic + + # Export all configuration values as environment variables + for key in "${!CONFIG_CACHE[@]}"; do + export "$key"="${CONFIG_CACHE[$key]}" + done + + # Export package list variables + export tool_list + export mpi_list + export math_list + export lib_list + export package_list +} + +# Get configuration value +# Usage: config_get "key" +config_get() { + local key="$1" + echo "${CONFIG_CACHE[$key]}" +} + +# Set configuration value +# Usage: config_set "key" "value" +config_set() { + local key="$1" + local value="$2" + CONFIG_CACHE["$key"]="$value" +} + +# Check if configuration key exists +# Usage: config_has "key" +config_has() { + local key="$1" + [[ -n "${CONFIG_CACHE[$key]}" ]] +} + +# Print configuration summary +# Usage: config_print_summary +config_print_summary() { + echo "Configuration Summary:" + echo "=====================" + echo "MPI Mode: ${CONFIG_CACHE[MPI_MODE]}" + echo "Math Mode: ${CONFIG_CACHE[MATH_MODE]}" + echo "Target CPU: ${CONFIG_CACHE[TARGET_CPU]}" + echo "GPU Version: ${CONFIG_CACHE[GPUVER]}" + echo "Parallel Jobs: ${CONFIG_CACHE[NPROCS_OVERWRITE]:-$(get_nprocs)}" + echo "Dry Run: ${CONFIG_CACHE[dry_run]}" + echo "" + + echo "Package Configuration:" + echo "=====================" + for pkg in ${package_list}; do + local status="${CONFIG_CACHE[with_${pkg}]}" + if [[ "$status" != "__DONTUSE__" ]]; then + printf "%-15s: %s\n" "$pkg" "$status" + fi + done + echo "" +} + +# Parse command line arguments (New unified version) +# Usage: config_parse_arguments "$@" +config_parse_arguments() { + local show_help=false + local show_version=false + + while [[ $# -gt 0 ]]; do + case $1 in + # Help and version + -h|--help) + show_help=true + shift + ;; + --version) + show_version=true + shift + ;; + --version-info) + if [[ -n "$2" && "$2" != -* ]]; then + CONFIG_CACHE["show_version_info"]="$2" + shift 2 + else + CONFIG_CACHE["show_version_info"]="all" + shift + fi + ;; + + # Build options + -j) + if [[ -n "$2" && "$2" =~ ^[0-9]+$ ]]; then + CONFIG_CACHE["NPROCS_OVERWRITE"]="$2" + shift 2 + else + report_error $LINENO "-j requires a number argument" + return 1 + fi + ;; + --dry-run) + CONFIG_CACHE["dry_run"]="__TRUE__" + shift + ;; + --pack-run) + CONFIG_CACHE["PACK_RUN"]="__TRUE__" + shift + ;; + + # Package version selection - Support multiple package:version pairs + --package-version) + local package_version_args="" + local processed_count=0 + + # Handle --package-version=value format + if [[ "$1" =~ ^--package-version=(.+)$ ]]; then + package_version_args="${BASH_REMATCH[1]}" + shift + processed_count=1 + else + # Collect all consecutive non-option arguments + shift # Skip --package-version + while [[ $# -gt 0 && "$1" != -* ]]; do + if [[ -n "$package_version_args" ]]; then + package_version_args="$package_version_args $1" + else + package_version_args="$1" + fi + shift + ((processed_count++)) + done + fi + + # Validate we have at least one argument + if [[ -z "$package_version_args" ]]; then + report_error $LINENO "--package-version requires at least one package:version argument" + return 1 + fi + + # Process each package:version pair + local pair_count=0 + for pair in $package_version_args; do + if [[ "$pair" =~ ^([a-zA-Z0-9_]+):(main|alt)$ ]]; then + local pkg="${BASH_REMATCH[1]}" + local ver="${BASH_REMATCH[2]}" + CONFIG_CACHE["PACKAGE_VERSION_${pkg^^}"]="$ver" + ((pair_count++)) + else + report_error $LINENO "Invalid package version format: '$pair'. Use format 'package:version' (e.g., openmpi:alt, openblas:main)" + return 1 + fi + done + + # Report successful processing + if [[ $pair_count -gt 1 ]]; then + echo "INFO: Processed $pair_count package version overrides" + fi + ;; + + # Configuration file + --config-file) + if [[ -n "$2" && "$2" != -* ]]; then + config_load_from_file "$2" + shift 2 + else + report_error $LINENO "--config-file requires an argument" + return 1 + fi + ;; + + # Mode selections - Support both --mode=value and --mode value formats + --mpi-mode|--mpi-mode=*) + local mpi_value="" + # Handle --mpi-mode=value format + if [[ "$1" =~ ^--mpi-mode=(.+)$ ]]; then + mpi_value="${BASH_REMATCH[1]}" + shift + elif [[ -n "$2" && "$2" != -* ]]; then + mpi_value="$2" + shift 2 + else + report_error $LINENO "--mpi-mode requires an argument" + return 1 + fi + + case "$mpi_value" in + mpich|openmpi|intelmpi|no) + CONFIG_CACHE["MPI_MODE"]="$mpi_value" + ;; + *) + report_error $LINENO "Invalid MPI mode: $mpi_value. Valid options: mpich, openmpi, intelmpi, no" + return 1 + ;; + esac + ;; + --math-mode|--math-mode=*) + local math_value="" + # Handle --math-mode=value format + if [[ "$1" =~ ^--math-mode=(.+)$ ]]; then + math_value="${BASH_REMATCH[1]}" + shift + elif [[ -n "$2" && "$2" != -* ]]; then + math_value="$2" + shift 2 + else + report_error $LINENO "--math-mode requires an argument" + return 1 + fi + + case "$math_value" in + mkl|aocl|openblas|cray|no) + CONFIG_CACHE["MATH_MODE"]="$math_value" + # Apply automatic parameter settings for specific math modes + case "$math_value" in + aocl) + CONFIG_CACHE["with_aocl"]="__SYSTEM__" + CONFIG_CACHE["with_fftw"]="__SYSTEM__" + CONFIG_CACHE["with_scalapack"]="__SYSTEM__" + ;; + esac + ;; + *) + report_error $LINENO "Invalid math mode: $math_value. Valid options: mkl, aocl, openblas, cray, no" + return 1 + ;; + esac + ;; + + # Package options + --with-*) + local option="${1#--with-}" + local value="__INSTALL__" + + # Handle --with-package=value format + if [[ "$option" =~ ^(.+)=(.+)$ ]]; then + option="${BASH_REMATCH[1]}" + value="${BASH_REMATCH[2]}" + fi + + # Special handling for --with-mpich-device=* (must be handled before general mpich) + if [[ "$option" == "mpich-device" ]]; then + CONFIG_CACHE["MPICH_DEVICE"]="$value" + CONFIG_CACHE["MPI_MODE"]="mpich" + shift + continue + fi + + # Convert value to internal format (matching original read_with function) + case "$value" in + system) + value="__SYSTEM__" + ;; + install) + value="__INSTALL__" + ;; + no) + value="__DONTUSE__" + ;; + __SYSTEM__|__INSTALL__|__DONTUSE__) + # Already in internal format, keep as is + ;; + *) + # For custom paths, expand ~ to $HOME (matching original behavior) + value="${value//\~/$HOME}" + ;; + esac + + # Handle special cases with specific processing logic + case "$option" in + 4th-openmpi) + # Handle --with-4th-openmpi parameter (only yes/no options) + case "$value" in + "yes"|"install"|"__INSTALL__") + CONFIG_CACHE["OPENMPI_4TH"]="yes" + ;; + "no"|"__DONTUSE__") + CONFIG_CACHE["OPENMPI_4TH"]="no" + ;; + *) + # Default to "no" for any other values + CONFIG_CACHE["OPENMPI_4TH"]="no" + ;; + esac + ;; + mpich) + CONFIG_CACHE["with_mpich"]="$value" + USER_EXPLICIT_MPI["with_mpich"]="true" # Mark as explicitly set by user + # Set MPI_MODE if not disabled + if [ "$value" != "__DONTUSE__" ]; then + CONFIG_CACHE["MPI_MODE"]="mpich" + fi + ;; + openmpi) + CONFIG_CACHE["with_openmpi"]="$value" + USER_EXPLICIT_MPI["with_openmpi"]="true" # Mark as explicitly set by user + # Set MPI_MODE if not disabled + if [ "$value" != "__DONTUSE__" ]; then + CONFIG_CACHE["MPI_MODE"]="openmpi" + fi + ;; + intelmpi) + CONFIG_CACHE["with_intelmpi"]="$value" + USER_EXPLICIT_MPI["with_intelmpi"]="true" # Mark as explicitly set by user + # Set MPI_MODE if not disabled + if [ "$value" != "__DONTUSE__" ]; then + CONFIG_CACHE["MPI_MODE"]="intelmpi" + fi + ;; + intel-classic) + # Special handling for intel-classic: only accepts yes/no values + case "$value" in + "__INSTALL__"|""|"__DONTUSE__") + value="no" # Default to "no" + ;; + "yes") + value="yes" + ;; + "no") + value="no" + ;; + *) + report_error $LINENO "Invalid value '$value' for --with-intel-classic. Only 'yes' or 'no' are allowed." "CONFIG_ERROR" + return 1 + ;; + esac + CONFIG_CACHE["intel_classic"]="$value" + ;; + intel-mpi-clas*) + # Special handling for intel-mpi-classic: only accepts yes/no values + case "$value" in + "__INSTALL__"|""|"__DONTUSE__") + value="no" # Default to "no" + ;; + "yes") + value="yes" + ;; + "no") + value="no" + ;; + *) + report_error $LINENO "Invalid value '$value' for --with-intel-mpi-classic. Only 'yes' or 'no' are allowed." "CONFIG_ERROR" + return 1 + ;; + esac + CONFIG_CACHE["INTELMPI_CLASSIC"]="$value" + ;; + intel) + CONFIG_CACHE["with_intel"]="$value" + ;; + ifx) + # Special handling for ifx: only accepts yes/no values + case "$value" in + "__INSTALL__"|""|"__DONTUSE__") + value="no" # Default to "no" + ;; + "yes") + value="yes" + ;; + "no") + value="no" + ;; + *) + report_error $LINENO "Invalid value '$value' for --with-ifx. Only 'yes' or 'no' are allowed." "CONFIG_ERROR" + return 1 + ;; + esac + CONFIG_CACHE["WITH_IFX"]="$value" + ;; + amd) + CONFIG_CACHE["with_amd"]="$value" + ;; + flang) + # Special handling for flang: only accepts yes/no values + case "$value" in + "__INSTALL__"|""|"__DONTUSE__") + value="no" # Default to "no" + ;; + "yes") + value="yes" + ;; + "no") + value="no" + ;; + *) + report_error $LINENO "Invalid value '$value' for --with-flang. Only 'yes' or 'no' are allowed." "CONFIG_ERROR" + return 1 + ;; + esac + CONFIG_CACHE["WITH_FLANG"]="$value" + ;; + aocl) + CONFIG_CACHE["with_aocl"]="$value" + USER_EXPLICIT_MATH["with_aocl"]="true" # Mark as explicitly set by user + ;; + mkl) + CONFIG_CACHE["with_mkl"]="$value" + USER_EXPLICIT_MATH["with_mkl"]="true" # Mark as explicitly set by user + # Set MATH_MODE if not disabled + if [ "$value" != "__DONTUSE__" ]; then + CONFIG_CACHE["MATH_MODE"]="mkl" + fi + ;; + openblas) + CONFIG_CACHE["with_openblas"]="$value" + USER_EXPLICIT_MATH["with_openblas"]="true" # Mark as explicitly set by user + # Set MATH_MODE if not disabled + if [ "$value" != "__DONTUSE__" ]; then + CONFIG_CACHE["MATH_MODE"]="openblas" + fi + ;; + fftw) + CONFIG_CACHE["with_fftw"]="$value" + USER_EXPLICIT_MATH["with_fftw"]="true" # Mark as explicitly set by user + ;; + scalapack) + CONFIG_CACHE["with_scalapack"]="$value" + USER_EXPLICIT_MATH["with_scalapack"]="true" # Mark as explicitly set by user + ;; + *) + # Convert to standard format for all other options + option="${option,,}" # Convert to lowercase + CONFIG_CACHE["with_${option}"]="$value" + ;; + esac + shift + ;; + + # Enable options + --enable-*) + local option="${1#--enable-}" + local value="__TRUE__" + + # Handle --enable-feature=value format + if [[ "$option" =~ ^(.+)=(.+)$ ]]; then + option="${BASH_REMATCH[1]}" + value="${BASH_REMATCH[2]}" + case "$value" in + yes|true|1) value="__TRUE__" ;; + no|false|0) value="__FALSE__" ;; + *) + report_error $LINENO "Invalid value for --enable-${option}: $value. Use yes/no" + return 1 + ;; + esac + fi + + CONFIG_CACHE["enable_${option}"]="$value" + shift + ;; + + # GPU version + --gpu-ver|--gpu-ver=*) + local gpu_value="" + # Handle --gpu-ver=value format + if [[ "$1" =~ ^--gpu-ver=(.+)$ ]]; then + gpu_value="${BASH_REMATCH[1]}" + shift + elif [[ -n "$2" && "$2" != -* ]]; then + gpu_value="$2" + shift 2 + else + report_error $LINENO "--gpu-ver requires an argument" + return 1 + fi + CONFIG_CACHE["GPUVER"]="$gpu_value" + ;; + + # Target CPU + --target-cpu|--target-cpu=*) + local cpu_value="" + # Handle --target-cpu=value format + if [[ "$1" =~ ^--target-cpu=(.+)$ ]]; then + cpu_value="${BASH_REMATCH[1]}" + shift + elif [[ -n "$2" && "$2" != -* ]]; then + cpu_value="$2" + shift 2 + else + report_error $LINENO "--target-cpu requires an argument" + return 1 + fi + CONFIG_CACHE["TARGET_CPU"]="$cpu_value" + ;; + + # Log lines + --log-lines|--log-lines=*) + local log_value="" + # Handle --log-lines=value format + if [[ "$1" =~ ^--log-lines=(.+)$ ]]; then + log_value="${BASH_REMATCH[1]}" + shift + elif [[ -n "$2" && "$2" =~ ^[0-9]+$ ]]; then + log_value="$2" + shift 2 + else + report_error $LINENO "--log-lines requires a number argument" + return 1 + fi + + if [[ "$log_value" =~ ^[0-9]+$ ]]; then + CONFIG_CACHE["LOG_LINES"]="$log_value" + else + report_error $LINENO "--log-lines requires a number argument" + return 1 + fi + ;; + + # Skip system checks + --skip-system-checks) + CONFIG_CACHE["skip_system_checks"]="__TRUE__" + shift + ;; + + # Positional arguments (ignored for now) + *) + shift + ;; + esac + done + + # Handle help and version requests + if [[ "$show_help" == true ]]; then + CONFIG_CACHE["show_help"]="__TRUE__" + return 0 + fi + + if [[ "$show_version" == true ]]; then + CONFIG_CACHE["show_version"]="__TRUE__" + return 0 + fi + + return 0 +} + +# Initialize configuration with command line arguments +# Usage: config_init "$@" +config_init() { + # Set defaults first + config_set_defaults + + # Initialize version helper to ensure VERSION_STRATEGY defaults are set + if command -v version_helper_init > /dev/null 2>&1; then + version_helper_init + fi + + # Load configuration from file (if available) - this will override defaults + config_load_from_file + + # Apply mode-based configurations from file - this will override defaults + config_apply_modes_from_file + + # Parse command line arguments - this will override file settings + config_parse_arguments "$@" + + # Apply mode-based configurations from command line + config_apply_modes + + # Validate configuration + config_validate + + return 0 +} + +# Apply configuration based on modes +# Usage: config_apply_modes +config_apply_modes() { + # Apply MPI mode settings (with output for user feedback) + local mpi_mode="${CONFIG_CACHE[MPI_MODE]}" + if [[ -n "$mpi_mode" ]]; then + if command -v ui_info &> /dev/null; then + ui_info "⚙️ Configuring MPI mode: $mpi_mode" + else + echo "Applying MPI mode: $mpi_mode" + fi + case "$mpi_mode" in + mpich) + # Only override if user hasn't explicitly set these values via command line + [[ -z "${USER_EXPLICIT_MPI[with_mpich]}" ]] && CONFIG_CACHE["with_mpich"]="__INSTALL__" + [[ -z "${USER_EXPLICIT_MPI[with_openmpi]}" ]] && CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MPI[with_intelmpi]}" ]] && CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + openmpi) + [[ -z "${USER_EXPLICIT_MPI[with_mpich]}" ]] && CONFIG_CACHE["with_mpich"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MPI[with_openmpi]}" ]] && CONFIG_CACHE["with_openmpi"]="__INSTALL__" + [[ -z "${USER_EXPLICIT_MPI[with_intelmpi]}" ]] && CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + intelmpi) + [[ -z "${USER_EXPLICIT_MPI[with_mpich]}" ]] && CONFIG_CACHE["with_mpich"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MPI[with_openmpi]}" ]] && CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MPI[with_intelmpi]}" ]] && CONFIG_CACHE["with_intelmpi"]="__INSTALL__" + ;; + no) + [[ -z "${USER_EXPLICIT_MPI[with_mpich]}" ]] && CONFIG_CACHE["with_mpich"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MPI[with_openmpi]}" ]] && CONFIG_CACHE["with_openmpi"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MPI[with_intelmpi]}" ]] && CONFIG_CACHE["with_intelmpi"]="__DONTUSE__" + ;; + esac + fi + + # Apply math mode settings (with output for user feedback) + local math_mode="${CONFIG_CACHE[MATH_MODE]}" + if [[ -n "$math_mode" ]]; then + if command -v ui_info &> /dev/null; then + ui_info "🧮 Configuring Math mode: $math_mode" + else + echo "Applying Math mode: $math_mode" + fi + case "$math_mode" in + mkl) + [[ -z "${USER_EXPLICIT_MATH[with_mkl]}" ]] && CONFIG_CACHE["with_mkl"]="__SYSTEM__" + [[ -z "${USER_EXPLICIT_MATH[with_aocl]}" ]] && CONFIG_CACHE["with_aocl"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_openblas]}" ]] && CONFIG_CACHE["with_openblas"]="__DONTUSE__" + # MKL provides FFTW and ScaLAPACK, so disable them (original logic) + [[ -z "${USER_EXPLICIT_MATH[with_fftw]}" ]] && CONFIG_CACHE["with_fftw"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_scalapack]}" ]] && CONFIG_CACHE["with_scalapack"]="__DONTUSE__" + ;; + aocl) + [[ -z "${USER_EXPLICIT_MATH[with_mkl]}" ]] && CONFIG_CACHE["with_mkl"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_aocl]}" ]] && CONFIG_CACHE["with_aocl"]="__SYSTEM__" + [[ -z "${USER_EXPLICIT_MATH[with_openblas]}" ]] && CONFIG_CACHE["with_openblas"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_fftw]}" ]] && CONFIG_CACHE["with_fftw"]="__SYSTEM__" + [[ -z "${USER_EXPLICIT_MATH[with_scalapack]}" ]] && CONFIG_CACHE["with_scalapack"]="__SYSTEM__" + ;; + openblas) + [[ -z "${USER_EXPLICIT_MATH[with_mkl]}" ]] && CONFIG_CACHE["with_mkl"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_aocl]}" ]] && CONFIG_CACHE["with_aocl"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_openblas]}" ]] && CONFIG_CACHE["with_openblas"]="__INSTALL__" + ;; + cray) + [[ -z "${USER_EXPLICIT_MATH[with_mkl]}" ]] && CONFIG_CACHE["with_mkl"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_aocl]}" ]] && CONFIG_CACHE["with_aocl"]="__DONTUSE__" + [[ -z "${USER_EXPLICIT_MATH[with_openblas]}" ]] && CONFIG_CACHE["with_openblas"]="__DONTUSE__" + ;; + esac + fi +} \ No newline at end of file diff --git a/toolchain/scripts/lib/config_validator.sh b/toolchain/scripts/lib/config_validator.sh new file mode 100644 index 0000000000..648a6ffbe9 --- /dev/null +++ b/toolchain/scripts/lib/config_validator.sh @@ -0,0 +1,531 @@ +#!/bin/bash + +# ============================================================================ +# ABACUS Toolchain Configuration Validator +# ============================================================================ +# Validates configuration settings and detects potential conflicts +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# ============================================================================ + +# Global validation state +VALIDATION_ERRORS=() +VALIDATION_WARNINGS=() +VALIDATION_ERROR_GROUPS=() +VALIDATION_WARNING_GROUPS=() +VALIDATION_INITIALIZED=false + +# Initialize configuration validator +# Usage: config_validator_init +config_validator_init() { + if [[ "$VALIDATION_INITIALIZED" == "true" ]]; then + return 0 + fi + + VALIDATION_ERRORS=() + VALIDATION_WARNINGS=() + VALIDATION_ERROR_GROUPS=() + VALIDATION_WARNING_GROUPS=() + VALIDATION_INITIALIZED=true + + return 0 +} + +# Add validation error +# Usage: add_validation_error "error message" +add_validation_error() { + local message="$1" + VALIDATION_ERRORS+=("ERROR: $message") +} + +# Add validation warning +# Usage: add_validation_warning "warning message" +add_validation_warning() { + local message="$1" + VALIDATION_WARNINGS+=("WARNING: $message") +} + +# Start a new error group (represents one logical error) +# Usage: start_error_group "group_name" +start_error_group() { + local group_name="$1" + VALIDATION_ERROR_GROUPS+=("$group_name") +} + +# Start a new warning group (represents one logical warning) +# Usage: start_warning_group "group_name" +start_warning_group() { + local group_name="$1" + VALIDATION_WARNING_GROUPS+=("$group_name") +} + +# Add validation error group (complete error with all details) +# Usage: add_validation_error_group "group_name" "line1" "line2" ... +add_validation_error_group() { + local group_name="$1" + shift + + # Add the group to track unique errors + VALIDATION_ERROR_GROUPS+=("$group_name") + + # Add all error lines + for line in "$@"; do + VALIDATION_ERRORS+=("ERROR: $line") + done +} + +# Add validation warning group (complete warning with all details) +# Usage: add_validation_warning_group "group_name" "line1" "line2" ... +add_validation_warning_group() { + local group_name="$1" + shift + + # Add the group to track unique warnings + VALIDATION_WARNING_GROUPS+=("$group_name") + + # Add all warning lines + for line in "$@"; do + VALIDATION_WARNINGS+=("WARNING: $line") + done +} + +# Add validation info (for successful validations) +# Usage: add_validation_info "info message" +add_validation_info() { + local message="$1" + # Info messages are displayed immediately during validation + echo "INFO: $message" +} + +# Check for conflicting math libraries +# Usage: validate_math_libraries +validate_math_libraries() { + local math_libs_enabled=0 + local enabled_libs=() + + # Check which math libraries are enabled + for lib in mkl aocl openblas; do + if [[ "${CONFIG_CACHE[with_${lib}]}" == "__INSTALL__" || "${CONFIG_CACHE[with_${lib}]}" == "__SYSTEM__" ]]; then + math_libs_enabled=$((math_libs_enabled + 1)) + enabled_libs+=("$lib") + fi + done + + # Validate math library configuration + if [[ $math_libs_enabled -gt 1 ]]; then + add_validation_error "Multiple math libraries enabled: ${enabled_libs[*]}. Only one should be active." + elif [[ $math_libs_enabled -eq 0 ]]; then + add_validation_warning "No math library enabled. This may cause compilation issues." + fi + + # Check MKL-specific conflicts + if [[ "${CONFIG_CACHE[with_mkl]}" == "__SYSTEM__" || "${CONFIG_CACHE[with_mkl]}" == "__INSTALL__" ]]; then + if [[ "${CONFIG_CACHE[with_fftw]}" == "__INSTALL__" ]]; then + add_validation_warning "MKL includes FFTW interface. Consider setting FFTW to __DONTUSE__ or __SYSTEM__." + fi + if [[ "${CONFIG_CACHE[with_scalapack]}" == "__INSTALL__" ]]; then + add_validation_warning "MKL includes ScaLAPACK. Consider setting ScaLAPACK to __DONTUSE__ or __SYSTEM__." + fi + fi +} + +# Check for conflicting MPI implementations +# Usage: validate_mpi_implementations +validate_mpi_implementations() { + local mpi_libs_enabled=0 + local enabled_mpis=() + + # Check which MPI implementations are enabled + for mpi in mpich openmpi intelmpi; do + if [[ "${CONFIG_CACHE[with_${mpi}]}" == "__INSTALL__" || "${CONFIG_CACHE[with_${mpi}]}" == "__SYSTEM__" ]]; then + mpi_libs_enabled=$((mpi_libs_enabled + 1)) + enabled_mpis+=("$mpi") + fi + done + + # Validate MPI configuration + if [[ $mpi_libs_enabled -gt 1 ]]; then + add_validation_error "Multiple MPI implementations enabled: ${enabled_mpis[*]}. Only one should be active." + elif [[ $mpi_libs_enabled -eq 0 ]]; then + add_validation_warning "No MPI implementation enabled. This may limit parallel functionality." + fi +} + +# Check for compiler consistency +# Usage: validate_compiler_consistency +validate_compiler_consistency() { + local compilers_enabled=0 + local enabled_compilers=() + + # Check which compilers are enabled + for compiler in gcc intel amd; do + if [[ "${CONFIG_CACHE[with_${compiler}]}" == "__INSTALL__" || "${CONFIG_CACHE[with_${compiler}]}" == "__SYSTEM__" ]]; then + compilers_enabled=$((compilers_enabled + 1)) + enabled_compilers+=("$compiler") + fi + done + + # Validate compiler configuration + if [[ $compilers_enabled -gt 1 ]]; then + add_validation_warning "Multiple compilers enabled: ${enabled_compilers[*]}. This may cause compatibility issues." + elif [[ $compilers_enabled -eq 0 ]]; then + add_validation_error "No compiler enabled. At least one compiler must be available." + fi + + # Check Intel-specific dependencies + if [[ "${CONFIG_CACHE[with_intel]}" == "__SYSTEM__" || "${CONFIG_CACHE[with_intel]}" == "__INSTALL__" ]]; then + if [[ "${CONFIG_CACHE[with_mkl]}" == "__DONTUSE__" ]]; then + add_validation_warning "Intel compiler is enabled but MKL is disabled. Consider enabling MKL for optimal performance." + fi + if [[ "${CONFIG_CACHE[with_intelmpi]}" == "__DONTUSE__" ]]; then + add_validation_warning "Intel compiler is enabled but Intel MPI is disabled. Consider using Intel MPI for better integration." + fi + fi +} + +# Check system requirements +# Usage: validate_system_requirements +validate_system_requirements() { + # ABACUS itself and some dependencies require cmake. + # Check cmake requirement based on configuration (mirrors original L768-772) + if [[ "${CONFIG_CACHE[with_cmake]}" == "__DONTUSE__" ]]; then + add_validation_error_group "cmake_required" \ + "CMake is required for ABACUS and some dependencies. Please enable it." \ + "" \ + "SOLUTION: Use '--with-cmake=install' to automatically install CMake:" \ + " ./install_abacus_toolchain_new.sh --with-cmake=install [other options]" + return + fi + + # Check for required system tools (excluding cmake which is handled above) + local required_tools=("make" "git" "wget" "tar" "gzip") + local missing_tools=() + + # Only check for system cmake if using system cmake + if [[ "${CONFIG_CACHE[with_cmake]}" == "__SYSTEM__" ]]; then + if ! command -v "cmake" &> /dev/null; then + missing_tools+=("cmake") + else + # Check cmake version if it exists + local cmake_min_version="3.16" + local cmake_version=$(cmake --version 2>/dev/null | head -n 1 | awk '{print $3}') + + if [[ -z "$cmake_version" ]]; then + add_validation_error "Failed to determine CMake version" + add_validation_error "This usually indicates a corrupted or non-standard CMake installation." + add_validation_error "" + add_validation_error "SOLUTION: Use '--with-cmake=install' to install a known-good CMake version:" + add_validation_error " ./install_abacus_toolchain_new.sh --with-cmake=install [other options]" + add_validation_error "" + add_validation_error "This will install CMake ${cmake_ver:-3.29.6} with proper version information." + return + fi + + # Extract major and minor version numbers for comparison + local cmake_major=$(echo "$cmake_version" | awk -F. '{print $1}') + local cmake_minor=$(echo "$cmake_version" | awk -F. '{print $2}') + local min_major=$(echo "$cmake_min_version" | awk -F. '{print $1}') + local min_minor=$(echo "$cmake_min_version" | awk -F. '{print $2}') + + # Validate version format + if ! [[ "$cmake_major" =~ ^[0-9]+$ ]] || ! [[ "$cmake_minor" =~ ^[0-9]+$ ]]; then + add_validation_error "Unable to parse CMake version from: $cmake_version" + add_validation_error "Expected format: X.Y.Z (e.g., 3.29.6), but got: $cmake_version" + add_validation_error "" + add_validation_error "SOLUTION: Use '--with-cmake=install' to install a standard CMake version:" + add_validation_error " ./install_abacus_toolchain_new.sh --with-cmake=install [other options]" + add_validation_error "" + add_validation_error "This will install CMake ${cmake_ver:-3.29.6} with standard version format." + return + fi + + # Compare versions (major.minor comparison) + local version_too_old=false + if [[ "$cmake_major" -lt "$min_major" ]]; then + version_too_old=true + elif [[ "$cmake_major" -eq "$min_major" ]] && [[ "$cmake_minor" -lt "$min_minor" ]]; then + version_too_old=true + fi + + if [[ "$version_too_old" == "true" ]]; then + add_validation_error "CMake version $cmake_version is too old" + add_validation_error "Minimum required: CMake $cmake_min_version or newer (ABACUS requirement)" + add_validation_error "" + add_validation_error_group "cmake_outdated" \ + "Your system CMake is outdated and may cause build failures." \ + "" \ + "SOLUTION: Use '--with-cmake=install' to install a modern CMake version:" \ + " ./install_abacus_toolchain_new.sh --with-cmake=install [other options]" \ + "" \ + "This will install CMake ${cmake_ver:-3.29.6} (>= $cmake_min_version) with full feature support." + return + fi + + # Success - add informational message + add_validation_info "System CMake validated: version $cmake_version (>= $cmake_min_version required)" + fi + fi + + for tool in "${required_tools[@]}"; do + if ! command -v "$tool" &> /dev/null; then + missing_tools+=("$tool") + fi + done + + if [[ ${#missing_tools[@]} -gt 0 ]]; then + # Check if cmake is in the missing tools and provide specific guidance + if [[ " ${missing_tools[*]} " =~ " cmake " ]]; then + add_validation_error_group "cmake_missing" \ + "CMake is not installed on your system" \ + "CMake is required for building ABACUS and its dependencies." \ + "" \ + "SOLUTION: Use '--with-cmake=install' to automatically install CMake:" \ + " ./install_abacus_toolchain_new.sh --with-cmake=install [other options]" \ + "" \ + "This will install CMake ${cmake_ver:-3.29.6} with full feature support." + + # Remove cmake from missing_tools to avoid duplicate error messages + local filtered_tools=() + for tool in "${missing_tools[@]}"; do + if [[ "$tool" != "cmake" ]]; then + filtered_tools+=("$tool") + fi + done + missing_tools=("${filtered_tools[@]}") + fi + + # Report other missing tools if any + if [[ ${#missing_tools[@]} -gt 0 ]]; then + add_validation_error "Missing required system tools: ${missing_tools[*]}" + fi + fi + + # Check for development packages (common names) + local dev_packages=("build-essential" "gcc" "g++" "gfortran") + local missing_dev=() + + for pkg in "${dev_packages[@]}"; do + if ! command -v "$pkg" &> /dev/null && ! dpkg -l | grep -q "$pkg" 2>/dev/null; then + missing_dev+=("$pkg") + fi + done + + if [[ ${#missing_dev[@]} -gt 0 ]]; then + add_validation_warning "Potentially missing development packages: ${missing_dev[*]}" + fi + + # Check GCC version consistency and minimum requirements + # This mirrors the logic from original install_abacus_toolchain.sh L636-667 + if [[ "${CONFIG_CACHE[with_gcc]}" != "__INSTALL__" ]]; then + local gcc_min_version=5 + + # Check if GCC tools are available + if ! command -v gcc &> /dev/null || ! command -v g++ &> /dev/null || ! command -v gfortran &> /dev/null; then + local missing_tools=() + ! command -v gcc &> /dev/null && missing_tools+=("gcc") + ! command -v g++ &> /dev/null && missing_tools+=("g++") + ! command -v gfortran &> /dev/null && missing_tools+=("gfortran") + + add_validation_error_group "gcc_incomplete" \ + "System GCC toolchain incomplete. Missing: ${missing_tools[*]}" \ + "" \ + "SOLUTION: Use '--with-gcc=install' to automatically download and install a complete GCC toolchain:" \ + " ./install_abacus_toolchain_new.sh --with-gcc=install [other options]" \ + "" \ + "This will install GCC ${gcc_ver:-13.2.0} with all required components (gcc, g++, gfortran)." + return + fi + + # Get versions of GCC components + local gcc_version=$(gcc --version 2>/dev/null | head -n 1 | awk '{print $NF}') + local gxx_version=$(g++ --version 2>/dev/null | head -n 1 | awk '{print $NF}') + local gfc_version=$(gfortran --version 2>/dev/null | head -n 1 | awk '{print $NF}') + + # Check if version extraction was successful + if [[ -z "$gcc_version" || -z "$gxx_version" || -z "$gfc_version" ]]; then + local failed_tools=() + [[ -z "$gcc_version" ]] && failed_tools+=("gcc") + [[ -z "$gxx_version" ]] && failed_tools+=("g++") + [[ -z "$gfc_version" ]] && failed_tools+=("gfortran") + + add_validation_error_group "gcc_version_failed" \ + "Failed to determine GCC toolchain versions for: ${failed_tools[*]}" \ + "This usually indicates corrupted or non-standard GCC installation." \ + "" \ + "SOLUTION: Use '--with-gcc=install' to install a known-good GCC version:" \ + " ./install_abacus_toolchain_new.sh --with-gcc=install [other options]" \ + "" \ + "This will install GCC ${gcc_ver:-13.2.0} with proper version information." + return + fi + + # Check version consistency + if [[ "$gcc_version" != "$gxx_version" ]] || [[ "$gcc_version" != "$gfc_version" ]]; then + add_validation_error_group "gcc_version_inconsistent" \ + "GCC toolchain versions are inconsistent:" \ + " gcc: $gcc_version" \ + " g++: $gxx_version" \ + " gfortran: $gfc_version" \ + "" \ + "All GCC components must have the same version for proper compilation." \ + "SOLUTION: Use '--with-gcc=install' to install a consistent GCC toolchain:" \ + " ./install_abacus_toolchain_new.sh --with-gcc=install [other options]" \ + "" \ + "This will install GCC ${gcc_ver:-13.2.0} with all components at the same version." + return + fi + + # Extract major version number + local gcc_major=$(echo "$gcc_version" | awk -F. '{print $1}') + + # Validate major version is numeric and meets minimum requirement + if ! [[ "$gcc_major" =~ ^[0-9]+$ ]]; then + add_validation_error_group "gcc_version_parse_failed" \ + "Unable to parse GCC major version from: $gcc_version" \ + "Expected format: X.Y.Z (e.g., 13.2.0), but got: $gcc_version" \ + "" \ + "SOLUTION: Use '--with-gcc=install' to install a standard GCC version:" \ + " ./install_abacus_toolchain_new.sh --with-gcc=install [other options]" \ + "" \ + "This will install GCC ${gcc_ver:-13.2.0} with standard version format." + return + fi + + if [[ "$gcc_major" -lt "$gcc_min_version" ]]; then + add_validation_error_group "gcc_version_too_old" \ + "GCC version $gcc_version is too old (major version: $gcc_major)" \ + "Minimum required: GCC $gcc_min_version.x or newer (ABACUS requirement)" \ + "" \ + "Your system GCC is outdated and may cause compilation failures." \ + "" \ + "SOLUTION: Use '--with-gcc=install' to install a modern GCC version:" \ + " ./install_abacus_toolchain_new.sh --with-gcc=install [other options]" \ + "" \ + "This will install GCC ${gcc_ver:-13.2.0} (>= $gcc_min_version.x) with full C++17/C++20 support." + return + fi + + # Success - add informational message + add_validation_info "System GCC toolchain validated: version $gcc_version (>= $gcc_min_version.x required)" + fi +} + +# Check for logical inconsistencies +# Usage: validate_logical_consistency +validate_logical_consistency() { + # Check if ELPA is enabled without ScaLAPACK + if [[ "${CONFIG_CACHE[with_elpa]}" == "__INSTALL__" || "${CONFIG_CACHE[with_elpa]}" == "__SYSTEM__" ]]; then + if [[ "${CONFIG_CACHE[with_scalapack]}" == "__DONTUSE__" ]]; then + # Check if MKL is enabled (MKL provides ScaLAPACK functionality) + if [[ "${CONFIG_CACHE[MATH_MODE]}" == "mkl" || "${CONFIG_CACHE[with_mkl]}" == "__SYSTEM__" || "${CONFIG_CACHE[with_mkl]}" == "__INSTALL__" ]]; then + # MKL provides ScaLAPACK, so this is acceptable + : + else + add_validation_error "ELPA requires ScaLAPACK but ScaLAPACK is disabled." + fi + fi + fi + + # Check if ScaLAPACK is enabled without MPI + if [[ "${CONFIG_CACHE[with_scalapack]}" == "__INSTALL__" || "${CONFIG_CACHE[with_scalapack]}" == "__SYSTEM__" ]]; then + local mpi_enabled=false + for mpi in mpich openmpi intelmpi; do + if [[ "${CONFIG_CACHE[with_${mpi}]}" == "__INSTALL__" || "${CONFIG_CACHE[with_${mpi}]}" == "__SYSTEM__" ]]; then + mpi_enabled=true + break + fi + done + + if [[ "$mpi_enabled" == "false" ]]; then + add_validation_warning "ScaLAPACK is enabled but no MPI implementation is active." + fi + fi + + # Check for GPU-related inconsistencies + if [[ "${CONFIG_CACHE[GPUVER]}" != "no" ]]; then + if [[ "${CONFIG_CACHE[with_elpa]}" == "__INSTALL__" ]]; then + add_validation_warning "GPU support is enabled. Ensure ELPA is compiled with GPU support." + fi + fi +} + +# Validate package versions compatibility +# Usage: validate_package_versions +validate_package_versions() { + # This is a placeholder for version compatibility checks + # In a real implementation, you would check for known incompatible version combinations + + # Example: Check for known problematic combinations + if [[ "${CONFIG_CACHE[with_gcc]}" == "__INSTALL__" ]]; then + local gcc_version="${gcc_ver:-unknown}" + if [[ "$gcc_version" == "unknown" ]]; then + add_validation_warning "GCC version not specified. Using default version." + fi + fi +} + +# Run all validation checks +# Usage: validate_configuration +validate_configuration() { + config_validator_init + + echo "Running configuration validation..." + + # Run all validation checks + validate_math_libraries + validate_mpi_implementations + validate_compiler_consistency + validate_system_requirements + validate_logical_consistency + validate_package_versions + + # Report results using error groups for accurate counting + local error_groups=${#VALIDATION_ERROR_GROUPS[@]} + local warning_groups=${#VALIDATION_WARNING_GROUPS[@]} + local total_issues=$((error_groups + warning_groups)) + + if [[ ${#VALIDATION_ERRORS[@]} -gt 0 ]]; then + echo "" + echo "Configuration Errors Found:" + echo "==========================" + for error in "${VALIDATION_ERRORS[@]}"; do + echo " $error" + done + fi + + if [[ ${#VALIDATION_WARNINGS[@]} -gt 0 ]]; then + echo "" + echo "Configuration Warnings:" + echo "======================" + for warning in "${VALIDATION_WARNINGS[@]}"; do + echo " $warning" + done + fi + + if [[ $total_issues -eq 0 ]]; then + echo "✓ Configuration validation passed with no issues." + return 0 + else + echo "" + echo "Configuration validation completed with $total_issues issue(s)." + echo " Errors: $error_groups" + echo " Warnings: $warning_groups" + + # Return error code if there are validation errors + if [[ $error_groups -gt 0 ]]; then + return 1 + else + return 0 + fi + fi +} + +# Check if validation should be skipped +# Usage: should_skip_validation +should_skip_validation() { + if [[ "${CONFIG_CACHE[SKIP_SYSTEM_CHECKS]}" == "true" ]]; then + echo "Skipping configuration validation (SKIP_SYSTEM_CHECKS=true)" + return 0 + fi + return 1 +} diff --git a/toolchain/scripts/lib/error_handler.sh b/toolchain/scripts/lib/error_handler.sh new file mode 100644 index 0000000000..322fb2a498 --- /dev/null +++ b/toolchain/scripts/lib/error_handler.sh @@ -0,0 +1,135 @@ +#!/bin/bash + +# ============================================================================ +# ABACUS Toolchain Error Handler +# ============================================================================ +# Provides error handling and reporting functions for the toolchain +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# ============================================================================ + +# Global error handling state +ERROR_HANDLER_INITIALIZED=false + +# Initialize error handler +# Usage: error_handler_init +error_handler_init() { + if [[ "$ERROR_HANDLER_INITIALIZED" == "true" ]]; then + return 0 + fi + + # Set up error trapping + set -e + trap 'error_handler ${LINENO}' ERR + + ERROR_HANDLER_INITIALIZED=true + return 0 +} + +# Main error handler function +# Usage: error_handler line_number [error_code] +error_handler() { + local line_number="$1" + local error_code="${2:-$?}" + local script_name="${SCRIPT_NAME:-${BASH_SOURCE[1]}}" + + echo "ERROR: Script failed at line $line_number in $script_name (exit code: $error_code)" >&2 + echo "Command that failed: ${BASH_COMMAND}" >&2 + + # Print stack trace + echo "Call stack:" >&2 + local frame=0 + while caller $frame; do + ((frame++)) + done | while read line func file; do + echo " at $func ($file:$line)" >&2 + done + + exit $error_code +} + +# Report error with context +# Usage: report_error line_number "error message" [context] +report_error() { + local line_number="" + local message="" + local context="" + + if [[ $# -gt 1 ]]; then + line_number="$1" + message="$2" + context="${3:-}" + else + message="$1" + fi + + local location="" + if [[ -n "$line_number" ]]; then + location=", line $line_number" + fi + + echo "ERROR: (${SCRIPT_NAME:-$0}${location}) $message" >&2 + if [[ -n "$context" ]]; then + echo "Context: $context" >&2 + fi +} + +# Report warning with context +# Usage: report_warning line_number "warning message" [context] +report_warning() { + local line_number="" + local message="" + local context="" + + if [[ $# -gt 1 ]]; then + line_number="$1" + message="$2" + context="${3:-}" + else + message="$1" + fi + + local location="" + if [[ -n "$line_number" ]]; then + location=", line $line_number" + fi + + echo "WARNING: (${SCRIPT_NAME:-$0}${location}) $message" >&2 + if [[ -n "$context" ]]; then + echo "Context: $context" >&2 + fi +} + +# Enhanced error reporting with error codes +# Usage: report_error_enhanced error_code "error message" [context] [line_number] +report_error_enhanced() { + local error_code="$1" + local error_message="$2" + local context="${3:-}" + local line_number="${4:-}" + + local location="" + if [[ -n "$line_number" ]]; then + location=", line $line_number" + fi + + echo "ERROR [$error_code]: (${SCRIPT_NAME:-$0}${location}) $error_message" >&2 + if [[ -n "$context" ]]; then + echo "Context: $context" >&2 + fi + + return $error_code +} + +# Enhanced warning reporting with severity levels +# Usage: report_warning_enhanced severity "warning message" [context] +report_warning_enhanced() { + local severity="$1" + local warning_message="$2" + local context="${3:-}" + + echo "$severity: (${SCRIPT_NAME:-$0}) $warning_message" >&2 + if [[ -n "$context" ]]; then + echo "Context: $context" >&2 + fi +} \ No newline at end of file diff --git a/toolchain/scripts/lib/package_manager.sh b/toolchain/scripts/lib/package_manager.sh new file mode 100644 index 0000000000..94a7c0b307 --- /dev/null +++ b/toolchain/scripts/lib/package_manager.sh @@ -0,0 +1,469 @@ +#!/bin/bash + +# ============================================================================ +# ABACUS Toolchain Package Manager +# ============================================================================ +# Handles package downloading, building, and installation +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# ============================================================================ + +# Global package manager variables +PACKAGE_MANAGER_INITIALIZED=false +declare -A PACKAGE_BUILD_STATUS +declare -A PACKAGE_DEPENDENCIES + +# Initialize package manager +# Usage: package_manager_init +package_manager_init() { + if [[ "$PACKAGE_MANAGER_INITIALIZED" == "true" ]]; then + return 0 + fi + + # Initialize dependencies + if ! command -v config_manager_init &> /dev/null; then + source "${SCRIPTDIR}/lib/config_manager.sh" + fi + + config_manager_init + + # Define package dependencies + package_manager_define_dependencies + + PACKAGE_MANAGER_INITIALIZED=true + return 0 +} + +# Define package dependencies +# Usage: package_manager_define_dependencies +package_manager_define_dependencies() { + # Stage 0: Compilers and Build Tools + PACKAGE_DEPENDENCIES["gcc"]="" + PACKAGE_DEPENDENCIES["intel"]="" + PACKAGE_DEPENDENCIES["amd"]="" + PACKAGE_DEPENDENCIES["cmake"]="" + + # Stage 1: MPI Implementations (depend on compilers) + PACKAGE_DEPENDENCIES["openmpi"]="gcc cmake" + PACKAGE_DEPENDENCIES["mpich"]="gcc cmake" + PACKAGE_DEPENDENCIES["intelmpi"]="intel" + + # Stage 2: Mathematical Libraries (depend on compilers and MPI) + PACKAGE_DEPENDENCIES["openblas"]="gcc cmake" + PACKAGE_DEPENDENCIES["mkl"]="intel" + PACKAGE_DEPENDENCIES["aocl"]="amd" + + # Stage 3: Scientific Computing Libraries (depend on math libs and MPI) + PACKAGE_DEPENDENCIES["fftw"]="gcc cmake" + PACKAGE_DEPENDENCIES["libxc"]="gcc cmake" + PACKAGE_DEPENDENCIES["scalapack"]="openblas" + PACKAGE_DEPENDENCIES["elpa"]="scalapack" + + # Stage 4: Advanced Libraries + PACKAGE_DEPENDENCIES["cereal"]="gcc cmake" + PACKAGE_DEPENDENCIES["rapidjson"]="gcc cmake" + PACKAGE_DEPENDENCIES["libtorch"]="gcc cmake" + PACKAGE_DEPENDENCIES["libnpy"]="gcc cmake" + PACKAGE_DEPENDENCIES["libri"]="gcc cmake" + PACKAGE_DEPENDENCIES["libcomm"]="gcc cmake" + PACKAGE_DEPENDENCIES["nep"]="gcc cmake" +} + +# Check if package is enabled for installation +# Usage: package_is_enabled "package_name" +package_is_enabled() { + local package="$1" + local status=$(config_get "with_${package}") + + case "$status" in + "__INSTALL__"|"__SYSTEM__") + return 0 + ;; + *) + return 1 + ;; + esac +} + +# Get package installation mode +# Usage: package_get_mode "package_name" +package_get_mode() { + local package="$1" + config_get "with_${package}" +} + + +# Mark package as built +# Usage: package_mark_built "package_name" +package_mark_built() { + local package="$1" + PACKAGE_BUILD_STATUS["$package"]="built" +} + +# Get package dependencies +# Usage: package_get_dependencies "package_name" +package_get_dependencies() { + local package="$1" + echo "${PACKAGE_DEPENDENCIES[$package]}" +} + +# Check if all dependencies are satisfied +# Usage: package_check_dependencies "package_name" +package_check_dependencies() { + local package="$1" + local deps=$(package_get_dependencies "$package") + + for dep in $deps; do + if package_is_enabled "$dep"; then + if ! package_is_built "$dep"; then + echo "Dependency $dep for $package is not built yet" + return 1 + fi + fi + done + + return 0 +} + +# Install package by stage +# Usage: package_install_stage "stage_number" +package_install_stage() { + local stage="$1" + local stage_script="${SCRIPTDIR}/stage${stage}/install_stage${stage}.sh" + + if [[ ! -f "$stage_script" ]]; then + report_error ${LINENO} "Stage script not found: $stage_script" + return 1 + fi + + echo "Installing Stage ${stage} packages..." + + # Check if dry run mode + if [[ "$(config_get dry_run)" == "__TRUE__" ]]; then + echo "Dry run: would execute $stage_script" + return 0 + fi + + # Export version configuration for stage scripts + package_export_version_config + + # Execute stage installation script + if ! "$stage_script"; then + report_error ${LINENO} "Failed to install Stage ${stage} packages" + return 1 + fi + + echo "Stage ${stage} installation completed successfully" + return 0 +} + +# Install all enabled packages +# Usage: package_install_all +package_install_all() { + echo "Starting package installation process..." + + # Check if pack run mode (only check and install required packages) + if [[ "$(config_get PACK_RUN)" == "__TRUE__" ]]; then + echo "Pack run mode: checking and installing required packages only" + package_check_system_requirements + return $? + fi + + # Install packages by stages + for stage in 0 1 2 3 4; do + if ! package_install_stage "$stage"; then + report_error ${LINENO} "Failed to install stage $stage" + return 1 + fi + done + + echo "All package installations completed successfully" + return 0 +} + +# Check system requirements +# Usage: package_check_system_requirements +package_check_system_requirements() { + echo "Checking system requirements..." + + # Check for essential system tools + local required_tools="wget curl tar gzip make" + local missing_tools="" + + for tool in $required_tools; do + if ! command -v "$tool" &> /dev/null; then + missing_tools="$missing_tools $tool" + fi + done + + if [[ -n "$missing_tools" ]]; then + report_error ${LINENO} "Missing required system tools:$missing_tools" + echo "Please install these tools using your system package manager" + return 1 + fi + + # Check for compiler availability + local compiler_mode=$(config_get MATH_MODE) + case "$compiler_mode" in + gcc) + if ! command -v gcc &> /dev/null; then + report_warning ${LINENO} "GCC compiler not found in system PATH" + fi + ;; + intel) + if ! command -v icc &> /dev/null && ! command -v icx &> /dev/null; then + report_warning ${LINENO} "Intel compiler not found in system PATH" + fi + ;; + amd) + if ! command -v clang &> /dev/null; then + report_warning ${LINENO} "AMD/Clang compiler not found in system PATH" + fi + ;; + esac + + echo "System requirements check completed" + return 0 +} + +# Generate package configuration file +# Usage: package_write_config "config_file" +package_write_config() { + local config_file="$1" + + echo "# ABACUS Toolchain Package Configuration" > "$config_file" + echo "# Generated on $(date)" >> "$config_file" + echo "" >> "$config_file" + + # Write package lists + echo "tool_list=\"${tool_list}\"" >> "$config_file" + echo "mpi_list=\"${mpi_list}\"" >> "$config_file" + echo "math_list=\"${math_list}\"" >> "$config_file" + echo "lib_list=\"${lib_list}\"" >> "$config_file" + echo "package_list=\"${package_list}\"" >> "$config_file" + echo "" >> "$config_file" + + # Write package configurations + for pkg in ${package_list}; do + local status=$(config_get "with_${pkg}") + echo "with_${pkg}=\"${status}\"" >> "$config_file" + done + + echo "" >> "$config_file" + echo "# Configuration modes" >> "$config_file" + echo "MPI_MODE=\"$(config_get MPI_MODE)\"" >> "$config_file" + echo "MATH_MODE=\"$(config_get MATH_MODE)\"" >> "$config_file" + echo "TARGET_CPU=\"$(config_get TARGET_CPU)\"" >> "$config_file" + echo "GPUVER=\"$(config_get GPUVER)\"" >> "$config_file" + + echo "Package configuration written to: $config_file" +} + +# Get list of packages to install +# Usage: package_list_to_install +package_list_to_install() { + local install_packages="" + + for pkg in ${package_list}; do + if [[ "$(config_get with_${pkg})" == "__INSTALL__" ]]; then + install_packages="$install_packages $pkg" + fi + done + + echo "$install_packages" +} + +# Get list of system packages +# Usage: package_get_system_list +package_get_system_list() { + local system_packages="" + + for pkg in ${package_list}; do + if [[ "$(config_get with_${pkg})" == "__SYSTEM__" ]]; then + system_packages="$system_packages $pkg" + fi + done + + echo "$system_packages" +} + +# Print package installation summary +# Usage: package_print_summary +package_print_summary() { + echo "Package Installation Summary:" + echo "============================" + echo "" + + local install_list=$(package_get_install_list) + local system_list=$(package_get_system_list) + + if [[ -n "$install_list" ]]; then + echo "Packages to be installed from source:" + for pkg in $install_list; do + echo " - $pkg" + done + echo "" + fi + + if [[ -n "$system_list" ]]; then + echo "Packages to be used from system:" + for pkg in $system_list; do + echo " - $pkg" + done + echo "" + fi + + echo "Installation modes:" + echo " MPI Mode: $(config_get MPI_MODE)" + echo " Math Mode: $(config_get MATH_MODE)" + echo " Target CPU: $(config_get TARGET_CPU)" + echo " GPU Version: $(config_get GPUVER)" + echo "" +} + +# Validate package configuration +# Usage: package_validate_config +package_validate_config() { + local errors=0 + + # Check for conflicting MPI implementations + local mpi_count=0 + for mpi in mpich openmpi intelmpi; do + if [[ "$(config_get with_${mpi})" != "__DONTUSE__" ]]; then + ((mpi_count++)) + fi + done + + if [[ $mpi_count -gt 1 ]]; then + report_error ${LINENO} "Multiple MPI implementations selected. Please choose only one." + ((errors++)) + fi + + # Check for conflicting math libraries + local math_count=0 + for math in mkl openblas aocl; do + if [[ "$(config_get with_${math})" != "__DONTUSE__" ]]; then + ((math_count++)) + fi + done + + if [[ $math_count -gt 1 ]]; then + report_warning ${LINENO} "Multiple math libraries selected. This may cause conflicts." + fi + + # Check for GPU support consistency + local gpu_ver=$(config_get GPUVER) + if [[ "$gpu_ver" != "no" ]]; then + if [[ "$(config_get enable_cuda)" != "__TRUE__" && "$(config_get enable_hip)" != "__TRUE__" ]]; then + report_warning ${LINENO} "GPU version specified but no GPU support enabled" + fi + fi + + return $errors +} + +# Clean build directories +# Usage: package_clean_build +package_clean_build() { + if [[ -d "$BUILDDIR" ]]; then + echo "Cleaning build directory: $BUILDDIR" + rm -rf "$BUILDDIR" + fi + mkdir -p "$BUILDDIR" +} + +# Clean install directories +# Usage: package_clean_install +package_clean_install() { + if [[ -d "$INSTALLDIR" ]]; then + echo "Cleaning install directory: $INSTALLDIR" + rm -rf "$INSTALLDIR" + fi + mkdir -p "$INSTALLDIR" +} + +# Install all packages in pack run mode +# Usage: package_install_all_pack_run +package_install_all_pack_run() { + echo "Pack run mode - installing all packages in sequence" + + # Execute all stages in sequence + for stage in 0 1 2 3 4; do + local stage_script="${SCRIPTDIR}/stage${stage}/install_stage${stage}.sh" + if [[ -f "$stage_script" ]]; then + echo "Executing stage $stage in pack run mode..." + if ! "$stage_script"; then + echo "Stage $stage failed in pack run mode" + return 1 + fi + fi + done + + return 0 +} + +# Export version configuration for stage scripts +# Usage: package_export_version_config +package_export_version_config() { + # Export version suffix configuration + local use_alt_versions=$(config_get use_alt_versions) + + if [[ "$use_alt_versions" == "__TRUE__" ]]; then + export ABACUS_TOOLCHAIN_VERSION_SUFFIX="alt" + else + export ABACUS_TOOLCHAIN_VERSION_SUFFIX="main" + fi + + # Build package versions string from CONFIG_CACHE PACKAGE_VERSION_* entries + local package_versions="" + for key in "${!CONFIG_CACHE[@]}"; do + if [[ "$key" =~ ^PACKAGE_VERSION_(.+)$ ]]; then + local pkg_name="${BASH_REMATCH[1]}" + local pkg_version="${CONFIG_CACHE[$key]}" + local pkg_lower=$(echo "$pkg_name" | tr '[:upper:]' '[:lower:]') + + if [[ -n "$package_versions" ]]; then + package_versions="${package_versions} ${pkg_lower}:${pkg_version}" + else + package_versions="${pkg_lower}:${pkg_version}" + fi + fi + done + + # Export individual package version overrides + if [[ -n "$package_versions" ]]; then + export ABACUS_TOOLCHAIN_PACKAGE_VERSIONS="$package_versions" + fi + + # Export for backward compatibility + export VERSION_SUFFIX="$ABACUS_TOOLCHAIN_VERSION_SUFFIX" +} + +# Execute stage-based installation (stage0 -> stage1 -> stage2 -> stage3 -> stage4) +# This follows the original toolchain logic exactly +package_install_all_stages() { + echo "Starting stage-based installation..." + + # Export version configuration before starting stages + package_export_version_config + + # Execute stages in order, just like the original fallback version + local stages=("stage0" "stage1" "stage2" "stage3" "stage4") + + for stage in "${stages[@]}"; do + local stage_script="${SCRIPTDIR}/${stage}/install_${stage}.sh" + + if [[ -f "$stage_script" ]]; then + echo "Executing ${stage}..." + if ! "$stage_script"; then + report_error ${LINENO} "Failed to execute ${stage}" + return 1 + fi + else + echo "Warning: Stage script not found: $stage_script" + fi + done + + echo "All stages completed successfully." + return 0 +} \ No newline at end of file diff --git a/toolchain/scripts/lib/user_interface.sh b/toolchain/scripts/lib/user_interface.sh new file mode 100644 index 0000000000..c942379daa --- /dev/null +++ b/toolchain/scripts/lib/user_interface.sh @@ -0,0 +1,1170 @@ +#!/bin/bash + +# ============================================================================ +# ABACUS Toolchain User Interface Module (Enhanced) +# ============================================================================ +# Provides beautiful and consistent user interaction, help messages, and progress display +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# Enhanced: Beautiful terminal output with harmonious color scheme +# ============================================================================ + +# Global UI variables +UI_INITIALIZED=false +UI_VERBOSE=false +UI_QUIET=false +UI_LOG_FILE="" + +# Enhanced color definitions with harmonious blue-based theme +if [[ -t 1 ]]; then + # Terminal supports colors - Enhanced color palette + readonly UI_RED='\033[38;5;196m' # Bright red for errors + readonly UI_GREEN='\033[38;5;46m' # Bright green for success + readonly UI_YELLOW='\033[38;5;226m' # Bright yellow for warnings + readonly UI_BLUE='\033[38;5;39m' # Bright blue for info (main theme) + readonly UI_PURPLE='\033[38;5;141m' # Purple for debug + readonly UI_CYAN='\033[38;5;51m' # Cyan for progress + readonly UI_WHITE='\033[38;5;255m' # Pure white + readonly UI_GRAY='\033[38;5;244m' # Gray for secondary text + readonly UI_ORANGE='\033[38;5;208m' # Orange for highlights + + # Style definitions + readonly UI_BOLD='\033[1m' + readonly UI_DIM='\033[2m' + readonly UI_ITALIC='\033[3m' + readonly UI_UNDERLINE='\033[4m' + readonly UI_BLINK='\033[5m' + readonly UI_REVERSE='\033[7m' + readonly UI_NC='\033[0m' # No Color + + # Background colors for special effects + readonly UI_BG_RED='\033[48;5;196m' + readonly UI_BG_GREEN='\033[48;5;46m' + readonly UI_BG_BLUE='\033[48;5;39m' + readonly UI_BG_YELLOW='\033[48;5;226m' +else + # No color support - fallback to empty strings + readonly UI_RED='' + readonly UI_GREEN='' + readonly UI_YELLOW='' + readonly UI_BLUE='' + readonly UI_PURPLE='' + readonly UI_CYAN='' + readonly UI_WHITE='' + readonly UI_GRAY='' + readonly UI_ORANGE='' + readonly UI_BOLD='' + readonly UI_DIM='' + readonly UI_ITALIC='' + readonly UI_UNDERLINE='' + readonly UI_BLINK='' + readonly UI_REVERSE='' + readonly UI_NC='' + readonly UI_BG_RED='' + readonly UI_BG_GREEN='' + readonly UI_BG_BLUE='' + readonly UI_BG_YELLOW='' +fi + +# Terminal Unicode support detection +UI_UNICODE_SUPPORT="" + +# Detect terminal Unicode support capability +# Returns: "full", "basic", or "none" +ui_detect_unicode_support() { + # Check if already detected + if [[ -n "$UI_UNICODE_SUPPORT" ]]; then + echo "$UI_UNICODE_SUPPORT" + return 0 + fi + + # Manual override via environment variable + if [[ "${ABACUS_UI_SIMPLE:-}" == "1" ]] || [[ "${ABACUS_UI_ASCII:-}" == "1" ]]; then + UI_UNICODE_SUPPORT="none" + echo "none" + return 0 + fi + + # Force Unicode mode if explicitly requested + if [[ "${ABACUS_UI_UNICODE:-}" == "1" ]]; then + UI_UNICODE_SUPPORT="full" + echo "full" + return 0 + fi + + # Check if we're in a non-interactive environment + if [[ ! -t 1 ]]; then + UI_UNICODE_SUPPORT="none" + echo "none" + return 0 + fi + + # Check locale settings for UTF-8 support + local locale_utf8=false + + # First check environment variables + if [[ "${LC_ALL:-}" =~ [Uu][Tt][Ff]-?8 ]] || \ + [[ "${LC_CTYPE:-}" =~ [Uu][Tt][Ff]-?8 ]] || \ + [[ "${LANG:-}" =~ [Uu][Tt][Ff]-?8 ]]; then + locale_utf8=true + fi + + # Additional check using locale command if available + if [[ "$locale_utf8" == "false" ]] && command -v locale &>/dev/null; then + local charset=$(locale charmap 2>/dev/null || echo "") + if [[ "$charset" =~ UTF-?8 ]]; then + locale_utf8=true + fi + fi + + # Check terminal type and capabilities + local term_support="none" + case "${TERM:-}" in + xterm*|screen*|tmux*|rxvt*|gnome*|konsole*|alacritty*|kitty*|iterm*) + if [[ "$locale_utf8" == "true" ]]; then + term_support="full" + else + term_support="basic" + fi + ;; + linux|vt*) + # Linux console - basic Unicode support + if [[ "$locale_utf8" == "true" ]]; then + term_support="basic" + else + term_support="none" + fi + ;; + dumb|unknown) + term_support="none" + ;; + *) + # Unknown terminal - be conservative + if [[ "$locale_utf8" == "true" ]]; then + term_support="basic" + else + term_support="none" + fi + ;; + esac + + # Final validation - ensure we have both locale and terminal support + if [[ "$term_support" != "none" && "$locale_utf8" == "false" ]]; then + # Terminal claims support but locale doesn't - be conservative + term_support="none" + fi + + UI_UNICODE_SUPPORT="$term_support" + echo "$term_support" +} + +# Unicode symbols with ASCII fallbacks +# These will be set based on terminal capability +UI_ICON_SUCCESS="" +UI_ICON_ERROR="" +UI_ICON_WARNING="" +UI_ICON_INFO="" +UI_ICON_PROGRESS="" +UI_ICON_ROCKET="" +UI_ICON_GEAR="" +UI_ICON_PACKAGE="" +UI_ICON_DOWNLOAD="" +UI_ICON_BUILD="" +UI_ICON_INSTALL="" +UI_ICON_CHECK="" +UI_ICON_CROSS="" +UI_ICON_ARROW="" +UI_ICON_STAR="" + +# Progress bar characters +UI_PROGRESS_FULL="" +UI_PROGRESS_PARTIAL1="" +UI_PROGRESS_PARTIAL2="" +UI_PROGRESS_EMPTY="" + +# Initialize Unicode icons based on terminal capability +ui_init_icons() { + local support=$(ui_detect_unicode_support) + + case "$support" in + "full") + # Full Unicode support - use emoji and special symbols + readonly UI_ICON_SUCCESS="✅" + readonly UI_ICON_ERROR="❌" + readonly UI_ICON_WARNING="⚠️" + readonly UI_ICON_INFO="ℹ️" + readonly UI_ICON_PROGRESS="🔄" + readonly UI_ICON_ROCKET="🚀" + readonly UI_ICON_GEAR="⚙️" + readonly UI_ICON_PACKAGE="📦" + readonly UI_ICON_DOWNLOAD="⬇️" + readonly UI_ICON_BUILD="🔨" + readonly UI_ICON_INSTALL="📥" + readonly UI_ICON_CHECK="✓" + readonly UI_ICON_CROSS="✗" + readonly UI_ICON_ARROW="→" + readonly UI_ICON_STAR="⭐" + # Full Unicode progress bar + readonly UI_PROGRESS_FULL="█" + readonly UI_PROGRESS_PARTIAL1="▓" + readonly UI_PROGRESS_PARTIAL2="▒" + readonly UI_PROGRESS_EMPTY="░" + ;; + "basic") + # Basic Unicode support - use simple Unicode symbols + readonly UI_ICON_SUCCESS="✓" + readonly UI_ICON_ERROR="✗" + readonly UI_ICON_WARNING="!" + readonly UI_ICON_INFO="i" + readonly UI_ICON_PROGRESS="*" + readonly UI_ICON_ROCKET="^" + readonly UI_ICON_GEAR="*" + readonly UI_ICON_PACKAGE="+" + readonly UI_ICON_DOWNLOAD="v" + readonly UI_ICON_BUILD="#" + readonly UI_ICON_INSTALL="+" + readonly UI_ICON_CHECK="✓" + readonly UI_ICON_CROSS="✗" + readonly UI_ICON_ARROW="→" + readonly UI_ICON_STAR="*" + # Basic Unicode progress bar + readonly UI_PROGRESS_FULL="█" + readonly UI_PROGRESS_PARTIAL1="▓" + readonly UI_PROGRESS_PARTIAL2="▒" + readonly UI_PROGRESS_EMPTY="░" + ;; + *) + # ASCII fallback - maximum compatibility + readonly UI_ICON_SUCCESS="[OK]" + readonly UI_ICON_ERROR="[ERR]" + readonly UI_ICON_WARNING="[WARN]" + readonly UI_ICON_INFO="[INFO]" + readonly UI_ICON_PROGRESS="[*]" + readonly UI_ICON_ROCKET="[^]" + readonly UI_ICON_GEAR="[*]" + readonly UI_ICON_PACKAGE="[+]" + readonly UI_ICON_DOWNLOAD="[v]" + readonly UI_ICON_BUILD="[#]" + readonly UI_ICON_INSTALL="[+]" + readonly UI_ICON_CHECK="[+]" + readonly UI_ICON_CROSS="[-]" + readonly UI_ICON_ARROW="->" + readonly UI_ICON_STAR="[*]" + # ASCII progress bar + readonly UI_PROGRESS_FULL="#" + readonly UI_PROGRESS_PARTIAL1="#" + readonly UI_PROGRESS_PARTIAL2="-" + readonly UI_PROGRESS_EMPTY="." + ;; + esac +} + +# Initialize user interface +# Usage: ui_init [verbose] [quiet] [log_file] +ui_init() { + if [[ "$UI_INITIALIZED" == "true" ]]; then + return 0 + fi + + # Parse options + while [[ $# -gt 0 ]]; do + case "$1" in + verbose) + UI_VERBOSE=true + ;; + quiet) + UI_QUIET=true + ;; + --log-file=*) + UI_LOG_FILE="${1#*=}" + ;; + *) + # Assume it's a log file path + UI_LOG_FILE="$1" + ;; + esac + shift + done + + # Initialize icons based on terminal capability + ui_init_icons + + UI_INITIALIZED=true + return 0 +} + +# Print colored message with enhanced formatting +# Usage: ui_print_color "color" "message" [icon] +ui_print_color() { + local color="$1" + local message="$2" + local icon="${3:-}" + + if [[ "$UI_QUIET" == "true" ]]; then + return 0 + fi + + if [[ -n "$icon" ]]; then + echo -e "${color}${icon} ${message}${UI_NC}" + else + echo -e "${color}${message}${UI_NC}" + fi + + # Log to file if specified + if [[ -n "$UI_LOG_FILE" ]]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $message" >> "$UI_LOG_FILE" + fi +} + +# Print info message with enhanced styling +# Usage: ui_info "message" +ui_info() { + ui_print_color "${UI_BLUE}${UI_BOLD}" "$1" "$UI_ICON_INFO" +} + +# Print success message with enhanced styling +# Usage: ui_success "message" +ui_success() { + ui_print_color "${UI_GREEN}${UI_BOLD}" "$1" "$UI_ICON_SUCCESS" +} + +# Print warning message with enhanced styling +# Usage: ui_warning "message" +ui_warning() { + ui_print_color "${UI_YELLOW}${UI_BOLD}" "$1" "$UI_ICON_WARNING" +} + +# Print error message with enhanced styling +# Usage: ui_error "message" +ui_error() { + ui_print_color "${UI_RED}${UI_BOLD}" "$1" "$UI_ICON_ERROR" >&2 +} + +# Print debug message (only in verbose mode) with enhanced styling +# Usage: ui_debug "message" +ui_debug() { + if [[ "$UI_VERBOSE" == "true" ]]; then + ui_print_color "${UI_PURPLE}${UI_DIM}" "DEBUG: $1" "🔍" + fi +} + +# Detect glibc version using multiple methods for maximum compatibility +# Usage: ui_get_glibc_version +ui_get_glibc_version() { + local glibc_version="unknown" + + # Skip on non-Linux systems + if [[ "$(uname -s)" != "Linux" ]]; then + echo "unknown" + return 0 + fi + + # Method 1: ldd --version (most reliable) + if command -v ldd &> /dev/null; then + local ldd_output=$(ldd --version 2>/dev/null | head -n1) + if [[ $? -eq 0 && -n "$ldd_output" ]]; then + # Extract version from ldd output (e.g., "ldd (Ubuntu GLIBC 2.35-0ubuntu3.1) 2.35") + glibc_version=$(echo "$ldd_output" | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?' | head -n1) + if [[ -n "$glibc_version" ]]; then + echo "$glibc_version" + return 0 + fi + fi + fi + + # Method 2: Direct library call (fallback) + local libc_paths=( + "/lib/x86_64-linux-gnu/libc.so.6" + "/lib64/libc.so.6" + "/lib/libc.so.6" + "/usr/lib/x86_64-linux-gnu/libc.so.6" + "/usr/lib64/libc.so.6" + ) + + for libc_path in "${libc_paths[@]}"; do + if [[ -x "$libc_path" ]]; then + local libc_output=$("$libc_path" 2>/dev/null | head -n1) + if [[ $? -eq 0 && -n "$libc_output" ]]; then + # Extract version from libc output + glibc_version=$(echo "$libc_output" | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?' | head -n1) + if [[ -n "$glibc_version" ]]; then + echo "$glibc_version" + return 0 + fi + fi + fi + done + + # Method 3: getconf GNU_LIBC_VERSION (alternative) + if command -v getconf &> /dev/null; then + local getconf_output=$(getconf GNU_LIBC_VERSION 2>/dev/null) + if [[ $? -eq 0 && -n "$getconf_output" ]]; then + # Extract version from getconf output (e.g., "glibc 2.35") + glibc_version=$(echo "$getconf_output" | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?' | head -n1) + if [[ -n "$glibc_version" ]]; then + echo "$glibc_version" + return 0 + fi + fi + fi + + # Method 4: Check /proc/version_signature or similar (last resort) + if [[ -r "/proc/version" ]]; then + local proc_version=$(cat /proc/version 2>/dev/null) + if [[ -n "$proc_version" ]]; then + # Look for glibc version in kernel version string (rare but possible) + glibc_version=$(echo "$proc_version" | grep -oE 'glibc[[:space:]]*[0-9]+\.[0-9]+(\.[0-9]+)?' | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?' | head -n1) + if [[ -n "$glibc_version" ]]; then + echo "$glibc_version" + return 0 + fi + fi + fi + + # All methods failed - return unknown + echo "unknown" + return 0 +} + +# Read version from VERSION file +# Usage: ui_get_version +ui_get_version() { + local version_file + local version="Unknown" + + # Try to find VERSION file in multiple locations + if [[ -n "${SCRIPT_DIR:-}" && -f "${SCRIPT_DIR}/VERSION" ]]; then + version_file="${SCRIPT_DIR}/VERSION" + elif [[ -f "$(dirname "${BASH_SOURCE[0]}")/../VERSION" ]]; then + version_file="$(dirname "${BASH_SOURCE[0]}")/../VERSION" + elif [[ -f "./scripts/VERSION" ]]; then + version_file="./scripts/VERSION" + elif [[ -f "./VERSION" ]]; then + version_file="./VERSION" + fi + + if [[ -n "$version_file" && -f "$version_file" ]]; then + # Extract version from VERSION file, handling different formats + version=$(grep -E '^VERSION=' "$version_file" 2>/dev/null | cut -d'=' -f2 | tr -d '"' | tr -d "'" | xargs) + if [[ -z "$version" ]]; then + # Fallback: try to read the entire file content if no VERSION= line + version=$(head -n 1 "$version_file" 2>/dev/null | grep -v '^#' | xargs) + fi + if [[ -z "$version" ]]; then + version="Unknown" + fi + fi + + echo "$version" +} + +# Print beautiful welcome banner +# Usage: ui_welcome_banner +ui_welcome_banner() { + if [[ "$UI_QUIET" == "true" ]]; then + return 0 + fi + + local banner_width=80 + local title="ABACUS Toolchain Installation" + local subtitle="DFT Calculation Made Simple" + local version=$(ui_get_version) + local version_text="Version $version" + + echo "" + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '═%.0s' $(seq 1 $banner_width))" + echo "" + ui_print_color "${UI_BLUE}${UI_BOLD}" "$(printf '%*s' $(((banner_width + ${#title}) / 2)) "$title")" + ui_print_color "${UI_GRAY}${UI_ITALIC}" "$(printf '%*s' $(((banner_width + ${#subtitle}) / 2)) "$subtitle")" + ui_print_color "${UI_GRAY}${UI_DIM}" "$(printf '%*s' $(((banner_width + ${#version_text}) / 2)) "$version_text")" + echo "" + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '═%.0s' $(seq 1 $banner_width))" + echo "" +} + +# Print beautiful section header with enhanced styling +# Usage: ui_section "title" +ui_section() { + local title="$1" + local line_width=60 + + if [[ "$UI_QUIET" != "true" ]]; then + echo "" + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '─%.0s' $(seq 1 $line_width))" + ui_print_color "${UI_BLUE}${UI_BOLD}" "${UI_ICON_GEAR} $title" + ui_print_color "${UI_CYAN}${UI_BOLD}" "$(printf '─%.0s' $(seq 1 $line_width))" + echo "" + fi +} + +# Print enhanced progress bar with percentage and ETA +# Usage: ui_progress "current" "total" "description" [eta_seconds] +ui_progress() { + local current="$1" + local total="$2" + local description="$3" + local eta_seconds="${4:-}" + + if [[ "$UI_QUIET" == "true" ]]; then + return 0 + fi + + local percent=$((current * 100 / total)) + local filled=$((percent / 2)) + local empty=$((50 - filled)) + + # Create progress bar with gradient effect using detected characters + local progress_bar="" + for ((i=1; i<=filled; i++)); do + if [[ $i -le 10 ]]; then + progress_bar+="$UI_PROGRESS_FULL" + elif [[ $i -le 30 ]]; then + progress_bar+="$UI_PROGRESS_PARTIAL1" + else + progress_bar+="$UI_PROGRESS_PARTIAL2" + fi + done + + for ((i=1; i<=empty; i++)); do + progress_bar+="$UI_PROGRESS_EMPTY" + done + + # Format ETA if provided + local eta_text="" + if [[ -n "$eta_seconds" && "$eta_seconds" -gt 0 ]]; then + local eta_min=$((eta_seconds / 60)) + local eta_sec=$((eta_seconds % 60)) + eta_text=" (ETA: ${eta_min}m${eta_sec}s)" + fi + + printf "\r${UI_CYAN}${UI_ICON_PROGRESS} Progress: ${UI_BOLD}[${progress_bar}]${UI_NC} ${UI_GREEN}${percent}%%${UI_NC} - ${UI_BLUE}%s${UI_NC}${eta_text}" "$description" + + if [[ $current -eq $total ]]; then + echo "" + ui_success "Task completed successfully!" + fi +} + +# Show enhanced help message with beautiful formatting +# Usage: ui_show_help +ui_show_help() { + ui_welcome_banner + + cat << 'EOF' +USAGE: + install_abacus_toolchain_new.sh [OPTIONS] + +DESCRIPTION: + This script installs the ABACUS toolchain and its dependencies with a beautiful + and user-friendly interface. It supports various compilers, MPI implementations, + and mathematical libraries for density functional theory calculations. + +RECOMMENDED WORKFLOW: + 1. 🔍 Run with --help to see all available options + 2. 🧪 Use --dry-run to preview what will be installed + 3. 🚀 Run the actual installation + 4. ⚙️ Source the setup file before building ABACUS + +BASIC OPTIONS: + -h, --help Show this comprehensive help message + --version Show version information + --version-info [PACKAGE] Show version information for specific package or all + --dry-run Preview installation without executing (recommended first) + --pack-run Only download packages without building + +CONFIGURATION OPTIONS: + --config-file Load configuration from specified file + 🎯 Default: ./install_abacus_toolchain.conf + + --mpi-mode MPI implementation to use + 📋 Options: mpich, openmpi, intelmpi, no + 🎯 Default: mpich (auto-detect available) + + --math-mode Mathematical library to use + 📋 Options: mkl, aocl, openblas, cray, no + 🎯 Default: openblas + + --gpu-ver GPU version for ELPA (CUDA compute capability) + 📋 Options: Numeric (7.0, 7.5, 8.0, etc.) or (70, 75, 80, etc.) + 🎯 Default: no (CPU-only) + + --target-cpu Target CPU architecture for optimizations + 🎯 Default: native (auto-detect) + + --log-lines Number of log lines to show during compilation + 🎯 Default: 200 + +PACKAGE CONTROL OPTIONS: + --package-version Set package version strategy + 📋 Format: package:version (e.g., openmpi:alt, openblas:main) + 📋 Versions: main (latest stable), alt (alternative/legacy) + + --with-= Fine-tune package installation modes + 📋 Modes: install, system, no, + 📦 Build Tools: gcc, cmake + 📦 Compilers: intel, intel-classic, ifx, amd, flang + 📦 MPI Libraries: openmpi, mpich, intelmpi + 📦 Math Libraries: mkl, aocl, openblas, scalapack + 📦 Scientific: libxc, fftw, elpa + 📦 Advanced: cereal, rapidjson, libri, libcomm, libtorch, libnpy, nep + +ADVANCED OPTIONS: + --enable-[=yes/no] Enable specific advanced features + 📋 Features: tsan, cuda, hip, opencl, cray + 🎯 Default: no for all features + + --with-intel-classic[=yes/no] Use Intel Classic compiler (icc/ifort) + 🎯 Default: no (uses OneAPI icx/ifx) + + --with-intel-mpi-classic[=yes/no] Use Intel MPI Classic + 🎯 Default: no + + --with-ifx[=yes/no] Use Intel Fortran compiler (ifx) + 🎯 Default: yes (when Intel is enabled) + + --with-flang[=yes/no] Use AMD Flang Fortran compiler + 🎯 Default: no + + --with-4th-openmpi[=yes/no] Use OpenMPI 4th generation (v4.x) + 🎯 Default: no (uses v5.x) + + --with-mpich-device= MPICH device type + 📋 Options: ch3, ch4 + 🎯 Default: ch4 + + --skip-system-checks Skip system validation checks + +EXAMPLES: + # 🎯 Basic installation with OpenMPI and OpenBLAS + ./install_abacus_toolchain_new.sh --mpi-mode openmpi --math-mode openblas + + # 🧪 Preview installation with all packages + ./install_abacus_toolchain_new.sh --dry-run --mpi-mode mpich + + # 🏭 Intel compiler with MKL (high performance) + ./install_abacus_toolchain_new.sh --with-intel=install --math-mode mkl + + # 🎮 GPU-enabled installation for CUDA compute capability 8.0 + ./install_abacus_toolchain_new.sh --enable-cuda --gpu-ver 8.0 + + # 🔧 Custom configuration with specific package versions + ./install_abacus_toolchain_new.sh --package-version openmpi:alt --with-fftw=system + + # 📁 Load configuration from file + ./install_abacus_toolchain_new.sh --config-file my_config.conf --dry-run + + # 🚀 Use pre-configured toolchain scripts (recommended) + ./toolchain_gnu.sh # GNU toolchain (GCC + OpenMPI + OpenBLAS) + ./toolchain_intel.sh # Intel toolchain (Intel + MPI + MKL) + ./toolchain_gcc-aocl.sh # GCC + AMD AOCL + ./toolchain_aocc-aocl.sh # AMD AOCC + AOCL + +ENVIRONMENT VARIABLES: + NPROCS_OVERWRITE=N Override parallel compilation processes + DOWNLOAD_CERT_POLICY Certificate verification policy (strict/smart/skip) + 🎯 Default: smart (try secure, fallback if needed) + +NOTES: + 📁 Build and install directories can be safely deleted after installation + 🔧 Source the setup file (install/setup) before building ABACUS + 🧪 Always use --dry-run first to preview changes + 📋 Check log files in build/PKG_NAME/make.log for compilation errors + 💡 For detailed information, see the documentation in README.md + 🎛️ Configuration files allow saving and reusing complex setups + 🚀 Use toolchain_*.sh scripts for easier pre-configured installations + +EOF + + ui_print_color "${UI_ORANGE}${UI_BOLD}" "${UI_ICON_STAR} For the best experience, start with: ${UI_WHITE}./install_abacus_toolchain_new.sh --dry-run${UI_NC}" + echo "" +} + +# Show enhanced package installation summary with beautiful table formatting +# Usage: ui_show_summary +ui_show_summary() { + ui_section "Installation Configuration Summary" + + # System information box + ui_print_color "${UI_BLUE}${UI_BOLD}" "🖥️ System Information:" + echo " ├─ OS: $(uname -s) $(uname -m)" + echo " ├─ Kernel: $(uname -r)" + echo " ├─ glibc: $(ui_get_glibc_version)" + echo " ├─ CPU Cores: $(nproc 2>/dev/null || echo "unknown")" + if command -v free &> /dev/null; then + local mem_gb=$(free -g | awk '/^Mem:/ {print $2}') + echo " ├─ Memory: ${mem_gb}GB" + else + echo " ├─ Memory: unknown" + fi + + # GPU detection with comprehensive vendor support + local gpu_info="no GPU detected" + local gpu_count=0 + local gpu_models="" + + # Method 1: Try nvidia-smi for NVIDIA GPUs + if command -v nvidia-smi &> /dev/null; then + local nvidia_output=$(nvidia-smi --query-gpu=name --format=csv,noheader,nounits 2>/dev/null) + if [[ $? -eq 0 && -n "$nvidia_output" ]]; then + gpu_count=$(echo "$nvidia_output" | wc -l) + if [[ $gpu_count -gt 0 ]]; then + local first_gpu=$(echo "$nvidia_output" | head -n1 | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + if [[ $gpu_count -eq 1 ]]; then + gpu_info="$first_gpu" + else + gpu_info="$first_gpu (${gpu_count} devices)" + fi + fi + fi + fi + + # Method 2: Try rocm-smi for AMD GPUs (if no NVIDIA found) + if [[ "$gpu_info" == "no GPU detected" ]] && command -v rocm-smi &> /dev/null; then + local amd_output=$(rocm-smi --showproductname 2>/dev/null | grep -E "GPU\[" | head -n1) + if [[ $? -eq 0 && -n "$amd_output" ]]; then + local amd_name=$(echo "$amd_output" | sed -n 's/.*: \(.*\)/\1/p' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + if [[ -n "$amd_name" ]]; then + local amd_count=$(rocm-smi --showproductname 2>/dev/null | grep -c "GPU\[" || echo "1") + if [[ $amd_count -eq 1 ]]; then + gpu_info="$amd_name" + else + gpu_info="$amd_name (${amd_count} devices)" + fi + fi + fi + fi + + # Method 3: Try lspci as fallback for any GPU vendor + if [[ "$gpu_info" == "no GPU detected" ]] && command -v lspci &> /dev/null; then + local pci_gpus=$(lspci 2>/dev/null | grep -i "vga\|3d\|display" | grep -v "audio") + if [[ -n "$pci_gpus" ]]; then + gpu_count=$(echo "$pci_gpus" | wc -l) + local first_gpu_line=$(echo "$pci_gpus" | head -n1) + + # Extract GPU name from lspci output + local gpu_name="" + if echo "$first_gpu_line" | grep -qi "nvidia"; then + gpu_name=$(echo "$first_gpu_line" | sed -n 's/.*NVIDIA Corporation \(.*\) (rev.*/\1/p' | sed 's/\[.*\]//g' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [[ -z "$gpu_name" ]] && gpu_name=$(echo "$first_gpu_line" | sed -n 's/.*NVIDIA \(.*\)/\1/p' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [[ -z "$gpu_name" ]] && gpu_name="NVIDIA GPU" + elif echo "$first_gpu_line" | grep -qi "amd\|ati\|radeon"; then + gpu_name=$(echo "$first_gpu_line" | sed -n 's/.*Advanced Micro Devices.*\[\(.*\)\].*/\1/p' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [[ -z "$gpu_name" ]] && gpu_name=$(echo "$first_gpu_line" | sed -n 's/.*AMD\/ATI \(.*\)/\1/p' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [[ -z "$gpu_name" ]] && gpu_name="AMD GPU" + elif echo "$first_gpu_line" | grep -qi "intel"; then + gpu_name=$(echo "$first_gpu_line" | sed -n 's/.*Intel Corporation \(.*\) (rev.*/\1/p' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [[ -z "$gpu_name" ]] && gpu_name="Intel GPU" + else + gpu_name="Unknown GPU" + fi + + if [[ -n "$gpu_name" && "$gpu_name" != "Unknown GPU" ]]; then + if [[ $gpu_count -eq 1 ]]; then + gpu_info="$gpu_name" + else + gpu_info="$gpu_name (${gpu_count} devices)" + fi + fi + fi + fi + + echo " └─ GPU: $gpu_info" + echo "" + + # Configuration box with aligned formatting + ui_print_color "${UI_BLUE}${UI_BOLD}" "⚙️ Build Configuration:" + printf " ├─ %-15s %s\n" "MPI Mode:" "$(config_get MPI_MODE)" + printf " ├─ %-15s %s\n" "Math Mode:" "$(config_get MATH_MODE)" + printf " ├─ %-15s %s\n" "Target CPU:" "$(config_get TARGET_CPU)" + printf " ├─ %-15s %s\n" "GPU Version:" "$(config_get GPUVER)" + printf " └─ %-15s %s\n" "Parallel Jobs:" "$(config_get NPROCS_OVERWRITE)" + + # Special modes + if [[ "$(config_get dry_run)" == "__TRUE__" ]]; then + ui_print_color "${UI_YELLOW}${UI_BOLD}" " 🧪 Dry Run Mode: Preview only (no actual installation)" + fi + if [[ "$(config_get PACK_RUN)" == "__TRUE__" ]]; then + ui_print_color "${UI_YELLOW}${UI_BOLD}" " 📦 Pack Run Mode: System packages only" + fi + echo "" + + # Package configuration table + ui_print_color "${UI_BLUE}${UI_BOLD}" "📦 Package Configuration:" + local has_packages=false + for pkg in ${package_list}; do + local status=$(config_get "with_${pkg}") + if [[ "$status" != "__DONTUSE__" ]]; then + has_packages=true + # Convert status for display + local display_status="$status" + local status_icon="" + case "$status" in + "__INSTALL__") + display_status="${UI_GREEN}install${UI_NC}" + status_icon="📥" + ;; + "__SYSTEM__") + display_status="${UI_BLUE}system${UI_NC}" + status_icon="🔗" + ;; + "__DONTUSE__") + display_status="${UI_GRAY}disabled${UI_NC}" + status_icon="❌" + ;; + *) + display_status="${UI_ORANGE}${status}${UI_NC}" + status_icon="📁" + ;; + esac + printf " ├─ %s %-12s %b %b\n" "$status_icon" "$pkg:" "$UI_ARROW" "$display_status" + fi + done + + if [[ "$has_packages" == "false" ]]; then + ui_print_color "${UI_GRAY}" " └─ No packages configured" + else + echo " └─ Configuration complete" + fi + echo "" + + # Installation summary + local install_list=$(package_list_to_install 2>/dev/null || echo "") + if [[ -n "$install_list" ]]; then + ui_print_color "${UI_GREEN}${UI_BOLD}" "🚀 Packages to be built from source:" + for pkg in $install_list; do + ui_print_color "${UI_GREEN}" " ${UI_ICON_BUILD} $pkg" + done + echo "" + fi + + # System packages + local system_packages="" + for pkg in ${package_list}; do + if [[ "$(config_get with_${pkg})" == "__SYSTEM__" ]]; then + system_packages="$system_packages $pkg" + fi + done + + if [[ -n "$system_packages" ]]; then + ui_print_color "${UI_BLUE}${UI_BOLD}" "🔗 System packages to be used:" + for pkg in $system_packages; do + ui_print_color "${UI_BLUE}" " ${UI_ICON_CHECK} $pkg" + done + echo "" + fi +} + +# Confirm installation with user +# Usage: ui_confirm_installation +ui_confirm_installation() { + # Skip confirmation in dry-run mode + if [[ "$(config_get dry_run)" == "__TRUE__" ]]; then + ui_info "Dry run mode - no actual installation will be performed" + return 0 + fi + + # Skip confirmation in quiet mode + if [[ "$UI_QUIET" == "true" ]]; then + return 0 + fi + + # Simple confirmation prompt + echo "" + echo -n "Proceed with installation? [y/N]: " + read -r response + + case "$response" in + [yY]|[yY][eE][sS]) + return 0 + ;; + *) + ui_info "Installation cancelled" + return 1 + ;; + esac +} + +# Show installation progress for a stage with enhanced styling +# Usage: ui_stage_progress "stage_number" "stage_name" +ui_stage_progress() { + local stage="$1" + local name="$2" + + ui_section "Stage $stage: $name" + ui_print_color "${UI_CYAN}${UI_BOLD}" "${UI_ICON_ROCKET} Installing packages for stage $stage..." +} + +# Show package build progress with enhanced visual feedback +# Usage: ui_package_progress "package_name" "action" +ui_package_progress() { + local package="$1" + local action="$2" + + case "$action" in + start) + ui_print_color "${UI_BLUE}${UI_BOLD}" "${UI_ICON_BUILD} Building $package..." + ;; + download) + ui_debug "${UI_ICON_DOWNLOAD} Downloading $package..." + ;; + extract) + ui_debug "${UI_ICON_PACKAGE} Extracting $package..." + ;; + configure) + ui_debug "${UI_ICON_GEAR} Configuring $package..." + ;; + compile) + ui_debug "${UI_ICON_BUILD} Compiling $package..." + ;; + install) + ui_debug "${UI_ICON_INSTALL} Installing $package..." + ;; + success) + ui_success "Successfully built $package" + ;; + skip) + ui_print_color "${UI_GRAY}${UI_BOLD}" "${UI_ICON_INFO} Skipping $package (already built or disabled)" + ;; + error) + ui_error "Failed to build $package" + ;; + esac +} + +# Show enhanced final installation results +# Usage: ui_show_results "success_count" "total_count" "failed_packages" +ui_show_results() { + local success_count="$1" + local total_count="$2" + local failed_packages="$3" + + ui_section "Installation Results" + + if [[ $success_count -eq $total_count ]]; then + ui_success "All packages installed successfully! ($success_count/$total_count)" + echo "" + ui_print_color "${UI_GREEN}${UI_BOLD}" "${UI_ICON_ROCKET} Ready to use ABACUS toolchain!" + echo "" + ui_print_color "${UI_BLUE}${UI_BOLD}" "🔧 To activate the toolchain environment:" + ui_print_color "${UI_WHITE}" " source ${SETUPFILE:-setup}" + ui_print_color "${UI_GRAY}" " # or alternatively:" + ui_print_color "${UI_WHITE}" " source ${INSTALLDIR:-install}/abacus_env.sh" + echo "" + ui_print_color "${UI_BLUE}${UI_BOLD}" "🚀 Build ABACUS with:" + ui_print_color "${UI_WHITE}" " ./build_abacus_gnu.sh # GNU toolchain" + ui_print_color "${UI_WHITE}" " ./build_abacus_intel.sh # Intel toolchain" + ui_print_color "${UI_WHITE}" " ./build_abacus_gcc-aocl.sh # AMD GCC+AOCL" + ui_print_color "${UI_WHITE}" " ./build_abacus_aocc-aocl.sh # AMD AOCC+AOCL" + else + local failed_count=$((total_count - success_count)) + ui_error "Installation completed with errors ($success_count/$total_count successful)" + echo "" + if [[ -n "$failed_packages" ]]; then + ui_print_color "${UI_RED}${UI_BOLD}" "❌ Failed packages:" + for pkg in $failed_packages; do + ui_print_color "${UI_RED}" " ${UI_ICON_CROSS} $pkg" + done + fi + echo "" + ui_print_color "${UI_YELLOW}${UI_BOLD}" "🔍 Troubleshooting tips:" + ui_print_color "${UI_YELLOW}" " • Check log files for detailed error information" + ui_print_color "${UI_YELLOW}" " • Install missing system dependencies" + ui_print_color "${UI_YELLOW}" " • Verify network connectivity for downloads" + ui_print_color "${UI_YELLOW}" " • Try with --install-all for problematic packages" + fi +} + +# Show enhanced environment setup instructions +# Usage: ui_show_env_setup +ui_show_env_setup() { + ui_section "Environment Setup Instructions" + + ui_print_color "${UI_BLUE}${UI_BOLD}" "🔧 To use the installed ABACUS toolchain:" + echo "" + ui_print_color "${UI_GREEN}${UI_BOLD}" " source ${SETUPFILE:-setup}" + ui_print_color "${UI_GRAY}" " # or" + ui_print_color "${UI_GREEN}${UI_BOLD}" " source ${INSTALLDIR:-install}/abacus_env.sh" + echo "" + ui_print_color "${UI_BLUE}${UI_BOLD}" "🚀 Then build ABACUS with:" + ui_print_color "${UI_GREEN}${UI_BOLD}" " make -j\$(nproc)" + echo "" + ui_warning "The environment setup needs to be done in each new shell session" +} + +# Handle user interruption with graceful cleanup +# Usage: ui_handle_interrupt +ui_handle_interrupt() { + echo "" + ui_warning "Installation interrupted by user (Ctrl+C)" + ui_print_color "${UI_BLUE}${UI_BOLD}" "${UI_ICON_GEAR} Cleaning up temporary files..." + + # Clean up any temporary files or processes + if [[ -n "$BUILDDIR" && -d "$BUILDDIR" ]]; then + ui_debug "Cleaning build directory: $BUILDDIR" + # Don't remove the entire build directory, just mark as interrupted + touch "$BUILDDIR/.interrupted" + fi + + ui_print_color "${UI_YELLOW}${UI_BOLD}" "${UI_ICON_INFO} Installation cancelled - you can resume later" + exit 130 +} + +# Set up signal handlers +# Usage: ui_setup_signals +ui_setup_signals() { + trap ui_handle_interrupt SIGINT SIGTERM +} + +# Enhanced input validation with helpful error messages +# Usage: ui_validate_input "input" "type" +ui_validate_input() { + local input="$1" + local type="$2" + + case "$type" in + number) + if [[ ! "$input" =~ ^[0-9]+$ ]]; then + ui_error "Invalid number: $input" + ui_info "Please provide a positive integer" + return 1 + fi + ;; + path) + if [[ ! -d "$input" ]]; then + ui_error "Directory does not exist: $input" + ui_info "Please provide a valid directory path" + return 1 + fi + ;; + file) + if [[ ! -f "$input" ]]; then + ui_error "File does not exist: $input" + ui_info "Please provide a valid file path" + return 1 + fi + ;; + mpi_mode) + case "$input" in + mpich|openmpi|intelmpi|no) + return 0 + ;; + *) + ui_error "Invalid MPI mode: $input" + ui_print_color "${UI_BLUE}${UI_BOLD}" "📋 Valid options:" + ui_print_color "${UI_GREEN}" " • mpich - MPICH implementation (recommended)" + ui_print_color "${UI_GREEN}" " • openmpi - Open MPI implementation" + ui_print_color "${UI_GREEN}" " • intelmpi - Intel MPI (requires Intel compiler)" + ui_print_color "${UI_GREEN}" " • no - Disable MPI support" + return 1 + ;; + esac + ;; + math_mode) + case "$input" in + cray|mkl|openblas|aocl) + return 0 + ;; + *) + ui_error "Invalid math mode: $input" + ui_print_color "${UI_BLUE}${UI_BOLD}" "📋 Valid options:" + ui_print_color "${UI_GREEN}" " • openblas - OpenBLAS (open source, recommended)" + ui_print_color "${UI_GREEN}" " • mkl - Intel Math Kernel Library (high performance)" + ui_print_color "${UI_GREEN}" " • aocl - AMD Optimizing CPU Libraries" + ui_print_color "${UI_GREEN}" " • cray - Cray LibSci (for Cray systems)" + return 1 + ;; + esac + ;; + gpu_version) + # Support only numeric formats for GPU versions + if [[ "$input" == "no" ]]; then + return 0 + fi + + # Check if it's a valid numeric format (like 8.0, 70, 80, etc.) + local arch_num="${input//.}" + if [[ "$arch_num" =~ ^[1-9][0-9]*$ ]]; then + return 0 + fi + + # Invalid format - show enhanced error message + ui_error "Invalid GPU version: $input" + ui_print_color "${UI_BLUE}${UI_BOLD}" "📋 Valid formats:" + ui_print_color "${UI_GREEN}" " • Decimal format: 6.0, 7.0, 8.0, 8.9, etc. (CUDA compute capability)" + ui_print_color "${UI_GREEN}" " • Integer format: 60, 70, 80, 89, etc." + ui_print_color "${UI_GREEN}" " • Disable GPU: no" + ui_print_color "${UI_ORANGE}${UI_BOLD}" "💡 Examples:" + ui_print_color "${UI_WHITE}" " • 8.0 or 80 for compute capability 8.0 (RTX 30xx series)" + ui_print_color "${UI_WHITE}" " • 7.5 or 75 for compute capability 7.5 (RTX 20xx series)" + return 1 + ;; + *) + ui_error "Unknown validation type: $type" + return 1 + ;; + esac + + return 0 +} + +# Enhanced system information display +# Usage: ui_show_system_info +ui_show_system_info() { + ui_section "System Information" + + ui_print_color "${UI_BLUE}${UI_BOLD}" "🖥️ Hardware & OS:" + printf " ├─ %-20s %s\n" "Operating System:" "$(uname -s)" + printf " ├─ %-20s %s\n" "Architecture:" "$(uname -m)" + printf " ├─ %-20s %s\n" "Kernel Version:" "$(uname -r)" + printf " ├─ %-20s %s\n" "CPU Cores:" "$(nproc 2>/dev/null || echo "unknown")" + + if command -v free &> /dev/null; then + local mem_gb=$(free -g | awk '/^Mem:/ {print $2}') + printf " ├─ %-20s %sGB\n" "Total Memory:" "$mem_gb" + fi + + printf " ├─ %-20s %s\n" "Shell:" "$SHELL" + printf " ├─ %-20s %s\n" "User:" "$(whoami)" + printf " └─ %-20s %s\n" "Working Directory:" "$(pwd)" + echo "" +} + +# Enhanced system requirements check +# Usage: ui_check_system_requirements +ui_check_system_requirements() { + ui_section "System Requirements Check" + + local missing_tools="" + local required_tools="wget curl tar gzip make" + local found_tools="" + + ui_print_color "${UI_BLUE}${UI_BOLD}" "🔍 Checking required system tools..." + + for tool in $required_tools; do + if ! command -v "$tool" &> /dev/null; then + missing_tools="$missing_tools $tool" + ui_print_color "${UI_RED}" " ${UI_ICON_CROSS} $tool - not found" + else + found_tools="$found_tools $tool" + ui_print_color "${UI_GREEN}" " ${UI_ICON_CHECK} $tool - found" + fi + done + + echo "" + + if [[ -n "$missing_tools" ]]; then + ui_error "Missing required system tools:$missing_tools" + echo "" + ui_print_color "${UI_YELLOW}${UI_BOLD}" "📦 Install missing tools using your package manager:" + ui_print_color "${UI_WHITE}" " # Ubuntu/Debian:" + ui_print_color "${UI_GREEN}" " sudo apt-get install$missing_tools" + ui_print_color "${UI_WHITE}" " # CentOS/RHEL:" + ui_print_color "${UI_GREEN}" " sudo yum install$missing_tools" + ui_print_color "${UI_WHITE}" " # Fedora:" + ui_print_color "${UI_GREEN}" " sudo dnf install$missing_tools" + return 1 + else + ui_success "All required system tools are available" + return 0 + fi +} \ No newline at end of file diff --git a/toolchain/scripts/lib/version_helper.sh b/toolchain/scripts/lib/version_helper.sh new file mode 100644 index 0000000000..46e7436cad --- /dev/null +++ b/toolchain/scripts/lib/version_helper.sh @@ -0,0 +1,338 @@ +#!/bin/bash + +# ============================================================================ +# ABACUS Toolchain Version Helper +# ============================================================================ +# Provides version display, help information, and user interaction for +# dual-version switching functionality +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# ============================================================================ + +# Source required modules +if [[ -f "${SCRIPTDIR}/package_versions.sh" ]]; then + source "${SCRIPTDIR}/package_versions.sh" +else + echo "Error: package_versions.sh not found in ${SCRIPTDIR}" >&2 + return 1 +fi + +# Display available package versions +# Usage: version_show_available [package_name] +version_show_available() { + local package_name="${1:-all}" + + echo "Available Package Versions:" + echo "==========================" + echo + + if [[ "${package_name}" == "all" ]]; then + # Show all packages + for pkg in gcc cmake openmpi mpich openblas elpa fftw libxc scalapack libtorch libnpy; do + version_show_package_info "${pkg}" + done + else + version_show_package_info "${package_name}" + fi +} + +# Helper function to show package information +version_show_package_info() { + local pkg="$1" + local main_var="${pkg}_main_ver" + local alt_var="${pkg}_alt_ver" + + # Get values using indirect variable expansion + local main_ver="${!main_var}" + local alt_ver="${!alt_var}" + + if [[ -n "$main_ver" || -n "$alt_ver" ]]; then + printf "%-12s: " "$pkg" + + if [[ -n "$main_ver" ]]; then + printf "%s (main)" "$main_ver" + fi + + if [[ -n "$alt_ver" ]]; then + if [[ -n "$main_ver" ]]; then + printf ", %s (alt)" "$alt_ver" + else + printf "%s (alt)" "$alt_ver" + fi + fi + + echo "" + fi +} + +# Display version switching help +# Usage: version_show_help +version_show_help() { + cat << 'EOF' +ABACUS Toolchain Version Switching Help +======================================= + +Command Line Options: + --use-alt-versions Use alternative versions for all packages + --package-version PKG:VER Set specific package version (PKG:main|alt) + Supports two writing styles: + 1. Multiple independent parameters: --package-version PKG1:VER1 --package-version PKG2:VER2 + 2. Single parameter with multiple key-value pairs: --package-version PKG1:VER1 PKG2:VER2 ... + --config-file FILE Load configuration from file + --version-info [PKG] Show version info for package (or all) + +Configuration File Support: + VERSION_STRATEGY=main|alt Global version strategy + PACKAGE_VERSION_PKG=main|alt Specific package version + +Examples: + # Use alternative versions for all packages + ./install_abacus_toolchain_new.sh --use-alt-versions + + # Use specific package versions (single) + ./install_abacus_toolchain_new.sh --package-version openmpi:alt + + # Use specific package versions (multiple independent parameters) + ./install_abacus_toolchain_new.sh --package-version openmpi:alt --package-version gcc:main --package-version cmake:alt + + # Use specific package versions (single parameter with multiple key-value pairs) + ./install_abacus_toolchain_new.sh --package-version openmpi:alt gcc:main cmake:alt + + # Mixed with other options + ./install_abacus_toolchain_new.sh --package-version elpa:alt cmake:alt --dry-run + + # Show version information + ./install_abacus_toolchain_new.sh --version-info openmpi + ./install_abacus_toolchain_new.sh --version-info + +Backward Compatibility: + --with-4th-openmpi Legacy flag (use --package-version openmpi=alt) + OPENMPI_4TH=yes Legacy variable (use PACKAGE_VERSION_OPENMPI=alt) + +Supported Packages: + gcc, cmake, openmpi, mpich, openblas, elpa, fftw, libxc, scalapack, libtorch, libnpy + +EOF +} + +# Interactive version selection interface +# Usage: version_interactive_select +version_interactive_select() { + echo "Interactive Version Selection" + echo "============================" + echo "" + + local packages=( + "openmpi" "mpich" "openblas" "elpa" "fftw" + "libxc" "scalapack" "libtorch" "libnpy" + ) + + echo "Select versions for packages (press Enter for default 'main'):" + echo "" + + for pkg in "${packages[@]}"; do + local main_var="${pkg^^}_main_ver" + local alt_var="${pkg^^}_alt_ver" + + # Skip if no versions available + if [[ -z "${!main_var}" && -z "${!alt_var}" ]]; then + continue + fi + + printf "%-12s: " "$pkg" + + # Show available versions + if [[ -n "${!main_var}" ]]; then + printf "main(%s)" "${!main_var}" + fi + if [[ -n "${!alt_var}" ]]; then + if [[ -n "${!main_var}" ]]; then + printf ", alt(%s)" "${!alt_var}" + else + printf "alt(%s)" "${!alt_var}" + fi + fi + + printf " [main]: " + read -r user_choice + + # Set user choice or default to main + case "$user_choice" in + alt|alternative) + if [[ -n "${!alt_var}" ]]; then + CONFIG_CACHE["PACKAGE_VERSION_${pkg^^}"]="alt" + echo " → Selected: alt (${!alt_var})" + else + echo " → Alternative version not available, using main" + CONFIG_CACHE["PACKAGE_VERSION_${pkg^^}"]="main" + fi + ;; + main|"") + CONFIG_CACHE["PACKAGE_VERSION_${pkg^^}"]="main" + if [[ -n "${!main_var}" ]]; then + echo " → Selected: main (${!main_var})" + else + echo " → Main version not available" + fi + ;; + *) + echo " → Invalid choice, using main" + CONFIG_CACHE["PACKAGE_VERSION_${pkg^^}"]="main" + ;; + esac + echo "" + done + + echo "Version selection completed!" + echo "" +} + +# Get effective version for a package +# Usage: version_get_effective PACKAGE_NAME +version_get_effective() { + local package_name="$1" + local pkg_upper="${package_name^^}" + + # Check for specific package version setting + local specific_version="${CONFIG_CACHE[PACKAGE_VERSION_${pkg_upper}]}" + if [[ -n "$specific_version" ]]; then + echo "$specific_version" + return 0 + fi + + # Check for global version strategy + local global_strategy="${CONFIG_CACHE[VERSION_STRATEGY]}" + if [[ -n "$global_strategy" ]]; then + echo "$global_strategy" + return 0 + fi + + # Check for legacy OPENMPI_4TH support + if [[ "$package_name" == "openmpi" && "${CONFIG_CACHE[OPENMPI_4TH]}" == "yes" ]]; then + echo "alt" + return 0 + fi + + # Default to main + echo "main" + return 0 +} + +# Load package variables with version selection +# Usage: version_load_package_vars PACKAGE_NAME +version_load_package_vars() { + local package_name="$1" + local effective_version + + effective_version=$(version_get_effective "$package_name") + + # Convert effective version to suffix for load_package_vars + local version_suffix="" + if [[ "$effective_version" == "alt" ]]; then + version_suffix="alt" + fi + + # Call the original load_package_vars function + load_package_vars "$package_name" "$version_suffix" + + # Export version information for debugging + export "${package_name^^}_EFFECTIVE_VERSION"="$effective_version" +} + +# Display current version configuration +# Usage: version_show_current +version_show_current() { + echo "Current Version Configuration:" + echo "=============================" + + # Show global strategy + local global_strategy="${CONFIG_CACHE[VERSION_STRATEGY]}" + if [[ -n "$global_strategy" ]]; then + echo "Global Strategy: $global_strategy" + else + echo "Global Strategy: main (default)" + fi + echo "" + + # Show per-package settings + echo "Package-specific settings:" + local packages=( + "gcc" "cmake" "openmpi" "mpich" "openblas" + "elpa" "fftw" "libxc" "scalapack" "libtorch" "libnpy" + ) + + local has_specific=false + for pkg in "${packages[@]}"; do + local pkg_upper="${pkg^^}" + local specific_version="${CONFIG_CACHE[PACKAGE_VERSION_${pkg_upper}]}" + + if [[ -n "$specific_version" ]]; then + printf " %-12s: %s\n" "$pkg" "$specific_version" + has_specific=true + fi + done + + if [[ "$has_specific" == false ]]; then + echo " (none - using global strategy)" + fi + + echo "" + + # Show legacy compatibility + if [[ "${CONFIG_CACHE[OPENMPI_4TH]}" == "yes" ]]; then + echo "Legacy compatibility: OPENMPI_4TH=yes (equivalent to openmpi:alt)" + echo "" + fi +} + +# Validate version configuration +# Usage: version_validate_config +version_validate_config() { + local validation_errors=0 + + # Check if requested versions are available + for key in "${!CONFIG_CACHE[@]}"; do + if [[ "$key" =~ ^PACKAGE_VERSION_(.+)$ ]]; then + local pkg_name="${BASH_REMATCH[1]}" + local requested_version="${CONFIG_CACHE[$key]}" + local pkg_lower=$(echo "$pkg_name" | tr '[:upper:]' '[:lower:]') + + # Check if the requested version exists + local version_var="${pkg_lower}_${requested_version}_ver" + if [[ -z "${!version_var}" ]]; then + echo "Warning: Requested version '$requested_version' for package '$pkg_lower' is not available" + validation_errors=$((validation_errors + 1)) + fi + fi + done + + return $validation_errors +} + +# Initialize version helper +# Usage: version_helper_init +version_helper_init() { + # Set default version strategy if not set + if [[ -z "${CONFIG_CACHE[VERSION_STRATEGY]}" ]]; then + CONFIG_CACHE["VERSION_STRATEGY"]="main" + fi + + # Handle legacy OPENMPI_4TH environment variable + if [[ "${OPENMPI_4TH}" == "yes" && -z "${CONFIG_CACHE[PACKAGE_VERSION_OPENMPI]}" ]]; then + CONFIG_CACHE["PACKAGE_VERSION_OPENMPI"]="alt" + CONFIG_CACHE["OPENMPI_4TH"]="yes" + echo "Notice: Legacy OPENMPI_4TH=yes detected, using openmpi:alt" + fi + + return 0 +} + +# Export functions for use by other modules +export -f version_show_available +export -f version_show_help +export -f version_interactive_select +export -f version_get_effective +export -f version_load_package_vars +export -f version_show_current +export -f version_validate_config +export -f version_helper_init \ No newline at end of file diff --git a/toolchain/scripts/lib/version_loader.sh b/toolchain/scripts/lib/version_loader.sh new file mode 100644 index 0000000000..8883a7e789 --- /dev/null +++ b/toolchain/scripts/lib/version_loader.sh @@ -0,0 +1,81 @@ +#!/bin/bash + +# ============================================================================ +# ABACUS Toolchain Version Loader Helper +# ============================================================================ +# Provides a unified version loading mechanism for all stage scripts +# Author: Quantum Misaka by Trae SOLO +# Date: 2025-10-16 +# ============================================================================ + +# Load package variables with version suffix support +# Usage: load_package_with_version "package_name" +load_package_with_version() { + local package_name="$1" + + # Check for version configuration from environment or individual package setting + local version_suffix="" + + if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "${package_name}:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "${package_name}:main"; then + version_suffix="main" + fi + fi + + # Fall back to global version suffix if no individual setting + if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" + fi + + # Load package variables with appropriate version + load_package_vars "$package_name" "$version_suffix" + + # Debug output if verbose mode is enabled + if [[ "${VERBOSE_MODE}" == "__TRUE__" ]]; then + echo "DEBUG: Loaded $package_name with version suffix: ${version_suffix:-main}" + fi +} + +# Check if version configuration is available +# Usage: has_version_config +has_version_config() { + [[ -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" || -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]] +} + +# Get effective version suffix for a package +# Usage: get_package_version_suffix "package_name" +get_package_version_suffix() { + local package_name="$1" + local version_suffix="" + + if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "${package_name}:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "${package_name}:main"; then + version_suffix="main" + fi + fi + + # Fall back to global version suffix if no individual setting + if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" + fi + + echo "${version_suffix:-main}" +} + +# Show version information for debugging +# Usage: show_version_debug "package_name" +show_version_debug() { + local package_name="$1" + local version_suffix=$(get_package_version_suffix "$package_name") + + echo "Version Debug for $package_name:" + echo " Global suffix: ${ABACUS_TOOLCHAIN_VERSION_SUFFIX:-none}" + echo " Package versions: ${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS:-none}" + echo " Effective suffix: $version_suffix" +} \ No newline at end of file diff --git a/toolchain/scripts/package_versions.sh b/toolchain/scripts/package_versions.sh new file mode 100644 index 0000000000..b8e689c367 --- /dev/null +++ b/toolchain/scripts/package_versions.sh @@ -0,0 +1,268 @@ +#!/bin/bash +# ABACUS Toolchain - Centralized Package Version Management +# +# This file contains all package versions, checksums, and URLs in one place. +# +# Usage: source this file in install scripts to get package version information +# Example: source "${SCRIPT_DIR}/package_versions.sh" && load_package_vars "openmpi" + +# ============================================================================= +# STAGE 0: Build Tools and Compilers +# ============================================================================= + +# GCC (supports dual versions) - Special case: main=13.2.0, alt=11.4.0 +gcc_main_ver="13.2.0" +gcc_main_sha256="8cb4be3796651976f94b9356fa08d833524f62420d6292c5033a9a26af315078" +gcc_alt_ver="11.4.0" +gcc_alt_sha256="af828619dd1970734dda3cfb792ea3f2cba61b5a00170ba8bce4910749d73c07" + +# CMake (supports dual versions) - main=3.31.7, alt=3.30.5 +cmake_main_ver="3.31.7" +cmake_main_sha256_x86_64="b7a5c909cdafc36042c8c9bd5765e92ff1f2528cf01720aa6dc4df294ec7e1a0" +cmake_main_sha256_aarch64="ce8e32b2c1c497dd7f619124c043ac5c28a88677e390c58748dd62fe460c62a2" +cmake_main_sha256_macos="1cb11aa2edae8551bb0f22807c6f5246bd0eb60ae9fa1474781eb4095d299aca" +cmake_alt_ver="3.30.5" +cmake_alt_sha256_x86_64="83de8839f3fb0d9caf982a0435da3fa8c4fbe2c817dfec99def310dc7e6a8404" +cmake_alt_sha256_aarch64="93c3b8920379585dece1314f113c6c9008eaedfe56023c78d856fc86dad5b8e2" +cmake_alt_sha256_macos="3d603e507c7579b13518ef752b4ffcf3ed479fba80ee171d7d85da8153e869d0" + +# ============================================================================= +# STAGE 1: MPI Implementations +# ============================================================================= + +# OpenMPI (supports dual versions) - main=5.0.8, alt=4.1.6 +openmpi_main_ver="5.0.8" +openmpi_main_sha256="53131e1a57e7270f645707f8b0b65ba56048f5b5ac3f68faabed3eb0d710e449" +openmpi_alt_ver="4.1.6" +openmpi_alt_sha256="f740994485516deb63b5311af122c265179f5328a0d857a567b85db00b11e415" + +# MPICH (supports dual versions) - main=4.3.1, alt=4.1.0 +mpich_main_ver="4.3.1" +mpich_main_sha256="acc11cb2bdc69678dc8bba747c24a28233c58596f81f03785bf2b7bb7a0ef7dc" +mpich_alt_ver="4.1.0" +mpich_alt_sha256="8b1ec63bc44c7caa2afbb457bc5b3cd4a70dbe46baba700123d67c48dc5ab6a0" + +# ============================================================================= +# STAGE 2: Math Libraries +# ============================================================================= + +# OpenBLAS (supports dual versions) - main=0.3.30, alt=0.3.27 +openblas_main_ver="0.3.30" +openblas_main_sha256="27342cff518646afb4c2b976d809102e368957974c250a25ccc965e53063c95d" +openblas_alt_ver="0.3.27" +openblas_alt_sha256="aa2d68b1564fe2b13bc292672608e9cdeeeb6dc34995512e65c3b10f4599e897" + +# ============================================================================= +# STAGE 3: Scientific Computing Libraries +# ============================================================================= + +# ELPA (supports dual versions) - main=2025.06.001, alt=2024.05.001 +elpa_main_ver="2025.06.001" +elpa_main_sha256="feeb1fea1ab4a8670b8d3240765ef0ada828062ef7ec9b735eecba2848515c94" +elpa_alt_ver="2024.05.001" +elpa_alt_sha256="9caf41a3e600e2f6f4ce1931bd54185179dade9c171556d0c9b41bbc6940f2f6" + +# FFTW (supports dual versions) - Special case: both main and alt are 3.3.10 +fftw_main_ver="3.3.10" +fftw_main_sha256="56c932549852cddcfafdab3820b0200c7742675be92179e59e6215b340e26467" +fftw_alt_ver="3.3.10" +fftw_alt_sha256="56c932549852cddcfafdab3820b0200c7742675be92179e59e6215b340e26467" + +# LibXC (supports dual versions) - main=7.0.0, alt=6.2.2 +libxc_main_ver="7.0.0" +libxc_main_sha256="e9ae69f8966d8de6b7585abd9fab588794ada1fab8f689337959a35abbf9527d" +libxc_alt_ver="6.2.2" +libxc_alt_sha256="f72ed08af7b9dff5f57482c5f97bff22c7dc49da9564bc93871997cbda6dacf3" + +# ScaLAPACK (supports dual versions) - main=2.2.2, alt=2.2.1 +scalapack_main_ver="2.2.2" +scalapack_main_sha256="a2f0c9180a210bf7ffe126c9cb81099cf337da1a7120ddb4cbe4894eb7b7d022" +scalapack_alt_ver="2.2.1" +scalapack_alt_sha256="4aede775fdb28fa44b331875730bcd5bab130caaec225fadeccf424c8fcb55aa" + +# ============================================================================= +# STAGE 4: Advanced Feature Libraries +# ============================================================================= + +# LibTorch (supports dual versions) - main=2.1.2, alt=1.12.1 +libtorch_main_ver="2.1.2" +libtorch_main_sha256="904b764df6106a8a35bef64c4b55b8c1590ad9d071eb276e680cf42abafe79e9" +libtorch_alt_ver="1.12.1" +libtorch_alt_sha256="82c7be80860f2aa7963f8700004a40af8205e1d721298f2e09b700e766a9d283" +# user can manually download higher version of libtorch by: +# wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-{libtorch_ver}%2Bcpu.zip +# 2.1.2 recommended for lower GLIBC support (lower than 3.4.26) + +# LibNPY (supports dual versions) - Special case: both main and alt are 1.0.1 +libnpy_main_ver="1.0.1" +libnpy_main_sha256="43452a4db1e8c1df606c64376ea1e32789124051d7640e7e4e8518ab4f0fba44" +libnpy_alt_ver="1.0.1" +libnpy_alt_sha256="43452a4db1e8c1df606c64376ea1e32789124051d7640e7e4e8518ab4f0fba44" + +# Master branch packages (no fixed versions) +cereal_ver="master" +cereal_sha256="--no-checksum" + +libcomm_ver="master" +libcomm_sha256="--no-checksum" + +libri_ver="master" +libri_sha256="--no-checksum" + +rapidjson_ver="master" +rapidjson_sha256="--no-checksum" + +# NEP (Neural Evolution Potential) - CPU version +nep_ver="main" +nep_sha256="--no-checksum" + +# ============================================================================= +# Package Variable Loading Function +# ============================================================================= + +load_package_vars() { + local package_name="$1" + local version_suffix="$2" # Optional version suffix for multi-version packages + + case "${package_name}" in + "gcc") + if [ "${version_suffix}" = "alt" ]; then + gcc_ver="${gcc_alt_ver}" + gcc_sha256="${gcc_alt_sha256}" + else + gcc_ver="${gcc_main_ver}" + gcc_sha256="${gcc_main_sha256}" + fi + ;; + "cmake") + # Determine architecture for SHA256 selection + local arch_suffix="" + if [ "${OPENBLAS_ARCH}" = "arm64" ]; then + if [ "$(uname -s)" = "Darwin" ]; then + arch_suffix="_macos" + else + arch_suffix="_aarch64" + fi + else + arch_suffix="_x86_64" + fi + + if [ "${version_suffix}" = "alt" ]; then + cmake_ver="${cmake_alt_ver}" + eval "cmake_sha256=\${cmake_alt_sha256${arch_suffix}}" + else + cmake_ver="${cmake_main_ver}" + eval "cmake_sha256=\${cmake_main_sha256${arch_suffix}}" + fi + ;; + "openmpi") + if [ "${OPENMPI_4TH}" = "yes" ]; then + echo "WARNING: OPENMPI_4TH=yes is deprecated. Please use 'alt' parameter instead." >&2 + openmpi_ver="${openmpi_alt_ver}" + openmpi_sha256="${openmpi_alt_sha256}" + elif [ "${version_suffix}" = "alt" ]; then + openmpi_ver="${openmpi_alt_ver}" + openmpi_sha256="${openmpi_alt_sha256}" + else + openmpi_ver="${openmpi_main_ver}" + openmpi_sha256="${openmpi_main_sha256}" + fi + ;; + "mpich") + if [ "${version_suffix}" = "alt" ]; then + mpich_ver="${mpich_alt_ver}" + mpich_sha256="${mpich_alt_sha256}" + else + mpich_ver="${mpich_main_ver}" + mpich_sha256="${mpich_main_sha256}" + fi + ;; + "openblas") + if [ "${version_suffix}" = "alt" ]; then + openblas_ver="${openblas_alt_ver}" + openblas_sha256="${openblas_alt_sha256}" + else + openblas_ver="${openblas_main_ver}" + openblas_sha256="${openblas_main_sha256}" + fi + ;; + "elpa") + if [ "${version_suffix}" = "alt" ]; then + elpa_ver="${elpa_alt_ver}" + elpa_sha256="${elpa_alt_sha256}" + else + elpa_ver="${elpa_main_ver}" + elpa_sha256="${elpa_main_sha256}" + fi + ;; + "fftw") + if [ "${version_suffix}" = "alt" ]; then + fftw_ver="${fftw_alt_ver}" + fftw_sha256="${fftw_alt_sha256}" + else + fftw_ver="${fftw_main_ver}" + fftw_sha256="${fftw_main_sha256}" + fi + ;; + "libxc") + if [ "${version_suffix}" = "alt" ]; then + libxc_ver="${libxc_alt_ver}" + libxc_sha256="${libxc_alt_sha256}" + else + libxc_ver="${libxc_main_ver}" + libxc_sha256="${libxc_main_sha256}" + fi + ;; + "scalapack") + if [ "${version_suffix}" = "alt" ]; then + scalapack_ver="${scalapack_alt_ver}" + scalapack_sha256="${scalapack_alt_sha256}" + else + scalapack_ver="${scalapack_main_ver}" + scalapack_sha256="${scalapack_main_sha256}" + fi + ;; + "libtorch") + if [ "${version_suffix}" = "alt" ]; then + libtorch_ver="${libtorch_alt_ver}" + libtorch_sha256="${libtorch_alt_sha256}" + else + libtorch_ver="${libtorch_main_ver}" + libtorch_sha256="${libtorch_main_sha256}" + fi + ;; + "libnpy") + if [ "${version_suffix}" = "alt" ]; then + libnpy_ver="${libnpy_alt_ver}" + libnpy_sha256="${libnpy_alt_sha256}" + else + libnpy_ver="${libnpy_main_ver}" + libnpy_sha256="${libnpy_main_sha256}" + fi + ;; + "cereal") + cereal_ver="${cereal_ver}" + cereal_sha256="${cereal_sha256}" + ;; + "libcomm") + libcomm_ver="${libcomm_ver}" + libcomm_sha256="${libcomm_sha256}" + ;; + "libri") + libri_ver="${libri_ver}" + libri_sha256="${libri_sha256}" + ;; + "rapidjson") + rapidjson_ver="${rapidjson_ver}" + rapidjson_sha256="${rapidjson_sha256}" + ;; + "nep") + nep_ver="${nep_ver}" + nep_sha256="${nep_sha256}" + ;; + *) + echo "Error: Unknown package '${package_name}'" + return 1 + ;; + esac +} \ No newline at end of file diff --git a/toolchain/scripts/stage0/install_amd.sh b/toolchain/scripts/stage0/install_amd.sh index 4ecfce526c..e3565a7dff 100755 --- a/toolchain/scripts/stage0/install_amd.sh +++ b/toolchain/scripts/stage0/install_amd.sh @@ -3,7 +3,6 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2025-0308 # NOTICE: flang cannot be used when compiling ELPA [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=${0} @@ -24,85 +23,83 @@ mkdir -p "${BUILDDIR}" cd "${BUILDDIR}" case "${with_amd}" in - __INSTALL__) - echo "==================== Installing the AMD compiler ======================" - echo "__INSTALL__ is not supported; please install the AMD compiler manually" - if [ "${PACK_RUN}" != "__TRUE__" ]; then + __INSTALL__) + echo "==================== Installing the AMD compiler ======================" + report_error ${LINENO} "To install the AMD compiler, please contact your system administrator." exit 1 - fi - ;; - __SYSTEM__) - echo "==================== Finding AMD compiler from system paths ====================" - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else - check_command clang "amd" && CC="$(realpath $(command -v clang))" || exit 1 - check_command clang++ "amd" && CXX="$(realpath $(command -v clang++))" || exit 1 - if [ "${WITH_FLANG}" = "yes" ]; then - check_command flang "amd" && FC="$(realpath $(command -v flang))" || exit 1 - else - check_command gfortran "gcc" && FC="gfortran" || exit 1 - add_lib_from_paths GCC_LDFLAGS "libgfortran.*" ${LIB_PATHS} - fi - fi - F90="${FC}" - F77="${FC}" - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking AMD compiler to user paths ====================" - pkg_install_dir="${with_amd}" - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - check_command ${pkg_install_dir}/bin/clang "amd" && CC="${pkg_install_dir}/bin/clang" || exit 1 - check_command ${pkg_install_dir}/bin/clang++ "amd" && CXX="${pkg_install_dir}/bin/clang++" || exit 1 + ;; + __SYSTEM__) + echo "==================== Finding AMD compiler from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_command clang "amd" && CC="$(realpath $(command -v clang))" || exit 1 + check_command clang++ "amd" && CXX="$(realpath $(command -v clang++))" || exit 1 + if [ "${WITH_FLANG}" = "yes" ]; then + check_command flang "amd" && FC="$(realpath $(command -v flang))" || exit 1 + else + check_command gfortran "gcc" && FC="gfortran" || exit 1 + add_lib_from_paths GCC_LDFLAGS "libgfortran.*" ${LIB_PATHS} + fi + F90="${FC}" + F77="${FC}" + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking AMD compiler to user paths ====================" + pkg_install_dir="${with_amd}" + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + check_command ${pkg_install_dir}/bin/clang "amd" && CC="${pkg_install_dir}/bin/clang" || exit 1 + check_command ${pkg_install_dir}/bin/clang++ "amd" && CXX="${pkg_install_dir}/bin/clang++" || exit 1 + if [ "${WITH_FLANG}" = "yes" ]; then + check_command ${pkg_install_dir}/bin/flang "amd" && FC="${pkg_install_dir}/bin/flang" || exit 1 + else + check_command gfortran "gcc" && FC="$(command -v gfortran)" || exit 1 + fi + F90="${FC}" + F77="${FC}" + AMD_CFLAGS="-I'${pkg_install_dir}/include'" + AMD_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; +esac +if [ "${with_amd}" != "__DONTUSE__" ]; then + echo "CC is ${CC}" + [ $(realpath $(command -v clang) | grep -e aocc-compiler) ] || echo "Check the AMD C compiler path" + echo "CXX is ${CXX}" + [ $(realpath $(command -v clang++) | grep -e aocc-compiler) ] || echo "Check the AMD C++ compiler path" + echo "FC is ${FC}" if [ "${WITH_FLANG}" = "yes" ]; then - check_command ${pkg_install_dir}/bin/flang "amd" && FC="${pkg_install_dir}/bin/flang" || exit 1 + [ $(realpath $(command -v flang) | grep -e aocc-compiler) ] || echo "Check the AMD Fortran compiler path" else - check_command gfortran "gcc" && FC="$(command -v gfortran)" || exit 1 + [ $(realpath $(command -v gfortran) | grep -e aocc-compiler) ] || echo "Check the GNU Fortran compiler path" fi - F90="${FC}" - F77="${FC}" - AMD_CFLAGS="-I'${pkg_install_dir}/include'" - AMD_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; -esac -if [ "${with_amd}" != "__DONTUSE__" ]; then - echo "CC is ${CC}" - [ $(realpath $(command -v clang) | grep -e aocc-compiler) ] || echo "Check the AMD C compiler path" - echo "CXX is ${CXX}" - [ $(realpath $(command -v clang++) | grep -e aocc-compiler) ] || echo "Check the AMD C++ compiler path" - echo "FC is ${FC}" - if [ "${WITH_FLANG}" = "yes" ]; then - [ $(realpath $(command -v flang) | grep -e aocc-compiler) ] || echo "Check the AMD Fortran compiler path" - else - [ $(realpath $(command -v gfortran) | grep -e aocc-compiler) ] || echo "Check the GNU Fortran compiler path" - fi - cat << EOF > "${BUILDDIR}/setup_amd" + cat << EOF > "${BUILDDIR}/setup_amd" export CC="${CC}" export CXX="${CXX}" export FC="${FC}" export F90="${F90}" export F77="${F77}" EOF - if [ "${with_amd}" != "__SYSTEM__" ]; then - cat << EOF >> "${BUILDDIR}/setup_amd" + if [ "${with_amd}" != "__SYSTEM__" ]; then + cat << EOF >> "${BUILDDIR}/setup_amd" prepend_path PATH "${pkg_install_dir}/bin" prepend_path LD_LIBRARY_PATH "${pkg_install_dir}/lib" prepend_path LD_RUN_PATH "${pkg_install_dir}/lib" prepend_path LIBRARY_PATH "${pkg_install_dir}/lib" prepend_path CPATH "${pkg_install_dir}/include" EOF - fi - cat << EOF >> "${BUILDDIR}/setup_amd" + fi + cat << EOF >> "${BUILDDIR}/setup_amd" export AMD_CFLAGS="${AMD_CFLAGS}" export AMD_LDFLAGS="${AMD_LDFLAGS}" export AMD_LIBS="${AMD_LIBS}" EOF - cat "${BUILDDIR}/setup_amd" >> ${SETUPFILE} + cat "${BUILDDIR}/setup_amd" >> ${SETUPFILE} fi load "${BUILDDIR}/setup_amd" diff --git a/toolchain/scripts/stage0/install_cmake.sh b/toolchain/scripts/stage0/install_cmake.sh index 0ef5158c97..8692bfb058 100755 --- a/toolchain/scripts/stage0/install_cmake.sh +++ b/toolchain/scripts/stage0/install_cmake.sh @@ -11,87 +11,137 @@ SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load CMake package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "cmake:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "cmake:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi + +# Ensure OPENBLAS_ARCH is set before loading package variables +# This is needed for architecture-specific SHA256 selection +# In --pack-run mode, openblas_arch file may contain empty values, so we need fallback +if [ -f "${BUILDDIR}/openblas_arch" ]; then + source "${BUILDDIR}/openblas_arch" +fi + +if [ -z "${OPENBLAS_ARCH}" ]; then + case "$(uname -m)" in + x86_64|amd64) OPENBLAS_ARCH="x86_64" ;; + aarch64|arm64) OPENBLAS_ARCH="arm64" ;; + *) OPENBLAS_ARCH="x86_64" ;; # default fallback + esac + echo "OPENBLAS_ARCH not set, using fallback: ${OPENBLAS_ARCH}" +fi + +# Export OPENBLAS_ARCH to ensure it's available throughout the script +export OPENBLAS_ARCH + +# Load package variables with appropriate version +load_package_vars "cmake" "$version_suffix" + source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env +# Re-apply architecture detection if OPENBLAS_ARCH is still empty after sourcing +if [ -z "${OPENBLAS_ARCH}" ]; then + case "$(uname -m)" in + x86_64|amd64) OPENBLAS_ARCH="x86_64" ;; + aarch64|arm64) OPENBLAS_ARCH="arm64" ;; + *) OPENBLAS_ARCH="x86_64" ;; # default fallback + esac + export OPENBLAS_ARCH +fi + [ -f "${BUILDDIR}/setup_cmake" ] && rm "${BUILDDIR}/setup_cmake" ! [ -d "${BUILDDIR}" ] && mkdir -p "${BUILDDIR}" cd "${BUILDDIR}" case "${with_cmake}" in - __INSTALL__) - echo "==================== Installing CMake ====================" - cmake_ver="3.31.7" - cmake_ext="sh" - if [ "${OPENBLAS_ARCH}" = "arm64" ]; then - if [ "$(uname -s)" = "Darwin" ]; then - cmake_arch="macos-universal" - cmake_sha256="1cb11aa2edae8551bb0f22807c6f5246bd0eb60ae9fa1474781eb4095d299aca" - cmake_ext="tar.gz" - elif [ "$(uname -s)" = "Linux" ]; then - cmake_arch="linux-aarch64" - cmake_sha256="ce8e32b2c1c497dd7f619124c043ac5c28a88677e390c58748dd62fe460c62a2" - else - report_error ${LINENO} \ - "cmake installation for ARCH=${OPENBLAS_ARCH} under $(uname -s) is not supported. You can try to use the system installation using the flag --with-cmake=system instead." - fi - elif [ "${OPENBLAS_ARCH}" = "x86_64" ]; then - cmake_arch="linux-x86_64" - cmake_sha256="b7a5c909cdafc36042c8c9bd5765e92ff1f2528cf01720aa6dc4df294ec7e1a0" - else - report_error ${LINENO} \ - "cmake installation for ARCH=${ARCH} is not supported. You can try to use the system installation using the flag --with-cmake=system instead." - exit 1 - fi - pkg_install_dir="${INSTALLDIR}/cmake-${cmake_ver}" - #pkg_install_dir="${HOME}/apps/cmake/${cmake_ver}" - install_lock_file="$pkg_install_dir/install_successful" - cmake_pkg="cmake-${cmake_ver}-${cmake_arch}.${cmake_ext}" - if verify_checksums "${install_lock_file}"; then - echo "cmake-${cmake_ver} is already installed, skipping it." - else - if [ -f $cmake_pkg ]; then - echo "$cmake_pkg is found" - else - #download_pkg_from_ABACUS_org "${cmake_sha256}" "$cmake_pkg" - url="https://cmake.org/files/v${cmake_ver%.*}/${cmake_pkg}" - download_pkg_from_url "${cmake_sha256}" "${cmake_pkg}" "${url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - mkdir -p ${pkg_install_dir} - if [ "${cmake_arch}" = "macos-universal" ]; then - tar --strip-components=3 -xvf $cmake_pkg -C ${pkg_install_dir} > install.log 2>&1 || tail -n ${LOG_LINES} install.log + __INSTALL__) + echo "==================== Installing CMake ====================" + cmake_ext="sh" + if [ "${OPENBLAS_ARCH}" = "arm64" ]; then + if [ "$(uname -s)" = "Darwin" ]; then + cmake_arch="macos-universal" + cmake_ext="tar.gz" + elif [ "$(uname -s)" = "Linux" ]; then + cmake_arch="linux-aarch64" + else + report_error ${LINENO} \ + "cmake installation for ARCH=${OPENBLAS_ARCH} under $(uname -s) is not supported. You can try to use the system installation using the flag --with-cmake=system instead." + fi + elif [ "${OPENBLAS_ARCH}" = "x86_64" ]; then + cmake_arch="linux-x86_64" else - /bin/sh $cmake_pkg --prefix=${pkg_install_dir} --skip-license > install.log 2>&1 || tail -n ${LOG_LINES} install.log + report_error ${LINENO} \ + "cmake installation for ARCH=${OPENBLAS_ARCH} is not supported. You can try to use the system installation using the flag --with-cmake=system instead." + exit 1 fi - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage0/$(basename ${SCRIPT_NAME})" - fi - fi - ;; - __SYSTEM__) - echo "==================== Finding CMake from system paths ====================" - check_command cmake "cmake" - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking CMake to user paths ====================" - pkg_install_dir="$with_cmake" - check_dir "${with_cmake}/bin" - ;; + pkg_install_dir="${INSTALLDIR}/cmake-${cmake_ver}" + #pkg_install_dir="${HOME}/apps/cmake/${cmake_ver}" + install_lock_file="$pkg_install_dir/install_successful" + cmake_pkg="cmake-${cmake_ver}-${cmake_arch}.${cmake_ext}" + if verify_checksums "${install_lock_file}"; then + echo "cmake-${cmake_ver} is already installed, skipping it." + else + if [ -f $cmake_pkg ]; then + echo "$cmake_pkg is found" + else + #download_pkg_from_ABACUS_org "${cmake_sha256}" "$cmake_pkg" + url="https://cmake.org/files/v${cmake_ver%.*}/${cmake_pkg}" + download_pkg_from_url "${cmake_sha256}" "${cmake_pkg}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + mkdir -p ${pkg_install_dir} + if [ "${cmake_arch}" = "macos-universal" ]; then + tar --strip-components=3 -xvf $cmake_pkg -C ${pkg_install_dir} > install.log 2>&1 || tail -n ${LOG_LINES} install.log + else + /bin/sh $cmake_pkg --prefix=${pkg_install_dir} --skip-license > install.log 2>&1 || tail -n ${LOG_LINES} install.log + fi + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage0/$(basename ${SCRIPT_NAME})" + fi + ;; + __SYSTEM__) + echo "==================== Finding CMake from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_command cmake "cmake" + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking CMake to user paths ====================" + pkg_install_dir="$with_cmake" + check_dir "${with_cmake}/bin" + ;; esac if [ "${with_cmake}" != "__DONTUSE__" ]; then - if [ "${with_cmake}" != "__SYSTEM__" ]; then - cat << EOF > "${BUILDDIR}/setup_cmake" + if [ "${with_cmake}" != "__SYSTEM__" ]; then + cat << EOF > "${BUILDDIR}/setup_cmake" prepend_path PATH "${pkg_install_dir}/bin" export PATH="${pkg_install_dir}/bin":\${PATH} EOF - cat "${BUILDDIR}/setup_cmake" >> $SETUPFILE - fi + cat "${BUILDDIR}/setup_cmake" >> $SETUPFILE + fi fi load "${BUILDDIR}/setup_cmake" diff --git a/toolchain/scripts/stage0/install_gcc.sh b/toolchain/scripts/stage0/install_gcc.sh index 65e9d93aa1..7930d6a9f7 100755 --- a/toolchain/scripts/stage0/install_gcc.sh +++ b/toolchain/scripts/stage0/install_gcc.sh @@ -7,16 +7,30 @@ [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -# gcc 13 is good -gcc_ver="13.2.0" -gcc_sha256="8cb4be3796651976f94b9356fa08d833524f62420d6292c5033a9a26af315078" -# use gcc 14 with caution, do not compatible with GPU-ABACUS -#gcc_ver="14.2.0" -#gcc_sha256="7d376d445f93126dc545e2c0086d0f647c3094aae081cdb78f42ce2bc25e7293" source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load GCC package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "gcc:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "gcc:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "gcc" "$version_suffix" + source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -29,101 +43,127 @@ TSANFLAGS="" cd "${BUILDDIR}" case "${with_gcc}" in - __INSTALL__) - echo "==================== Installing GCC ====================" - pkg_install_dir="${INSTALLDIR}/gcc-${gcc_ver}" - #pkg_install_dir="${HOME}/apps/gcc/${gcc_ver}" - install_lock_file="$pkg_install_dir/install_successful" - if verify_checksums "${install_lock_file}"; then - echo "gcc-${gcc_ver} is already installed, skipping it." - else - if [ -f gcc-${gcc_ver}.tar.gz ]; then - echo "gcc-${gcc_ver}.tar.gz is found" - else - #download_pkg_from_ABACUS_org "${gcc_sha256}" "gcc-${gcc_ver}.tar.gz" - url=https://mirrors.tuna.tsinghua.edu.cn/gnu/gcc/gcc-${gcc_ver}/gcc-${gcc_ver}.tar.gz - download_pkg_from_url "${gcc_sha256}" "gcc-${gcc_ver}.tar.gz" "${url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - [ -d gcc-${gcc_ver} ] && rm -rf gcc-${gcc_ver} - tar -xzf gcc-${gcc_ver}.tar.gz - - echo "Installing GCC from scratch into ${pkg_install_dir}" - cd gcc-${gcc_ver} - - # Download prerequisites from cp2k.org because gcc.gnu.org returns 403 when queried from GCP. - sed -i 's|http://gcc.gnu.org/pub/gcc/infrastructure/|https://cp2k.org/static/downloads/|' ./contrib/download_prerequisites - ./contrib/download_prerequisites > prereq.log 2>&1 || tail -n ${LOG_LINES} prereq.log - GCCROOT=${PWD} - mkdir obj - cd obj - # TODO: Maybe use --disable-libquadmath-support to improve static linking: - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46539 - # - # TODO: Maybe use --disable-gnu-unique-object to improve static linking: - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60348#c13 - # https://stackoverflow.com/questions/11931420 - # - # TODO: Unfortunately, we can not simply use --disable-shared, because - # it would break OpenBLAS build and probably others too. - COMMON_FLAGS="-O2 -fPIC -fno-omit-frame-pointer -fopenmp -g" - CFLAGS="${COMMON_FLAGS}" - CXXFLAGS="${CFLAGS}" - FCFLAGS="${COMMON_FLAGS} -fbacktrace" - ${GCCROOT}/configure --prefix="${pkg_install_dir}" \ - --libdir="${pkg_install_dir}/lib" \ - --enable-languages=c,c++,fortran \ - --disable-multilib --disable-bootstrap \ - --enable-lto \ - --enable-plugins \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - make -j $(get_nprocs) \ - CFLAGS="${CFLAGS}" \ - CXXFLAGS="${CXXFLAGS}" \ - FCFLAGS="${FCFLAGS}" \ - > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make -j $(get_nprocs) install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - # thread sanitizer - if [ ${ENABLE_TSAN} = "__TRUE__" ]; then - # now the tricky bit... we need to recompile in particular - # libgomp with -fsanitize=thread.. there is not configure - # option for this (as far as I know). we need to go in - # the build tree and recompile / reinstall with proper - # options... this is likely to break for later version of - # gcc, tested with 5.1.0 based on - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55374#c10 - cd x86_64*/libgfortran - make clean > clean.log 2>&1 || tail -n ${LOG_LINES} clean.log - CFLAGS="${CFLAGS} -fsanitize=thread" - CXXFLAGS="${CXXFLAGS} -fsanitize=thread" - FCFLAGS="${FCFLAGS} -fsanitize=thread" - make -j $(get_nprocs) \ - CFLAGS="${CFLAGS}" \ - CXXFLAGS="${CXXFLAGS}" \ - FCFLAGS="${FCFLAGS}" \ - LDFLAGS="-B$(pwd)/../libsanitizer/tsan/.libs/ -Wl,-rpath,$(pwd)/../libsanitizer/tsan/.libs/ -fsanitize=thread" \ - > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd ../libgomp - make clean > clean.log 2>&1 || tail -n ${LOG_LINES} clean.log - make -j $(get_nprocs) \ - CFLAGS="${CFLAGS}" \ - CXXFLAGS="${CXXFLAGS}" \ - FCFLAGS="${FCFLAGS}" \ - LDFLAGS="-B$(pwd)/../libsanitizer/tsan/.libs/ -Wl,-rpath,$(pwd)/../libsanitizer/tsan/.libs/ -fsanitize=thread" \ - > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd ${GCCROOT}/obj/ - fi - cd ../.. - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage0/$(basename ${SCRIPT_NAME})" - fi - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else + __INSTALL__) + echo "==================== Installing GCC ====================" + pkg_install_dir="${INSTALLDIR}/gcc-${gcc_ver}" + repack_filename="gcc-${gcc_ver}-with-prereq.tar.gz" + repkg_install_dir="${INSTALLDIR}/${repack_filename}" + #pkg_install_dir="${HOME}/apps/gcc/${gcc_ver}" + install_lock_file="$pkg_install_dir/install_successful" + if verify_checksums "${install_lock_file}"; then + echo "gcc-${gcc_ver} is already installed, skipping it." + else + # Check if repackaged tarball exists for offline installation + if [ -f "${repack_filename}" ]; then + echo "${repack_filename} is found, extracting it..." + tar -xzf "${repack_filename}" + echo "Successfully extracted ${repack_filename}" + else + if [ -f gcc-${gcc_ver}.tar.gz ]; then + echo "gcc-${gcc_ver}.tar.gz is found" + else + #download_pkg_from_ABACUS_org "${gcc_sha256}" "gcc-${gcc_ver}.tar.gz" + url=https://mirrors.tuna.tsinghua.edu.cn/gnu/gcc/gcc-${gcc_ver}/gcc-${gcc_ver}.tar.gz + download_pkg_from_url "${gcc_sha256}" "gcc-${gcc_ver}.tar.gz" "${url}" + fi + fi + [ -d gcc-${gcc_ver} ] && rm -rf gcc-${gcc_ver} + tar -xzf gcc-${gcc_ver}.tar.gz + + echo "Installing GCC from scratch into ${pkg_install_dir}" + cd gcc-${gcc_ver} + + # Check network connectivity before downloading prerequisites + if curl -s --connect-timeout 5 https://gcc.gnu.org > /dev/null 2>&1; then + echo "Downloading prerequisites from official GCC site..." + # Try official site first + if ./contrib/download_prerequisites > prereq.log 2>&1; then + echo "Prerequisites downloaded successfully from official site" + else + echo "Official site failed, trying mirror site..." + # Fallback to cp2k.org mirror + sed -i 's|http://gcc.gnu.org/pub/gcc/infrastructure/|https://cp2k.org/static/downloads/|' ./contrib/download_prerequisites + ./contrib/download_prerequisites > prereq.log 2>&1 || tail -n ${LOG_LINES} prereq.log + fi + else + echo "Network unavailable, skipping prerequisites download (offline mode)" + fi + + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode: repackaging GCC with prerequisites for offline installation" + cd .. + # Create a new tarball with prerequisites included + repack_filename="gcc-${gcc_ver}-with-prereq.tar.gz" + echo "Creating ${repack_filename}..." + tar -czf "${repack_filename}" gcc-${gcc_ver}/ + echo "Successfully created ${repack_filename} for offline installation" + echo "This package contains GCC source code with all prerequisites ready for offline build" + exit 0 + fi + GCCROOT=${PWD} + mkdir obj + cd obj + # TODO: Maybe use --disable-libquadmath-support to improve static linking: + # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46539 + # + # TODO: Maybe use --disable-gnu-unique-object to improve static linking: + # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60348#c13 + # https://stackoverflow.com/questions/11931420 + # + # TODO: Unfortunately, we can not simply use --disable-shared, because + # it would break OpenBLAS build and probably others too. + COMMON_FLAGS="-O2 -fPIC -fno-omit-frame-pointer -fopenmp -g" + CFLAGS="${COMMON_FLAGS}" + CXXFLAGS="${CFLAGS}" + FCFLAGS="${COMMON_FLAGS} -fbacktrace" + ${GCCROOT}/configure --prefix="${pkg_install_dir}" \ + --libdir="${pkg_install_dir}/lib" \ + --enable-languages=c,c++,fortran \ + --disable-multilib --disable-bootstrap \ + --enable-lto \ + --enable-plugins \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + make -j $(get_nprocs) \ + CFLAGS="${CFLAGS}" \ + CXXFLAGS="${CXXFLAGS}" \ + FCFLAGS="${FCFLAGS}" \ + > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make -j $(get_nprocs) install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + # thread sanitizer + if [ "${ENABLE_TSAN}" = "__TRUE__" ]; then + # now the tricky bit... we need to recompile in particular + # libgomp with -fsanitize=thread.. there is not configure + # option for this (as far as I know). we need to go in + # the build tree and recompile / reinstall with proper + # options... this is likely to break for later version of + # gcc, tested with 5.1.0 based on + # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55374#c10 + cd x86_64*/libgfortran + make clean > clean.log 2>&1 || tail -n ${LOG_LINES} clean.log + CFLAGS="${CFLAGS} -fsanitize=thread" + CXXFLAGS="${CXXFLAGS} -fsanitize=thread" + FCFLAGS="${FCFLAGS} -fsanitize=thread" + make -j $(get_nprocs) \ + CFLAGS="${CFLAGS}" \ + CXXFLAGS="${CXXFLAGS}" \ + FCFLAGS="${FCFLAGS}" \ + LDFLAGS="-B$(pwd)/../libsanitizer/tsan/.libs/ -Wl,-rpath,$(pwd)/../libsanitizer/tsan/.libs/ -fsanitize=thread" \ + > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd ../libgomp + make clean > clean.log 2>&1 || tail -n ${LOG_LINES} clean.log + make -j $(get_nprocs) \ + CFLAGS="${CFLAGS}" \ + CXXFLAGS="${CXXFLAGS}" \ + FCFLAGS="${FCFLAGS}" \ + LDFLAGS="-B$(pwd)/../libsanitizer/tsan/.libs/ -Wl,-rpath,$(pwd)/../libsanitizer/tsan/.libs/ -fsanitize=thread" \ + > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd ${GCCROOT}/obj/ + fi + cd ../.. + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage0/$(basename ${SCRIPT_NAME})" + fi check_install ${pkg_install_dir}/bin/gcc "gcc" && CC="${pkg_install_dir}/bin/gcc" || exit 1 check_install ${pkg_install_dir}/bin/g++ "gcc" && CXX="${pkg_install_dir}/bin/g++" || exit 1 check_install ${pkg_install_dir}/bin/gfortran "gcc" && FC="${pkg_install_dir}/bin/gfortran" || exit 1 @@ -131,52 +171,55 @@ case "${with_gcc}" in F77="${FC}" GCC_CFLAGS="-I'${pkg_install_dir}/include'" GCC_LDFLAGS="-L'${pkg_install_dir}/lib64' -L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib64' -Wl,-rpath,'${pkg_install_dir}/lib64'" - fi - ;; - __SYSTEM__) - echo "==================== Finding GCC from system paths ====================" - check_command gcc "gcc" && CC="$(command -v gcc)" || exit 1 - check_command g++ "gcc" && CXX="$(command -v g++)" || exit 1 - check_command gfortran "gcc" && FC="$(command -v gfortran)" || exit 1 - F90="${FC}" - F77="${FC}" - add_include_from_paths -p GCC_CFLAGS "c++" ${INCLUDE_PATHS} - add_lib_from_paths GCC_LDFLAGS "libgfortran.*" ${LIB_PATHS} - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking GCC to user paths ====================" - pkg_install_dir="${with_gcc}" - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/lib64" - check_dir "${pkg_install_dir}/include" - check_command ${pkg_install_dir}/bin/gcc "gcc" && CC="${pkg_install_dir}/bin/gcc" || exit 1 - check_command ${pkg_install_dir}/bin/g++ "gcc" && CXX="${pkg_install_dir}/bin/g++" || exit 1 - check_command ${pkg_install_dir}/bin/gfortran "gcc" && FC="${pkg_install_dir}/bin/gfortran" || exit 1 - F90="${FC}" - F77="${FC}" - GCC_CFLAGS="-I'${pkg_install_dir}/include'" - GCC_LDFLAGS="-L'${pkg_install_dir}/lib64' -L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib64' -Wl,-rpath,'${pkg_install_dir}/lib64'" - ;; + ;; + __SYSTEM__) + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + echo "==================== Finding GCC from system paths ====================" + check_command gcc "gcc" && CC="$(command -v gcc)" || exit 1 + check_command g++ "gcc" && CXX="$(command -v g++)" || exit 1 + check_command gfortran "gcc" && FC="$(command -v gfortran)" || exit 1 + F90="${FC}" + F77="${FC}" + add_include_from_paths -p GCC_CFLAGS "c++" ${INCLUDE_PATHS} + add_lib_from_paths GCC_LDFLAGS "libgfortran.*" ${LIB_PATHS} + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking GCC to user paths ====================" + pkg_install_dir="${with_gcc}" + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/lib64" + check_dir "${pkg_install_dir}/include" + check_command ${pkg_install_dir}/bin/gcc "gcc" && CC="${pkg_install_dir}/bin/gcc" || exit 1 + check_command ${pkg_install_dir}/bin/g++ "gcc" && CXX="${pkg_install_dir}/bin/g++" || exit 1 + check_command ${pkg_install_dir}/bin/gfortran "gcc" && FC="${pkg_install_dir}/bin/gfortran" || exit 1 + F90="${FC}" + F77="${FC}" + GCC_CFLAGS="-I'${pkg_install_dir}/include'" + GCC_LDFLAGS="-L'${pkg_install_dir}/lib64' -L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib64' -Wl,-rpath,'${pkg_install_dir}/lib64'" + ;; esac if [ "${ENABLE_TSAN}" = "__TRUE__" ]; then - TSANFLAGS="-fsanitize=thread" + TSANFLAGS="-fsanitize=thread" else - TSANFLAGS="" + TSANFLAGS="" fi if [ "${with_gcc}" != "__DONTUSE__" ]; then - cat << EOF > "${BUILDDIR}/setup_gcc" + cat << EOF > "${BUILDDIR}/setup_gcc" export CC="${CC}" export CXX="${CXX}" export FC="${FC}" export F90="${F90}" export F77="${F77}" EOF - if [ "${with_gcc}" != "__SYSTEM__" ]; then - cat << EOF >> "${BUILDDIR}/setup_gcc" + if [ "${with_gcc}" != "__SYSTEM__" ]; then + cat << EOF >> "${BUILDDIR}/setup_gcc" # needs full path for mpich/openmpi builds, triggers openblas bug prepend_path PATH "${pkg_install_dir}/bin" prepend_path LD_LIBRARY_PATH "${pkg_install_dir}/lib" @@ -194,13 +237,13 @@ export LIBRARY_PATH="${pkg_install_dir}/lib":\${LIBRARY_PATH} export LIBRARY_PATH="${pkg_install_dir}/lib64":\${LIBRARY_PATH} export CPATH="${pkg_install_dir}/include":\${CPATH} EOF - fi - cat << EOF >> "${BUILDDIR}/setup_gcc" + fi + cat << EOF >> "${BUILDDIR}/setup_gcc" export GCC_CFLAGS="${GCC_CFLAGS}" export GCC_LDFLAGS="${GCC_LDFLAGS}" export TSANFLAGS="${TSANFLAGS}" EOF - cat "${BUILDDIR}/setup_gcc" >> ${SETUPFILE} + cat "${BUILDDIR}/setup_gcc" >> ${SETUPFILE} fi # ---------------------------------------------------------------------- diff --git a/toolchain/scripts/stage0/install_intel.sh b/toolchain/scripts/stage0/install_intel.sh index afec2e312f..53ef5527ea 100755 --- a/toolchain/scripts/stage0/install_intel.sh +++ b/toolchain/scripts/stage0/install_intel.sh @@ -23,95 +23,87 @@ mkdir -p "${BUILDDIR}" cd "${BUILDDIR}" case "${with_intel}" in - __INSTALL__) - echo "==================== Installing the Intel compiler ====================" - echo "__INSTALL__ is not supported; please install the Intel compiler manually" - if [ "${PACK_RUN}" != "__TRUE__" ]; then + __INSTALL__) + echo "==================== Installing the Intel compiler ====================" + report_error ${LINENO} "To install the Intel compiler, please contact your system administrator." exit 1 - fi - exit 1 - ;; - __SYSTEM__) - echo "==================== Finding Intel compiler from system paths ====================" - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else - if [ "${intel_classic}" = "yes" ]; then - check_command icc "intel" && CC="$(realpath $(command -v icc))" || exit 1 - check_command icpc "intel" && CXX="$(realpath $(command -v icpc))" || exit 1 - check_command ifort "intel" && FC="$(realpath $(command -v ifort))" || exit 1 - else - check_command icx "intel" && CC="$(realpath $(command -v icx))" || exit 1 - check_command icpx "intel" && CXX="$(realpath $(command -v icpx))" || exit 1 - if [ "${WITH_IFX}" = "yes" ]; then - check_command ifx "intel" && FC="$(realpath $(command -v ifx))" || exit 1 - else - check_command ifort "intel" && FC="$(realpath $(command -v ifort))" || exit 1 - fi - fi - fi - F90="${FC}" - F77="${FC}" - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking Intel compiler to user paths ====================" - pkg_install_dir="${with_intel}" - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - if [ "${intel_classic}" = "yes" ]; then - check_command ${pkg_install_dir}/bin/icc "intel" && CC="${pkg_install_dir}/bin/icc" || exit 1 - check_command ${pkg_install_dir}/bin/icpc "intel" && CXX="${pkg_install_dir}/bin/icpc" || exit 1 - check_command ${pkg_install_dir}/bin/ifort "intel" && FC="${pkg_install_dir}/bin/ifort" || exit 1 - else - check_command ${pkg_install_dir}/bin/icx "intel" && CC="${pkg_install_dir}/bin/icx" || exit 1 - check_command ${pkg_install_dir}/bin/icpx "intel" && CXX="${pkg_install_dir}/bin/icpx" || exit 1 - if [ "${WITH_IFX}" = "yes" ]; then - check_command ${pkg_install_dir}/bin/ifx "intel" && FC="${pkg_install_dir}/bin/ifx" || exit 1 - else - check_command ${pkg_install_dir}/bin/ifort "intel" && FC="${pkg_install_dir}/bin/ifort" || exit 1 - fi - fi - F90="${FC}" - F77="${FC}" - INTEL_CFLAGS="-I'${pkg_install_dir}/include'" - INTEL_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + ;; + __SYSTEM__) + echo "==================== Finding Intel compiler from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + if [ "${intel_classic}" = "yes" ]; then + check_command icc "intel" && CC="$(realpath $(command -v icc))" || exit 1 + check_command icpc "intel" && CXX="$(realpath $(command -v icpc))" || exit 1 + check_command ifort "intel" && FC="$(realpath $(command -v ifort))" || exit 1 + else + check_command icx "intel" && CC="$(realpath $(command -v icx))" || exit 1 + check_command icpx "intel" && CXX="$(realpath $(command -v icpx))" || exit 1 + if [ "${WITH_IFX}" = "yes" ]; then + check_command ifx "intel" && FC="$(realpath $(command -v ifx))" || exit 1 + else + check_command ifort "intel" && FC="$(realpath $(command -v ifort))" || exit 1 + fi + fi + F90="${FC}" + F77="${FC}" + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking Intel compiler to user paths ====================" + pkg_install_dir="${with_intel}" + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + if [ "${intel_classic}" = "yes" ]; then + check_command ${pkg_install_dir}/bin/icc "intel" && CC="${pkg_install_dir}/bin/icc" || exit 1 + check_command ${pkg_install_dir}/bin/icpc "intel" && CXX="${pkg_install_dir}/bin/icpc" || exit 1 + check_command ${pkg_install_dir}/bin/ifort "intel" && FC="${pkg_install_dir}/bin/ifort" || exit 1 + else + check_command ${pkg_install_dir}/bin/icx "intel" && CC="${pkg_install_dir}/bin/icx" || exit 1 + check_command ${pkg_install_dir}/bin/icpx "intel" && CXX="${pkg_install_dir}/bin/icpx" || exit 1 + if [ "${WITH_IFX}" = "yes" ]; then + check_command ${pkg_install_dir}/bin/ifx "intel" && FC="${pkg_install_dir}/bin/ifx" || exit 1 + else + check_command ${pkg_install_dir}/bin/ifort "intel" && FC="${pkg_install_dir}/bin/ifort" || exit 1 + fi + fi + F90="${FC}" + F77="${FC}" + INTEL_CFLAGS="-I'${pkg_install_dir}/include'" + INTEL_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac if [ "${with_intel}" != "__DONTUSE__" ]; then - echo "CC is ${CC}" - echo "CXX is ${CXX}" - echo "FC is ${FC}" - cat << EOF > "${BUILDDIR}/setup_intel" + echo "CC is ${CC}" + echo "CXX is ${CXX}" + echo "FC is ${FC}" + cat << EOF > "${BUILDDIR}/setup_intel" export CC="${CC}" export CXX="${CXX}" export FC="${FC}" export F90="${F90}" export F77="${F77}" EOF - if [ "${with_intel}" != "__SYSTEM__" ]; then - cat << EOF >> "${BUILDDIR}/setup_intel" + if [ "${with_intel}" != "__SYSTEM__" ]; then + cat << EOF >> "${BUILDDIR}/setup_intel" prepend_path PATH "${pkg_install_dir}/bin" prepend_path LD_LIBRARY_PATH "${pkg_install_dir}/lib" prepend_path LD_RUN_PATH "${pkg_install_dir}/lib" prepend_path LIBRARY_PATH "${pkg_install_dir}/lib" prepend_path CPATH "${pkg_install_dir}/include" -export PATH="${pkg_install_dir}/bin":\${PATH} -export LD_LIBRARY_PATH="${pkg_install_dir}/lib":\${LD_LIBRARY_PATH} -export LD_RUN_PATH="${pkg_install_dir}/lib":\${LD_RUN_PATH} -export LIBRARY_PATH="${pkg_install_dir}/lib":\${LIBRARY_PATH} -export CPATH="${pkg_install_dir}/include":\${CPATH} EOF - fi - cat << EOF >> "${BUILDDIR}/setup_intel" + fi + cat << EOF >> "${BUILDDIR}/setup_intel" export INTEL_CFLAGS="${INTEL_CFLAGS}" export INTEL_LDFLAGS="${INTEL_LDFLAGS}" export INTEL_LIBS="${INTEL_LIBS}" EOF - cat "${BUILDDIR}/setup_intel" >> ${SETUPFILE} + cat "${BUILDDIR}/setup_intel" >> ${SETUPFILE} fi load "${BUILDDIR}/setup_intel" diff --git a/toolchain/scripts/stage1/install_intelmpi.sh b/toolchain/scripts/stage1/install_intelmpi.sh index 0c350f69ae..5f311c4b5e 100755 --- a/toolchain/scripts/stage1/install_intelmpi.sh +++ b/toolchain/scripts/stage1/install_intelmpi.sh @@ -24,107 +24,105 @@ mkdir -p "${BUILDDIR}" cd "${BUILDDIR}" case "${with_intelmpi}" in - __INSTALL__) - echo "==================== Installing Intel MPI ====================" - echo '__INSTALL__ is not supported; please manually install Intel MPI' - if [ "${PACK_RUN}" != "__TRUE__" ]; then + __INSTALL__) + echo "==================== Installing Intel MPI ====================" + report_error ${LINENO} "To install Intel MPI, please contact your system administrator." exit 1 - fi - ;; - __SYSTEM__) - echo "==================== Finding Intel MPI from system paths ====================" - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else - check_command mpiexec "intelmpi" && MPIRUN="$(realpath $(command -v mpiexec))" - if [ "${intel_classic}" = "yes" ]; then - # if intel compiler used as classic, so as intelmpi - export INTELMPI_CLASSIC="yes" - fi - if [ "${with_intel}" != "__DONTUSE__" ]; then - if [ "${INTELMPI_CLASSIC}" = "yes" ]; then - check_command mpiicc "intelmpi" && MPICC="$(realpath $(command -v mpiicc))" || exit 1 - check_command mpiicpc "intelmpi" && MPICXX="$(realpath $(command -v mpiicpc))" || exit 1 - check_command mpiifort "intelmpi" && MPIFC="$(realpath $(command -v mpiifort))" || exit 1 + ;; + __SYSTEM__) + echo "==================== Finding Intel MPI from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" else - check_command mpiicx "intelmpi" && MPICC="$(realpath $(command -v mpiicx))" || exit 1 - check_command mpiicpx "intelmpi" && MPICXX="$(realpath $(command -v mpiicpx))" || exit 1 - if [ "${WITH_IFX}" == "yes" ]; then - check_command mpiifx "intelmpi" && MPIFC="$(realpath $(command -v mpiifx))" || exit 1 + check_command mpiexec "intelmpi" && MPIRUN="$(realpath $(command -v mpiexec))" + if [ "${intel_classic}" = "yes" ]; then + # if intel compiler used as classic, so as intelmpi + export INTELMPI_CLASSIC="yes" + fi + if [ "${with_intel}" != "__DONTUSE__" ]; then + if [ "${INTELMPI_CLASSIC}" = "yes" ]; then + check_command mpiicc "intelmpi" && MPICC="$(realpath $(command -v mpiicc))" || exit 1 + check_command mpiicpc "intelmpi" && MPICXX="$(realpath $(command -v mpiicpc))" || exit 1 + check_command mpiifort "intelmpi" && MPIFC="$(realpath $(command -v mpiifort))" || exit 1 + else + check_command mpiicx "intelmpi" && MPICC="$(realpath $(command -v mpiicx))" || exit 1 + check_command mpiicpx "intelmpi" && MPICXX="$(realpath $(command -v mpiicpx))" || exit 1 + if [ "${WITH_IFX}" == "yes" ]; then + check_command mpiifx "intelmpi" && MPIFC="$(realpath $(command -v mpiifx))" || exit 1 + else + check_command mpiifort "intelmpi" && MPIFC="$(realpath $(command -v mpiifort))" || exit 1 + fi + fi else - check_command mpiifort "intelmpi" && MPIFC="$(realpath $(command -v mpiifort))" || exit 1 + echo "The use of Intel MPI is only supported with the Intel compiler" + exit 1 fi + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + # include path is already handled by compiler wrapper scripts (can cause wrong mpi.mod with GNU Fortran) + # add_include_from_paths INTELMPI_CFLAGS "mpi.h" $INCLUDE_PATHS + add_lib_from_paths INTELMPI_LDFLAGS "libmpi.*" $LIB_PATHS + check_lib -lmpi "intelmpi" + check_lib -lmpicxx "intelmpi" fi - else - echo "The use of Intel MPI is only supported with the Intel compiler" - exit 1 - fi - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - # include path is already handled by compiler wrapper scripts (can cause wrong mpi.mod with GNU Fortran) - # add_include_from_paths INTELMPI_CFLAGS "mpi.h" $INCLUDE_PATHS - add_lib_from_paths INTELMPI_LDFLAGS "libmpi.*" $LIB_PATHS - check_lib -lmpi "intelmpi" - check_lib -lmpicxx "intelmpi" - fi - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking INTELMPI to user paths ====================" - pkg_install_dir="${with_intelmpi}" - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - check_command ${pkg_install_dir}/bin/mpiexec "intel" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 - if [ "${with_intel}" != "__DONTUSE__" ]; then - if [ "${INTELMPI_CLASSIC}" = "yes" ]; then - check_command ${pkg_install_dir}/bin/mpiicc "intel" && MPICC="${pkg_install_dir}/bin/mpiicc" || exit 1 - check_command ${pkg_install_dir}/bin/mpiicpc "intel" && MPICXX="${pkg_install_dir}/bin/mpiicpc" || exit 1 - check_command ${pkg_install_dir}/bin/mpiifort "intel" && MPIFC="${pkg_install_dir}/bin/mpiifort" || exit 1 - else - check_command ${pkg_install_dir}/bin/mpiicx "intel" && MPICC="${pkg_install_dir}/bin/mpiicx" || exit 1 - check_command ${pkg_install_dir}/bin/mpiicpx "intel" && MPICXX="${pkg_install_dir}/bin/mpiicpx" || exit 1 - if [ "${WITH_IFX}" = "yes" ]; then - check_command ${pkg_install_dir}/bin/mpiifx "intel" && MPIFC="${pkg_install_dir}/bin/mpiifx" || exit 1 - else + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking INTELMPI to user paths ====================" + pkg_install_dir="${with_intelmpi}" + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + check_command ${pkg_install_dir}/bin/mpiexec "intel" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 + if [ "${with_intel}" != "__DONTUSE__" ]; then + if [ "${INTELMPI_CLASSIC}" = "yes" ]; then + check_command ${pkg_install_dir}/bin/mpiicc "intel" && MPICC="${pkg_install_dir}/bin/mpiicc" || exit 1 + check_command ${pkg_install_dir}/bin/mpiicpc "intel" && MPICXX="${pkg_install_dir}/bin/mpiicpc" || exit 1 check_command ${pkg_install_dir}/bin/mpiifort "intel" && MPIFC="${pkg_install_dir}/bin/mpiifort" || exit 1 + else + check_command ${pkg_install_dir}/bin/mpiicx "intel" && MPICC="${pkg_install_dir}/bin/mpiicx" || exit 1 + check_command ${pkg_install_dir}/bin/mpiicpx "intel" && MPICXX="${pkg_install_dir}/bin/mpiicpx" || exit 1 + if [ "${WITH_IFX}" = "yes" ]; then + check_command ${pkg_install_dir}/bin/mpiifx "intel" && MPIFC="${pkg_install_dir}/bin/mpiifx" || exit 1 + else + check_command ${pkg_install_dir}/bin/mpiifort "intel" && MPIFC="${pkg_install_dir}/bin/mpiifort" || exit 1 + fi fi + else + echo "The use of Intel MPI is only supported with the Intel compiler" + exit 1 fi - else - echo "The use of Intel MPI is only supported with the Intel compiler" - exit 1 - fi - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - # include path is already handled by compiler wrapper scripts (can cause wrong mpi.mod with GNU Fortran) - INTELMPI_CFLAGS="-I'${pkg_install_dir}/include'" - INTELMPI_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + # include path is already handled by compiler wrapper scripts (can cause wrong mpi.mod with GNU Fortran) + INTELMPI_CFLAGS="-I'${pkg_install_dir}/include'" + INTELMPI_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac if [ "${with_intelmpi}" != "__DONTUSE__" ]; then - if [ "${intel_classic}" = "yes" ]; then - I_MPI_CXX="icpc" - I_MPI_CC="icc" - I_MPI_FC="ifort" - else - I_MPI_CXX="icpx" - I_MPI_CC="icx" - if [ "${WITH_IFX}" = "yes" ]; then - I_MPI_FC="ifx" + if [ "${intel_classic}" = "yes" ]; then + I_MPI_CXX="icpc" + I_MPI_CC="icc" + I_MPI_FC="ifort" else - I_MPI_FC="ifort" + I_MPI_CXX="icpx" + I_MPI_CC="icx" + if [ "${WITH_IFX}" = "yes" ]; then + I_MPI_FC="ifx" + else + I_MPI_FC="ifort" + fi fi - fi - INTELMPI_LIBS="-lmpi -lmpicxx" - echo "I_MPI_CXX is ${I_MPI_CXX}" - echo "I_MPI_CC is ${I_MPI_CC}" - echo "I_MPI_FC is ${I_MPI_FC}" - echo "MPICXX is ${MPICXX}" - echo "MPICC is ${MPICC}" - echo "MPIFC is ${MPIFC}" - cat << EOF > "${BUILDDIR}/setup_intelmpi" + INTELMPI_LIBS="-lmpi -lmpicxx" + echo "I_MPI_CXX is ${I_MPI_CXX}" + echo "I_MPI_CC is ${I_MPI_CC}" + echo "I_MPI_FC is ${I_MPI_FC}" + echo "MPICXX is ${MPICXX}" + echo "MPICC is ${MPICC}" + echo "MPIFC is ${MPIFC}" + cat << EOF > "${BUILDDIR}/setup_intelmpi" export I_MPI_CXX="${I_MPI_CXX}" export I_MPI_CC="${I_MPI_CC}" export I_MPI_FC="${I_MPI_FC}" @@ -146,8 +144,8 @@ export CP_CFLAGS="\${CP_CFLAGS} IF_MPI(${INTELMPI_CFLAGS}|)" export CP_LDFLAGS="\${CP_LDFLAGS} IF_MPI(${INTELMPI_LDFLAGS}|)" export CP_LIBS="\${CP_LIBS} IF_MPI(${INTELMPI_LIBS}|)" EOF - if [ "${with_intelmpi}" != "__SYSTEM__" ]; then - cat << EOF >> "${BUILDDIR}/setup_intelmpi" + if [ "${with_intelmpi}" != "__SYSTEM__" ]; then + cat << EOF >> "${BUILDDIR}/setup_intelmpi" prepend_path PATH "${pkg_install_dir}/bin" export PATH="${pkg_install_dir}/bin":\${PATH} export LD_LIBRARY_PATH="${pkg_install_dir}/lib":\${LD_LIBRARY_PATH} @@ -155,8 +153,8 @@ export LD_RUN_PATH="${pkg_install_dir}/lib":\${LD_RUN_PATH} export LIBRARY_PATH="${pkg_install_dir}/lib":\${LIBRARY_PATH} export CPATH="${pkg_install_dir}/include":\${CPATH} EOF - fi - cat "${BUILDDIR}/setup_intelmpi" >> ${SETUPFILE} + fi + cat "${BUILDDIR}/setup_intelmpi" >> ${SETUPFILE} fi load "${BUILDDIR}/setup_intelmpi" diff --git a/toolchain/scripts/stage1/install_mpich.sh b/toolchain/scripts/stage1/install_mpich.sh index 2b80c73892..cafda9200d 100755 --- a/toolchain/scripts/stage1/install_mpich.sh +++ b/toolchain/scripts/stage1/install_mpich.sh @@ -3,17 +3,29 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2025-0308 - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -# mpich_ver="4.0.3" -# mpich_sha256="17406ea90a6ed4ecd5be39c9ddcbfac9343e6ab4f77ac4e8c5ebe4a3e3b6c501" -# mpich_ver="4.1.2" -# mpich_sha256="3492e98adab62b597ef0d292fb2459b6123bc80070a8aa0a30be6962075a12f0" -mpich_ver="4.3.0" -mpich_sha256="5e04132984ad83cab9cc53f76072d2b5ef5a6d24b0a9ff9047a8ff96121bcc63" +# Load centralized version management +source "${SCRIPT_DIR}"/package_versions.sh + +# Load MPICH package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "mpich:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "mpich:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "mpich" "$version_suffix" mpich_pkg="mpich-${mpich_ver}.tar.gz" source "${SCRIPT_DIR}"/common_vars.sh @@ -32,111 +44,115 @@ MPICH_LIBS="" cd "${BUILDDIR}" case "${with_mpich}" in - __INSTALL__) - echo "==================== Installing MPICH ====================" - pkg_install_dir="${INSTALLDIR}/mpich-${mpich_ver}" - #pkg_install_dir="${HOME}/apps/mpich/${mpich_ver}-intel" - install_lock_file="$pkg_install_dir/install_successful" - url="https://www.mpich.org/static/downloads/${mpich_ver}/${mpich_pkg}" - if verify_checksums "${install_lock_file}"; then - echo "mpich-${mpich_ver} is already installed, skipping it." - else - if [ -f ${mpich_pkg} ]; then - echo "${mpich_pkg} is found" - else - #download_pkg_from_ABACUS_org "${mpich_sha256}" "${mpich_pkg}" - download_pkg_from_url "${mpich_sha256}" "${mpich_pkg}" "${url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir} for MPICH device ${MPICH_DEVICE}" - [ -d mpich-${mpich_ver} ] && rm -rf mpich-${mpich_ver} - tar -xzf ${mpich_pkg} - cd mpich-${mpich_ver} - unset F90 - unset F90FLAGS + __INSTALL__) + echo "==================== Installing MPICH ====================" + pkg_install_dir="${INSTALLDIR}/mpich-${mpich_ver}" + #pkg_install_dir="${HOME}/apps/mpich/${mpich_ver}-intel" + install_lock_file="$pkg_install_dir/install_successful" + url="https://www.mpich.org/static/downloads/${mpich_ver}/${mpich_pkg}" + if verify_checksums "${install_lock_file}"; then + echo "mpich-${mpich_ver} is already installed, skipping it." + else + if [ -f ${mpich_pkg} ]; then + echo "${mpich_pkg} is found" + else + #download_pkg_from_ABACUS_org "${mpich_sha256}" "${mpich_pkg}" + download_pkg_from_url "${mpich_sha256}" "${mpich_pkg}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir} for MPICH device ${MPICH_DEVICE}" + [ -d mpich-${mpich_ver} ] && rm -rf mpich-${mpich_ver} + tar -xzf ${mpich_pkg} + cd mpich-${mpich_ver} + unset F90 + unset F90FLAGS - # workaround for compilation with GCC >= 10, until properly fixed: - # https://github.com/pmodels/mpich/issues/4300 - if ("${FC}" --version | grep -q 'GNU'); then - compat_flag=$(allowed_gfortran_flags "-fallow-argument-mismatch") - fi - ./configure \ - --prefix="${pkg_install_dir}" \ - --libdir="${pkg_install_dir}/lib" \ - MPICC="" \ - FFLAGS="${FCFLAGS} ${compat_flag}" \ - FCFLAGS="${FCFLAGS} ${compat_flag}" \ - --with-device=${MPICH_DEVICE} \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd .. - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage1/$(basename ${SCRIPT_NAME})" - fi - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - check_install ${pkg_install_dir}/bin/mpiexec "mpich" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 - check_install ${pkg_install_dir}/bin/mpicc "mpich" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 - check_install ${pkg_install_dir}/bin/mpicxx "mpich" && MPICXX="${pkg_install_dir}/bin/mpicxx" || exit 1 - check_install ${pkg_install_dir}/bin/mpifort "mpich" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - MPICH_CFLAGS="-I'${pkg_install_dir}/include'" - MPICH_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - fi - ;; - __SYSTEM__) - echo "==================== Finding MPICH from system paths ====================" - check_command mpiexec "mpich" && MPIRUN="$(command -v mpiexec)" - check_command mpicc "mpich" && MPICC="$(command -v mpicc)" || exit 1 - if [ $(command -v mpic++ > /dev/null 2>&1) ]; then - check_command mpic++ "mpich" && MPICXX="$(command -v mpic++)" || exit 1 - else - check_command mpicxx "mpich" && MPICXX="$(command -v mpicxx)" || exit 1 - fi - check_command mpifort "mpich" && MPIFC="$(command -v mpifort)" || exit 1 - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - check_lib -lmpifort "mpich" - check_lib -lmpicxx "mpich" - check_lib -lmpi "mpich" - add_include_from_paths MPICH_CFLAGS "mpi.h" ${INCLUDE_PATHS} - add_lib_from_paths MPICH_LDFLAGS "libmpi.*" ${LIB_PATHS} - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking MPICH to user paths ====================" - pkg_install_dir="${with_mpich}" - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - check_command ${pkg_install_dir}/bin/mpiexec "mpich" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 - check_command ${pkg_install_dir}/bin/mpicc "mpich" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 - check_command ${pkg_install_dir}/bin/mpicxx "mpich" && MPICXX="${pkg_install_dir}/bin/mpicxx" || exit 1 - check_command ${pkg_install_dir}/bin/mpifort "mpich" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - MPICH_CFLAGS="-I'${pkg_install_dir}/include'" - MPICH_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + # workaround for compilation with GCC >= 10, until properly fixed: + # https://github.com/pmodels/mpich/issues/4300 + if ("${FC}" --version | grep -q 'GNU'); then + compat_flag=$(allowed_gfortran_flags "-fallow-argument-mismatch") + fi + ./configure \ + --prefix="${pkg_install_dir}" \ + --libdir="${pkg_install_dir}/lib" \ + MPICC="" \ + FFLAGS="${FCFLAGS} ${compat_flag}" \ + FCFLAGS="${FCFLAGS} ${compat_flag}" \ + --with-device=${MPICH_DEVICE} \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd .. + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage1/$(basename ${SCRIPT_NAME})" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + else + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + check_install ${pkg_install_dir}/bin/mpiexec "mpich" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 + check_install ${pkg_install_dir}/bin/mpicc "mpich" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 + check_install ${pkg_install_dir}/bin/mpicxx "mpich" && MPICXX="${pkg_install_dir}/bin/mpicxx" || exit 1 + check_install ${pkg_install_dir}/bin/mpifort "mpich" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + MPICH_CFLAGS="-I'${pkg_install_dir}/include'" + MPICH_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + fi + ;; + __SYSTEM__) + echo "==================== Finding MPICH from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_command mpiexec "mpich" && MPIRUN="$(command -v mpiexec)" + check_command mpicc "mpich" && MPICC="$(command -v mpicc)" || exit 1 + if [ $(command -v mpic++ > /dev/null 2>&1) ]; then + check_command mpic++ "mpich" && MPICXX="$(command -v mpic++)" || exit 1 + else + check_command mpicxx "mpich" && MPICXX="$(command -v mpicxx)" || exit 1 + fi + check_command mpifort "mpich" && MPIFC="$(command -v mpifort)" || exit 1 + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + check_lib -lmpifort "mpich" + check_lib -lmpicxx "mpich" + check_lib -lmpi "mpich" + add_include_from_paths MPICH_CFLAGS "mpi.h" ${INCLUDE_PATHS} + add_lib_from_paths MPICH_LDFLAGS "libmpi.*" ${LIB_PATHS} + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking MPICH to user paths ====================" + pkg_install_dir="${with_mpich}" + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + check_command ${pkg_install_dir}/bin/mpiexec "mpich" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 + check_command ${pkg_install_dir}/bin/mpicc "mpich" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 + check_command ${pkg_install_dir}/bin/mpicxx "mpich" && MPICXX="${pkg_install_dir}/bin/mpicxx" || exit 1 + check_command ${pkg_install_dir}/bin/mpifort "mpich" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + MPICH_CFLAGS="-I'${pkg_install_dir}/include'" + MPICH_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac if [ "${with_mpich}" != "__DONTUSE__" ]; then - if [ "${with_mpich}" != "__SYSTEM__" ]; then - mpi_bin="${pkg_install_dir}/bin/mpiexec" - else - mpi_bin="mpiexec" - fi - MPICH_LIBS="-lmpifort -lmpicxx -lmpi" - cat << EOF > "${BUILDDIR}/setup_mpich" + if [ "${with_mpich}" != "__SYSTEM__" ]; then + mpi_bin="${pkg_install_dir}/bin/mpiexec" + else + mpi_bin="mpiexec" + fi + MPICH_LIBS="-lmpifort -lmpicxx -lmpi" + cat << EOF > "${BUILDDIR}/setup_mpich" export MPI_MODE="${MPI_MODE}" export MPIRUN="${MPIRUN}" export MPICC="${MPICC}" @@ -155,18 +171,17 @@ export CP_CFLAGS="\${CP_CFLAGS} IF_MPI(${MPICH_CFLAGS}|)" export CP_LDFLAGS="\${CP_LDFLAGS} IF_MPI(${MPICH_LDFLAGS}|)" export CP_LIBS="\${CP_LIBS} IF_MPI(${MPICH_LIBS}|)" EOF - if [ "${with_mpich}" != "__SYSTEM__" ]; then - cat << EOF >> "${BUILDDIR}/setup_mpich" + if [ "${with_mpich}" != "__SYSTEM__" ]; then + cat << EOF >> "${BUILDDIR}/setup_mpich" prepend_path PATH "${pkg_install_dir}/bin" export PATH="${pkg_install_dir}/bin":\${PATH} export LD_LIBRARY_PATH="${pkg_install_dir}/lib":\${LD_LIBRARY_PATH} -export LD_RUN_PATH="${pkg_install_dir}/lib":\${LD_RUN_PATH} -export LIBRARY_PATH="${pkg_install_dir}/lib":\${LIBRARY_PATH} -export CPATH="${pkg_install_dir}/include":\${CPATH} - +export LD_RUN_PATH "${pkg_install_dir}/lib":\${LD_RUN_PATH} +export LIBRARY_PATH "${pkg_install_dir}/lib":\${LIBRARY_PATH} +export CPATH "${pkg_install_dir}/include":\${CPATH} EOF - fi - cat "${BUILDDIR}/setup_mpich" >> ${SETUPFILE} + fi + cat "${BUILDDIR}/setup_mpich" >> ${SETUPFILE} fi # Update leak suppression file diff --git a/toolchain/scripts/stage1/install_openmpi.sh b/toolchain/scripts/stage1/install_openmpi.sh index faad08eb48..6656fb9a97 100755 --- a/toolchain/scripts/stage1/install_openmpi.sh +++ b/toolchain/scripts/stage1/install_openmpi.sh @@ -3,20 +3,29 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2025-0504 -# Change default version to openmpi 5 -# allow user to choose openmpi 4 in used scripts - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -if [ "${OPENMPI_4TH}" = "yes" ]; then - openmpi_ver="4.1.6" - openmpi_sha256="f740994485516deb63b5311af122c265179f5328a0d857a567b85db00b11e415" -else - openmpi_ver="5.0.7" - openmpi_sha256="119f2009936a403334d0df3c0d74d5595a32d99497f9b1d41e90019fee2fc2dd" +# Load centralized version management +source "${SCRIPT_DIR}"/package_versions.sh + +# Load OpenMPI package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "openmpi:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "openmpi:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" fi +# Load package variables with appropriate version +load_package_vars "openmpi" "$version_suffix" openmpi_pkg="openmpi-${openmpi_ver}.tar.bz2" source "${SCRIPT_DIR}"/common_vars.sh @@ -35,133 +44,136 @@ OPENMPI_LIBS="" cd "${BUILDDIR}" case "${with_openmpi}" in - __INSTALL__) - echo "==================== Installing OpenMPI ====================" - pkg_install_dir="${INSTALLDIR}/openmpi-${openmpi_ver}" - #pkg_install_dir="${HOME}/apps/openmpi/${openmpi_ver}-gcc8" - install_lock_file="$pkg_install_dir/install_successful" - url="https://download.open-mpi.org/release/open-mpi/v${openmpi_ver%.*}/${openmpi_pkg}" - if verify_checksums "${install_lock_file}"; then - echo "openmpi-${openmpi_ver} is already installed, skipping it." - else - if [ -f ${openmpi_pkg} ]; then - echo "${openmpi_pkg} is found" - else - download_pkg_from_url "${openmpi_sha256}" "${openmpi_pkg}" "${url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d openmpi-${openmpi_ver} ] && rm -rf openmpi-${openmpi_ver} - tar -xjf ${openmpi_pkg} - cd openmpi-${openmpi_ver} - if [ "${OPENBLAS_ARCH}" = "x86_64" ]; then - # can have issue with older glibc libraries, in which case - # we need to add the -fgnu89-inline to CFLAGS. We can check - # the version of glibc using ldd --version, as ldd is part of - # glibc package - glibc_version=$(ldd --version | awk '/ldd/{print $NF}') - glibc_major_ver=${glibc_version%%.*} - glibc_minor_ver=${glibc_version##*.} - if [ $glibc_major_ver -lt 2 ] || - [ $glibc_major_ver -eq 2 -a $glibc_minor_ver -lt 12 ]; then - CFLAGS="${CFLAGS} -fgnu89-inline" - fi - fi + __INSTALL__) + echo "==================== Installing OpenMPI ====================" + pkg_install_dir="${INSTALLDIR}/openmpi-${openmpi_ver}" + install_lock_file="$pkg_install_dir/install_successful" + url="https://download.open-mpi.org/release/open-mpi/v${openmpi_ver%.*}/${openmpi_pkg}" + if verify_checksums "${install_lock_file}"; then + echo "openmpi-${openmpi_ver} is already installed, skipping it." + else + if [ -f ${openmpi_pkg} ]; then + echo "${openmpi_pkg} is found" + else + download_pkg_from_url "${openmpi_sha256}" "${openmpi_pkg}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d openmpi-${openmpi_ver} ] && rm -rf openmpi-${openmpi_ver} + tar -xjf ${openmpi_pkg} + cd openmpi-${openmpi_ver} + if [ "${OPENBLAS_ARCH}" = "x86_64" ]; then + # can have issue with older glibc libraries, in which case + # we need to add the -fgnu89-inline to CFLAGS. We can check + # the version of glibc using ldd --version, as ldd is part of + # glibc package + glibc_version=$(ldd --version | awk '/ldd/{print $NF}') + glibc_major_ver=${glibc_version%%.*} + glibc_minor_ver=${glibc_version##*.} + if [ $glibc_major_ver -lt 2 ] || + [ $glibc_major_ver -eq 2 -a $glibc_minor_ver -lt 12 ]; then + CFLAGS="${CFLAGS} -fgnu89-inline" + fi + fi - # Notice for RHEL8 refer https://github.com/open-mpi/ompi/issues/13103 + # Notice for RHEL8 refer https://github.com/open-mpi/ompi/issues/13103 - # OpenMPI 5.0 only supports PMIx - # PMI support is required for Slurm, but not for other schedulers - # default not use - # for OpenMPI 4.1 with pmi slurm, we can open this setting manually - # if [ $(command -v srun) ]; then - # echo "Slurm installation found. OpenMPI will be configured with --with-pmi." - # EXTRA_CONFIGURE_FLAGS="--with-pmi" - # else - # EXTRA_CONFIGURE_FLAGS="" - # fi - ./configure CFLAGS="${CFLAGS}" \ - --prefix=${pkg_install_dir} \ - --libdir="${pkg_install_dir}/lib" \ - --with-libevent=internal \ - ${EXTRA_CONFIGURE_FLAGS} \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make -j $(get_nprocs) install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd .. - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage1/$(basename ${SCRIPT_NAME})" - fi - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - check_install ${pkg_install_dir}/bin/mpiexec "openmpi" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 - check_install ${pkg_install_dir}/bin/mpicc "openmpi" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 - check_install ${pkg_install_dir}/bin/mpicxx "openmpi" && MPICXX="${pkg_install_dir}/bin/mpicxx" || exit 1 - check_install ${pkg_install_dir}/bin/mpifort "openmpi" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - OPENMPI_CFLAGS="-I'${pkg_install_dir}/include'" - OPENMPI_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - fi - ;; - __SYSTEM__) - echo "==================== Finding OpenMPI from system paths ====================" - check_command mpiexec "openmpi" && MPIRUN="$(command -v mpiexec)" - check_command mpicc "openmpi" && MPICC="$(command -v mpicc)" || exit 1 - check_command mpic++ "openmpi" && MPICXX="$(command -v mpic++)" || exit 1 - check_command mpifort "openmpi" && MPIFC="$(command -v mpifort)" || exit 1 - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - # libraries and linker flags for C/C++-based MPI codepaths, pull them in at this point. - OPENMPI_CFLAGS="$(mpicxx --showme:compile)" - OPENMPI_LDFLAGS="$(mpicxx --showme:link)" - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking OpenMPI to user paths ====================" - pkg_install_dir="${with_openmpi}" - check_dir "${pkg_install_dir}/bin" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - check_command ${pkg_install_dir}/bin/mpiexec "openmpi" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 - check_command ${pkg_install_dir}/bin/mpicc "openmpi" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 - check_command ${pkg_install_dir}/bin/mpic++ "openmpi" && MPICXX="${pkg_install_dir}/bin/mpic++" || exit 1 - check_command ${pkg_install_dir}/bin/mpifort "openmpi" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 - MPIFORT="${MPIFC}" - MPIF77="${MPIFC}" - OPENMPI_CFLAGS="-I'${pkg_install_dir}/include'" - OPENMPI_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + # OpenMPI 5.0 only supports PMIx + # PMI support is required for Slurm, but not for other schedulers + # default not use + # for OpenMPI 4.1 with pmi slurm, we can open this setting manually + # if [ $(command -v srun) ]; then + # echo "Slurm installation found. OpenMPI will be configured with --with-pmi." + # EXTRA_CONFIGURE_FLAGS="--with-pmi" + # else + # EXTRA_CONFIGURE_FLAGS="" + # fi + ./configure CFLAGS="${CFLAGS}" \ + --prefix=${pkg_install_dir} \ + --libdir="${pkg_install_dir}/lib" \ + --with-libevent=internal \ + ${EXTRA_CONFIGURE_FLAGS} \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make -j $(get_nprocs) install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd .. + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage1/$(basename ${SCRIPT_NAME})" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + else + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + check_install ${pkg_install_dir}/bin/mpiexec "openmpi" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 + check_install ${pkg_install_dir}/bin/mpicc "openmpi" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 + check_install ${pkg_install_dir}/bin/mpicxx "openmpi" && MPICXX="${pkg_install_dir}/bin/mpicxx" || exit 1 + check_install ${pkg_install_dir}/bin/mpifort "openmpi" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + OPENMPI_CFLAGS="-I'${pkg_install_dir}/include'" + OPENMPI_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + fi + ;; + __SYSTEM__) + echo "==================== Finding OpenMPI from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_command mpiexec "openmpi" && MPIRUN="$(command -v mpiexec)" + check_command mpicc "openmpi" && MPICC="$(command -v mpicc)" || exit 1 + check_command mpicxx "openmpi" && MPICXX="$(command -v mpicxx)" || exit 1 + check_command mpifort "openmpi" && MPIFC="$(command -v mpifort)" || exit 1 + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking OpenMPI to user paths ====================" + pkg_install_dir="${with_openmpi}" + check_dir "${pkg_install_dir}/bin" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + check_command ${pkg_install_dir}/bin/mpiexec "openmpi" && MPIRUN="${pkg_install_dir}/bin/mpiexec" || exit 1 + check_command ${pkg_install_dir}/bin/mpicc "openmpi" && MPICC="${pkg_install_dir}/bin/mpicc" || exit 1 + check_command ${pkg_install_dir}/bin/mpicxx "openmpi" && MPICXX="${pkg_install_dir}/bin/mpicxx" || exit 1 + check_command ${pkg_install_dir}/bin/mpifort "openmpi" && MPIFC="${pkg_install_dir}/bin/mpifort" || exit 1 + MPIFORT="${MPIFC}" + MPIF77="${MPIFC}" + OPENMPI_CFLAGS="-I'${pkg_install_dir}/include'" + OPENMPI_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac -if [ "${with_openmpi}" != "__DONTUSE__" ]; then - if [ "${with_openmpi}" != "__SYSTEM__" ]; then - mpi_bin="${pkg_install_dir}/bin/mpiexec" - mpicxx_bin="${pkg_install_dir}/bin/mpicxx" - else - mpi_bin="mpiexec" - mpicxx_bin="mpicxx" - fi - # check openmpi version as reported by mpiexec - raw_version=$(${mpi_bin} --version 2>&1 | - grep "(Open MPI)" | awk '{print $4}') - major_version=$(echo ${raw_version} | cut -d '.' -f 1) - minor_version=$(echo ${raw_version} | cut -d '.' -f 2) - OPENMPI_LIBS="" - # grab additional runtime libs (for C/C++) from the mpicxx wrapper, - # and remove them from the LDFLAGS if present - for lib in $("${mpicxx_bin}" --showme:libs); do - OPENMPI_LIBS+=" -l${lib}" - OPENMPI_LDFLAGS="${OPENMPI_LDFLAGS//-l${lib}/}" - done - cat << EOF > "${BUILDDIR}/setup_openmpi" +if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip env setting" +else + if [ "${with_openmpi}" != "__DONTUSE__" ]; then + if [ "${with_openmpi}" != "__SYSTEM__" ]; then + mpi_bin="${pkg_install_dir}/bin/mpiexec" + mpicxx_bin="${pkg_install_dir}/bin/mpicxx" + else + mpi_bin="mpiexec" + mpicxx_bin="mpicxx" + fi + # check openmpi version as reported by mpiexec + raw_version=$(${mpi_bin} --version 2>&1 | + grep "(Open MPI)" | awk '{print $4}') + major_version=$(echo ${raw_version} | cut -d '.' -f 1) + minor_version=$(echo ${raw_version} | cut -d '.' -f 2) + OPENMPI_LIBS="" + # grab additional runtime libs (for C/C++) from the mpicxx wrapper, + # and remove them from the LDFLAGS if present + for lib in $("${mpicxx_bin}" --showme:libs); do + OPENMPI_LIBS+=" -l${lib}" + OPENMPI_LDFLAGS="${OPENMPI_LDFLAGS//-l${lib}/}" + done + cat << EOF > "${BUILDDIR}/setup_openmpi" export MPI_MODE="${MPI_MODE}" export MPIRUN="${MPIRUN}" export MPICC="${MPICC}" @@ -178,31 +190,31 @@ export MPI_LIBS="${OPENMPI_LIBS}" export CP_DFLAGS="\${CP_DFLAGS} IF_MPI(-D__parallel|)" # For proper mpi_f08 support, we need at least GCC version 9 (asynchronous keyword) # Other compilers should work - if ! [ "$(gfortran -dumpversion | cut -d. -f1)" -lt 9 ]; then - export CP_DFLAGS="\${CP_DFLAGS} IF_MPI(-D__MPI_F08|)" - fi + if ! [ "$(gfortran -dumpversion | cut -d. -f1)" -lt 9 ]; then + export CP_DFLAGS="\${CP_DFLAGS} IF_MPI(-D__MPI_F08|)" + fi export CP_CFLAGS="\${CP_CFLAGS} IF_MPI(${OPENMPI_CFLAGS}|)" export CP_LDFLAGS="\${CP_LDFLAGS} IF_MPI(${OPENMPI_LDFLAGS}|)" export CP_LIBS="\${CP_LIBS} IF_MPI(${OPENMPI_LIBS}|)" EOF - if [ "${with_openmpi}" != "__SYSTEM__" ]; then - cat << EOF >> "${BUILDDIR}/setup_openmpi" + if [ "${with_openmpi}" != "__SYSTEM__" ]; then + cat << EOF >> "${BUILDDIR}/setup_openmpi" prepend_path PATH "${pkg_install_dir}/bin" export PATH="${pkg_install_dir}/bin":\${PATH} export LD_LIBRARY_PATH="${pkg_install_dir}/lib":\${LD_LIBRARY_PATH} export LD_RUN_PATH="${pkg_install_dir}/lib":\${LD_RUN_PATH} export LIBRARY_PATH="${pkg_install_dir}/lib":\${LIBRARY_PATH} export CPATH="${pkg_install_dir}/include":\${CPATH} -export MANPATH="${pkg_install_dir}/share/man":\${MANPATH} + EOF - fi - cat "${BUILDDIR}/setup_openmpi" >> ${SETUPFILE} -fi + fi + cat "${BUILDDIR}/setup_openmpi" >> ${SETUPFILE} + fi -# ---------------------------------------------------------------------- -# Suppress reporting of known leaks -# ---------------------------------------------------------------------- -cat << EOF >> ${INSTALLDIR}/valgrind.supp + # ---------------------------------------------------------------------- + # Suppress reporting of known leaks + # ---------------------------------------------------------------------- + cat << EOF >> ${INSTALLDIR}/valgrind.supp { Memcheck:Leak @@ -249,7 +261,7 @@ cat << EOF >> ${INSTALLDIR}/valgrind.supp fun:ompi_comm_activate } EOF -cat << EOF >> ${INSTALLDIR}/lsan.supp + cat << EOF >> ${INSTALLDIR}/lsan.supp # leaks related to OpenMPI leak:query_2_0_0 leak:ompi_init_f @@ -258,6 +270,7 @@ leak:ompi_file_open_f leak:progress_engine leak:__GI___strdup EOF +fi load "${BUILDDIR}/setup_openmpi" write_toolchain_env "${INSTALLDIR}" diff --git a/toolchain/scripts/stage2/install_aocl.sh b/toolchain/scripts/stage2/install_aocl.sh index d3f0d70dd5..e085e9877f 100755 --- a/toolchain/scripts/stage2/install_aocl.sh +++ b/toolchain/scripts/stage2/install_aocl.sh @@ -1,9 +1,6 @@ #!/bin/bash -e -# TODO: Review and if possible fix shellcheck errors. -# shellcheck disable=all - -# Last Update in 2025-0308 +# developer: QuantumMisaka [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" @@ -24,18 +21,17 @@ AOCL_ROOT="" cd "${BUILDDIR}" case "${with_aocl}" in - __INSTALL__) - echo "==================== Installing AOCL ====================" - report_error ${LINENO} "To install AOCL, please contact your system administrator." - if [ "${PACK_RUN}" != "__TRUE__" ]; then + __INSTALL__) + echo "==================== Installing AOCL ====================" + report_error ${LINENO} "To install AOCL, please contact your system administrator." exit 1 - fi - ;; - __SYSTEM__) - echo "==================== Finding AOCL from system paths ====================" - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else + ;; + __SYSTEM__) + echo "==================== Finding AOCL from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi check_lib -lblis "AOCL" check_lib -lflame "AOCL" AOCL_LIBS="-lblis -lflame" @@ -43,23 +39,26 @@ case "${with_aocl}" in add_lib_from_paths AOCL_LDFLAGS "libblis.*" $LIB_PATHS add_include_from_paths AOCL_CFLAGS "lapack.h" $INCLUDE_PATHS add_lib_from_paths AOCL_LDFLAGS "libflame.*" $LIB_PATHS - fi - ;; - __DONTUSE__) ;; - - *) - echo "==================== Linking AOCL to user paths ====================" - pkg_install_dir="$with_aocl" - check_dir "${pkg_install_dir}/include" - check_dir "${pkg_install_dir}/lib" - AOCL_CFLAGS="-I'${pkg_install_dir}/include'" - AOCL_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - AOCL_LIBS="-lblis -lflame" - ;; + ;; + __DONTUSE__) + ;; + *) + echo "==================== Linking AOCL to user paths ====================" + pkg_install_dir="$with_aocl" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_dir "${pkg_install_dir}/include" + check_dir "${pkg_install_dir}/lib" + AOCL_CFLAGS="-I'${pkg_install_dir}/include'" + AOCL_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + AOCL_LIBS="-lblis -lflame" + ;; esac if [ "$with_aocl" != "__DONTUSE__" ]; then - if [ "$with_aocl" != "__SYSTEM__" ]; then - cat << EOF > "${BUILDDIR}/setup_aocl" + if [ "$with_aocl" != "__SYSTEM__" ]; then + cat << EOF > "${BUILDDIR}/setup_aocl" prepend_path LD_LIBRARY_PATH "$pkg_install_dir/lib" prepend_path LD_RUN_PATH "$pkg_install_dir/lib" prepend_path LIBRARY_PATH "$pkg_install_dir/lib" @@ -74,20 +73,16 @@ export PKG_CONFIG_PATH="$pkg_install_dir/lib/pkgconfig:"\${PKG_CONFIG_PATH} export CMAKE_PREFIX_PATH="$pkg_install_dir:"\${CMAKE_PREFIX_PATH} export AOCL_ROOT=${pkg_install_dir} EOF - cat "${BUILDDIR}/setup_aocl" >> $SETUPFILE - fi -cat << EOF >> "${BUILDDIR}/setup_aocl" + cat "${BUILDDIR}/setup_aocl" >> $SETUPFILE + fi + cat << EOF >> "${BUILDDIR}/setup_aocl" export AOCL_ROOT="${pkg_install_dir}" export AOCL_CFLAGS="${AOCL_CFLAGS}" export AOCL_LDFLAGS="${AOCL_LDFLAGS}" export AOCL_LIBS="${AOCL_LIBS}" -export MATH_CFLAGS="\${MATH_CFLAGS} ${AOCL_CFLAGS}" -export MATH_LDFLAGS="\${MATH_LDFLAGS} ${AOCL_LDFLAGS}" -export MATH_LIBS="\${MATH_LIBS} ${AOCL_LIBS}" -export PKG_CONFIG_PATH="${pkg_install_dir}/lib/pkgconfig" -export CMAKE_PREFIX_PATH="${pkg_install_dir}" -prepend_path PKG_CONFIG_PATH "$pkg_install_dir/lib/pkgconfig" -prepend_path CMAKE_PREFIX_PATH "$pkg_install_dir" +export FAST_MATH_CFLAGS="\${FAST_MATH_CFLAGS} ${AOCL_CFLAGS}" +export FAST_MATH_LDFLAGS="\${FAST_MATH_LDFLAGS} ${AOCL_LDFLAGS}" +export FAST_MATH_LIBS="\${FAST_MATH_LIBS} ${AOCL_LIBS}" EOF fi diff --git a/toolchain/scripts/stage2/install_mathlibs.sh b/toolchain/scripts/stage2/install_mathlibs.sh index 3ad3aef336..f747101c80 100755 --- a/toolchain/scripts/stage2/install_mathlibs.sh +++ b/toolchain/scripts/stage2/install_mathlibs.sh @@ -21,24 +21,24 @@ export MATH_LIBS='' write_toolchain_env "${INSTALLDIR}" case "$MATH_MODE" in - mkl) - "${SCRIPTDIR}"/stage2/install_mkl.sh "${with_mkl}" - load "${BUILDDIR}/setup_mkl" - ;; - aocl) - "${SCRIPTDIR}"/stage2/install_aocl.sh "${with_aocl}" - load "${BUILDDIR}/setup_aocl" - ;; - openblas) - "${SCRIPTDIR}"/stage2/install_openblas.sh "${with_openblas}" - load "${BUILDDIR}/setup_openblas" - ;; - cray) - # note the space is intentional so that the variable is - # non-empty and can pass require_env checks - export MATH_LDFLAGS="${MATH_LDFLAGS} " - export MATH_LIBS="${MATH_LIBS} ${CRAY_EXTRA_LIBS}" - ;; + mkl) + "${SCRIPTDIR}"/stage2/install_mkl.sh "${with_mkl}" + load "${BUILDDIR}/setup_mkl" + ;; + aocl) + "${SCRIPTDIR}"/stage2/install_aocl.sh "${with_aocl}" + load "${BUILDDIR}/setup_aocl" + ;; + openblas) + "${SCRIPTDIR}"/stage2/install_openblas.sh "${with_openblas}" + load "${BUILDDIR}/setup_openblas" + ;; + cray) + # note the space is intentional so that the variable is + # non-empty and can pass require_env checks + export MATH_LDFLAGS="${MATH_LDFLAGS} " + export MATH_LIBS="${MATH_LIBS} ${CRAY_EXTRA_LIBS}" + ;; esac export CP_CFLAGS="${CP_CFLAGS} ${MATH_CFLAGS}" diff --git a/toolchain/scripts/stage2/install_mkl.sh b/toolchain/scripts/stage2/install_mkl.sh index 4262b202bf..34248284e4 100755 --- a/toolchain/scripts/stage2/install_mkl.sh +++ b/toolchain/scripts/stage2/install_mkl.sh @@ -3,8 +3,6 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2023-0901 - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" @@ -25,90 +23,96 @@ MKL_FFTW="yes" cd "${BUILDDIR}" case "${with_mkl}" in - __INSTALL__) - echo "==================== Installing MKL ====================" - report_error ${LINENO} "To install MKL, please contact your system administrator." - if [ "${PACK_RUN}" != "__TRUE__" ]; then - exit 1 - fi - ;; - __SYSTEM__) - echo "==================== Finding MKL from system paths ====================" - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip system check" - else - if ! [ -z "${MKLROOT}" ]; then - echo "MKLROOT is found to be ${MKLROOT}" - else - report_error ${LINENO} "Cannot find env variable MKLROOT, the script relies on it being set. Please check in MKL installation and use --with-mkl= to pass the path to MKL root directory to this script." + __INSTALL__) + echo "==================== Installing MKL ====================" + report_error ${LINENO} "To install MKL, please contact your system administrator." exit 1 - fi - fi - check_lib -lm - check_lib -ldl - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking MKL to user paths ====================" - check_dir "${with_mkl}" - MKLROOT="${with_mkl}" - ;; + ;; + __SYSTEM__) + echo "==================== Finding MKL from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + if ! [ -z "${MKLROOT}" ]; then + echo "MKLROOT is found to be ${MKLROOT}" + else + report_error ${LINENO} "Cannot find env variable MKLROOT, the script relies on it being set. Please check in MKL installation and use --with-mkl= to pass the path to MKL root directory to this script." + exit 1 + fi + check_lib -lm + check_lib -ldl + ;; + __DONTUSE__) + # Nothing to do + ;; + *) + echo "==================== Linking MKL to user paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_dir "${with_mkl}" + MKLROOT="${with_mkl}" + ;; esac if [ "${with_mkl}" != "__DONTUSE__" ]; then - case ${OPENBLAS_ARCH} in - x86_64) - mkl_arch_dir="intel64" - MKL_CFLAGS="-m64" - ;; - i386) - mkl_arch_dir="ia32" - MKL_CFLAGS="-m32" - ;; - *) - report_error $LINENO "MKL only supports intel64 (x86_64) and ia32 (i386) at the moment, and your arch obtained from OpenBLAS prebuild is $OPENBLAS_ARCH" - exit 1 - ;; - esac - mkl_lib_dir="${MKLROOT}/lib/${mkl_arch_dir}" - # check we have required libraries - mkl_required_libs="libmkl_gf_lp64.so libmkl_sequential.so libmkl_core.so" - for ii in $mkl_required_libs; do - if [ ! -f "$mkl_lib_dir/${ii}" ]; then - report_error $LINENO "missing MKL library ${ii}" - exit 1 - fi - done + case ${OPENBLAS_ARCH} in + x86_64) + mkl_arch_dir="intel64" + MKL_CFLAGS="-m64" + ;; + i386) + mkl_arch_dir="ia32" + MKL_CFLAGS="-m32" + ;; + *) + report_error $LINENO "MKL only supports intel64 (x86_64) and ia32 (i386) at the moment, and your arch obtained from OpenBLAS prebuild is $OPENBLAS_ARCH" + exit 1 + ;; + esac + mkl_lib_dir="${MKLROOT}/lib/${mkl_arch_dir}" + # check we have required libraries + mkl_required_libs="libmkl_gf_lp64.so libmkl_sequential.so libmkl_core.so" + for ii in $mkl_required_libs; do + if [ ! -f "$mkl_lib_dir/${ii}" ]; then + report_error $LINENO "missing MKL library ${ii}" + exit 1 + fi + done + + + case ${MPI_MODE} in + intelmpi | mpich) + mkl_scalapack_lib="IF_MPI(-lmkl_scalapack_lp64|)" + mkl_blacs_lib="IF_MPI(-lmkl_blacs_intelmpi_lp64|)" + ;; + openmpi) + mkl_scalapack_lib="IF_MPI(-lmkl_scalapack_lp64|)" + mkl_blacs_lib="IF_MPI(-lmkl_blacs_openmpi_lp64|)" + ;; + *) + echo "Not using MKL provided ScaLAPACK and BLACS" + mkl_scalapack_lib="" + mkl_blacs_lib="" + ;; + esac - case ${MPI_MODE} in - intelmpi | mpich) - mkl_scalapack_lib="IF_MPI(-lmkl_scalapack_lp64|)" - mkl_blacs_lib="IF_MPI(-lmkl_blacs_intelmpi_lp64|)" - ;; - openmpi) - mkl_scalapack_lib="IF_MPI(-lmkl_scalapack_lp64|)" - mkl_blacs_lib="IF_MPI(-lmkl_blacs_openmpi_lp64|)" - ;; - *) - echo "Not using MKL provided ScaLAPACK and BLACS" - mkl_scalapack_lib="" - mkl_blacs_lib="" - ;; - esac - # set the correct lib flags from MLK link adviser - MKL_LIBS="-L${mkl_lib_dir} -Wl,-rpath,${mkl_lib_dir} ${mkl_scalapack_lib}" - MKL_LIBS+=" -Wl,--start-group -lmkl_gf_lp64 -lmkl_sequential -lmkl_core" - MKL_LIBS+=" ${mkl_blacs_lib} -Wl,--end-group -lpthread -lm -ldl" - # setup_mkl disables using separate FFTW library (see below) - MKL_CFLAGS="${MKL_CFLAGS} -I${MKLROOT}/include" - if [ "${MKL_FFTW}" != "no" ]; then - MKL_CFLAGS+=" -I${MKLROOT}/include/fftw" - fi + # added by trae + MKL_LDFLAGS="-L'${mkl_lib_dir}' -Wl,-rpath,'${mkl_lib_dir}'" + # set the correct lib flags from MLK link adviser + MKL_LIBS="-L${mkl_lib_dir} -Wl,-rpath,${mkl_lib_dir} ${mkl_scalapack_lib}" + MKL_LIBS+=" -Wl,--start-group -lmkl_gf_lp64 -lmkl_sequential -lmkl_core" + MKL_LIBS+=" ${mkl_blacs_lib} -Wl,--end-group -lpthread -lm -ldl" + # setup_mkl disables using separate FFTW library (see below) + MKL_CFLAGS="${MKL_CFLAGS} -I${MKLROOT}/include" + if [ "${MKL_FFTW}" != "no" ]; then + MKL_CFLAGS+=" -I${MKLROOT}/include/fftw" + fi # write setup files - cat << EOF > "${BUILDDIR}/setup_mkl" + cat << EOF > "${BUILDDIR}/setup_mkl" export MKLROOT="${MKLROOT}" export MKL_CFLAGS="${MKL_CFLAGS}" export MKL_LIBS="${MKL_LIBS}" @@ -116,14 +120,14 @@ export MATH_CFLAGS="\${MATH_CFLAGS} ${MKL_CFLAGS}" export MATH_LIBS="\${MATH_LIBS} ${MKL_LIBS}" export CP_DFLAGS="\${CP_DFLAGS} -D__MKL -D__FFTW3 IF_COVERAGE(IF_MPI(|-U__FFTW3)|)" EOF - if [ -n "${mkl_scalapack_lib}" ]; then - cat << EOF >> "${BUILDDIR}/setup_mkl" + if [ -n "${mkl_scalapack_lib}" ]; then + cat << EOF >> "${BUILDDIR}/setup_mkl" export CP_DFLAGS="\${CP_DFLAGS} IF_MPI(-D__SCALAPACK|)" export with_scalapack="__DONTUSE__" EOF - fi - if [ "${MKL_FFTW}" != "no" ]; then - cat << EOF >> "${BUILDDIR}/setup_mkl" + fi + if [ "${MKL_FFTW}" != "no" ]; then + cat << EOF >> "${BUILDDIR}/setup_mkl" export with_fftw="__DONTUSE__" export FFTW3_INCLUDES="${MKL_CFLAGS}" export FFTW3_LIBS="${MKL_LIBS}" @@ -131,8 +135,8 @@ export FFTW_CFLAGS="${MKL_CFLAGS}" export FFTW_LDFLAGS="${MKL_LDFLAGS}" export FFTW_LIBS="${MKL_LIBS}" EOF - fi - cat "${BUILDDIR}/setup_mkl" >> ${SETUPFILE} + fi + cat "${BUILDDIR}/setup_mkl" >> ${SETUPFILE} fi load "${BUILDDIR}/setup_mkl" diff --git a/toolchain/scripts/stage2/install_openblas.sh b/toolchain/scripts/stage2/install_openblas.sh index fb4e0c563e..f2a24b1be4 100755 --- a/toolchain/scripts/stage2/install_openblas.sh +++ b/toolchain/scripts/stage2/install_openblas.sh @@ -3,18 +3,33 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2024-0811 - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -openblas_ver="0.3.29" # Keep in sync with get_openblas_arch.sh -openblas_sha256="38240eee1b29e2bde47ebb5d61160207dc68668a54cac62c076bb5032013b1eb" -openblas_pkg="OpenBLAS-${openblas_ver}.tar.gz" - source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load OpenBLAS package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "openblas:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "openblas:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "openblas" "$version_suffix" +openblas_pkg="openblas-${openblas_ver}.tar.gz" + source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -28,133 +43,135 @@ OPENBLAS_ROOT="" cd "${BUILDDIR}" case "${with_openblas}" in - __INSTALL__) - echo "==================== Installing OpenBLAS ====================" - pkg_install_dir="${INSTALLDIR}/openblas-${openblas_ver}" - #pkg_install_dir="${HOME}/lib/openblas/${openblas_ver}-gcc8" - install_lock_file="$pkg_install_dir/install_successful" - if verify_checksums "${install_lock_file}"; then - echo "openblas-${openblas_ver} is already installed, skipping it." - else - if [ -f ${openblas_pkg} ]; then - echo "${openblas_pkg} is found" - else - #download_pkg_from_ABACUS_org "${openblas_sha256}" "${openblas_pkg}" - # using codeload.github - url="https://codeload.github.com/OpenMathLib/OpenBLAS/tar.gz/v${openblas_ver}" - download_pkg_from_url "${openblas_sha256}" "${openblas_pkg}" "${url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d OpenBLAS-${openblas_ver} ] && rm -rf OpenBLAS-${openblas_ver} - tar -zxf ${openblas_pkg} - cd OpenBLAS-${openblas_ver} + __INSTALL__) + echo "==================== Installing OpenBLAS ====================" + pkg_install_dir="${INSTALLDIR}/openblas-${openblas_ver}" + install_lock_file="$pkg_install_dir/install_successful" + if verify_checksums "${install_lock_file}"; then + echo "openblas-${openblas_ver} is already installed, skipping it." + else + if [ -f ${openblas_pkg} ]; then + echo "${openblas_pkg} is found" + else + # using codeload.github + url="https://codeload.github.com/OpenMathLib/OpenBLAS/tar.gz/v${openblas_ver}" + download_pkg_from_url "${openblas_sha256}" "${openblas_pkg}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d OpenBLAS-${openblas_ver} ] && rm -rf OpenBLAS-${openblas_ver} + tar -zxf ${openblas_pkg} + cd OpenBLAS-${openblas_ver} - # First attempt to make openblas using auto detected - # TARGET, if this fails, then make with forced - # TARGET=NEHALEM - # - # wrt NUM_THREADS=64: this is what the most common Linux distros seem to choose atm - # for a good compromise between memory usage and scalability - # - # Unfortunately, NO_SHARED=1 breaks ScaLAPACK build. - case "${TARGET_CPU}" in - "generic") - TARGET="NEHALEM" - ;; - "native") - TARGET=${OPENBLAS_LIBCORE} - ;; - "broadwell" | "skylake") - TARGET="HASWELL" - ;; - "skylake-avx512") - TARGET="SKYLAKEX" - ;; - *) - TARGET=${TARGET_CPU} - ;; - esac - TARGET=$(echo ${TARGET} | tr '[:lower:]' '[:upper:]') - echo "Installing OpenBLAS library for target ${TARGET}" - ( - make -j $(get_nprocs) \ - MAKE_NB_JOBS=0 \ - TARGET=${TARGET} \ - NUM_THREADS=64 \ - USE_THREAD=1 \ - USE_OPENMP=1 \ - NO_AFFINITY=1 \ - CC="${CC}" \ - FC="${FC}" \ - PREFIX="${pkg_install_dir}" \ - > make.log 2>&1 || tail -n ${LOG_LINES} make.log - ) || ( - make -j $(get_nprocs) \ - MAKE_NB_JOBS=0 \ - TARGET=NEHALEM \ - NUM_THREADS=64 \ - USE_THREAD=1 \ - USE_OPENMP=1 \ - NO_AFFINITY=1 \ - CC="${CC}" \ - FC="${FC}" \ - PREFIX="${pkg_install_dir}" \ - > make.nehalem.log 2>&1 || tail -n ${LOG_LINES} make.nehalem.log - ) - make -j $(get_nprocs) \ - MAKE_NB_JOBS=0 \ - TARGET=${TARGET} \ - NUM_THREADS=64 \ - USE_THREAD=1 \ - USE_OPENMP=1 \ - NO_AFFINITY=1 \ - CC="${CC}" \ - FC="${FC}" \ - PREFIX="${pkg_install_dir}" \ - install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd .. - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage2/$(basename ${SCRIPT_NAME})" - fi - fi - OPENBLAS_CFLAGS="-I'${pkg_install_dir}/include'" - OPENBLAS_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - OPENBLAS_ROOT="${pkg_install_dir}" - OPENBLAS_LIBS="-lopenblas" - ;; - __SYSTEM__) - echo "==================== Finding LAPACK from system paths ====================" - # assume that system openblas is threaded - check_lib -lopenblas "OpenBLAS" - OPENBLAS_LIBS="-lopenblas" - # detect separate omp builds - check_lib -lopenblas_openmp 2> /dev/null && OPENBLAS_LIBS="-lopenblas_openmp" - check_lib -lopenblas_omp 2> /dev/null && OPENBLAS_LIBS="-lopenblas_omp" - add_include_from_paths OPENBLAS_CFLAGS "openblas_config.h" $INCLUDE_PATHS - add_lib_from_paths OPENBLAS_LDFLAGS "libopenblas.*" $LIB_PATHS - ;; - __DONTUSE__) ;; + # First attempt to make openblas using auto detected + # TARGET, if this fails, then make with forced + # TARGET=NEHALEM + # + # wrt NUM_THREADS=64: this is what the most common Linux distros seem to choose atm + # for a good compromise between memory usage and scalability + # + # Unfortunately, NO_SHARED=1 breaks ScaLAPACK build. + case "${TARGET_CPU}" in + "generic") + TARGET="NEHALEM" + ;; + "native") + TARGET=${OPENBLAS_LIBCORE} + ;; + "broadwell" | "skylake") + TARGET="HASWELL" + ;; + "skylake-avx512") + TARGET="SKYLAKEX" + ;; + *) + TARGET=${TARGET_CPU} + ;; + esac + TARGET=$(echo ${TARGET} | tr '[:lower:]' '[:upper:]') + echo "Installing OpenBLAS library for target ${TARGET}" + ( + make -j $(get_nprocs) \ + MAKE_NB_JOBS=0 \ + TARGET=${TARGET} \ + NUM_THREADS=64 \ + USE_THREAD=1 \ + USE_OPENMP=1 \ + NO_AFFINITY=1 \ + CC="${CC}" \ + FC="${FC}" \ + PREFIX="${pkg_install_dir}" \ + > make.log 2>&1 || tail -n ${LOG_LINES} make.log + ) || ( + make -j $(get_nprocs) \ + MAKE_NB_JOBS=0 \ + TARGET=NEHALEM \ + NUM_THREADS=64 \ + USE_THREAD=1 \ + USE_OPENMP=1 \ + NO_AFFINITY=1 \ + CC="${CC}" \ + FC="${FC}" \ + PREFIX="${pkg_install_dir}" \ + > make.nehalem.log 2>&1 || tail -n ${LOG_LINES} make.nehalem.log + ) + make -j $(get_nprocs) \ + MAKE_NB_JOBS=0 \ + TARGET=${TARGET} \ + NUM_THREADS=64 \ + USE_THREAD=1 \ + USE_OPENMP=1 \ + NO_AFFINITY=1 \ + CC="${CC}" \ + FC="${FC}" \ + PREFIX="${pkg_install_dir}" \ + install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd .. + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage2/$(basename ${SCRIPT_NAME})" + fi + OPENBLAS_CFLAGS="-I'${pkg_install_dir}/include'" + OPENBLAS_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + OPENBLAS_ROOT="${pkg_install_dir}" + OPENBLAS_LIBS="-lopenblas" + ;; + __SYSTEM__) + echo "==================== Finding OpenBLAS/LAPACK from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + # assume that system openblas is threaded + check_lib -lopenblas "OpenBLAS" + OPENBLAS_LIBS="-lopenblas" + # detect separate omp builds + check_lib -lopenblas_openmp 2> /dev/null && OPENBLAS_LIBS="-lopenblas_openmp" + check_lib -lopenblas_omp 2> /dev/null && OPENBLAS_LIBS="-lopenblas_omp" + add_include_from_paths OPENBLAS_CFLAGS "openblas_config.h" $INCLUDE_PATHS + add_lib_from_paths OPENBLAS_LDFLAGS "libopenblas.*" $LIB_PATHS + ;; + __DONTUSE__) ;; - *) - echo "==================== Linking LAPACK to user paths ====================" - pkg_install_dir="$with_openblas" - check_dir "${pkg_install_dir}/include" - check_dir "${pkg_install_dir}/lib" - OPENBLAS_CFLAGS="-I'${pkg_install_dir}/include'" - OPENBLAS_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - OPENBLAS_LIBS="-lopenblas" - # detect separate omp builds - (__libdir="${pkg_install_dir}/lib" LIB_PATHS="__libdir" check_lib -lopenblas_openmp 2> /dev/null) && - OPENBLAS_LIBS="-lopenblas_openmp" - (__libdir="${pkg_install_dir}/lib" LIB_PATHS="__libdir" check_lib -lopenblas_omp 2> /dev/null) && - OPENBLAS_LIBS="-lopenblas_omp" - ;; + *) + echo "==================== Linking OpenBLAS/LAPACK to user paths ====================" + pkg_install_dir="$with_openblas" + check_dir "${pkg_install_dir}/include" + check_dir "${pkg_install_dir}/lib" + OPENBLAS_CFLAGS="-I'${pkg_install_dir}/include'" + OPENBLAS_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + OPENBLAS_LIBS="-lopenblas" + # detect separate omp builds + (__libdir="${pkg_install_dir}/lib" LIB_PATHS="__libdir" check_lib -lopenblas_openmp 2> /dev/null) && + OPENBLAS_LIBS="-lopenblas_openmp" + (__libdir="${pkg_install_dir}/lib" LIB_PATHS="__libdir" check_lib -lopenblas_omp 2> /dev/null) && + OPENBLAS_LIBS="-lopenblas_omp" + ;; esac if [ "$with_openblas" != "__DONTUSE__" ]; then - if [ "$with_openblas" != "__SYSTEM__" ]; then - cat << EOF > "${BUILDDIR}/setup_openblas" + if [ "$with_openblas" != "__SYSTEM__" ]; then + cat << EOF > "${BUILDDIR}/setup_openblas" prepend_path LD_LIBRARY_PATH "$pkg_install_dir/lib" prepend_path LD_RUN_PATH "$pkg_install_dir/lib" prepend_path LIBRARY_PATH "$pkg_install_dir/lib" @@ -169,9 +186,9 @@ export PKG_CONFIG_PATH="$pkg_install_dir/lib/pkgconfig:"\${PKG_CONFIG_PATH} export CMAKE_PREFIX_PATH="$pkg_install_dir:"\${CMAKE_PREFIX_PATH} export OPENBLAS_ROOT=${pkg_install_dir} EOF - cat "${BUILDDIR}/setup_openblas" >> $SETUPFILE - fi - cat << EOF >> "${BUILDDIR}/setup_openblas" + cat "${BUILDDIR}/setup_openblas" >> $SETUPFILE + fi + cat << EOF >> "${BUILDDIR}/setup_openblas" export OPENBLAS_ROOT="${pkg_install_dir}" export OPENBLAS_CFLAGS="${OPENBLAS_CFLAGS}" export OPENBLAS_LDFLAGS="${OPENBLAS_LDFLAGS}" diff --git a/toolchain/scripts/stage3/install_elpa.sh b/toolchain/scripts/stage3/install_elpa.sh index 56bd30c966..c5ffc77063 100755 --- a/toolchain/scripts/stage3/install_elpa.sh +++ b/toolchain/scripts/stage3/install_elpa.sh @@ -3,22 +3,36 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2025-0504 +# Last Update in 2025-01-04 # other contributor: Benrui Tang [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -# From https://elpa.mpcdf.mpg.de/software/tarball-archive/ELPA_TARBALL_ARCHIVE.html -# elpa_ver="2024.05.001" -# elpa_sha256="9caf41a3e600e2f6f4ce1931bd54185179dade9c171556d0c9b41bbc6940f2f6" -# newer version of elpa may have problem in GPU-ELPA compliation -elpa_ver="2025.01.001" -elpa_sha256="3ef0c6aed9a3e05db6efafe6e14d66eb88b2a1354d61e765b7cde0d3d5f3951e" - source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load ELPA package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "elpa:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "elpa:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "elpa" "$version_suffix" +elpa_pkg="elpa-${elpa_ver}.tar.gz" + source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -34,196 +48,204 @@ cd "${BUILDDIR}" # elpa only works with MPI switched on if [ $MPI_MODE = no ]; then - report_warning $LINENO "MPI is disabled, skipping elpa installation" - exit 0 + report_warning $LINENO "MPI is disabled, skipping elpa installation" + exit 0 fi case "$with_elpa" in - __INSTALL__) - echo "==================== Installing ELPA ====================" - pkg_install_dir="${INSTALLDIR}/elpa-${elpa_ver}" - #pkg_install_dir="${HOME}/lib/elpa/${elpa_ver}-gcc8" - install_lock_file="$pkg_install_dir/install_successful" - enable_openmp="yes" + __INSTALL__) + echo "==================== Installing ELPA ====================" + pkg_install_dir="${INSTALLDIR}/elpa-${elpa_ver}" + #pkg_install_dir="${HOME}/lib/elpa/${elpa_ver}-gcc8" + install_lock_file="$pkg_install_dir/install_successful" + enable_openmp="yes" - # specific settings needed on CRAY Linux Environment - if [ "$ENABLE_CRAY" = "__TRUE__" ]; then - if [ ${CRAY_PRGENVCRAY} ]; then - # extra LDFLAGS needed - cray_ldflags="-dynamic" - fi - # enable_openmp="no" - fi + # specific settings needed on CRAY Linux Environment + if [ "$ENABLE_CRAY" = "__TRUE__" ]; then + if [ ${CRAY_PRGENVCRAY} ]; then + # extra LDFLAGS needed + cray_ldflags="-dynamic" + fi + # enable_openmp="no" + fi - if verify_checksums "${install_lock_file}"; then - echo "elpa-${elpa_ver} is already installed, skipping it." - else - require_env MATH_LIBS - elpa_pkg="elpa-${elpa_ver}.tar.gz" - url="https://elpa.mpcdf.mpg.de/software/tarball-archive/Releases/${elpa_ver}/${elpa_pkg}" - if [ -f ${elpa_pkg} ]; then - echo "${elpa_pkg} is found" - else - download_pkg_from_url "${elpa_sha256}" "${elpa_pkg}" "${url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - [ -d elpa-${elpa_ver} ] && rm -rf elpa-${elpa_ver} - tar -xzf ${elpa_pkg} + if verify_checksums "${install_lock_file}"; then + echo "elpa-${elpa_ver} is already installed, skipping it." + else + require_env MATH_LIBS + url="https://elpa.mpcdf.mpg.de/software/tarball-archive/Releases/${elpa_ver}/${elpa_pkg}" + if [ -f ${elpa_pkg} ]; then + echo "${elpa_pkg} is found" + else + download_pkg_from_url "${elpa_sha256}" "${elpa_pkg}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + [ -d elpa-${elpa_ver} ] && rm -rf elpa-${elpa_ver} + tar -xzf ${elpa_pkg} - # elpa expect FC to be an mpi fortran compiler that is happy - # with long lines, and that a bunch of libs can be found - cd elpa-${elpa_ver} + # elpa expect FC to be an mpi fortran compiler that is happy + # with long lines, and that a bunch of libs can be found + cd elpa-${elpa_ver} - # ELPA-2017xxxx enables AVX2 by default, switch off if machine doesn't support it. - AVX_flag="" - AVX512_flags="" - FMA_flag="" - SSE4_flag="" - config_flags="--disable-avx-kernels --disable-avx2-kernels --disable-avx512-kernels --disable-sse-kernels --disable-sse-assembly-kernels" - if [ "${TARGET_CPU}" = "native" ]; then - if [ -f /proc/cpuinfo ] && [ "${OPENBLAS_ARCH}" = "x86_64" ]; then - has_AVX=$(grep '\bavx\b' /proc/cpuinfo 1> /dev/null && echo 'yes' || echo 'no') - [ "${has_AVX}" = "yes" ] && AVX_flag="-mavx" || AVX_flag="" - has_AVX2=$(grep '\bavx2\b' /proc/cpuinfo 1> /dev/null && echo 'yes' || echo 'no') - [ "${has_AVX2}" = "yes" ] && AVX_flag="-mavx2" - has_AVX512=$(grep '\bavx512f\b' /proc/cpuinfo 1> /dev/null && echo 'yes' || echo 'no') - [ "${has_AVX512}" = "yes" ] && AVX512_flags="-mavx512f" - FMA_flag=$(grep '\bfma\b' /proc/cpuinfo 1> /dev/null && echo '-mfma' || echo '-mno-fma') - SSE4_flag=$(grep '\bsse4_1\b' /proc/cpuinfo 1> /dev/null && echo '-msse4' || echo '-mno-sse4') - grep '\bavx512dq\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512dq" - grep '\bavx512cd\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512cd" - grep '\bavx512bw\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512bw" - grep '\bavx512vl\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512vl" - config_flags="--enable-avx-kernels=${has_AVX} --enable-avx2-kernels=${has_AVX2} --enable-avx512-kernels=${has_AVX512}" - fi - fi - for TARGET in "cpu" "nvidia"; do - [ "$TARGET" = "nvidia" ] && [ "$ENABLE_CUDA" != "__TRUE__" ] && continue - # disable cpu if cuda is enabled, only install one - [ "$TARGET" != "nvidia" ] && [ "$ENABLE_CUDA" = "__TRUE__" ] && continue - # extend the pkg_install_dir by TARGET - # this linking method is totally different from cp2k toolchain - # for cp2k, ref https://github.com/cp2k/cp2k/commit/6fe2fc105b8cded84256248f68c74139dd8fc2e9 - pkg_install_dir="${pkg_install_dir}/${TARGET}" + # ELPA-2017xxxx enables AVX2 by default, switch off if machine doesn't support it. + AVX_flag="" + AVX512_flags="" + FMA_flag="" + SSE4_flag="" + config_flags="--disable-avx-kernels --disable-avx2-kernels --disable-avx512-kernels --disable-sse-kernels --disable-sse-assembly-kernels" + if [ "${TARGET_CPU}" = "native" ]; then + if [ -f /proc/cpuinfo ] && [ "${OPENBLAS_ARCH}" = "x86_64" ]; then + has_AVX=$(grep '\bavx\b' /proc/cpuinfo 1> /dev/null && echo 'yes' || echo 'no') + [ "${has_AVX}" = "yes" ] && AVX_flag="-mavx" || AVX_flag="" + has_AVX2=$(grep '\bavx2\b' /proc/cpuinfo 1> /dev/null && echo 'yes' || echo 'no') + [ "${has_AVX2}" = "yes" ] && AVX_flag="-mavx2" + has_AVX512=$(grep '\bavx512f\b' /proc/cpuinfo 1> /dev/null && echo 'yes' || echo 'no') + [ "${has_AVX512}" = "yes" ] && AVX512_flags="-mavx512f" + FMA_flag=$(grep '\bfma\b' /proc/cpuinfo 1> /dev/null && echo '-mfma' || echo '-mno-fma') + SSE4_flag=$(grep '\bsse4_1\b' /proc/cpuinfo 1> /dev/null && echo '-msse4' || echo '-mno-sse4') + grep '\bavx512dq\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512dq" + grep '\bavx512cd\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512cd" + grep '\bavx512bw\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512bw" + grep '\bavx512vl\b' /proc/cpuinfo 1> /dev/null && AVX512_flags+=" -mavx512vl" + config_flags="--enable-avx-kernels=${has_AVX} --enable-avx2-kernels=${has_AVX2} --enable-avx512-kernels=${has_AVX512}" + fi + fi + for TARGET in "cpu" "nvidia"; do + [ "$TARGET" = "nvidia" ] && [ "$ENABLE_CUDA" != "__TRUE__" ] && continue + # disable cpu if cuda is enabled, only install one + [ "$TARGET" != "nvidia" ] && [ "$ENABLE_CUDA" = "__TRUE__" ] && continue + # extend the pkg_install_dir by TARGET + # this linking method is totally different from cp2k toolchain + # for cp2k, ref https://github.com/cp2k/cp2k/commit/6fe2fc105b8cded84256248f68c74139dd8fc2e9 + pkg_install_dir="${pkg_install_dir}/${TARGET}" - echo "Installing from scratch into ${pkg_install_dir}" - mkdir -p "build_${TARGET}" - cd "build_${TARGET}" - if [ "${with_amd}" != "__DONTUSE__" ] && [ "${WITH_FLANG}" = "yes" ] ; then - echo "AMD fortran compiler detected, enable special option operation" - ../configure --prefix="${pkg_install_dir}" \ - --libdir="${pkg_install_dir}/lib" \ - --enable-openmp=${enable_openmp} \ - --enable-static=no \ - --enable-shared=yes \ - --disable-c-tests \ - --disable-cpp-tests \ - ${config_flags} \ - --with-cuda-path=${CUDA_PATH:-${CUDA_HOME:-/CUDA_HOME-notset}} \ - --enable-nvidia-gpu-kernels=$([ "$TARGET" = "nvidia" ] && echo "yes" || echo "no") \ - --with-NVIDIA-GPU-compute-capability=$([ "$TARGET" = "nvidia" ] && echo "sm_$ARCH_NUM" || echo "sm_70") \ - OMPI_MCA_plm_rsh_agent=/bin/false \ - FC=${MPIFC} \ - CC=${MPICC} \ - CXX=${MPICXX} \ - CPP="cpp -E" \ - FCFLAGS="${FCFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ - CFLAGS="${CFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ - CXXFLAGS="${CXXFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ - LDFLAGS="${MATH_LDFLAGS} ${SCALAPACK_LDFLAGS} ${cray_ldflags} -lstdc++" \ - LIBS="${SCALAPACK_LIBS} $(resolve_string "${MATH_LIBS}" "MPI")" \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - # remove unsupported compile option in libtool - sed -i ./libtool \ - -e 's/\\$wl-soname //g' \ - -e 's/\\$wl--whole-archive\\$convenience \\$wl--no-whole-archive//g' \ - -e 's/\\$wl\\$soname //g' - else - # normal installation - ../configure --prefix="${pkg_install_dir}/" \ - --libdir="${pkg_install_dir}/lib" \ - --enable-openmp=${enable_openmp} \ - --enable-static=no \ - --enable-shared=yes \ - --disable-c-tests \ - --disable-cpp-tests \ - ${config_flags} \ - --enable-nvidia-gpu-kernels=$([ "$TARGET" = "nvidia" ] && echo "yes" || echo "no") \ - --with-cuda-path=${CUDA_PATH:-${CUDA_HOME:-/CUDA_HOME-notset}} \ - --with-NVIDIA-GPU-compute-capability=$([ "$TARGET" = "nvidia" ] && echo "sm_$ARCH_NUM" || echo "sm_70") \ - FC=${MPIFC} \ - CC=${MPICC} \ - CXX=${MPICXX} \ - CPP="cpp -E" \ - FCFLAGS="${FCFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ - CFLAGS="${CFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ - CXXFLAGS="${CXXFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ - LDFLAGS="-Wl,--allow-multiple-definition -Wl,--enable-new-dtags ${MATH_LDFLAGS} ${SCALAPACK_LDFLAGS} ${cray_ldflags} -lstdc++" \ - LIBS="${SCALAPACK_LIBS} $(resolve_string "${MATH_LIBS}" "MPI")" \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + echo "Installing from scratch into ${pkg_install_dir}" + mkdir -p "build_${TARGET}" + cd "build_${TARGET}" + if [ "${with_amd}" != "__DONTUSE__" ] && [ "${WITH_FLANG}" = "yes" ] ; then + # special option for flang compiler + echo "AMD fortran compiler detected, enable special option operation" + ../configure --prefix="${pkg_install_dir}" \ + --libdir="${pkg_install_dir}/lib" \ + --enable-openmp=${enable_openmp} \ + --enable-static=no \ + --enable-shared=yes \ + --disable-c-tests \ + --disable-cpp-tests \ + ${config_flags} \ + --with-cuda-path=${CUDA_PATH:-${CUDA_HOME:-/CUDA_HOME-notset}} \ + --enable-nvidia-gpu-kernels=$([ "$TARGET" = "nvidia" ] && echo "yes" || echo "no") \ + --with-NVIDIA-GPU-compute-capability=$([ "$TARGET" = "nvidia" ] && echo "sm_$ARCH_NUM" || echo "sm_70") \ + OMPI_MCA_plm_rsh_agent=/bin/false \ + FC=${MPIFC} \ + CC=${MPICC} \ + CXX=${MPICXX} \ + CPP="cpp -E" \ + FCFLAGS="${FCFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ + CFLAGS="${CFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ + CXXFLAGS="${CXXFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ + LDFLAGS="${MATH_LDFLAGS} ${SCALAPACK_LDFLAGS} ${cray_ldflags} -lstdc++" \ + LIBS="${SCALAPACK_LIBS} $(resolve_string "${MATH_LIBS}" "MPI") ${MPI_LIBS}" \ + SCALAPACK_LDFLAGS="${SCALAPACK_LDFLAGS}" \ + SCALAPACK_FCFLAGS="${SCALAPACK_CFLAGS}" \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + # remove unsupported compile option in libtool + sed -i ./libtool \ + -e 's/\\$wl-soname //g' \ + -e 's/\\$wl--whole-archive\\$convenience \\$wl--no-whole-archive//g' \ + -e 's/\\$wl\\$soname //g' + else + # normal installation + ../configure --prefix="${pkg_install_dir}/" \ + --libdir="${pkg_install_dir}/lib" \ + --enable-openmp=${enable_openmp} \ + --enable-static=no \ + --enable-shared=yes \ + --disable-c-tests \ + --disable-cpp-tests \ + ${config_flags} \ + --enable-nvidia-gpu-kernels=$([ "$TARGET" = "nvidia" ] && echo "yes" || echo "no") \ + --with-cuda-path=${CUDA_PATH:-${CUDA_HOME:-/CUDA_HOME-notset}} \ + --with-NVIDIA-GPU-compute-capability=$([ "$TARGET" = "nvidia" ] && echo "sm_$ARCH_NUM" || echo "sm_70") \ + FC=${MPIFC} \ + CC=${MPICC} \ + CXX=${MPICXX} \ + CPP="cpp -E" \ + FCFLAGS="${FCFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ + CFLAGS="${CFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ + CXXFLAGS="${CXXFLAGS} ${MATH_CFLAGS} ${SCALAPACK_CFLAGS} ${AVX_flag} ${FMA_flag} ${SSE4_flag} ${AVX512_flags} -fno-lto" \ + LDFLAGS="-Wl,--allow-multiple-definition -Wl,--enable-new-dtags ${MATH_LDFLAGS} ${SCALAPACK_LDFLAGS} ${cray_ldflags} -lstdc++" \ + LIBS="${SCALAPACK_LIBS} $(resolve_string "${MATH_LIBS}" "MPI")" \ + SCALAPACK_LDFLAGS="${SCALAPACK_LDFLAGS}" \ + SCALAPACK_FCFLAGS="${SCALAPACK_CFLAGS}" \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + fi + make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd .. + # link elpa + link=${pkg_install_dir}/include/elpa + if [[ ! -d $link ]]; then + ln -s ${pkg_install_dir}/include/elpa_openmp-${elpa_ver}/elpa $link + fi + done + cd .. + + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" fi - make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd .. - # link elpa - link=${pkg_install_dir}/include/elpa - if [[ ! -d $link ]]; then - ln -s ${pkg_install_dir}/include/elpa_openmp-${elpa_ver}/elpa $link + [ "$enable_openmp" != "yes" ] && elpa_dir_openmp="" + ELPA_CFLAGS="-I'${pkg_install_dir}/include/elpa${elpa_dir_openmp}-${elpa_ver}/modules' -I'${pkg_install_dir}/include/elpa${elpa_dir_openmp}-${elpa_ver}/elpa'" + ELPA_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; + __SYSTEM__) + echo "==================== Finding ELPA from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 fi - done - cd .. - - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" - fi - fi - [ "$enable_openmp" != "yes" ] && elpa_dir_openmp="" - ELPA_CFLAGS="-I'${pkg_install_dir}/include/elpa${elpa_dir_openmp}-${elpa_ver}/modules' -I'${pkg_install_dir}/include/elpa${elpa_dir_openmp}-${elpa_ver}/elpa'" - ELPA_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; - __SYSTEM__) - echo "==================== Finding ELPA from system paths ====================" - check_lib -lelpa_openmp "ELPA" - # get the include paths - elpa_include="$(find_in_paths "elpa_openmp-*" $INCLUDE_PATHS)" - if [ "$elpa_include" != "__FALSE__" ]; then - echo "ELPA include directory threaded version is found to be $elpa_include" - ELPA_CFLAGS="-I'$elpa_include/modules' -I'$elpa_include/elpa'" - else - echo "Cannot find elpa_openmp-${elpa_ver} from paths $INCLUDE_PATHS" - exit 1 - fi - # get the lib paths - add_lib_from_paths ELPA_LDFLAGS "libelpa.*" $LIB_PATHS - ;; - __DONTUSE__) ;; + check_lib -lelpa_openmp "ELPA" + # get the include paths + elpa_include="$(find_in_paths "elpa_openmp-*" $INCLUDE_PATHS)" + if [ "$elpa_include" != "__FALSE__" ]; then + echo "ELPA include directory threaded version is found to be $elpa_include" + ELPA_CFLAGS="-I'$elpa_include/modules' -I'$elpa_include/elpa'" + else + echo "Cannot find elpa_openmp-${elpa_ver} from paths $INCLUDE_PATHS" + exit 1 + fi + # get the lib paths + add_lib_from_paths ELPA_LDFLAGS "libelpa.*" $LIB_PATHS + ;; + __DONTUSE__) ;; - *) - echo "==================== Linking ELPA to user paths ====================" - pkg_install_dir="$with_elpa" - check_dir "${pkg_install_dir}/include" - check_dir "${pkg_install_dir}/lib" - user_include_path="${pkg_install_dir}/include" - elpa_include="$(find_in_paths "elpa_openmp-*" user_include_path)" - if [ "$elpa_include" != "__FALSE__" ]; then - echo "ELPA include directory threaded version is found to be $elpa_include/modules" - check_dir "$elpa_include/modules" - ELPA_CFLAGS="-I'$elpa_include/modules' -I'$elpa_include/elpa'" - else - echo "Cannot find elpa_openmp-* from path $user_include_path" - exit 1 - fi - ELPA_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + *) + echo "==================== Linking ELPA to user paths ====================" + pkg_install_dir="$with_elpa" + check_dir "${pkg_install_dir}/include" + check_dir "${pkg_install_dir}/lib" + user_include_path="${pkg_install_dir}/include" + elpa_include="$(find_in_paths "elpa_openmp-*" user_include_path)" + if [ "$elpa_include" != "__FALSE__" ]; then + echo "ELPA include directory threaded version is found to be $elpa_include/modules" + check_dir "$elpa_include/modules" + ELPA_CFLAGS="-I'$elpa_include/modules' -I'$elpa_include/elpa'" + else + echo "Cannot find elpa_openmp-* from path $user_include_path" + exit 1 + fi + ELPA_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac if [ "$with_elpa" != "__DONTUSE__" ]; then - ELPA_LIBS="-lelpa${elpa_dir_openmp}" - cat << EOF > "${BUILDDIR}/setup_elpa" + ELPA_LIBS="-lelpa${elpa_dir_openmp}" + cat << EOF > "${BUILDDIR}/setup_elpa" prepend_path CPATH "$elpa_include" EOF - if [ "$with_elpa" != "__SYSTEM__" ]; then - cat << EOF >> "${BUILDDIR}/setup_elpa" + if [ "$with_elpa" != "__SYSTEM__" ]; then + cat << EOF >> "${BUILDDIR}/setup_elpa" prepend_path PATH "$pkg_install_dir/bin" prepend_path LD_LIBRARY_PATH "$pkg_install_dir/lib" prepend_path CPATH "$pkg_install_dir/include" @@ -240,9 +262,9 @@ export PKG_CONFIG_PATH="$pkg_install_dir/lib/pkgconfig":\${PKG_CONFIG_PATH} export CMAKE_PREFIX_PATH="$pkg_install_dir":\${CMAKE_PREFIX_PATH} export ELPA_ROOT="$pkg_install_dir" EOF - fi - cat "${BUILDDIR}/setup_elpa" >> $SETUPFILE - cat << EOF >> "${BUILDDIR}/setup_elpa" + cat "${BUILDDIR}/setup_elpa" >> $SETUPFILE + fi + cat << EOF >> "${BUILDDIR}/setup_elpa" export ELPA_CFLAGS="${ELPA_CFLAGS}" export ELPA_LDFLAGS="${ELPA_LDFLAGS}" export ELPA_LIBS="${ELPA_LIBS}" diff --git a/toolchain/scripts/stage3/install_fftw.sh b/toolchain/scripts/stage3/install_fftw.sh index fac3c2040d..7f1a05dbcf 100755 --- a/toolchain/scripts/stage3/install_fftw.sh +++ b/toolchain/scripts/stage3/install_fftw.sh @@ -3,112 +3,131 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2023-0901 - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -fftw_ver="3.3.10" -fftw_sha256="56c932549852cddcfafdab3820b0200c7742675be92179e59e6215b340e26467" -fftw_pkg="fftw-${fftw_ver}.tar.gz" - source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load FFTW package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "fftw:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "fftw:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "fftw" "$version_suffix" +fftw_pkg="fftw-${fftw_ver}.tar.gz" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env [ -f "${BUILDDIR}/setup_fftw" ] && rm "${BUILDDIR}/setup_fftw" -FFTW_CFLAGS='' -FFTW_LDFLAGS='' -FFTW_LIBS='' +FFTW_CFLAGS="" +FFTW_LDFLAGS="" +FFTW_LIBS="" ! [ -d "${BUILDDIR}" ] && mkdir -p "${BUILDDIR}" cd "${BUILDDIR}" case "$with_fftw" in - __INSTALL__) - require_env MPI_LIBS - echo "==================== Installing FFTW ====================" - pkg_install_dir="${INSTALLDIR}/fftw-${fftw_ver}" - #pkg_install_dir="${HOME}/lib/fftw/${fftw_ver}-gcc8" - install_lock_file="$pkg_install_dir/install_successful" - - if verify_checksums "${install_lock_file}"; then - echo "fftw-${fftw_ver} is already installed, skipping it." - else - if [ -f ${fftw_pkg} ]; then - echo "${fftw_pkg} is found" - else - #download_pkg_from_ABACUS_org "${fftw_sha256}" "${fftw_pkg}" - url="http://www.fftw.org/${fftw_pkg}" - download_pkg_from_url "${fftw_sha256}" "${fftw_pkg}" "${url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d fftw-${fftw_ver} ] && rm -rf fftw-${fftw_ver} - tar -xzf ${fftw_pkg} - cd fftw-${fftw_ver} - FFTW_FLAGS="--enable-openmp --enable-shared" - # fftw has mpi support but not compiled by default. so compile it if we build with mpi. - # it will create a second library to link with if needed - [ "${MPI_MODE}" != "no" ] && FFTW_FLAGS="--enable-mpi ${FFTW_FLAGS}" - if [ "${TARGET_CPU}" = "native" ]; then - if [ -f /proc/cpuinfo ]; then - grep '\bavx\b' /proc/cpuinfo 1> /dev/null && FFTW_FLAGS="${FFTW_FLAGS} --enable-avx" - grep '\bavx2\b' /proc/cpuinfo 1> /dev/null && FFTW_FLAGS="${FFTW_FLAGS} --enable-avx2" - grep '\bavx512f\b' /proc/cpuinfo 1> /dev/null && FFTW_FLAGS="${FFTW_FLAGS} --enable-avx512" + __INSTALL__) + require_env MPI_LIBS + echo "==================== Installing FFTW ====================" + pkg_install_dir="${INSTALLDIR}/fftw-${fftw_ver}" + install_lock_file="$pkg_install_dir/install_successful" + if verify_checksums "${install_lock_file}"; then + echo "fftw-${fftw_ver} is already installed, skipping it." + else + if [ -f ${fftw_pkg} ]; then + echo "${fftw_pkg} is found" + else + url="http://www.fftw.org/${fftw_pkg}" + download_pkg_from_url "${fftw_sha256}" "${fftw_pkg}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d fftw-${fftw_ver} ] && rm -rf fftw-${fftw_ver} + tar -xzf ${fftw_pkg} + cd fftw-${fftw_ver} + FFTW_FLAGS="--enable-openmp --enable-shared" + # fftw has mpi support but not compiled by default. so compile it if we build with mpi. + # it will create a second library to link with if needed + [ "${MPI_MODE}" != "no" ] && FFTW_FLAGS="--enable-mpi ${FFTW_FLAGS}" + if [ "${TARGET_CPU}" = "native" ]; then + if [ -f /proc/cpuinfo ]; then + grep '\bavx\b' /proc/cpuinfo 1> /dev/null && FFTW_FLAGS="${FFTW_FLAGS} --enable-avx" + grep '\bavx2\b' /proc/cpuinfo 1> /dev/null && FFTW_FLAGS="${FFTW_FLAGS} --enable-avx2" + grep '\bavx512f\b' /proc/cpuinfo 1> /dev/null && FFTW_FLAGS="${FFTW_FLAGS} --enable-avx512" + fi + fi + # ABACUS need float version and double version fftw at the same time + # install float version fftw + echo "install float version fftw" + ./configure --prefix="${pkg_install_dir}" \ + --libdir="${pkg_install_dir}/lib" \ + --enable-float \ + ${FFTW_FLAGS} \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + # install double version fftw + echo "clean" + make distclean > /dev/null 2>&1 || true + echo "install double version fftw" + ./configure --prefix="${pkg_install_dir}" \ + --libdir="${pkg_install_dir}/lib" \ + ${FFTW_FLAGS} \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd .. + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" fi - fi - # ABACUS need float version and double version fftw at the same time - # install float version fftw - echo "install float version fftw" - ./configure --prefix=${pkg_install_dir} --libdir="${pkg_install_dir}/lib" ${FFTW_FLAGS} --enable-float \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - # install double version fftw - echo "clean" - make distclean > /dev/null 2>&1 || true - echo "install double version fftw" - ./configure --prefix=${pkg_install_dir} --libdir="${pkg_install_dir}/lib" ${FFTW_FLAGS} \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd .. - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" - fi - fi - FFTW_CFLAGS="-I'${pkg_install_dir}/include'" - FFTW_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; - __SYSTEM__) - echo "==================== Finding FFTW from system paths ====================" - check_lib -lfftw3 "FFTW" - check_lib -lfftw3_omp "FFTW" - [ "${MPI_MODE}" != "no" ] && check_lib -lfftw3_mpi "FFTW" - add_include_from_paths FFTW_CFLAGS "fftw3.h" FFTW_INC ${INCLUDE_PATHS} - add_lib_from_paths FFTW_LDFLAGS "libfftw3.*" ${LIB_PATHS} - ;; - __DONTUSE__) - # Nothing to do - ;; - *) - echo "==================== Linking FFTW to user paths ====================" - pkg_install_dir="$with_fftw" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - FFTW_CFLAGS="-I'${pkg_install_dir}/include'" - FFTW_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + FFTW_CFLAGS="-I'${pkg_install_dir}/include'" + FFTW_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; + __SYSTEM__) + echo "==================== Finding FFTW from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_lib -lfftw3 "FFTW" + check_lib -lfftw3_omp "FFTW" + [ "${MPI_MODE}" != "no" ] && check_lib -lfftw3_mpi "FFTW" + add_include_from_paths FFTW_CFLAGS "fftw3.h" FFTW_INC ${INCLUDE_PATHS} + add_lib_from_paths FFTW_LDFLAGS "libfftw3.*" ${LIB_PATHS} + ;; + __DONTUSE__) + ;; + *) + echo "==================== Linking FFTW to user paths ====================" + pkg_install_dir="$with_fftw" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + FFTW_CFLAGS="-I'${pkg_install_dir}/include'" + FFTW_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac if [ "$with_fftw" != "__DONTUSE__" ]; then - [ "$MPI_MODE" != "no" ] && FFTW_LIBS="IF_MPI(-lfftw3_mpi|)" - FFTW_LIBS+="-lfftw3 -lfftw3_omp" - if [ "$with_fftw" != "__SYSTEM__" ]; then - cat << EOF > "${BUILDDIR}/setup_fftw" + [ "$MPI_MODE" != "no" ] && FFTW_LIBS="IF_MPI(-lfftw3_mpi|)" + FFTW_LIBS+="-lfftw3 -lfftw3_omp" + if [ "$with_fftw" != "__SYSTEM__" ]; then + cat << EOF > "${BUILDDIR}/setup_fftw" prepend_path LD_LIBRARY_PATH "$pkg_install_dir/lib" prepend_path LD_RUN_PATH "$pkg_install_dir/lib" prepend_path LIBRARY_PATH "$pkg_install_dir/lib" @@ -122,9 +141,9 @@ export CPATH="$pkg_install_dir/include":\${CPATH} export PKG_CONFIG_PATH="$pkg_install_dir/lib/pkgconfig":\${PKG_CONFIG_PATH} export CMAKE_PREFIX_PATH="$pkg_install_dir":\${CMAKE_PREFIX_PATH} EOF - fi - # we may also want to cover FFT_SG - cat << EOF >> "${BUILDDIR}/setup_fftw" + fi + # we may also want to cover FFT_SG + cat << EOF >> "${BUILDDIR}/setup_fftw" export FFTW3_INCLUDES="${FFTW_CFLAGS}" export FFTW3_LIBS="${FFTW_LIBS}" export FFTW_CFLAGS="${FFTW_CFLAGS}" @@ -137,7 +156,7 @@ export CP_LIBS="${FFTW_LIBS} \${CP_LIBS}" export FFTW_ROOT=${FFTW_ROOT:-${pkg_install_dir}} export FFTW3_ROOT=${pkg_install_dir} EOF - cat "${BUILDDIR}/setup_fftw" >> $SETUPFILE + cat "${BUILDDIR}/setup_fftw" >> $SETUPFILE fi cd "${ROOTDIR}" diff --git a/toolchain/scripts/stage3/install_libxc.sh b/toolchain/scripts/stage3/install_libxc.sh index 3eeac7a658..4fc7ff6609 100755 --- a/toolchain/scripts/stage3/install_libxc.sh +++ b/toolchain/scripts/stage3/install_libxc.sh @@ -8,13 +8,29 @@ [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -# libxc_ver="6.2.2" -# libxc_sha256="a0f6f1bba7ba5c0c85b2bfe65aca1591025f509a7f11471b4cd651a79491b045" -libxc_ver="7.0.0" -libxc_sha256="e9ae69f8966d8de6b7585abd9fab588794ada1fab8f689337959a35abbf9527d" source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load LibXC package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libxc:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libxc:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "libxc" "$version_suffix" +libxc_pkg="libxc-${libxc_ver}.tar.bz2" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -27,72 +43,76 @@ LIBXC_LIBS="" cd "${BUILDDIR}" case "$with_libxc" in - __INSTALL__) - echo "==================== Installing LIBXC ====================" - pkg_install_dir="${INSTALLDIR}/libxc-${libxc_ver}" - #pkg_install_dir="${HOME}/lib/libxc/${libxc_ver}-gcc8" - install_lock_file="$pkg_install_dir/install_successful" - libxc_pkg="libxc-${libxc_ver}.tar.bz2" - if verify_checksums "${install_lock_file}"; then - echo "libxc-${libxc_ver} is already installed, skipping it." - else - if [ -f ${libxc_pkg} ]; then - echo "${libxc_pkg} is found" - else - #download_pkg_from_ABACUS_org "${libxc_sha256}" "${libxc_pkg}" - libxc_url="https://gitlab.com/libxc/libxc/-/archive/${libxc_ver}/${libxc_pkg}" - download_pkg_from_url "${libxc_sha256}" "${libxc_pkg}" "${libxc_url}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d libxc-${libxc_ver} ] && rm -rf libxc-${libxc_ver} - tar -xjf ${libxc_pkg} - cd libxc-${libxc_ver} - # using cmake method to install libxc is neccessary for abacus - mkdir build - cd build - cmake \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_INSTALL_PREFIX=${pkg_install_dir} \ - -DBUILD_SHARED_LIBS=YES \ - -DCMAKE_INSTALL_LIBDIR=lib \ - -DCMAKE_VERBOSE_MAKEFILE=ON \ - -DENABLE_FORTRAN=ON \ - -DENABLE_PYTHON=OFF \ - -DBUILD_TESTING=OFF .. \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd ../.. - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" - fi - fi - LIBXC_CFLAGS="-I'${pkg_install_dir}/include'" - LIBXC_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; - __SYSTEM__) - echo "==================== Finding LIBXC from system paths ====================" - check_lib -lxcf03 "libxc" - check_lib -lxc "libxc" - add_include_from_paths LIBXC_CFLAGS "xc.h" $INCLUDE_PATHS - add_lib_from_paths LIBXC_LDFLAGS "libxc.*" $LIB_PATHS - ;; - __DONTUSE__) ;; - *) - echo "==================== Linking LIBXC to user paths ====================" - pkg_install_dir="$with_libxc" - check_dir "${pkg_install_dir}/lib" - check_dir "${pkg_install_dir}/include" - LIBXC_CFLAGS="-I'${pkg_install_dir}/include'" - LIBXC_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + __INSTALL__) + echo "==================== Installing LIBXC ====================" + pkg_install_dir="${INSTALLDIR}/libxc-${libxc_ver}" + #pkg_install_dir="${HOME}/lib/libxc/${libxc_ver}-gcc8" + install_lock_file="$pkg_install_dir/install_successful" + if verify_checksums "${install_lock_file}"; then + echo "libxc-${libxc_ver} is already installed, skipping it." + else + if [ -f ${libxc_pkg} ]; then + echo "${libxc_pkg} is found" + else + #download_pkg_from_ABACUS_org "${libxc_sha256}" "${libxc_pkg}" + libxc_url="https://gitlab.com/libxc/libxc/-/archive/${libxc_ver}/${libxc_pkg}" + download_pkg_from_url "${libxc_sha256}" "${libxc_pkg}" "${libxc_url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d libxc-${libxc_ver} ] && rm -rf libxc-${libxc_ver} + tar -xjf ${libxc_pkg} + cd libxc-${libxc_ver} + # using cmake method to install libxc is neccessary for abacus + mkdir build + cd build + cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=${pkg_install_dir} \ + -DBUILD_SHARED_LIBS=YES \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_VERBOSE_MAKEFILE=ON \ + -DENABLE_FORTRAN=ON \ + -DENABLE_PYTHON=OFF \ + -DBUILD_TESTING=OFF .. \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + cd ../.. + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" + fi + LIBXC_CFLAGS="-I'${pkg_install_dir}/include'" + LIBXC_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; + __SYSTEM__) + echo "==================== Finding LIBXC from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_lib -lxcf03 "libxc" + check_lib -lxc "libxc" + add_include_from_paths LIBXC_CFLAGS "xc.h" $INCLUDE_PATHS + add_lib_from_paths LIBXC_LDFLAGS "libxc.*" $LIB_PATHS + ;; + __DONTUSE__) ;; + + *) + echo "==================== Linking LIBXC to user paths ====================" + pkg_install_dir="$with_libxc" + check_dir "${pkg_install_dir}/lib" + check_dir "${pkg_install_dir}/include" + LIBXC_CFLAGS="-I'${pkg_install_dir}/include'" + LIBXC_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac if [ "$with_libxc" != "__DONTUSE__" ]; then - LIBXC_LIBS="-lxcf03 -lxc" - if [ "$with_libxc" != "__SYSTEM__" ]; then - cat << EOF > "${BUILDDIR}/setup_libxc" + LIBXC_LIBS="-lxcf03 -lxc" + if [ "$with_libxc" != "__SYSTEM__" ]; then + cat << EOF > "${BUILDDIR}/setup_libxc" prepend_path LD_LIBRARY_PATH "$pkg_install_dir/lib" prepend_path LD_RUN_PATH "$pkg_install_dir/lib" prepend_path LIBRARY_PATH "$pkg_install_dir/lib" @@ -106,9 +126,9 @@ export CPATH="$pkg_install_dir/include":\${CPATH} export PKG_CONFIG_PATH="$pkg_install_dir/lib/pkgconfig":\${PKG_CONFIG_PATH} export CMAKE_PREFIX_PATH="$pkg_install_dir":\${CMAKE_PREFIX_PATH} EOF - cat "${BUILDDIR}/setup_libxc" >> $SETUPFILE - fi - cat << EOF >> "${BUILDDIR}/setup_libxc" + cat "${BUILDDIR}/setup_libxc" >> $SETUPFILE + fi + cat << EOF >> "${BUILDDIR}/setup_libxc" export LIBXC_CFLAGS="${LIBXC_CFLAGS}" export LIBXC_LDFLAGS="${LIBXC_LDFLAGS}" export LIBXC_LIBS="${LIBXC_LIBS}" diff --git a/toolchain/scripts/stage3/install_scalapack.sh b/toolchain/scripts/stage3/install_scalapack.sh index c2d6690bd1..3ffbf42c7d 100755 --- a/toolchain/scripts/stage3/install_scalapack.sh +++ b/toolchain/scripts/stage3/install_scalapack.sh @@ -3,100 +3,117 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2024-0504 - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -scalapack_ver="2.2.2" -scalapack_sha256="a2f0c9180a210bf7ffe126c9cb81099cf337da1a7120ddb4cbe4894eb7b7d022" -scalapack_pkg="scalapack-${scalapack_ver}.tar.gz" - source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load ScaLAPACK package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "scalapack:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "scalapack:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "scalapack" "$version_suffix" +scalapack_pkg="scalapack-${scalapack_ver}.tar.gz" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env [ -f "${BUILDDIR}/setup_scalapack" ] && rm "${BUILDDIR}/setup_scalapack" -SCALAPACK_CFLAGS='' -SCALAPACK_LDFLAGS='' -SCALAPACK_LIBS='' +SCALAPACK_CFLAGS="" +SCALAPACK_LDFLAGS="" +SCALAPACK_LIBS="" ! [ -d "${BUILDDIR}" ] && mkdir -p "${BUILDDIR}" cd "${BUILDDIR}" -case "$with_scalapack" in - __INSTALL__) - echo "==================== Installing ScaLAPACK ====================" - pkg_install_dir="${INSTALLDIR}/scalapack-${scalapack_ver}" - #pkg_install_dir="${HOME}/lib/scalapack/${scalapack_ver}-gcc8" - install_lock_file="$pkg_install_dir/install_successful" - if verify_checksums "${install_lock_file}"; then - echo "scalapack-${scalapack_ver} is already installed, skipping it." - else - require_env MATH_LIBS - if [ -f ${scalapack_pkg} ]; then - echo "${scalapack_pkg} is found" - else - url="https://codeload.github.com/Reference-ScaLAPACK/scalapack/tar.gz/v${scalapack_ver}" - download_pkg_from_url "${scalapack_sha256}" "${scalapack_pkg}" "${url}" - #download_pkg_from_ABACUS_org "${scalapack_sha256}" "${scalapack_pkg}" - fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d scalapack-${scalapack_ver} ] && rm -rf scalapack-${scalapack_ver} - tar -xzf ${scalapack_pkg} +case "${with_scalapack}" in + __INSTALL__) + echo "==================== Installing ScaLAPACK ====================" + pkg_install_dir="${INSTALLDIR}/scalapack-${scalapack_ver}" + install_lock_file="$pkg_install_dir/install_successful" + if verify_checksums "${install_lock_file}"; then + echo "scalapack-${scalapack_ver} is already installed, skipping it." + else + require_env MATH_LIBS + if [ -f ${scalapack_pkg} ]; then + echo "${scalapack_pkg} is found" + else + url="https://codeload.github.com/Reference-ScaLAPACK/scalapack/tar.gz/v${scalapack_ver}" + download_pkg_from_url "${scalapack_sha256}" "${scalapack_pkg}" "${url}" + #download_pkg_from_ABACUS_org "${scalapack_sha256}" "${scalapack_pkg}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d scalapack-${scalapack_ver} ] && rm -rf scalapack-${scalapack_ver} + tar -xzf ${scalapack_pkg} - mkdir -p "scalapack-${scalapack_ver}/build" - pushd "scalapack-${scalapack_ver}/build" > /dev/null + mkdir -p "scalapack-${scalapack_ver}/build" + pushd "scalapack-${scalapack_ver}/build" > /dev/null - cflags="" - fflags="" - if ("${FC}" --version | grep -q 'GNU'); then - cflags="-fpermissive" - fflags=$(allowed_gfortran_flags "-fallow-argument-mismatch") - fi - CFLAGS=${cflags} FFLAGS=${fflags} \ - cmake -DCMAKE_FIND_ROOT_PATH="$ROOTDIR" \ - -DCMAKE_INSTALL_PREFIX="${pkg_install_dir}" \ - -DCMAKE_INSTALL_LIBDIR="lib" \ - -DCMAKE_POSITION_INDEPENDENT_CODE=ON \ - -DCMAKE_VERBOSE_MAKEFILE=ON \ - -DBUILD_SHARED_LIBS=YES \ - -DCMAKE_BUILD_TYPE=Release .. \ - -DBUILD_TESTING=NO \ - -DSCALAPACK_BUILD_TESTS=NO \ - > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log - make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make install >> make.log 2>&1 || tail -n ${LOG_LINES} make.log + cflags="" + fflags="" + if ("${FC}" --version | grep -q 'GNU'); then + cflags="-fpermissive" + fflags=$(allowed_gfortran_flags "-fallow-argument-mismatch") + fi + CFLAGS=${cflags} FFLAGS=${fflags} \ + cmake -DCMAKE_FIND_ROOT_PATH="$ROOTDIR" \ + -DCMAKE_INSTALL_PREFIX="${pkg_install_dir}" \ + -DCMAKE_INSTALL_LIBDIR="lib" \ + -DCMAKE_POSITION_INDEPENDENT_CODE=ON \ + -DCMAKE_VERBOSE_MAKEFILE=ON \ + -DBUILD_SHARED_LIBS=YES \ + -DCMAKE_BUILD_TYPE=Release .. \ + -DBUILD_TESTING=NO \ + -DSCALAPACK_BUILD_TESTS=NO \ + > configure.log 2>&1 || tail -n ${LOG_LINES} configure.log + make -j $(get_nprocs) > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make install >> make.log 2>&1 || tail -n ${LOG_LINES} make.log - popd > /dev/null - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" - fi - fi - SCALAPACK_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; - __SYSTEM__) - echo "==================== Finding ScaLAPACK from system paths ====================" - check_lib -lscalapack "ScaLAPACK" - add_lib_from_paths SCALAPACK_LDFLAGS "libscalapack.*" $LIB_PATHS - ;; - __DONTUSE__) ;; + popd > /dev/null + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage3/$(basename ${SCRIPT_NAME})" + fi + SCALAPACK_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; + __SYSTEM__) + echo "==================== Finding ScaLAPACK from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_lib -lscalapack "ScaLAPACK" + add_lib_from_paths SCALAPACK_LDFLAGS "libscalapack.*" $LIB_PATHS + ;; + __DONTUSE__) ;; - *) - echo "==================== Linking ScaLAPACK to user paths ====================" - pkg_install_dir="$with_scalapack" - check_dir "${pkg_install_dir}/lib" - SCALAPACK_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" - ;; + *) + echo "==================== Linking ScaLAPACK to user paths ====================" + pkg_install_dir="$with_scalapack" + check_dir "${pkg_install_dir}/lib" + SCALAPACK_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" + ;; esac if [ "$with_scalapack" != "__DONTUSE__" ]; then - SCALAPACK_LIBS="-lscalapack" - if [ "$with_scalapack" != "__SYSTEM__" ]; then - cat << EOF > "${BUILDDIR}/setup_scalapack" + SCALAPACK_LIBS="-lscalapack" + if [ "$with_scalapack" != "__SYSTEM__" ]; then + cat << EOF > "${BUILDDIR}/setup_scalapack" prepend_path LD_LIBRARY_PATH "${pkg_install_dir}/lib" prepend_path LD_RUN_PATH "${pkg_install_dir}/lib" prepend_path LIBRARY_PATH "${pkg_install_dir}/lib" @@ -109,9 +126,9 @@ export PKG_CONFIG_PATH="${pkg_install_dir}/lib/pkgconfig":\${PKG_CONFIG_PATH} export CMAKE_PREFIX_PATH="${pkg_install_dir}":\${CMAKE_PREFIX_PATH} export SCALAPACK_ROOT="${pkg_install_dir}" EOF - cat "${BUILDDIR}/setup_scalapack" >> $SETUPFILE - fi - cat << EOF >> "${BUILDDIR}/setup_scalapack" + cat "${BUILDDIR}/setup_scalapack" >> $SETUPFILE + fi + cat << EOF >> "${BUILDDIR}/setup_scalapack" export SCALAPACK_LDFLAGS="${SCALAPACK_LDFLAGS}" export SCALAPACK_LIBS="${SCALAPACK_LIBS}" export SCALAPACK_ROOT="${pkg_install_dir}" diff --git a/toolchain/scripts/stage4/install_cereal.sh b/toolchain/scripts/stage4/install_cereal.sh index acadef15ab..5d5852a337 100755 --- a/toolchain/scripts/stage4/install_cereal.sh +++ b/toolchain/scripts/stage4/install_cereal.sh @@ -5,16 +5,33 @@ # CEREAL is not need any complex setting # Only problem is the installation from github.com -# Last Update in 2025-0504 - [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -cereal_ver="master" # latest version, instead of "1.3.2" -cereal_sha256="--no-checksum" # latest version cannot maintain checksum source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load Cereal package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "cereal:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "cereal:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "cereal" "$version_suffix" +dirname="cereal-${cereal_ver}" +filename="cereal-${cereal_ver}.tar.gz" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -25,72 +42,97 @@ CEREAL_CFLAGS="" cd "${BUILDDIR}" case "$with_cereal" in - __INSTALL__) - echo "==================== Installing CEREAL ====================" - dirname="cereal-${cereal_ver}" - pkg_install_dir="${INSTALLDIR}/$dirname" - #pkg_install_dir="${HOME}/lib/cereal/${cereal_ver}" - install_lock_file="$pkg_install_dir/install_successful" - url="https://codeload.github.com/USCiLab/cereal/tar.gz/${cereal_ver}" - filename="cereal-${cereal_ver}.tar.gz" - if verify_checksums "${install_lock_file}"; then - echo "$dirname is already installed, skipping it." - else - if [ -f $filename ]; then - echo "$filename is found" + __INSTALL__) + echo "==================== Installing CEREAL ====================" + pkg_install_dir="${INSTALLDIR}/$dirname" + #pkg_install_dir="${HOME}/lib/cereal/${cereal_ver}" + install_lock_file="$pkg_install_dir/install_successful" + # url construction rules: + # - Branch names (master, main, develop) without v prefix + # - Version tags (e.g., 1.0.0) with v prefix + if [[ "${cereal_ver}" =~ ^(master|main|develop)$ ]]; then + url="https://codeload.github.com/USCiLab/cereal/tar.gz/${cereal_ver}" else - # download from github.com and checksum - echo "===> Notice: This version of CEREAL is downloaded in GitHub master repository <===" - download_pkg_from_url "${cereal_sha256}" "${filename}" "${url}" + url="https://codeload.github.com/USCiLab/cereal/tar.gz/v${cereal_ver}" fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d $dirname ] && rm -rf $dirname - tar -xzf $filename - #unzip -q $filename - # apply patch files for libri installation in issue #6190, Kai Luo - # echo ${SCRIPT_DIR} - cd $dirname && pwd && patch -p1 < ${SCRIPT_DIR}/patches/6190.patch - cd "${BUILDDIR}" - # - mkdir -p "${pkg_install_dir}" - cp -r $dirname/* "${pkg_install_dir}/" - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" - fi - fi - CEREAL_CFLAGS="-I'${pkg_install_dir}'" + if verify_checksums "${install_lock_file}"; then + echo "$dirname is already installed, skipping it." + else + if [ -f $filename ]; then + echo "$filename is found" + else + # download from github.com and checksum + echo "===> Notice: This version of CEREAL is downloaded in GitHub master repository <===" + download_pkg_from_url "${cereal_sha256}" "${filename}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d $dirname ] && rm -rf $dirname + tar -xzf $filename + #unzip -q $filename + # apply patch files for libri installation in issue #6190, Kai Luo + # echo ${SCRIPT_DIR} + cd $dirname && pwd && patch -p1 < ${SCRIPT_DIR}/patches/6190.patch + cd "${BUILDDIR}" + # + mkdir -p "${pkg_install_dir}" + cp -r $dirname/* "${pkg_install_dir}/" + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" + fi + CEREAL_CFLAGS="-I'${pkg_install_dir}'" ;; __SYSTEM__) - echo "==================== CANNOT Finding CEREAL from system paths NOW ====================" - recommend_offline_installation $filename $url - # How to do it in cereal? -- Zhaoqing in 2023/08/23 - # check_lib -lxcf03 "libxc" - # check_lib -lxc "libxc" - # add_include_from_paths LIBXC_CFLAGS "xc.h" $INCLUDE_PATHS - # add_lib_from_paths LIBXC_LDFLAGS "libxc.*" $LIB_PATHS + echo "==================== Finding CEREAL from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + # Find cereal header file and derive package root directory + cereal_header_path="$(find_in_paths "cereal/cereal.hpp" $INCLUDE_PATHS)" + if [ "$cereal_header_path" != "__FALSE__" ]; then + # Derive pkg_install_dir from found header path + # cereal/cereal.hpp -> remove /cereal/cereal.hpp -> get include dir -> get parent dir + cereal_include_dir="$(dirname "$(dirname "$cereal_header_path")")" + pkg_install_dir="$(dirname "$cereal_include_dir")" + echo "Found cereal at: $pkg_install_dir" + CEREAL_CFLAGS="-I'${cereal_include_dir}'" + else + report_error "Cannot find cereal/cereal.hpp in system paths" + exit 1 + fi ;; __DONTUSE__) ;; *) - echo "==================== Linking CEREAL to user paths ====================" - check_dir "${pkg_install_dir}" - CEREAL_CFLAGS="-I'${pkg_install_dir}'" - ;; + echo "==================== Linking CEREAL to user paths ====================" + pkg_install_dir="${with_cereal}" + check_dir "${pkg_install_dir}" + CEREAL_CFLAGS="-I'${pkg_install_dir}'" + ;; esac if [ "$with_cereal" != "__DONTUSE__" ]; then if [ "$with_cereal" != "__SYSTEM__" ]; then - # LibRI deps should find cereal include in CPATH cat << EOF > "${BUILDDIR}/setup_cereal" prepend_path CPATH "$pkg_install_dir/include" +prepend_path CMAKE_PREFIX_PATH "${pkg_install_dir}/include" export CPATH="${pkg_install_dir}/include":\${CPATH} +export CMAKE_PREFIX_PATH="${pkg_install_dir}/include":\${CMAKE_PREFIX_PATH} +export CEREAL_ROOT="$pkg_install_dir" +EOF + else + cat << EOF > "${BUILDDIR}/setup_cereal" +export CEREAL_ROOT="$pkg_install_dir" EOF - cat "${BUILDDIR}/setup_cereal" >> $SETUPFILE fi + cat "${BUILDDIR}/setup_cereal" >> $SETUPFILE cat << EOF >> "${BUILDDIR}/setup_cereal" export CEREAL_CFLAGS="${CEREAL_CFLAGS}" -export CEREAL_ROOT="$pkg_install_dir" +export CP_DFLAGS="\${CP_DFLAGS} -D__CEREAL" +export CP_CFLAGS="\${CP_CFLAGS} ${CEREAL_CFLAGS}" +export CEREAL_VERSION="${cereal_ver}" EOF fi diff --git a/toolchain/scripts/stage4/install_libcomm.sh b/toolchain/scripts/stage4/install_libcomm.sh index ccb39061ea..f5b0c22715 100755 --- a/toolchain/scripts/stage4/install_libcomm.sh +++ b/toolchain/scripts/stage4/install_libcomm.sh @@ -12,11 +12,30 @@ [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -libcomm_ver="master" -libcomm_sha256="--no-checksum" source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load LibComm package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libcomm:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libcomm:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "libcomm" "$version_suffix" +dirname="LibComm-${libcomm_ver}" +filename="LibComm-${libcomm_ver}.tar.gz" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -27,54 +46,71 @@ libcomm_CFLAGS="" cd "${BUILDDIR}" case "$with_libcomm" in - __INSTALL__) - echo "==================== Installing LIBCOMM ====================" - dirname="LibComm-${libcomm_ver}" - pkg_install_dir="${INSTALLDIR}/$dirname" - #pkg_install_dir="${HOME}/lib/libcomm/${libcomm_ver}" - install_lock_file="$pkg_install_dir/install_successful" - # url="https://github.com/abacusmodeling/LibComm/archive/refs/tags/v${libcomm_ver}.tar.gz" - filename="LibComm-${libcomm_ver}.tar.gz" - url="https://codeload.github.com/abacusmodeling/LibComm/tar.gz/${libcomm_ver}" - if verify_checksums "${install_lock_file}"; then - echo "$dirname is already installed, skipping it." - else - if [ -f $filename ]; then - echo "$filename is found" + __INSTALL__) + echo "==================== Installing LIBCOMM ====================" + pkg_install_dir="${INSTALLDIR}/$dirname" + #pkg_install_dir="${HOME}/lib/libcomm/${libcomm_ver}" + install_lock_file="$pkg_install_dir/install_successful" + # url="https://github.com/abacusmodeling/LibComm/archive/refs/tags/v${libcomm_ver}.tar.gz" + # url construction rules: + # - Branch names (master, main, develop) without v prefix + # - Version tags (e.g., 1.0.0) with v prefix + if [[ "${libcomm_ver}" =~ ^(master|main|develop)$ ]]; then + url="https://codeload.github.com/abacusmodeling/LibComm/tar.gz/${libcomm_ver}" else - # download from github.com and checksum - echo "===> Notice: This version of LibComm is downloaded in GitHub master repository <===" - download_pkg_from_url "${libcomm_sha256}" "${filename}" "${url}" + url="https://codeload.github.com/abacusmodeling/LibComm/tar.gz/v${libcomm_ver}" fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d $dirname ] && rm -rf $dirname - tar -xzf $filename - # unzip -q $filename - cp -r $dirname "${pkg_install_dir}/" - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" - fi - fi - LIBCOMM_CFLAGS="-I'${pkg_install_dir}'" + if verify_checksums "${install_lock_file}"; then + echo "$dirname is already installed, skipping it." + else + if [ -f $filename ]; then + echo "$filename is found" + else + # download from github.com and checksum + echo "===> Notice: This version of LibComm is downloaded in GitHub master repository <===" + download_pkg_from_url "${libcomm_sha256}" "${filename}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d $dirname ] && rm -rf $dirname + tar -xzf $filename + mkdir -p "${pkg_install_dir}" + cp -r $dirname/* "${pkg_install_dir}/" + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" + fi + LIBCOMM_CFLAGS="-I'${pkg_install_dir}'" ;; __SYSTEM__) - echo "==================== CANNOT Finding LIBCOMM from system paths NOW ====================" - recommend_offline_installation $filename $url - # How to do it in libcomm? -- Zhaoqing in 2023/08/23 - # check_lib -lxcf03 "libxc" - # check_lib -lxc "libxc" - # add_include_from_paths LIBXC_CFLAGS "xc.h" $INCLUDE_PATHS - # add_lib_from_paths LIBXC_LDFLAGS "libxc.*" $LIB_PATHS + echo "==================== Finding LIBCOMM from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + # Find libcomm header file and derive package root directory + libcomm_header_path="$(find_in_paths "Comm/Comm_Tools.h" $INCLUDE_PATHS)" + if [ "$libcomm_header_path" != "__FALSE__" ]; then + # Derive pkg_install_dir from found header path + # Comm/Comm_Tools.h -> remove /Comm/Comm_Tools.h -> get include dir -> get parent dir + libcomm_include_dir="$(dirname "$(dirname "$libcomm_header_path")")" + pkg_install_dir="$(dirname "$libcomm_include_dir")" + echo "Found libcomm at: $pkg_install_dir" + LIBCOMM_CFLAGS="-I'${libcomm_include_dir}'" + else + report_error "Cannot find Comm/Comm_Tools.h in system paths" + exit 1 + fi ;; __DONTUSE__) ;; *) - echo "==================== Linking LIBCOMM to user paths ====================" - check_dir "${pkg_install_dir}" - LIBCOMM_CFLAGS="-I'${pkg_install_dir}'" - ;; + echo "==================== Linking LIBCOMM to user paths ====================" + pkg_install_dir="${with_libcomm}" + check_dir "${pkg_install_dir}" + LIBCOMM_CFLAGS="-I'${pkg_install_dir}'" + ;; esac if [ "$with_libcomm" != "__DONTUSE__" ]; then if [ "$with_libcomm" != "__SYSTEM__" ]; then diff --git a/toolchain/scripts/stage4/install_libnpy.sh b/toolchain/scripts/stage4/install_libnpy.sh index ebca6214e7..aa2f59df32 100755 --- a/toolchain/scripts/stage4/install_libnpy.sh +++ b/toolchain/scripts/stage4/install_libnpy.sh @@ -11,11 +11,30 @@ [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -libnpy_ver="1.0.1" -libnpy_sha256="43452a4db1e8c1df606c64376ea1e32789124051d7640e7e4e8518ab4f0fba44" source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load LibNPY package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libnpy:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libnpy:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "libnpy" "$version_suffix" +dirname="libnpy-${libnpy_ver}" +filename="libnpy-${libnpy_ver}.tar.gz" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -26,53 +45,63 @@ LIBNPY_CFLAGS="" cd "${BUILDDIR}" case "$with_libnpy" in - __INSTALL__) - echo "==================== Installing LIBNPY ====================" - dirname="libnpy-${libnpy_ver}" - pkg_install_dir="${INSTALLDIR}/$dirname" - #pkg_install_dir="${HOME}/lib/libnpy/${libnpy_ver}" - install_lock_file="$pkg_install_dir/install_successful" - url="https://codeload.github.com/llohse/libnpy/tar.gz/v${libnpy_ver}" - filename="libnpy-${libnpy_ver}.tar.gz" - if verify_checksums "${install_lock_file}"; then - echo "$dirname is already installed, skipping it." - else - if [ -f $filename ]; then - echo "$filename is found" + __INSTALL__) + echo "==================== Installing LIBNPY ====================" + pkg_install_dir="${INSTALLDIR}/$dirname" + #pkg_install_dir="${HOME}/lib/libnpy/${libnpy_ver}" + install_lock_file="$pkg_install_dir/install_successful" + url="https://codeload.github.com/llohse/libnpy/tar.gz/v${libnpy_ver}" + if verify_checksums "${install_lock_file}"; then + echo "$dirname is already installed, skipping it." else - # download from github.com and checksum - echo "===> Notice: This version of Libnpy is downloaded in GitHub Release <===" - download_pkg_from_url "${libnpy_sha256}" "${filename}" "${url}" + if [ -f $filename ]; then + echo "$filename is found" + else + # download from github.com and checksum + echo "===> Notice: This version of Libnpy is downloaded in GitHub Release <===" + download_pkg_from_url "${libnpy_sha256}" "${filename}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d $dirname ] && rm -rf $dirname + tar -xzf $filename + mkdir -p "${pkg_install_dir}" + cp -r $dirname/* "${pkg_install_dir}/" + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d $dirname ] && rm -rf $dirname - tar -xzf $filename - mkdir -p "${pkg_install_dir}" - cp -r $dirname/* "${pkg_install_dir}/" - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" - fi - fi - LIBNPY_CFLAGS="-I'${pkg_install_dir}'" + LIBNPY_CFLAGS="-I'${pkg_install_dir}'" ;; __SYSTEM__) - echo "==================== CANNOT Finding LIBNPY from system paths NOW ====================" - recommend_offline_installation $filename $url - # How to do it in libnpy? -- Zhaoqing in 2023/08/23 - # check_lib -lxcf03 "libxc" - # check_lib -lxc "libxc" - # add_include_from_paths LIBXC_CFLAGS "xc.h" $INCLUDE_PATHS - # add_lib_from_paths LIBXC_LDFLAGS "libxc.*" $LIB_PATHS + echo "==================== Finding LIBNPY from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + # Find libnpy header file and derive package root directory + libnpy_header_path="$(find_in_paths "npy.hpp" $INCLUDE_PATHS)" + if [ "$libnpy_header_path" != "__FALSE__" ]; then + # Derive pkg_install_dir from found header path + # npy.hpp -> get include dir -> get parent dir + libnpy_include_dir="$(dirname "$libnpy_header_path")" + pkg_install_dir="$(dirname "$libnpy_include_dir")" + echo "Found libnpy at: $pkg_install_dir" + LIBNPY_CFLAGS="-I'${libnpy_include_dir}'" + else + report_error "Cannot find npy.hpp in system paths" + exit 1 + fi ;; __DONTUSE__) ;; *) - echo "==================== Linking LIBNPY to user paths ====================" - check_dir "${pkg_install_dir}" - LIBNPY_CFLAGS="-I'${pkg_install_dir}'" - ;; + echo "==================== Linking LIBNPY to user paths ====================" + pkg_install_dir="${with_libnpy}" + check_dir "${pkg_install_dir}" + LIBNPY_CFLAGS="-I'${pkg_install_dir}'" + ;; esac if [ "$with_libnpy" != "__DONTUSE__" ]; then if [ "$with_libnpy" != "__SYSTEM__" ]; then diff --git a/toolchain/scripts/stage4/install_libri.sh b/toolchain/scripts/stage4/install_libri.sh index 70507388a2..ba0ee25c46 100755 --- a/toolchain/scripts/stage4/install_libri.sh +++ b/toolchain/scripts/stage4/install_libri.sh @@ -11,15 +11,29 @@ [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -# libri 0.2.0 and above need to be used in newer ABACUS -# libri_ver="master" -# libri_sha256="--no-checksum" -libri_ver=0.2.1.1 -libri_sha256="cd33fd5428400ea696b82c9132878c07bf785847b3f56b1979e25a3a5fc0b311" -# libri_sha256="66a5540daba36effdad6ce2fe5e8368b96ddd4a7e148af90894ef21dc20ff29f" + source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load LibRI package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libri:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libri:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "libri" "$version_suffix" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -30,53 +44,72 @@ libri_CFLAGS="" cd "${BUILDDIR}" case "$with_libri" in - __INSTALL__) - echo "==================== Installing LIBRI ====================" - dirname="LibRI-${libri_ver}" - pkg_install_dir="${INSTALLDIR}/$dirname" - #pkg_install_dir="${HOME}/lib/libri/${libri_ver}" - install_lock_file="$pkg_install_dir/install_successful" - url="https://codeload.github.com/abacusmodeling/LibRI/tar.gz/v${libri_ver}" - filename="LibRI-${libri_ver}.tar.gz" - if verify_checksums "${install_lock_file}"; then - echo "$dirname is already installed, skipping it." - else - if [ -f $filename ]; then - echo "$filename is found" + __INSTALL__) + echo "==================== Installing LIBRI ====================" + dirname="LibRI-${libri_ver}" + pkg_install_dir="${INSTALLDIR}/$dirname" + #pkg_install_dir="${HOME}/lib/libri/${libri_ver}" + install_lock_file="$pkg_install_dir/install_successful" + # url construction rules: + # - Branch names (master, main, develop) without v prefix + # - Version tags (e.g., 1.0.0) with v prefix + if [[ "${libri_ver}" =~ ^(master|main|develop)$ ]]; then + url="https://codeload.github.com/abacusmodeling/LibRI/tar.gz/${libri_ver}" else - # download from github.com and checksum - echo "===> Notice: This version of LibRI is downloaded in GitHub Release <===" - download_pkg_from_url "${libri_sha256}" "${filename}" "${url}" + url="https://codeload.github.com/abacusmodeling/LibRI/tar.gz/v${libri_ver}" fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d $dirname ] && rm -rf $dirname - #unzip -q $filename - tar -xzf $filename - cp -r $dirname "${pkg_install_dir}/" - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" - fi - fi - LIBRI_CFLAGS="-I'${pkg_install_dir}'" + filename="LibRI-${libri_ver}.tar.gz" + if verify_checksums "${install_lock_file}"; then + echo "$dirname is already installed, skipping it." + else + if [ -f $filename ]; then + echo "$filename is found" + else + # download from github.com and checksum + echo "===> Notice: This version of LibRI is downloaded in GitHub Release <===" + download_pkg_from_url "${libri_sha256}" "${filename}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d $dirname ] && rm -rf $dirname + tar -xzf $filename + mkdir -p "${pkg_install_dir}" + cp -r $dirname/* "${pkg_install_dir}/" + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" + fi + LIBRI_CFLAGS="-I'${pkg_install_dir}'" ;; __SYSTEM__) - echo "==================== CANNOT Finding LIBRI from system paths NOW ====================" - recommend_offline_installation $filename $url - # How to do it in libri? -- Zhaoqing in 2023/08/23 - # check_lib -lxcf03 "libxc" - # check_lib -lxc "libxc" - # add_include_from_paths LIBXC_CFLAGS "xc.h" $INCLUDE_PATHS - # add_lib_from_paths LIBXC_LDFLAGS "libxc.*" $LIB_PATHS + echo "==================== Finding LIBRI from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + # Find libri header file and derive package root directory + libri_header_path="$(find_in_paths "RI/version.h" $INCLUDE_PATHS)" + if [ "$libri_header_path" != "__FALSE__" ]; then + # Derive pkg_install_dir from found header path + # RI/version.h -> remove /RI/version.h -> get include dir -> get parent dir + libri_include_dir="$(dirname "$(dirname "$libri_header_path")")" + pkg_install_dir="$(dirname "$libri_include_dir")" + echo "Found libri at: $pkg_install_dir" + LIBRI_CFLAGS="-I'${libri_include_dir}'" + else + report_error "Cannot find RI/version.h in system paths" + exit 1 + fi ;; __DONTUSE__) ;; - + *) - echo "==================== Linking LIBRI to user paths ====================" - check_dir "${pkg_install_dir}" - LIBRI_CFLAGS="-I'${pkg_install_dir}'" - ;; + echo "==================== Linking LIBRI to user paths ====================" + pkg_install_dir="${with_libri}" + check_dir "${pkg_install_dir}" + LIBRI_CFLAGS="-I'${pkg_install_dir}'" + ;; esac if [ "$with_libri" != "__DONTUSE__" ]; then if [ "$with_libri" != "__SYSTEM__" ]; then diff --git a/toolchain/scripts/stage4/install_libtorch.sh b/toolchain/scripts/stage4/install_libtorch.sh index 84fcbe8e10..ed05c17833 100755 --- a/toolchain/scripts/stage4/install_libtorch.sh +++ b/toolchain/scripts/stage4/install_libtorch.sh @@ -9,22 +9,29 @@ [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -# From https://pytorch.org/get-started/locally/ -# libtorch_ver="1.12.1" -# libtorch_sha256="82c7be80860f2aa7963f8700004a40af8205e1d721298f2e09b700e766a9d283" -# libtorch_ver="2.0.1" -# libtorch_sha256="137a842d1cf1e9196b419390133a1623ef92f8f84dc7a072f95ada684f394afd" -libtorch_ver="2.1.2" -libtorch_sha256="904b764df6106a8a35bef64c4b55b8c1590ad9d071eb276e680cf42abafe79e9" - -# user can manually download higher version of libtorch by: -# wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-{libtorch_ver}%2Bcpu.zip -# 2.4.0 latest, 2.1.2 recommended for lower GLIBC support (lower than 3.4.26) - # shellcheck source=/dev/null source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load LibTorch package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libtorch:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "libtorch:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "libtorch" "$version_suffix" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -34,70 +41,71 @@ source "${INSTALLDIR}"/toolchain.env cd "${BUILDDIR}" case "${with_libtorch}" in - __INSTALL__) - echo "==================== Installing libtorch ====================" - dirname="libtorch-${libtorch_ver}" - pkg_install_dir="${INSTALLDIR}/${dirname}" - #pkg_install_dir="${HOME}/lib/libtorch/${libtorch_ver}" - install_lock_file="${pkg_install_dir}/install_successful" - archive_file="libtorch-cxx11-abi-shared-with-deps-${libtorch_ver}%2Bcpu.zip" - filename="${dirname}.zip" - - if verify_checksums "${install_lock_file}"; then - echo "${dirname} is already installed, skipping it." - else - if [ -f ${filename} ]; then - echo "${filename} is found" + __INSTALL__) + echo "==================== Installing libtorch ====================" + dirname="libtorch-${libtorch_ver}" + pkg_install_dir="${INSTALLDIR}/${dirname}" + #pkg_install_dir="${HOME}/lib/libtorch/${libtorch_ver}" + install_lock_file="${pkg_install_dir}/install_successful" + archive_file="libtorch-cxx11-abi-shared-with-deps-${libtorch_ver}%2Bcpu.zip" + filename="${dirname}.zip" + + if verify_checksums "${install_lock_file}"; then + echo "${dirname} is already installed, skipping it." else - # download from pytorch.com and checksum - url=https://download.pytorch.org/libtorch/cpu/${archive_file} - download_pkg_from_url "${libtorch_sha256}" "${filename}" "${url}" + if [ -f ${filename} ]; then + echo "${filename} is found" + else + # download from pytorch.com and checksum + url=https://download.pytorch.org/libtorch/cpu/${archive_file} + download_pkg_from_url "${libtorch_sha256}" "${filename}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + unzip -q $filename + mkdir -p "${pkg_install_dir}" + mv libtorch/* "${pkg_install_dir}/" + + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d libtorch ] && rm -rf libtorch - [ -d ${pkg_install_dir} ] && rm -rf ${pkg_install_dir} - unzip -q $filename # to libtorch - mkdir -p "${pkg_install_dir}" - mv libtorch/* "${pkg_install_dir}/" - - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" - fi - fi - LIBTORCH_CXXFLAGS="-I${pkg_install_dir}/include" - LIBTORCH_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath='${pkg_install_dir}/lib'" - ;; - __SYSTEM__) - echo "==================== Finding libtorch from system paths ====================" - check_lib -ltorch "libtorch" - add_include_from_paths LIBTORCH_CXXFLAGS "libtorch.h" $INCLUDE_PATHS - add_lib_from_paths LIBTORCH_LDFLAGS "libtorch.*" "$LIB_PATHS" - ;; - __DONTUSE__) ;; - - *) - echo "==================== Linking libtorch to user paths ====================" - pkg_install_dir="${with_libtorch}" - - # use the lib64 directory if present (multi-abi distros may link lib/ to lib32/ instead) - LIBTORCH_LIBDIR="${pkg_install_dir}/lib" - [ -d "${pkg_install_dir}/lib64" ] && LIBTORCH_LIBDIR="${pkg_install_dir}/lib64" - - check_dir "${LIBTORCH_LIBDIR}" - LIBTORCH_CXXFLAGS="-I${pkg_install_dir}/include" - if [ "$ENABLE_CUDA" = "__TRUE__" ]; then - LIBTORCH_LDFLAGS="-Wl,--no-as-needed,-L'${LIBTORCH_LIBDIR}' -Wl,--no-as-needed,-rpath='${LIBTORCH_LIBDIR}'" - else - LIBTORCH_LDFLAGS="-L'${LIBTORCH_LIBDIR}' -Wl,-rpath='${LIBTORCH_LIBDIR}'" - fi - ;; + LIBTORCH_CXXFLAGS="-I${pkg_install_dir}/include" + LIBTORCH_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath='${pkg_install_dir}/lib'" + ;; + __SYSTEM__) + echo "==================== Finding libtorch from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + check_lib -ltorch "libtorch" + add_include_from_paths LIBTORCH_CXXFLAGS "libtorch.h" $INCLUDE_PATHS + add_lib_from_paths LIBTORCH_LDFLAGS "libtorch.*" "$LIB_PATHS" + ;; + __DONTUSE__) ;; + + *) + echo "==================== Linking libtorch to user paths ====================" + pkg_install_dir="${with_libtorch}" + + # use the lib64 directory if present (multi-abi distros may link lib/ to lib32/ instead) + LIBTORCH_LIBDIR="${pkg_install_dir}/lib" + [ -d "${pkg_install_dir}/lib64" ] && LIBTORCH_LIBDIR="${pkg_install_dir}/lib64" + + check_dir "${LIBTORCH_LIBDIR}" + LIBTORCH_CXXFLAGS="-I${pkg_install_dir}/include" + if [ "$ENABLE_CUDA" = "__TRUE__" ]; then + LIBTORCH_LDFLAGS="-Wl,--no-as-needed,-L'${LIBTORCH_LIBDIR}' -Wl,--no-as-needed,-rpath='${LIBTORCH_LIBDIR}'" + LIBTORCH_LDFLAGS="-L'${LIBTORCH_LIBDIR}' -Wl,-rpath='${LIBTORCH_LIBDIR}'" + fi + ;; esac if [ "$with_libtorch" != "__DONTUSE__" ]; then - if [ "$with_libtorch" != "__SYSTEM__" ]; then - cat << EOF > "${BUILDDIR}/setup_libtorch" + if [ "$with_libtorch" != "__SYSTEM__" ]; then + cat << EOF > "${BUILDDIR}/setup_libtorch" prepend_path LD_LIBRARY_PATH "${pkg_install_dir}/lib" prepend_path LD_RUN_PATH "${pkg_install_dir}/lib" prepend_path LIBRARY_PATH "${pkg_install_dir}/lib" @@ -110,24 +118,23 @@ export CPATH="${pkg_install_dir}/include":\${CPATH} export PKG_CONFIG_PATH="${pkg_install_dir}/lib/pkgconfig":\${PKG_CONFIG_PATH} export CMAKE_PREFIX_PATH="${pkg_install_dir}":\${CMAKE_PREFIX_PATH} EOF - fi - if [ "$ENABLE_CUDA" = "__TRUE__" ]; then - cat << EOF >> "${BUILDDIR}/setup_libtorch" + fi + if [ "$ENABLE_CUDA" = "__TRUE__" ]; then + cat << EOF >> "${BUILDDIR}/setup_libtorch" export CP_DFLAGS="\${CP_DFLAGS} -D__LIBTORCH" export CXXFLAGS="\${CXXFLAGS} ${LIBTORCH_CXXFLAGS}" export CP_LDFLAGS="\${CP_LDFLAGS} ${LIBTORCH_LDFLAGS}" export CP_LIBS="\${CP_LIBS} -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch" EOF - cat "${BUILDDIR}/setup_libtorch" >> "${SETUPFILE}" - else - cat << EOF >> "${BUILDDIR}/setup_libtorch" + cat "${BUILDDIR}/setup_libtorch" >> "${SETUPFILE}" + cat << EOF >> "${BUILDDIR}/setup_libtorch" export CP_DFLAGS="\${CP_DFLAGS} -D__LIBTORCH" export CXXFLAGS="\${CXXFLAGS} ${LIBTORCH_CXXFLAGS}" export CP_LDFLAGS="\${CP_LDFLAGS} ${LIBTORCH_LDFLAGS}" export CP_LIBS="\${CP_LIBS} -lc10 -ltorch_cpu -ltorch" EOF - cat "${BUILDDIR}/setup_libtorch" >> "${SETUPFILE}" - fi + cat "${BUILDDIR}/setup_libtorch" >> "${SETUPFILE}" + fi fi load "${BUILDDIR}/setup_libtorch" diff --git a/toolchain/scripts/stage4/install_nep.sh b/toolchain/scripts/stage4/install_nep.sh index 42ac0b94ca..3a5ad8eb6f 100755 --- a/toolchain/scripts/stage4/install_nep.sh +++ b/toolchain/scripts/stage4/install_nep.sh @@ -3,13 +3,14 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# Last Update in 2025-10-10 +# contributor: MoseyQAQ (Denan Li) [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -nep_ver="main" -nep_sha256="--no-checksum" +# Load version information from centralized package_versions.sh +source "${SCRIPT_DIR}/package_versions.sh" +load_package_vars "nep" source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh @@ -45,13 +46,14 @@ case "$with_nep" in if [ "${PACK_RUN}" = "__TRUE__" ]; then echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d $dirname ] && rm -rf $dirname - tar -xzf $filename - cd $dirname + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d $dirname ] && rm -rf $dirname + tar -xzf $filename + cd $dirname - cat << EOF > Makefile + cat << EOF > Makefile CXX ?= g++ # Compiler flags @@ -92,12 +94,11 @@ install: cp src/nep.h \$(PREFIX)/include/ EOF - make > make.log 2>&1 || tail -n ${LOG_LINES} make.log - make PREFIX="${pkg_install_dir}" install > install.log 2>&1 || tail -n ${LOG_LINES} install.log + make > make.log 2>&1 || tail -n ${LOG_LINES} make.log + make PREFIX="${pkg_install_dir}" install > install.log 2>&1 || tail -n ${LOG_LINES} install.log - cd .. - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" - fi + cd .. + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" fi NEP_CFLAGS="-I'${pkg_install_dir}/include'" NEP_LDFLAGS="-L'${pkg_install_dir}/lib' -Wl,-rpath,'${pkg_install_dir}/lib'" diff --git a/toolchain/scripts/stage4/install_rapidjson.sh b/toolchain/scripts/stage4/install_rapidjson.sh index 27fb94edd3..da6727ac40 100755 --- a/toolchain/scripts/stage4/install_rapidjson.sh +++ b/toolchain/scripts/stage4/install_rapidjson.sh @@ -2,20 +2,36 @@ # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all -# RAPIDJSON is not need any complex setting +# RapidJSON is not need any complex setting # Only problem is the installation from github.com -# Last Update in 2025-0504 # other contributor: Kai Luo, XingLiang Peng [ "${BASH_SOURCE[0]}" ] && SCRIPT_NAME="${BASH_SOURCE[0]}" || SCRIPT_NAME=$0 SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_NAME")/.." && pwd -P)" -rapidjson_ver="master" # latest version, instead of "1.1.0" fixing issue of #5518 -rapidjson_sha256="--no-checksum" # latest version cannot maintain checksum source "${SCRIPT_DIR}"/common_vars.sh source "${SCRIPT_DIR}"/tool_kit.sh source "${SCRIPT_DIR}"/signal_trap.sh +source "${SCRIPT_DIR}"/package_versions.sh + +# Load RapidJSON package variables with version suffix support +# Check for version configuration from environment or individual package setting +version_suffix="" +if [[ -n "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" ]]; then + # Check for individual package version override + if echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "rapidjson:alt"; then + version_suffix="alt" + elif echo "${ABACUS_TOOLCHAIN_PACKAGE_VERSIONS}" | grep -q "rapidjson:main"; then + version_suffix="main" + fi +fi +# Fall back to global version suffix if no individual setting +if [[ -z "$version_suffix" && -n "${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" ]]; then + version_suffix="${ABACUS_TOOLCHAIN_VERSION_SUFFIX}" +fi +# Load package variables with appropriate version +load_package_vars "rapidjson" "$version_suffix" source "${INSTALLDIR}"/toolchain.conf source "${INSTALLDIR}"/toolchain.env @@ -26,77 +42,100 @@ RAPIDJSON_CFLAGS="" cd "${BUILDDIR}" case "$with_rapidjson" in - __INSTALL__) - echo "==================== Installing RAPIDJSON ====================" - dirname="rapidjson-${rapidjson_ver}" - pkg_install_dir="${INSTALLDIR}/$dirname" - #pkg_install_dir="${HOME}/lib/rapidjson/${rapidjson_ver}" - install_lock_file="$pkg_install_dir/install_successful" - #url="https://github.com/Tencent/rapidjson/archive/refs/tags/v${rapidjson_ver}.tar.gz" # commented by Kai Luo in 2025/04/28 - url="https://codeload.github.com/Tencent/rapidjson/tar.gz/${rapidjson_ver}" - # changed by Kai Luo in 2025/04/28， modified by Zhaoqing Liu in 2025/05/01 - filename="rapidjson-${rapidjson_ver}.tar.gz" - if verify_checksums "${install_lock_file}"; then - echo "$dirname is already installed, skipping it." - else - if [ -f $filename ]; then - echo "$filename is found" + __INSTALL__) + echo "==================== Installing RapidJSON ====================" + dirname="rapidjson-${rapidjson_ver}" + pkg_install_dir="${INSTALLDIR}/$dirname" + #pkg_install_dir="${HOME}/lib/rapidjson/${rapidjson_ver}" + install_lock_file="$pkg_install_dir/install_successful" + # url construction rules: + # - Branch names (master, main, develop) without v prefix + # - Version tags (e.g., 1.0.0) with v prefix + if [[ "${rapidjson_ver}" =~ ^(master|main|develop)$ ]]; then + url="https://codeload.github.com/Tencent/rapidjson/tar.gz/${rapidjson_ver}" else - # download from github.com and checksum - echo "===> Notice: This version of rapidjson is downloaded in GitHub master repository <===" - download_pkg_from_url "${rapidjson_sha256}" "${filename}" "${url}" - # wget "${url}" -O "${filename}" --no-check-certificate # use wget directly instead of download_pkg_from_url + url="https://codeload.github.com/Tencent/rapidjson/tar.gz/v${rapidjson_ver}" fi - if [ "${PACK_RUN}" = "__TRUE__" ]; then - echo "--pack-run mode specified, skip installation" - else - echo "Installing from scratch into ${pkg_install_dir}" - [ -d $dirname ] && rm -rf $dirname - #unzip -q $filename # use -q to suppress output - tar -xzf $filename - mkdir -p "${pkg_install_dir}" - cp -r $dirname/* "${pkg_install_dir}/" - # for CMake to find rapidjson - #cp ${pkg_install_dir}/RapidJSONConfig.cmake.in ${pkg_install_dir}/RapidJSONConfig.cmake - cat << EOF > "${pkg_install_dir}/RapidJSONConfig.cmake" + filename="rapidjson-${rapidjson_ver}.tar.gz" + if verify_checksums "${install_lock_file}"; then + echo "$dirname is already installed, skipping it." + else + if [ -f $filename ]; then + echo "$filename is found" + else + # download from github.com and checksum + echo "===> Notice: This version of RapidJSON is downloaded in GitHub master repository <===" + download_pkg_from_url "${rapidjson_sha256}" "${filename}" "${url}" + fi + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip installation" + exit 0 + fi + echo "Installing from scratch into ${pkg_install_dir}" + [ -d $dirname ] && rm -rf $dirname + tar -xzf $filename + #unzip -q $filename + mkdir -p "${pkg_install_dir}" + cp -r $dirname/* "${pkg_install_dir}/" + # for rapidjson found in cmake + cat << EOF > "${pkg_install_dir}/RapidJSONConfig.cmake" get_filename_component(RAPIDJSON_CMAKE_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH) set(RAPIDJSON_INCLUDE_DIRS "@INCLUDE_INSTALL_DIR@") message(STATUS "RapidJSON found. Headers: ${RAPIDJSON_INCLUDE_DIRS}") EOF - write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" - fi - fi - RAPIDJSON_CFLAGS="-I'${pkg_install_dir}'" + write_checksums "${install_lock_file}" "${SCRIPT_DIR}/stage4/$(basename ${SCRIPT_NAME})" + fi + RAPIDJSON_CFLAGS="-I'${pkg_install_dir}/include'" ;; __SYSTEM__) - echo "==================== CANNOT Finding RAPIDJSON from system paths NOW ====================" - recommend_offline_installation $filename $url - # How to do it in rapidjson? -- Zhaoqing in 2023/08/23 - # check_lib -lxcf03 "libxc" - # check_lib -lxc "libxc" - # add_include_from_paths LIBXC_CFLAGS "xc.h" $INCLUDE_PATHS - # add_lib_from_paths LIBXC_LDFLAGS "libxc.*" $LIB_PATHS + echo "==================== Finding RapidJSON from system paths ====================" + if [ "${PACK_RUN}" = "__TRUE__" ]; then + echo "--pack-run mode specified, skip system check" + exit 0 + fi + # Find rapidjson header file and derive package root directory + rapidjson_header_path="$(find_in_paths "rapidjson/rapidjson.h" $INCLUDE_PATHS)" + if [ "$rapidjson_header_path" != "__FALSE__" ]; then + # Derive pkg_install_dir from found header path + # rapidjson/rapidjson.h -> remove /rapidjson/rapidjson.h -> get include dir -> get parent dir + rapidjson_include_dir="$(dirname "$(dirname "$rapidjson_header_path")")" + pkg_install_dir="$(dirname "$rapidjson_include_dir")" + echo "Found rapidjson at: $pkg_install_dir" + RAPIDJSON_CFLAGS="-I'${rapidjson_include_dir}'" + else + report_error "Cannot find rapidjson/rapidjson.h in system paths" + exit 1 + fi ;; __DONTUSE__) ;; - + *) - echo "==================== Linking RAPIDJSON to user paths ====================" - check_dir "${pkg_install_dir}" - RAPIDJSON_CFLAGS="-I'${pkg_install_dir}'" - ;; + echo "==================== Linking RapidJSON to user paths ====================" + pkg_install_dir="${with_rapidjson}" + check_dir "${pkg_install_dir}" + RAPIDJSON_CFLAGS="-I'${pkg_install_dir}/include'" + ;; esac if [ "$with_rapidjson" != "__DONTUSE__" ]; then if [ "$with_rapidjson" != "__SYSTEM__" ]; then - # LibRI deps should find rapidjson include in CPATH cat << EOF > "${BUILDDIR}/setup_rapidjson" prepend_path CPATH "$pkg_install_dir/include" -export CPATH="${pkg_install_dir}/include":\${CPATH} +prepend_path CMAKE_PREFIX_PATH "$pkg_install_dir" +export CPATH="$pkg_install_dir/include":\${CPATH} +export CMAKE_PREFIX_PATH="$pkg_install_dir":\${CMAKE_PREFIX_PATH} +export RAPIDJSON_ROOT="$pkg_install_dir" +EOF + else + cat << EOF > "${BUILDDIR}/setup_rapidjson" +export RAPIDJSON_ROOT="$pkg_install_dir" EOF - cat "${BUILDDIR}/setup_rapidjson" >> $SETUPFILE fi + cat "${BUILDDIR}/setup_rapidjson" >> $SETUPFILE cat << EOF >> "${BUILDDIR}/setup_rapidjson" export RAPIDJSON_CFLAGS="${RAPIDJSON_CFLAGS}" -export RAPIDJSON_ROOT="$pkg_install_dir" +export CP_DFLAGS="\${CP_DFLAGS} -D__RAPIDJSON" +export CP_CFLAGS="\${CP_CFLAGS} ${RAPIDJSON_CFLAGS}" +export RAPIDJSON_VERSION="${rapidjson_ver}" EOF fi diff --git a/toolchain/scripts/tool_kit.sh b/toolchain/scripts/tool_kit.sh index c7fa483ffa..15d97d3e2d 100755 --- a/toolchain/scripts/tool_kit.sh +++ b/toolchain/scripts/tool_kit.sh @@ -1,23 +1,52 @@ +# ABACUS Toolchain Utility Functions # A set of tools used in the toolchain installer, intended to be used +# by sourcing this file inside other scripts. +# Enhanced with modular architecture support +# Author: ABACUS Development Team +# Date: 2025-01-12 # TODO: Review and if possible fix shellcheck errors. # shellcheck disable=all # shellcheck shell=bash -# by sourcing this file inside other scripts. - SYS_INCLUDE_PATH=${SYS_INCLUDE_PATH:-"/usr/local/include:/usr/include"} SYS_LIB_PATH=${SYS_LIB_PATH:-"/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib"} INCLUDE_PATHS=${INCLUDE_PATHS:-"CPATH SYS_INCLUDE_PATH"} LIB_PATHS=${LIB_PATHS:-"LIBRARY_PATH LD_LIBRARY_PATH LD_RUN_PATH SYS_LIB_PATH"} time_start=$(date +%s) +# ============================================================================ +# Timing and Reporting Functions +# ============================================================================ + # report timing report_timing() { time_stop=$(date +%s) printf "Step %s took %0.2f seconds.\n" $1 $((time_stop - time_start)) } +# Enhanced timing function with better formatting +report_timing_enhanced() { + local step_name="$1" + local start_time="$2" + local end_time="${3:-$(date +%s)}" + local duration=$((end_time - start_time)) + + if [[ $duration -gt 3600 ]]; then + printf "Step '%s' completed in %d hours, %d minutes, %d seconds.\n" \ + "$step_name" $((duration / 3600)) $(((duration % 3600) / 60)) $((duration % 60)) + elif [[ $duration -gt 60 ]]; then + printf "Step '%s' completed in %d minutes, %d seconds.\n" \ + "$step_name" $((duration / 60)) $((duration % 60)) + else + printf "Step '%s' completed in %d seconds.\n" "$step_name" $duration + fi +} + +# ============================================================================ +# Error and Warning Reporting Functions +# ============================================================================ + # report a warning message with script name and line number report_warning() { if [ $# -gt 1 ]; then @@ -42,8 +71,46 @@ report_error() { echo "ERROR: (${SCRIPT_NAME}${__lineno}) $__message" >&2 } +# Enhanced error reporting with context +report_error_enhanced() { + local error_code="$1" + local error_message="$2" + local context="${3:-}" + local lineno="${4:-}" + + local location="" + if [[ -n "$lineno" ]]; then + location=", line $lineno" + fi + + echo "ERROR [$error_code]: (${SCRIPT_NAME}${location}) $error_message" >&2 + if [[ -n "$context" ]]; then + echo "Context: $context" >&2 + fi + + return $error_code +} + +# Enhanced warning reporting with severity levels +report_warning_enhanced() { + local severity="${1:-INFO}" # INFO, WARNING, CRITICAL + local message="$2" + local context="${3:-}" + local lineno="${4:-}" + + local location="" + if [[ -n "$lineno" ]]; then + location=", line $lineno" + fi + + echo "$severity: (${SCRIPT_NAME}${location}) $message" >&2 + if [[ -n "$context" ]]; then + echo "Context: $context" >&2 + fi +} + # recommend users to use offline installation when download failed -# zhaoqing in 2023.09.15 +# zhaoqing in 2025.10.15 recommend_offline_installation(){ __filename=$1 __url=$2 @@ -54,11 +121,13 @@ By download $__filename from $__url, Rename it as $__filename and put it into ${BUILDDIR}, And re-run toolchain installation script. +Also. the --pack-run option can help in OFFLINE installation. + You can manually install requirements packages via: +0. Download by 'wget $__url -O $__filename' manually 1. Download from www.cp2k.org/static/downloads (for OpenBLAS, OpenMPI and Others) -2. Download from github.com (for CEREAL, RapidJSON, libnpy, LibRI and others stage4 packages) -3. wget https://bohrium-api.dp.tech/ds-dl/abacus-deps-93wi-v2 -O abacus-deps.zip (a mirror in Bohrium) -4. for Intel-oneAPI and AMD AOCC/AOCL, please contact your server manager our visit Intel official website +2. Download from github.com (especially for CEREAL, RapidJSON, libnpy, LibRI and other stage4 packages) +3. for Intel-oneAPI and AMD AOCC/AOCL, please contact your server manager or visit their official website EOF } @@ -332,6 +401,10 @@ add_lib_from_paths() { fi } +# ============================================================================ +# Environment and System Validation Functions +# ============================================================================ + # check if environment variable is assigned and non-empty # https://serverfault.com/questions/7503/how-to-determine-if-a-bash-variable-is-empty require_env() { @@ -343,6 +416,51 @@ require_env() { fi } +# Enhanced environment variable validation +validate_env_vars() { + local required_vars=("$@") + local missing_vars=() + + for var in "${required_vars[@]}"; do + if [[ -z "${!var:-}" ]]; then + missing_vars+=("$var") + fi + done + + if [[ ${#missing_vars[@]} -gt 0 ]]; then + report_error_enhanced 1 "Missing required environment variables" "$(printf '%s ' "${missing_vars[@]}")" + return 1 + fi + + return 0 +} + +# Check system requirements +check_system_requirements() { + local min_disk_space_gb="${1:-10}" # Default 10GB + local required_commands=("${@:2}") + + # Check disk space + local available_space=$(df . | awk 'NR==2 {print int($4/1024/1024)}') + if [[ $available_space -lt $min_disk_space_gb ]]; then + report_warning_enhanced "WARNING" "Low disk space: ${available_space}GB available, ${min_disk_space_gb}GB recommended" + fi + + # Check required commands + for cmd in "${required_commands[@]}"; do + if ! command -v "$cmd" >/dev/null 2>&1; then + report_error_enhanced 1 "Required command not found: $cmd" + return 1 + fi + done + + return 0 +} + +# ============================================================================ +# String and Configuration Processing Functions +# ============================================================================ + resolve_string() { local __to_resolve=$1 shift @@ -351,6 +469,39 @@ resolve_string() { echo $("${SCRIPTDIR}/parse_if.py" $__flags <<< "${__to_resolve}") } +# Enhanced string processing with validation +resolve_string_safe() { + local __to_resolve="$1" + shift + local __flags=("$@") + + if [[ -z "$__to_resolve" ]]; then + report_error_enhanced 1 "Empty string provided for resolution" + return 1 + fi + + if [[ ! -f "${SCRIPTDIR}/parse_if.py" ]]; then + report_error_enhanced 1 "Parser script not found" "${SCRIPTDIR}/parse_if.py" + return 1 + fi + + local result + result=$("${SCRIPTDIR}/parse_if.py" "${__flags[@]}" <<< "${__to_resolve}" 2>/dev/null) + local exit_code=$? + + if [[ $exit_code -ne 0 ]]; then + report_error_enhanced $exit_code "String resolution failed" "$__to_resolve" + return $exit_code + fi + + echo "$result" + return 0 +} + +# ============================================================================ +# Command and Package Validation Functions +# ============================================================================ + # check if a command is available check_command() { local __command=${1} @@ -367,6 +518,51 @@ check_command() { fi } +# Enhanced command checking with version validation +check_command_version() { + local command_name="$1" + local min_version="$2" + local package_name="${3:-$command_name}" + + if ! command -v "$command_name" >/dev/null 2>&1; then + report_error_enhanced 1 "Command not found: $command_name" "Package: $package_name" + return 1 + fi + + local command_path + command_path=$(command -v "$command_name") + echo "Found $command_name at: $(realpath "$command_path")" + + if [[ -n "$min_version" ]]; then + # This is a placeholder for version checking logic + # Different commands have different version output formats + echo "Version check for $command_name (minimum: $min_version) - implement as needed" + fi + + return 0 +} + +# Batch command checking +check_commands_batch() { + local commands=("$@") + local failed_commands=() + + for cmd in "${commands[@]}"; do + if ! command -v "$cmd" >/dev/null 2>&1; then + failed_commands+=("$cmd") + else + echo "✓ Found: $cmd at $(command -v "$cmd")" + fi + done + + if [[ ${#failed_commands[@]} -gt 0 ]]; then + report_error_enhanced 1 "Missing commands" "$(printf '%s ' "${failed_commands[@]}")" + return 1 + fi + + return 0 +} + # check if directory exists # add more error msg by QuantumMisaka in 2025.03.19 check_dir() { @@ -576,6 +772,10 @@ append_path() { eval export $1 } +# ============================================================================ +# Configuration Parsing Functions +# ============================================================================ + # helper routine for reading --enable=* input options read_enable() { local __input_var="${1#*=}" @@ -623,6 +823,44 @@ read_with() { esac } +# Enhanced configuration parsing with validation +parse_config_option() { + local option="$1" + local default_value="${2:-}" + local valid_values="${3:-}" # Space-separated list of valid values + + local parsed_value + if [[ "$option" =~ ^--enable ]]; then + parsed_value=$(read_enable "$option") + elif [[ "$option" =~ ^--with ]]; then + parsed_value=$(read_with "$option" "$default_value") + else + parsed_value="$option" + fi + + # Validate against allowed values if provided + if [[ -n "$valid_values" && "$parsed_value" != "__INVALID__" ]]; then + local is_valid=false + for valid in $valid_values; do + if [[ "$parsed_value" == "$valid" ]]; then + is_valid=true + break + fi + done + + if [[ "$is_valid" == false ]]; then + report_warning_enhanced "WARNING" "Invalid configuration value: $parsed_value" "Valid options: $valid_values" + parsed_value="__INVALID__" + fi + fi + + echo "$parsed_value" +} + +# ============================================================================ +# File Download and Integrity Functions +# ============================================================================ + # helper routine to check integrity of downloaded files checksum() { local __filename=$1 @@ -641,9 +879,54 @@ checksum() { fi } +# Enhanced checksum verification with multiple hash algorithms +verify_file_integrity() { + local filename="$1" + local expected_hash="$2" + local hash_type="${3:-sha256}" # Default to SHA256 + + if [[ ! -f "$filename" ]]; then + report_error_enhanced 1 "File not found for integrity check" "$filename" + return 1 + fi + + local hash_command + case "$hash_type" in + sha256) + hash_command="sha256sum" + command -v "$hash_command" >/dev/null 2>&1 || hash_command="shasum -a 256" + ;; + sha1) + hash_command="sha1sum" + command -v "$hash_command" >/dev/null 2>&1 || hash_command="shasum -a 1" + ;; + md5) + hash_command="md5sum" + command -v "$hash_command" >/dev/null 2>&1 || hash_command="md5" + ;; + *) + report_error_enhanced 1 "Unsupported hash type" "$hash_type" + return 1 + ;; + esac + + local computed_hash + computed_hash=$($hash_command "$filename" | cut -d' ' -f1) + + if [[ "$computed_hash" == "$expected_hash" ]]; then + echo "✓ Integrity check passed for $filename ($hash_type)" + return 0 + else + report_error_enhanced 1 "Integrity check failed for $filename" "Expected: $expected_hash, Got: $computed_hash" + return 1 + fi +} + # downloader for the package tars, includes checksum -download_pkg_from_ABACUS_org() { - # usage: download_pkg_from_ABACUS_org sha256 filename +# backup and deprecated +download_pkg_from_org() { + # usage: download_pkg_from_org sha256 filename + echo "use cp2k mirror to download $__filename" local __sha256="$1" local __filename="$2" local __url="https://www.cp2k.org/static/downloads/$__filename" @@ -667,18 +950,50 @@ download_pkg_from_url() { local __sha256="$1" # if set to "--no-checksum", do not check checksum local __filename="$2" local __url="$3" - # download - #echo "wget ${DOWNLOADER_FLAGS} --quiet $__url -O $__filename" - #if ! wget ${DOWNLOADER_FLAGS} --quiet $__url -O $__filename; then - echo "wget ${DOWNLOADER_FLAGS} $__url -O $__filename --no-check-certificate" - if ! wget ${DOWNLOADER_FLAGS} $__url -O $__filename --no-check-certificate; then - report_error "failed to download $__url" - recommend_offline_installation $__filename $__url - if [ "${PACK_RUN}" != "__TRUE__" ]; then - return 1 - fi - fi - # checksum + + # Smart certificate validation strategy + case "${DOWNLOAD_CERT_POLICY:-smart}" in + "strict") + echo "Downloading with strict certificate validation: $__url" + if ! wget ${DOWNLOADER_FLAGS} "$__url" -O "$__filename"; then + report_error "failed to download $__url (strict certificate validation)" + recommend_offline_installation "$__filename" "$__url" + if [ "${PACK_RUN}" != "__TRUE__" ]; then + return 1 + fi + fi + ;; + "skip") + echo "Downloading with certificate validation disabled: $__url" + if ! wget ${DOWNLOADER_FLAGS} "$__url" -O "$__filename" --no-check-certificate; then + report_error "failed to download $__url" + recommend_offline_installation "$__filename" "$__url" + if [ "${PACK_RUN}" != "__TRUE__" ]; then + return 1 + fi + fi + ;; + "smart"|*) + # Smart fallback: try with certificate validation first, then without + echo "Attempting secure download: $__url" + if wget ${DOWNLOADER_FLAGS} "$__url" -O "$__filename" 2>/dev/null; then + echo "Download successful with certificate validation" + else + echo "Certificate validation failed, retrying without certificate check..." + if ! wget ${DOWNLOADER_FLAGS} "$__url" -O "$__filename" --no-check-certificate; then + report_error "failed to download $__url (both secure and insecure attempts failed)" + recommend_offline_installation "$__filename" "$__url" + if [ "${PACK_RUN}" != "__TRUE__" ]; then + return 1 + fi + else + echo "Download successful without certificate validation" + fi + fi + ;; + esac + + # checksum validation (unchanged) if [ "$__sha256" != "--no-checksum" ]; then checksum "$__filename" "$__sha256" fi diff --git a/toolchain/toolchain_aocc-aocl.sh b/toolchain/toolchain_aocc-aocl.sh index 575be14d5b..32a80fe177 100755 --- a/toolchain/toolchain_aocc-aocl.sh +++ b/toolchain/toolchain_aocc-aocl.sh @@ -5,34 +5,112 @@ #SBATCH -o compile.log #SBATCH -e compile.err -# JamesMisaka in 2025-05-05 -# install abacus dependency by aocc-aocl toolchain -# openmpi is recommended to use -# libtorch and libnpy are for deepks support, which can be =no -# if you want to run EXX calculation, you should set --with-libri=install -# gpu-lcao supporting modify: CUDA_PATH and --enable-cuda -# export CUDA_PATH=/usr/local/cuda - -./install_abacus_toolchain.sh \ ---with-amd=system \ ---math-mode=aocl \ ---with-intel=no \ ---with-gcc=no \ ---with-openmpi=install \ ---with-cmake=install \ ---with-scalapack=system \ ---with-libxc=install \ ---with-fftw=system \ ---with-elpa=install \ ---with-cereal=install \ ---with-rapidjson=install \ ---with-libtorch=no \ ---with-nep=no \ ---with-libnpy=no \ ---with-libri=no \ ---with-libcomm=no \ ---with-4th-openmpi=no \ ---with-flang=no \ -| tee compile.log -# to use openmpi-version4: set --with-4th-openmpi=yes -# flang is not recommended to use in this stage \ No newline at end of file +# Users can easily modify these parameters to customize the build +# Before running this script, ensure you have loaded your system packages + +# Compiler Configuration +TOOLCHAIN_COMPILER="aocc-aocl" +WITH_AMD="system" +WITH_INTEL="no" +WITH_GCC="no" + +# Math Libraries (AMD AOCL recommended) +MATH_MODE="aocl" +WITH_AOCL="system" + +# MPI Implementation (OpenMPI recommended) +MPI_MODE="openmpi" +WITH_OPENMPI="install" +WITH_MPICH="no" +WITH_4TH_OPENMPI="no" # Set to "yes" for OpenMPI v4, deprecated + +# Core Dependencies +WITH_CMAKE="install" +WITH_SCALAPACK="system" # AOCL provides ScaLAPACK +WITH_FFTW="system" # AOCL provides FFTW +WITH_LIBXC="install" +WITH_ELPA="install" + +# Utility Libraries +WITH_CEREAL="install" +WITH_RAPIDJSON="install" + +# Advanced Features (EXX calculations) +WITH_LIBRI="install" +WITH_LIBCOMM="install" + +# Optional Features (DeepKS support) +WITH_LIBTORCH="no" # Set to "install" for DeepKS support +WITH_LIBNPY="no" # Set to "install" for DeepKS support +WITH_NEP="no" + +# AMD Compiler Options +WITH_FLANG="no" # Flang is not recommended in current stage + +# ELPA-GPU Support (uncomment and modify as needed) +# ENABLE_CUDA="yes" +# GPU_VERSION="75" # Check your GPU compute capability +# export CUDA_PATH="/usr/local/cuda" + +# ============================================================================ +# Execution Mode Control +# ============================================================================ +# Dry-run mode: Show what would be done without actually executing +DRY_RUN_MODE="no" # Set to "yes" to enable dry-run mode + +# Pack-run mode: Only check and install required packages +PACK_RUN_MODE="no" # Set to "yes" to enable pack-run mode + +# ============================================================================ +# Package Version Selection (main/alt versions) +# ============================================================================ +# Choose between main (latest stable) and alt (alternative/legacy) versions +# Refer to scripts/package_versions.sh for specific version numbers + +CMAKE_VERSION="main" # main=3.31.7, alt=3.30.5 +OPENMPI_VERSION="main" # main=5.0.8, alt=4.1.6 +ELPA_VERSION="main" # main=2025.06.001, alt=2024.05.001 +LIBXC_VERSION="main" # main=7.0.0, alt=6.2.2 +LIBTORCH_VERSION="main" # main=2.1.2, alt=1.12.1 (use alt for older GLIBC) + +# Note: AOCC-AOCL toolchain uses AOCL for math libraries (FFTW, ScaLAPACK) +# so OpenBLAS and ScaLAPACK version selections are not applicable + +# ============================================================================ +# Execute Installation (DO NOT MODIFY BELOW THIS LINE) +# ============================================================================ + +# Call the main installation script with configured parameters +exec ./install_abacus_toolchain_new.sh \ + --with-amd="$WITH_AMD" \ + --with-gcc="$WITH_GCC" \ + --math-mode="$MATH_MODE" \ + --mpi-mode="$MPI_MODE" \ + --with-aocl="$WITH_AOCL" \ + --with-openmpi="$WITH_OPENMPI" \ + --with-mpich="$WITH_MPICH" \ + --with-cmake="$WITH_CMAKE" \ + --with-scalapack="$WITH_SCALAPACK" \ + --with-libxc="$WITH_LIBXC" \ + --with-fftw="$WITH_FFTW" \ + --with-elpa="$WITH_ELPA" \ + --with-cereal="$WITH_CEREAL" \ + --with-rapidjson="$WITH_RAPIDJSON" \ + --with-libtorch="$WITH_LIBTORCH" \ + --with-nep="$WITH_NEP" \ + --with-libnpy="$WITH_LIBNPY" \ + --with-libri="$WITH_LIBRI" \ + --with-libcomm="$WITH_LIBCOMM" \ + --with-flang="$WITH_FLANG" \ + --with-4th-openmpi="$WITH_4TH_OPENMPI" \ + --package-version cmake:"$CMAKE_VERSION" \ + --package-version openmpi:"$OPENMPI_VERSION" \ + --package-version elpa:"$ELPA_VERSION" \ + --package-version libxc:"$LIBXC_VERSION" \ + --package-version libtorch:"$LIBTORCH_VERSION" \ + ${DRY_RUN_MODE:+$([ "$DRY_RUN_MODE" = "yes" ] && echo "--dry-run")} \ + ${PACK_RUN_MODE:+$([ "$PACK_RUN_MODE" = "yes" ] && echo "--pack-run")} \ + ${ENABLE_CUDA:+--enable-cuda} \ + ${GPU_VERSION:+--gpu-ver="$GPU_VERSION"} \ + "$@" \ + | tee compile.log diff --git a/toolchain/toolchain_gcc-aocl.sh b/toolchain/toolchain_gcc-aocl.sh index 426be22c59..bd342a85ab 100755 --- a/toolchain/toolchain_gcc-aocl.sh +++ b/toolchain/toolchain_gcc-aocl.sh @@ -5,33 +5,108 @@ #SBATCH -o compile.log #SBATCH -e compile.err -# JamesMisaka in 2025-05-05 -# install abacus dependency by gcc-aocl toolchain -# openmpi is recommended to use -# libtorch and libnpy are for deepks support, which can be =no -# if you want to run EXX calculation, you should set --with-libri=install - -# gpu-lcao supporting modify: CUDA_PATH and --enable-cuda -# export CUDA_PATH=/usr/local/cuda - -./install_abacus_toolchain.sh \ ---with-gcc=system \ ---math-mode=aocl \ ---with-amd=no \ ---with-intel=no \ ---with-openmpi=install \ ---with-cmake=install \ ---with-scalapack=system \ ---with-libxc=install \ ---with-fftw=system \ ---with-elpa=install \ ---with-cereal=install \ ---with-rapidjson=install \ ---with-libtorch=no \ ---with-nep=no \ ---with-libnpy=no \ ---with-libri=no \ ---with-libcomm=no \ ---with-4th-openmpi=no \ -| tee compile.log -# to use openmpi-version4: set --with-4th-openmpi=yes \ No newline at end of file +# Users can easily modify these parameters to customize the build +# Before running this script, ensure you have loaded your system packages + +# Compiler Configuration +TOOLCHAIN_COMPILER="gcc-aocl" +WITH_GCC="system" +WITH_AMD="no" +WITH_INTEL="no" + +# Math Libraries (AMD AOCL recommended) +MATH_MODE="aocl" +WITH_AOCL="system" + +# MPI Implementation (OpenMPI recommended) +MPI_MODE="openmpi" +WITH_OPENMPI="install" +WITH_4TH_OPENMPI="no" # Set to "yes" for OpenMPI v4, deprecated +WITH_MPICH="no" + +# Core Dependencies +WITH_CMAKE="install" +WITH_SCALAPACK="system" # AOCL provides ScaLAPACK +WITH_FFTW="system" # AOCL provides FFTW +WITH_LIBXC="install" +WITH_ELPA="install" + +# Utility Libraries +WITH_CEREAL="install" +WITH_RAPIDJSON="install" + +# Advanced Features (EXX calculations) +WITH_LIBRI="install" +WITH_LIBCOMM="install" + +# Optional Features (MLALGO support) +WITH_LIBTORCH="no" +WITH_LIBNPY="no" +WITH_NEP="no" + +# ELPA-GPU Support (uncomment and modify as needed) +# ENABLE_CUDA="yes" +# GPU_VERSION="75" # Check your GPU compute capability +# export CUDA_PATH="/usr/local/cuda" + +# ============================================================================ +# Execution Mode Control +# ============================================================================ +# Dry-run mode: Show what would be done without actually executing +DRY_RUN_MODE="no" # Set to "yes" to enable dry-run mode + +# Pack-run mode: Only check and install required packages +PACK_RUN_MODE="no" # Set to "yes" to enable pack-run mode + +# ============================================================================ +# Package Version Selection (main/alt versions) +# ============================================================================ +# Choose between main (latest stable) and alt (alternative/legacy) versions +# Refer to scripts/package_versions.sh for specific version numbers + +CMAKE_VERSION="main" # main=3.31.7, alt=3.30.5 +OPENMPI_VERSION="main" # main=5.0.8, alt=4.1.6 +ELPA_VERSION="main" # main=2025.06.001, alt=2024.05.001 +LIBXC_VERSION="main" # main=7.0.0, alt=6.2.2 +LIBTORCH_VERSION="main" # main=2.1.2, alt=1.12.1 (use alt for older GLIBC) + +# Note: GCC-AOCL toolchain uses AOCL for math libraries (FFTW, ScaLAPACK) +# so OpenBLAS and ScaLAPACK version selections are not applicable + +# ============================================================================ +# Execute Installation (DO NOT MODIFY BELOW THIS LINE) +# ============================================================================ + +# Call the main installation script with configured parameters +exec ./install_abacus_toolchain_new.sh \ + --with-gcc="$WITH_GCC" \ + --with-amd="$WITH_AMD" \ + --math-mode="$MATH_MODE" \ + --mpi-mode="$MPI_MODE" \ + --with-aocl="$WITH_AOCL" \ + --with-openmpi="$WITH_OPENMPI" \ + --with-mpich="$WITH_MPICH" \ + --with-cmake="$WITH_CMAKE" \ + --with-scalapack="$WITH_SCALAPACK" \ + --with-libxc="$WITH_LIBXC" \ + --with-fftw="$WITH_FFTW" \ + --with-elpa="$WITH_ELPA" \ + --with-cereal="$WITH_CEREAL" \ + --with-rapidjson="$WITH_RAPIDJSON" \ + --with-libtorch="$WITH_LIBTORCH" \ + --with-nep="$WITH_NEP" \ + --with-libnpy="$WITH_LIBNPY" \ + --with-libri="$WITH_LIBRI" \ + --with-libcomm="$WITH_LIBCOMM" \ + --with-4th-openmpi="$WITH_4TH_OPENMPI" \ + --package-version cmake:"$CMAKE_VERSION" \ + --package-version openmpi:"$OPENMPI_VERSION" \ + --package-version elpa:"$ELPA_VERSION" \ + --package-version libxc:"$LIBXC_VERSION" \ + --package-version libtorch:"$LIBTORCH_VERSION" \ + ${DRY_RUN_MODE:+$([ "$DRY_RUN_MODE" = "yes" ] && echo "--dry-run")} \ + ${PACK_RUN_MODE:+$([ "$PACK_RUN_MODE" = "yes" ] && echo "--pack-run")} \ + ${ENABLE_CUDA:+--enable-cuda} \ + ${GPU_VERSION:+--gpu-ver="$GPU_VERSION"} \ + "$@" \ + | tee compile.log diff --git a/toolchain/toolchain_gnu.sh b/toolchain/toolchain_gnu.sh index 9d4d220c1f..df41de31fa 100755 --- a/toolchain/toolchain_gnu.sh +++ b/toolchain/toolchain_gnu.sh @@ -4,38 +4,112 @@ #SBATCH -n 16 #SBATCH -o compile.log #SBATCH -e compile.err +# Users can easily modify these parameters to customize the build -# JamesMisaka in 2025-05-05 -# install abacus dependency by gnu-toolchain -# one can use mpich or openmpi. openmpi will be faster and less compatiable -# libtorch and libnpy are for deepks support, which can be =no -# if you want to run EXX calculation, you should set --with-libri=install - -# gpu-lcao supporting modify: CUDA_PATH and --enable-cuda -# export CUDA_PATH=/usr/local/cuda - -./install_abacus_toolchain.sh \ ---with-gcc=system \ ---with-intel=no \ ---with-openblas=install \ ---with-openmpi=install \ ---with-cmake=install \ ---with-scalapack=install \ ---with-libxc=install \ ---with-fftw=install \ ---with-elpa=install \ ---with-cereal=install \ ---with-rapidjson=install \ ---with-libtorch=no \ ---with-nep=no \ ---with-libnpy=no \ ---with-libri=no \ ---with-libcomm=no \ ---with-4th-openmpi=no \ -| tee compile.log -# to use openmpi-version4: set --with-4th-openmpi=yes -# to enable gpu-lcao, add the following lines: -# --enable-cuda \ -# --gpu-ver=75 \ -# one should check your gpu compute capability number -# and use it in --gpu-ver +# Before running this script, ensure you have loaded your system packages + +# Compiler Configuration +TOOLCHAIN_COMPILER="gnu" +WITH_GCC="system" +WITH_INTEL="no" +WITH_AMD="no" + +# Math Libraries +MATH_MODE="openblas" +WITH_OPENBLAS="install" + +# MPI Implementation (OpenMPI recommended) +MPI_MODE="openmpi" +WITH_OPENMPI="install" +WITH_4TH_OPENMPI="no" # Set to "yes" for OpenMPI v4, deprecated +WITH_MPICH="no" + +# Core Dependencies +WITH_CMAKE="install" +WITH_SCALAPACK="install" +WITH_LIBXC="install" +WITH_FFTW="install" +WITH_ELPA="install" + +# Utility Libraries +WITH_CEREAL="install" +WITH_RAPIDJSON="install" + +# Advanced Features (EXX calculations) +WITH_LIBRI="install" +WITH_LIBCOMM="install" + +# Optional Features (MLALGO support) +WITH_LIBTORCH="no" +WITH_LIBNPY="no" +WITH_NEP="no" + +# ELPA-GPU Support (uncomment and modify as needed) +# ENABLE_CUDA="yes" +# GPU_VERSION="75" # Check your GPU compute capability +# export CUDA_PATH="/usr/local/cuda" + +# ============================================================================ +# Execution Mode Control +# ============================================================================ +# Dry-run mode: Show what would be done without actually executing +DRY_RUN_MODE="no" # Set to "yes" to enable dry-run mode + +# Pack-run mode: Only check and install required packages +PACK_RUN_MODE="no" # Set to "yes" to enable pack-run mode + +# ============================================================================ +# Package Version Selection (main/alt versions) +# ============================================================================ +# Choose between main (latest stable) and alt (alternative/legacy) versions +# Refer to scripts/package_versions.sh for specific version numbers + +CMAKE_VERSION="main" # main=3.31.7, alt=3.30.5 +OPENMPI_VERSION="main" # main=5.0.8, alt=4.1.6 +OPENBLAS_VERSION="main" # main=0.3.30, alt=0.3.27 +ELPA_VERSION="main" # main=2025.06.001, alt=2024.05.001 +LIBXC_VERSION="main" # main=7.0.0, alt=6.2.2 +SCALAPACK_VERSION="main" # main=2.2.2, alt=2.2.1 +# Optional Libraries +LIBTORCH_VERSION="main" # main=2.1.2, alt=1.12.1 (use alt for older GLIBC) + +# ============================================================================ +# Execute Installation (DO NOT MODIFY BELOW THIS LINE) +# ============================================================================ + +# Call the main installation script with configured parameters +exec ./install_abacus_toolchain_new.sh \ + --with-gcc="$WITH_GCC" \ + --with-intel="$WITH_INTEL" \ + --with-amd="$WITH_AMD" \ + --math-mode="$MATH_MODE" \ + --mpi-mode="$MPI_MODE" \ + --with-openblas="$WITH_OPENBLAS" \ + --with-openmpi="$WITH_OPENMPI" \ + --with-mpich="$WITH_MPICH" \ + --with-cmake="$WITH_CMAKE" \ + --with-scalapack="$WITH_SCALAPACK" \ + --with-libxc="$WITH_LIBXC" \ + --with-fftw="$WITH_FFTW" \ + --with-elpa="$WITH_ELPA" \ + --with-cereal="$WITH_CEREAL" \ + --with-rapidjson="$WITH_RAPIDJSON" \ + --with-libtorch="$WITH_LIBTORCH" \ + --with-nep="$WITH_NEP" \ + --with-libnpy="$WITH_LIBNPY" \ + --with-libri="$WITH_LIBRI" \ + --with-libcomm="$WITH_LIBCOMM" \ + --with-4th-openmpi="$WITH_4TH_OPENMPI" \ + --package-version cmake:"$CMAKE_VERSION" \ + --package-version openmpi:"$OPENMPI_VERSION" \ + --package-version openblas:"$OPENBLAS_VERSION" \ + --package-version elpa:"$ELPA_VERSION" \ + --package-version libxc:"$LIBXC_VERSION" \ + --package-version scalapack:"$SCALAPACK_VERSION" \ + --package-version libtorch:"$LIBTORCH_VERSION" \ + ${DRY_RUN_MODE:+$([ "$DRY_RUN_MODE" = "yes" ] && echo "--dry-run")} \ + ${PACK_RUN_MODE:+$([ "$PACK_RUN_MODE" = "yes" ] && echo "--pack-run")} \ + ${ENABLE_CUDA:+--enable-cuda} \ + ${GPU_VERSION:+--gpu-ver="$GPU_VERSION"} \ + "$@" \ + | tee compile.log diff --git a/toolchain/toolchain_intel.sh b/toolchain/toolchain_intel.sh index 45e72a2181..acd6f38ead 100755 --- a/toolchain/toolchain_intel.sh +++ b/toolchain/toolchain_intel.sh @@ -5,38 +5,129 @@ #SBATCH -o compile.log #SBATCH -e compile.err -# JamesMisaka in 2025-05-05 -# install abacus dependency by intel-toolchain -# use mkl and intelmpi -# but mpich and openmpi can also be tried -# libtorch and libnpy are for deepks support, which can be =no -# gpu-lcao supporting modify: CUDA_PATH and --enable-cuda -# export CUDA_PATH=/usr/local/cuda +# Users can easily modify these parameters to customize the build +# Before running this script, ensure you have loaded your system packages # module load mkl mpi compiler -./install_abacus_toolchain.sh \ ---with-intel=system \ ---math-mode=mkl \ ---with-gcc=no \ ---with-intelmpi=system \ ---with-cmake=install \ ---with-scalapack=no \ ---with-libxc=install \ ---with-fftw=no \ ---with-elpa=install \ ---with-cereal=install \ ---with-rapidjson=install \ ---with-libtorch=no \ ---with-nep=no \ ---with-libnpy=no \ ---with-libri=install \ ---with-libcomm=install \ ---with-intel-classic=no \ -| tee compile.log -# for using AMD-CPU or GPU-version: set --with-intel-classic=yes -# to enable gpu-lcao, add the following lines: -# --enable-cuda \ -# --gpu-ver=75 \ -# one should check your gpu compute capability number -# and use it in --gpu-ver +# Compiler Configuration +TOOLCHAIN_COMPILER="intel" +WITH_INTEL="system" +WITH_GCC="no" + +# Math Libraries (Intel MKL recommended) +MATH_MODE="mkl" +WITH_MKL="system" # Use system MKL when MATH_MODE is mkl + +# MPI Implementation (Intel MPI recommended, but OpenMPI/MPICH also supported) +MPI_MODE="intelmpi" +WITH_INTELMPI="system" +WITH_OPENMPI="no" +WITH_MPICH="no" + +# Core Dependencies +WITH_CMAKE="install" +WITH_SCALAPACK="no" # MKL provides ScaLAPACK +WITH_FFTW="no" # MKL provides FFTW interface +WITH_LIBXC="install" +WITH_ELPA="install" + +# Utility Libraries +WITH_CEREAL="install" +WITH_RAPIDJSON="install" + +# Advanced Features (EXX calculations) +WITH_LIBRI="install" +WITH_LIBCOMM="install" + +# Optional Features (MLALGO support) +WITH_LIBTORCH="no" +WITH_LIBNPY="no" +WITH_NEP="no" + +# ELPA-GPU Support (uncomment and modify as needed) +# ENABLE_CUDA="yes" +# GPU_VERSION="75" # Check your GPU compute capability +# export CUDA_PATH="/usr/local/cuda" + +# ============================================================================ +# Execution Mode Control +# ============================================================================ +# Dry-run mode: Show what would be done without actually executing +DRY_RUN_MODE="no" # Set to "yes" to enable dry-run mode + +# Pack-run mode: Only check and install required packages +PACK_RUN_MODE="no" # Set to "yes" to enable pack-run mode + +# ============================================================================ +# Intel Compiler and MPI Options +# ============================================================================ +# Intel Compiler Version Selection +WITH_INTEL_CLASSIC="no" # Set to "yes" to use classic Intel compilers (icc/icpc/ifort) + # Set to "no" to use new Intel compilers (icx/icpx/ifx) + # Classic compilers needed for AMD-CPU or GPU-version + +WITH_IFX="yes" # Set to "yes" to use new Fortran compiler ifx (default) + # Set to "no" to use traditional ifort + # Only applies when WITH_INTEL_CLASSIC="no" + +# Intel MPI Version Selection +INTELMPI_CLASSIC="no" # Set to "yes" to use classic Intel MPI wrappers (mpiicc/mpiicpc/mpiifort) + # Set to "no" to use new Intel MPI wrappers (mpiicx/mpiicpx/mpiifx) + # Should match WITH_INTEL_CLASSIC setting for consistency + # Classic wrappers recommended for older Intel OneAPI versions + + +# ============================================================================ +# Package Version Selection (main/alt versions) +# ============================================================================ +# Choose between main (latest stable) and alt (alternative/legacy) versions +# Refer to scripts/package_versions.sh for specific version numbers + +CMAKE_VERSION="main" # main=3.31.7, alt=3.30.5 +ELPA_VERSION="alt" # main=2025.06.001, alt=2024.05.001 for intel oneapi<2024.2 +LIBXC_VERSION="main" # main=7.0.0, alt=6.2.2 +LIBTORCH_VERSION="main" # main=2.1.2, alt=1.12.1 (use alt for older GLIBC) + +# Note: Intel toolchain uses MKL for math libraries (FFTW, ScaLAPACK) +# so OpenBLAS, FFTW, and ScaLAPACK version selections are not applicable + +# ============================================================================ +# Execute Installation (DO NOT MODIFY BELOW THIS LINE) +# ============================================================================ + +# Call the main installation script with configured parameters +exec ./install_abacus_toolchain_new.sh \ + --with-intel="$WITH_INTEL" \ + --with-gcc="$WITH_GCC" \ + --math-mode="$MATH_MODE" \ + --mpi-mode="$MPI_MODE" \ + --with-mkl="$WITH_MKL" \ + --with-openmpi="$WITH_OPENMPI" \ + --with-mpich="$WITH_MPICH" \ + --with-intelmpi="$WITH_INTELMPI" \ + --with-cmake="$WITH_CMAKE" \ + --with-scalapack="$WITH_SCALAPACK" \ + --with-libxc="$WITH_LIBXC" \ + --with-fftw="$WITH_FFTW" \ + --with-elpa="$WITH_ELPA" \ + --with-cereal="$WITH_CEREAL" \ + --with-rapidjson="$WITH_RAPIDJSON" \ + --with-libtorch="$WITH_LIBTORCH" \ + --with-nep="$WITH_NEP" \ + --with-libnpy="$WITH_LIBNPY" \ + --with-libri="$WITH_LIBRI" \ + --with-libcomm="$WITH_LIBCOMM" \ + --with-intel-classic="$WITH_INTEL_CLASSIC" \ + --with-ifx="$WITH_IFX" \ + --with-intel-mpi-classic="$INTELMPI_CLASSIC" \ + --package-version cmake:"$CMAKE_VERSION" \ + --package-version elpa:"$ELPA_VERSION" \ + --package-version libxc:"$LIBXC_VERSION" \ + --package-version libtorch:"$LIBTORCH_VERSION" \ + ${DRY_RUN_MODE:+$([ "$DRY_RUN_MODE" = "yes" ] && echo "--dry-run")} \ + ${PACK_RUN_MODE:+$([ "$PACK_RUN_MODE" = "yes" ] && echo "--pack-run")} \ + ${ENABLE_CUDA:+--enable-cuda} \ + ${GPU_VERSION:+--gpu-ver="$GPU_VERSION"} \ + "$@" \ + | tee compile.log