-
Notifications
You must be signed in to change notification settings - Fork 59
How to build Proteus using Spack
These are the steps required to build Proteus using the Spack build tool. Spack has many similarities to HashDist.
Spack works only on Linux and MacOS platforms. Windows is not supported.
Before installing Spack, make sure your system supports Environment Modules package. It can be easily installed.
Ensure your system has the minimum Spack prerequisites. Follow the instructions in the Spack documentation, specifically the installation page.
Use Spack's shell support:
# For bash/zsh users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh
# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh
Confirm that which spack finds the Spack Python script executable.
Note that the filesystem that contains your Spack repo must be flock enabled, otherwise Spack will throw file-lock errors and be usable on your system. After Spack installation, test if the command spack find throws a flock error. Before any package installations, a successful spack find will return the message: ==> 0 installed packages.
After executing certain Spack commands, Spack will eventually create and populate a $HOME/.spack directory in which Spack behavior can be modified by editing existing files there or adding new ones.
Each Spack installation holds one software repo. Multiple Spack installations are required if, for whatever reasons, you wish to have multiple separate software stacks. Yet, a single Spack installation can accommodate many compiler and software option variants. See the Variants subsection in the Spack basic usage page.
Follow the compiler configuration section to find and add system compilers for use in the Spack build process.
If you are not satisfied with the available system compilers, you can easily build your own. I would suggest building new compilers with a separate Spack installation or repository, e.g. named spack-system or spack-compilers. By doing this, you can then access these disk-space-hog compilers for other purposes besides building Proteus.
Old system compilers usually have old system binutils, which Spack does not like. So, it may be needed to use Spack to install recent versions of binutils and curl into your spack-system repo. For example, if the command spack -d fetch libelf hangs or fails, then your system curl is too old for Spack usage. Another example is: if spack install libelf is giving link errors, then your binutils is probably too old and needs to be updated.
Here is an example of building a custom compiler. My HPC development node had several compiler options, but I found that only gcc-4.4.7 was able to install Spack packages successfully. For instance, spack install libelf %gcc@4.4.7 worked when other system compilers failed for various reasons. However, Proteus needs a later version of gcc. After finding a Spack-compatible compiler, I now had an option to install a base spack-system of my preference, namely, a gcc-7.3.0 environment. Below are the commands I used to install gcc-7.3.0 into my spack-system repo along with compatible up-to-date system utilities.
# install spack
git clone https://github.com/spack/spack.git spack-system
cd spack-system
export SPACK_ROOT=/<path...>/spack-system
source $SPACK_ROOT/share/spack/setup-env.sh
which spack
# install compilers
spack compiler find
spack compilers
# Experiment with different system compilers to install libelf.
spack install libelf %gcc@4.4.7 # this worked
# if you are happy with a system compiler, skip the remaining steps
# install a custom compiler
spack install gcc@7.3.0 %gcc@4.4.7
spack load gcc@7.3.0 # this is the Spack version of the 'module load' command
spack compiler find
# install system utilities
# my system /usr/local/ld was not linking properly, so update binutils
spack install binutils %gcc@7.3.0
spack load binutils
# my system curl was not fetching some archives properly, so update curl
spack install curl %gcc@7.3.0
spack load curl
spack find # lists all packages successfully installed in this repo
Appending %gcc@7.3.0 (or your preferred compiler) to each spack install command is a pain. Spack allows one to set a default compiler. To do this, create the file $HOME/.spack/packages.yaml with the following contents:
packages:
all:
compiler: [gcc@7.3.0]
At this point, you should be ready to start installing Proteus via Spack.
The host node should have secure shell access (ssh, scp, etc). Spack and Proteus will need to be installed using tarballs and Spack Mirrors. Below are two options of how to repeat steps 1 and 2 above on a machine without internet access.
Option 1. The hard way: Creating tarballs from a machine with internet access
Download an archive of Spack from github, or work through step 1 above on a machine having internet access and zip up the $SPACK_ROOT directory into an archive. scp this archive to the host node.
After setting up a Spack installation on the machine having internet access, and work through step 2 above. Spack will cache each of the downloaded packages into the $SPACK_ROOT/var/spack/cache directory. Archive this cache directory, and scp it to the host node.
On the host node, unarchive the Spack tarball and set it up according to step 1. Unarchive the cache tarball into some directory, e.g., $SPACK_ROOT/thirdparty, and create the file $HOME/.spack/mirrors.yaml with the contents
mirrors:
local_filesystem: file:///work/user/spack-system/thirdparty/ # or whatever your path is
Now, work through step 2 on the host node, and Spack should locate the required packages in your local filesystem mirror directory.
Option 2. The easy way: Grab tarballs from a remote site
This approach has certain guarantees that your installed packages are consistent and vetted by Proteus maintainers. Get the archives:
$ wget http://192.237.213.149/spack_src/1.4/spack.tar.gz
$ wget http://192.237.213.149/spack_src/1.4/spack-system.tar.gz
scp these archives to the host node, and follow the last half of directions from Option 1 above.
Step 4a. Setup a new Spack repo
Follow Step 1 above, but name your Spack repo proteus-stack. The spack find command should return the message: ==> 0 installed packages.
Step 4b. Setup your preferred compilers
If you are using system compilers, something as simple as module load <your_preferred_system_compiler> should suffice. If you have installed custom compilers, then you should setup a script file setup-compilers.sh with the following contents:
details coming
Then, source setup-compilers.sh, and check that the compiler is properly set, e.g., gcc --version.
Step 4c. Install the Proteus stack -- the remote-tarball way
Inside the proteus-stack repo:
$ wget http://192.237.213.149/spack_src/1.4/spack-packages.tar.gz
$ wget http://192.237.213.149/spack_src/1.4/thirdparty-proteus-stack-sources.tar.gz
$ tar xfv spack-packages.tar.gz
$ tar xfv thirdparty-proteus-stack-sources.tar.gz
Create or modify the file $HOME/.spack/mirrors.yaml with the contents
mirrors:
local_filesystem: file:///work/user/proteus-stack/thirdparty/ # or whatever your path is
Then, install the packages required by Proteus:
$ spack spec proteus-stack # just to ensure things are working
$ spack install proteus-stack
# ... The build should take several hours.
$ spack find # to see which packages have been installed
Alternate Step 4c. Install the Proteus stack -- the fetch-latest-packages-from-the-internet way
Details coming soon.
Details coming soon.