-
Notifications
You must be signed in to change notification settings - Fork 41
Add documentation on how to run and build NAMD on Eiger #86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
a747132
0f19d9c
9738893
774bbba
665cc37
608cbda
d8b746b
a755f99
67f3c86
be973be
79cffbe
bd006db
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,10 @@ | |
| !!! note "Prefer the single-node build and exploit GPU-resident mode" | ||
| Unless you have good reasons to use the multi-node build, we recommend using the single-node build with the GPU-resident mode. | ||
|
|
||
| !!! warning "Eiger" | ||
|
|
||
| The multi-node version is the only version of NAMD available on [Eiger][ref-cluster-eiger] - single-node is not provided. | ||
|
|
||
| ## Single-node build | ||
|
|
||
| The single-node build provides the following views: | ||
|
|
@@ -37,7 +41,7 @@ | |
| #!/bin/bash | ||
| #SBATCH --job-name="namd-example" | ||
| #SBATCH --time=00:10:00 | ||
| #SBATCH --account=<ACCOUNT> | ||
| #SBATCH --account=<ACCOUNT> (6) | ||
| #SBATCH --nodes=1 (1) | ||
| #SBATCH --ntasks-per-node=1 (2) | ||
| #SBATCH --cpus-per-task=288 | ||
|
|
@@ -46,19 +50,17 @@ | |
| #SBATCH --view=namd-single-node (5) | ||
|
|
||
|
|
||
| srun namd3 +p 29 +pmeps 5 +setcpuaffinity +devices 0,1,2,3 <NAMD_CONFIG_FILE> | ||
| srun namd3 +p 29 +pmeps 5 +setcpuaffinity +devices 0,1,2,3 <NAMD_CONFIG_FILE> # (7)! | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
pmeps is not a recognized word. (unrecognized-spelling)
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
setcpuaffinity is not a recognized word. (unrecognized-spelling)
|
||
| ``` | ||
|
|
||
| 1. You can only use one node with the `single-node` build | ||
| 2. You can only use one task per node with the `single-node` build | ||
| 3. Make all GPUs visible to NAMD (by automatically setting `CUDA_VISIBLE_DEVICES=0,1,2,3`) | ||
| 4. Load the NAMD UENV (UENV name or path to the UENV) | ||
| 4. Load the NAMD UENV (UENV name or path to the UENV). Change `<NAMD_UENV>` to the name (or path) of the actual NAMD UENV you want to use | ||
| 5. Load the `namd-single-node` view | ||
|
|
||
| * Change `<ACCOUNT>` to your project account | ||
| * Change `<NAMD_UENV>` to the name (or path) of the actual NAMD UENV you want to use | ||
| * Change `<NAMD_CONFIG_FILE>` to the name (or path) of the NAMD configuration file for your simulation | ||
| * Make sure you set `+p`, `+pmeps`, and other NAMD options optimally for your calculation | ||
| 6. Change `<ACCOUNT>` to your project account | ||
| 7. Make sure you set `+p`, `+pmeps`, and other NAMD options optimally for your calculation. | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
pmeps is not a recognized word. (unrecognized-spelling)
|
||
| Change `<NAMD_CONFIG_FILE>` to the name (or path) of the NAMD configuration file for your simulation | ||
|
|
||
| ??? example "Scaling of STMV benchmark with GPU-resident mode from 1 to 4 GPUs" | ||
|
|
||
|
|
@@ -205,52 +207,137 @@ | |
| !!! note "GPU-resident mode" | ||
| The multi-node build based on [Charm++]'s MPI backend can't take advantage of the new GPU-resident mode. Unless you require the multi-node | ||
| build or you can prove it is faster for your use case, we recommend using the single-node build with the GPU-resident mode. | ||
|
|
||
|
|
||
| ### Running NAMD on Eiger | ||
|
|
||
| The following sbatch script shows how to run NAMD on Eiger: | ||
|
||
|
|
||
| ```bash | ||
| #!/bin/bash -l | ||
| #SBATCH --job-name=namd-test | ||
| #SBATCH --time=00:30:00 | ||
| #SBATCH --nodes=4 | ||
| #SBATCH --ntasks-per-core=1 | ||
| #SBATCH --ntasks-per-node=128 | ||
| #SBATCH --account=<ACCOUNT> (1) | ||
| #SBATCH --hint=nomultithread | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
nomultithread is not a recognized word. (unrecognized-spelling)
|
||
| #SBATCH --hint=exclusive | ||
| #SBATCH --constraint=mc | ||
| #SBATCH --uenv=namd/3.0:v1 (2) | ||
| #SBATCH --view=namd (3) | ||
|
|
||
| export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK | ||
| export OMP_PROC_BIND=spread | ||
| export OMP_PLACES=threads | ||
|
|
||
| srun --cpu-bind=cores namd3 +setcpuaffinity ++ppn 4 <NAMD_CONFIG_FILE> # (4)! | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
setcpuaffinity is not a recognized word. (unrecognized-spelling)
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
ppn is not a recognized word. (unrecognized-spelling)
|
||
| ``` | ||
|
|
||
| 1. Change `<ACCOUNT>` to your project account | ||
| 2. Load the NAMD UENV (UENV name or path to the UENV). Change `<NAMD_UENV>` to the name (or path) of the actual NAMD UENV you want to use | ||
| 3. Load the `namd` view | ||
|
||
| 4. Make sure you set `++ppn`, and other NAMD options optimally for your calculation. | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
ppn is not a recognized word. (unrecognized-spelling)
|
||
msimberg marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Change `<NAMD_CONFIG_FILE>` to the name (or path) of the NAMD configuration file for your simulation | ||
|
|
||
|
|
||
| ### Building NAMD from source with Charm++'s MPI backend | ||
|
|
||
| !!! warning "TCL Version" | ||
| According to the NAMD 3.0 release notes, TCL `8.6` is required. However, the source code for the `3.0` release still contains hard-coded | ||
| flags for TCL `8.5`. The UENV provides `[email protected]`, therefore you need to manually modify NAMD 3.0's `arch/Linux-ARM64.tcl` file as follows: | ||
| change `-ltcl8.5` to `-ltcl8.6` in the definition of the `TCLLIB` variable. | ||
| According to the NAMD 3.0 release notes, TCL `8.6` is required. | ||
| However, the source code for some (beta) releases still contains hard-coded flags for TCL `8.5`. | ||
| The UENV provides `[email protected]`, therefore you need to manually modify NAMD's `arch/Linux-<ARCH>.tcl` file: | ||
| change `-ltcl8.5` to `-ltcl8.6` in the definition of the `TCLLIB` variable, if needed. | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
TCLLIB is not a recognized word. (unrecognized-spelling)
|
||
|
||
|
|
||
| The [NAMD] `uenv` provides all the dependencies required to build [NAMD] from source. You can follow these steps to build [NAMD] from source: | ||
|
|
||
| ```bash | ||
| export DEV_VIEW_NAME="develop" | ||
| export PATH_TO_NAMD_SOURCE=<PATH_TO_NAMD_SOURCE> | ||
| === "gh200 build" | ||
|
|
||
| # Start uenv and load develop view | ||
| uenv start --view=${DEV_VIEW_NAME} <NAMD_UENV> | ||
| ```bash | ||
| export DEV_VIEW_NAME="develop" | ||
| export PATH_TO_NAMD_SOURCE=<PATH_TO_NAMD_SOURCE> # (1)! | ||
|
|
||
| # Set variable VIEW_PATH to the view | ||
| export DEV_VIEW_PATH=/user-environment/env/${DEV_VIEW_NAME} | ||
| # Start uenv and load develop view | ||
| uenv start --view=${DEV_VIEW_NAME} <NAMD_UENV> # (2)! | ||
|
|
||
| cd ${PATH_TO_NAMD_SOURCE} | ||
| ``` | ||
| # Set variable VIEW_PATH to the view | ||
| export DEV_VIEW_PATH=/user-environment/env/${DEV_VIEW_NAME} | ||
|
|
||
| !!! info "Action required" | ||
| Modify the `<PATH_TO_NAMD_SOURCE>/arch/Linux-ARM64.tcl` file now. | ||
| Change `-ltcl8.5` with `-ltcl8.6` in the definition of the `TCLLIB` variable. | ||
| cd ${PATH_TO_NAMD_SOURCE} | ||
| ``` | ||
|
|
||
| ```bash | ||
| # Build bundled Charm++ | ||
| tar -xvf charm-8.0.0.tar && cd charm-8.0.0 | ||
| env MPICXX=mpicxx ./build charm++ mpi-linux-arm8 smp --with-production -j 32 | ||
|
|
||
| # Configure NAMD build for GPU | ||
| cd .. | ||
| ./config Linux-ARM64-g++.cuda \ | ||
| --charm-arch mpi-linux-arm8-smp --charm-base $PWD/charm-8.0.0 \ | ||
| --with-tcl --tcl-prefix ${DEV_VIEW_PATH} \ | ||
| --with-fftw --with-fftw3 --fftw-prefix ${DEV_VIEW_PATH} \ | ||
| --cuda-gencode arch=compute_90,code=sm_90 --with-single-node-cuda --with-cuda --cuda-prefix ${DEV_VIEW_PATH} | ||
| cd Linux-ARM64-g++.cuda && make -j 32 | ||
|
|
||
| # The namd3 executable (GPU-accelerated) will be built in the Linux-ARM64-g++.cuda directory | ||
| ``` | ||
| 1. Substitute `<PATH_TO_NAMD_SOURCE>` with the actual path to the NAMD source code | ||
| 2. Substitute `<NAMD_UENV>` with the actual name (or path) of the NAMD UENV you want to use. | ||
|
|
||
|
|
||
| !!! info "Action required" | ||
| Modify the `${PATH_TO_NAMD_SOURCE}/arch/Linux-ARM64.tcl` file now. | ||
| Change `-ltcl8.5` with `-ltcl8.6` in the definition of the `TCLLIB` variable, if needed. | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
TCLLIB is not a recognized word. (unrecognized-spelling)
|
||
|
||
|
|
||
|
|
||
| Build [Charm++] bundled with NAMD: | ||
|
|
||
| ```bash | ||
| tar -xvf charm-8.0.0.tar && cd charm-8.0.0 | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
xvf is not a recognized word. (unrecognized-spelling)
|
||
| env MPICXX=mpicxx ./build charm++ mpi-linux-arm8 smp --with-production -j 32 | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
MPICXX is not a recognized word. (unrecognized-spelling)
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
mpicxx is not a recognized word. (unrecognized-spelling)
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
smp is not a recognized word. (unrecognized-spelling)
|
||
| ``` | ||
|
|
||
| Finally, you can configure and build NAMD (with GPU accereration): | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
accereration is not a recognized word. (unrecognized-spelling)
|
||
RMeli marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ```bash | ||
| cd .. | ||
| ./config Linux-ARM64-g++.cuda \ | ||
| --charm-arch mpi-linux-arm8-smp --charm-base $PWD/charm-8.0.0 \ | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
smp is not a recognized word. (unrecognized-spelling)
|
||
| --with-tcl --tcl-prefix ${DEV_VIEW_PATH} \ | ||
| --with-fftw --with-fftw3 --fftw-prefix ${DEV_VIEW_PATH} \ | ||
| --cuda-gencode arch=compute_90,code=sm_90 --with-single-node-cuda --with-cuda --cuda-prefix ${DEV_VIEW_PATH} | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
gencode is not a recognized word. (unrecognized-spelling)
|
||
| cd Linux-ARM64-g++.cuda && make -j 32 | ||
| ``` | ||
|
|
||
| The `namd3` executable (GPU-accelerated) will be built in the `Linux-ARM64-g++.cuda` directory. | ||
|
||
|
|
||
| === "zen2 build" | ||
|
|
||
| ```bash | ||
| export DEV_VIEW_NAME="develop" | ||
| export PATH_TO_NAMD_SOURCE=<PATH_TO_NAMD_SOURCE> # (1)! | ||
|
|
||
| # Start uenv and load develop view | ||
| uenv start --view=${DEV_VIEW_NAME} <NAMD_UENV> # (2)! | ||
|
|
||
| # Set variable VIEW_PATH to the view | ||
| export DEV_VIEW_PATH=/user-environment/env/${DEV_VIEW_NAME} | ||
|
|
||
| cd ${PATH_TO_NAMD_SOURCE} | ||
| ``` | ||
|
|
||
| 1. Substitute `<PATH_TO_NAMD_SOURCE>` with the actual path to the NAMD source code | ||
| 2. Substitute `<NAMD_UENV>` with the actual name (or path) of the NAMD UENV you want to use. | ||
|
|
||
|
|
||
| !!! info "Action required" | ||
| Modify the `${PATH_TO_NAMD_SOURCE}/arch/Linux-x86_64.tcl` file now. | ||
| Change `-ltcl8.5` with `-ltcl8.6` in the definition of the `TCLLIB` variable, if needed. | ||
|
|
||
| Build [Charm++] bundled with NAMD: | ||
|
|
||
| ```bash | ||
| tar -xvf charm-8.0.0.tar && cd charm-8.0.0 | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
xvf is not a recognized word. (unrecognized-spelling)
|
||
| env MPICXX=mpicxx ./build charm++ mpi-linux-x86_64 smp --with-production -j 32 | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
MPICXX is not a recognized word. (unrecognized-spelling)
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
mpicxx is not a recognized word. (unrecognized-spelling)
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
smp is not a recognized word. (unrecognized-spelling)
|
||
| ``` | ||
|
|
||
| Finally, you can configure and build NAMD: | ||
|
|
||
| ```bash | ||
| cd .. | ||
| ./config Linux-x86_64-g++ \ | ||
| --charm-arch mpi-linux-x86_64-smp --charm-base $PWD/charm-8.0.0 \ | ||
Check failureCode scanning / check-spelling Unrecognized Spelling Error documentation
smp is not a recognized word. (unrecognized-spelling)
|
||
| --with-tcl --tcl-prefix ${DEV_VIEW_PATH} \ | ||
| --with-fftw --with-fftw3 --fftw-prefix ${DEV_VIEW_PATH} | ||
| cd Linux-x86_64-g++ && make -j 32 | ||
| ``` | ||
|
|
||
| * Change `<PATH_TO_NAMD_SOURCE>` to the path where you have the NAMD source code | ||
| * Change `<NAMD_UENV>` to the name (or path) of the actual NAMD UENV you want to use | ||
| The `namd3` executable will be built in the `Linux-x86_64-g++` directory. | ||
|
|
||
| To run NAMD, make sure you load the same UENV and view you used to build NAMD, and set the following variable: | ||
|
|
||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.