Skip to content

Commit 3863bb6

Browse files
added an update regarding GPU binding with kokkos
1 parent 8299e6e commit 3863bb6

File tree

1 file changed

+9
-19
lines changed

1 file changed

+9
-19
lines changed

docs/software/sciapps/lammps.md

Lines changed: 9 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,9 @@ The submission script is the following:
8181
#SBATCH --time=01:00:00 (1)
8282
#SBATCH --nodes=2
8383
#SBATCH --ntasks-per-node=4 (2)
84-
#SBATCH --gres=gpu:4
84+
#SBATCH --gpus-per-node=4
85+
#SBATCH --gpus-per-task=1
86+
#SBATCH --gpu-bind=per_task:1
8587
#SBATCH --account=<ACCOUNT> (3)
8688
#SBATCH --uenv=<LAMMPS_UENV>:/user-environment (4)
8789
#SBATCH --view=kokkos (5)
@@ -90,7 +92,7 @@ export MPICH_GPU_SUPPORT_ENABLED=1
9092

9193
ulimit -s unlimited
9294

93-
srun ./wrapper.sh lmp -in lj_kokkos.in -k on g 1 -sf kk -pk kokkos gpu/aware on
95+
srun lmp -in lj_kokkos.in -k on g 1 -sf kk -pk kokkos gpu/aware on
9496
```
9597

9698
1. Time format: `HH:MM:SS`.
@@ -99,23 +101,11 @@ srun ./wrapper.sh lmp -in lj_kokkos.in -k on g 1 -sf kk -pk kokkos gpu/aware on
99101
4. Change `<LAMMPS_UENV>` to the name (or path) of the LAMMPS uenv you want to use.
100102
5. Load the `kokkos` uenv view.
101103

102-
the `numactl` wrapper is the following:
104+
!!! Note
105+
Using `-k on g 1` specifies that we want 1 GPU per MPI-rank.
106+
This is contrary to what is mentioned in the official LAMMPS documentation, however this is required to achieve the propper configuration on Alps.
103107

104-
```bash title="wrapper.sh"
105-
#!/bin/bash
106-
107-
export LOCAL_RANK=$SLURM_LOCALID
108-
export GLOBAL_RANK=$SLURM_PROCID
109-
export GPUS=(0 1 2 3)
110-
export NUMA_NODE=$(echo "$LOCAL_RANK % 4" | bc)
111-
export CUDA_VISIBLE_DEVICES=${GPUS[$NUMA_NODE]}
112-
113-
export MPICH_GPU_SUPPORT_ENABLED=1
114-
115-
numactl --cpunodebind=$NUMA_NODE --membind=$NUMA_NODE "$@"
116-
```
117-
118-
With the above scripts, you can launch a LAMMPS + Kokkos calculation on 2 nodes, using 4 MPI ranks per node and 4 GPUs per node with:
108+
With the above script, you can launch a LAMMPS + Kokkos calculation on 2 nodes, using 4 MPI ranks and 1 GPU per MPI rank with:
119109

120110
```bash
121111
sbatch run_lammps_kokkos.sh
@@ -176,7 +166,7 @@ To start a job, two bash scripts are required: a [Slurm][ref-slurm] submission s
176166
#SBATCH --time=01:00:00 (1)
177167
#SBATCH --nodes=2 (2)
178168
#SBATCH --ntasks-per-node=32
179-
#SBATCH --gres=gpu:4
169+
#SBATCH --gpus-per-node=4
180170
#SBATCH --account=<ACCOUNT> (3)
181171
#SBATCH --uenv=<LAMMPS_UENV>:/user-environment (4)
182172
#SBATCH --view=gpu (5)

0 commit comments

Comments
 (0)