@@ -81,7 +81,9 @@ The submission script is the following:
8181# SBATCH --time=01:00:00 (1)
8282# SBATCH --nodes=2
8383# SBATCH --ntasks-per-node=4 (2)
84- # SBATCH --gres=gpu:4
84+ # SBATCH --gpus-per-node=4
85+ # SBATCH --gpus-per-task=1
86+ # SBATCH --gpu-bind=per_task:1
8587# SBATCH --account=<ACCOUNT> (3)
8688# SBATCH --uenv=<LAMMPS_UENV>:/user-environment (4)
8789# SBATCH --view=kokkos (5)
@@ -90,7 +92,7 @@ export MPICH_GPU_SUPPORT_ENABLED=1
9092
9193ulimit -s unlimited
9294
93- srun ./wrapper.sh lmp -in lj_kokkos.in -k on g 1 -sf kk -pk kokkos gpu/aware on
95+ srun lmp -in lj_kokkos.in -k on g 1 -sf kk -pk kokkos gpu/aware on
9496```
9597
96981 . Time format: ` HH:MM:SS ` .
@@ -99,23 +101,11 @@ srun ./wrapper.sh lmp -in lj_kokkos.in -k on g 1 -sf kk -pk kokkos gpu/aware on
991014 . Change ` <LAMMPS_UENV> ` to the name (or path) of the LAMMPS uenv you want to use.
1001025 . Load the ` kokkos ` uenv view.
101103
102- the ` numactl ` wrapper is the following:
104+ !!! Note
105+ Using ` -k on g 1 ` specifies that we want 1 GPU per MPI-rank.
106+ This is contrary to what is mentioned in the official LAMMPS documentation, however this is required to achieve the propper configuration on Alps.
103107
104- ``` bash title="wrapper.sh"
105- #! /bin/bash
106-
107- export LOCAL_RANK=$SLURM_LOCALID
108- export GLOBAL_RANK=$SLURM_PROCID
109- export GPUS=(0 1 2 3)
110- export NUMA_NODE=$( echo " $LOCAL_RANK % 4" | bc)
111- export CUDA_VISIBLE_DEVICES=${GPUS[$NUMA_NODE]}
112-
113- export MPICH_GPU_SUPPORT_ENABLED=1
114-
115- numactl --cpunodebind=$NUMA_NODE --membind=$NUMA_NODE " $@ "
116- ```
117-
118- With the above scripts, you can launch a LAMMPS + Kokkos calculation on 2 nodes, using 4 MPI ranks per node and 4 GPUs per node with:
108+ With the above script, you can launch a LAMMPS + Kokkos calculation on 2 nodes, using 4 MPI ranks and 1 GPU per MPI rank with:
119109
120110``` bash
121111sbatch run_lammps_kokkos.sh
@@ -176,7 +166,7 @@ To start a job, two bash scripts are required: a [Slurm][ref-slurm] submission s
176166# SBATCH --time=01:00:00 (1)
177167# SBATCH --nodes=2 (2)
178168# SBATCH --ntasks-per-node=32
179- # SBATCH --gres=gpu: 4
169+ # SBATCH --gpus-per-node= 4
180170# SBATCH --account=<ACCOUNT> (3)
181171# SBATCH --uenv=<LAMMPS_UENV>:/user-environment (4)
182172# SBATCH --view=gpu (5)
0 commit comments