Using MIT's Satori cluster

[glwagner]

This top-level post is a WIP.

Here's a description of MIT's Satori cluster: https://mit-satori.github.io/

Example snippet from a slurm script:

#!/bin/bash
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --cpus-per-task=16
#SBATCH --mem=100GB
#SBATCH --time 24:00:00

# Upload modules
module purge all
module add spack
module add cuda/11.4
module load openmpi/3.1.6-cuda-pmi-ucx-slurm-jhklron

# MPI specific exports
export OMPI_MCA_pml=^ucx
export OMPI_MCA_osc=^ucx
export OMPI_MCA_btl_openib_allow_ib=true

# Julia specific enviromental variables
export COMMON="/nobackup/users/ssilvest/perlmutter-test"
export JULIA_DEPOT_PATH=":${COMMON}/depot"
export JULIA_LOAD_PATH="${JULIA_LOAD_PATH}:$(pwd)/satori"
export JULIA_CUDA_MEMORY_POOL=none
export JULIA=julia

# Profile specific variable
export JULIA_NVTX_CALLBACKS=gc

# Number of threads in SLURM mode
export JULIA_NUM_THREADS=${SLURM_CPUS_PER_TASK:=1}

cat > launch.sh << EoF_s
#! /bin/sh
export CUDA_VISIBLE_DEVICES=0,1,2,3
exec \$*
EoF_s
chmod +x launch.sh

if $PROFILE; then
   NSYS="nsys profile --trace=nvtx,cuda,mpi --output=${COMMON}/report_N${SLURM_JOB_NUM_NODES}_R${RESOLUTION}_${PRECISION}"
fi

$NSYS srun --mpi=pmi2 ./launch.sh $JULIA --check-bounds=no --project experiments/run.jl ${RESOLUTION:=3}

[ali-ramadhan]

Is it worth mentioning that Satori is a PowerPC cluster and that

CUDA.jl v5.3 is the last version with support for PowerPC (removed in v5.4)

according to the CUDA.jl README?

I haven't run Julia or Oceananigans.jl on Satori in a while but I remember seeing some related warnings, although my code did run. So maybe it'll all be fine as long as long as Oceananigans.jl is compatible with CUDA.jl v5.3?

[glwagner]

Nice, thank you @masonrogers14 ! Do you have any code or slurm scripts to share?

[masonrogers14]

Instructions for getting Oceananigans working on a brand-new Satori account:

1. install julia. The juliaup installer didn't work for me, possibly because the server was just down, so I downloaded the v1.11.1 binary for linux on PowerPC and unpacked with $ tar -xzf path/to/julia/tarball
2. add the CUDA module with $ module add cuda/11.8.0
3. (optional) create a project julia environment with $ julia --project
4. in pkg mode, install Oceananigans.jl and CUDA.jl with ] add Oceananigans and ] add CUDA
5. if that does not suffice, you have to tell CUDA.jl about the Satori CUDA module. In the REPL, type

> using CUDA
> CUDA.set_runtime_version(v"11.8.0", local_toolkit=true)

and then restart the REPL. I think local_toolkit=true is telling CUDA.jl to use the loaded Satori CUDA driver.

[masonrogers14]

Batch script for submitting Oceananigans run:

#!/bin/bash
#SBATCH -J inertia
#SBATCH -o stdout
#SBATCH -e stderr
#SBATCH --gres=gpu:1
#SBATCH --nodes=1
#SBATCH --mem=1T
#SBATCH --time=12:00:00
#SBATCH --exclusive
#SBATCH [email protected]
#SBATCH --mail-type=END,FAIL

date
module add cuda/11.8.0
julia --project whatever_your_script_is_called.jl
date

[seamanticscience]

These steps worked for me using cuda/11.8.0 and then

> using CUDA
> CUDA.set_runtime_version(v"11.8.0", local_toolkit=true)

However, for runs using ClimaOcean.jl, it was easy to exceed the 4KiB parameter memory limit. @jagoosw on another discussion suggested that the limiting factor was the version of PTX (v7.8) that comes with cuda/11.8.0 and that the the parameter memory limit is more generous (32KiB) with newer versions >8.1.

Wasnt sure how to patch PTX separately from CUDA toolkit, so headed to the NVIDIA website.

nvidia-smi reports driver version 535.86.10 and CUDA version 12.2 is supported for Satori's Tesla V100s. Table 3 on NVIDIA's website lists cuda 12.2.2 as the highest version supported directly by the 535 series drivers, so I went with that one.

This is the official site for the latest CUDA releases, but I wanted a previous version and Satori's ppe64le architecture is not supported, so you have to go under "Resources" > "Archive of Previous CUDA Releases" > "CUDA Toolkit 12.2.2" > "Linux" > "ppe64le" > "RHEL" > "8". I chose installer type "runfile (local)", which reveals a wget command, and then a sh command to run the installer. I made a directory (~/Applications/cude-12.2.2) and did:

>> sh cuda_12.2.2_535.104.05_linux_ppc64le.run --installpath=~/Applications/cuda-12.2.2/ --toolkit

The toolkit option just lets you install the software, because I wasn't particularly keen on fiddling around with the driver directly! Actually, this starts a terminal-based GUI type thing, where you have to accept the EULA and can customize install options.

It was pretty smooth sailing, and once the install was complete, I repeated the set up instructions in Julia REPL:

> using CUDA
> CUDA.set_runtime_version(v"12.2.2", local_toolkit=true)

And everything appears to be working great!

[seamanticscience]

I'll put in a request that ORCD install this newer version when I get round to it.