Skip to content

Commit 2800d49

Browse files
oestebanoesteban
committed
enh: add a sbatch.slurm submission file for SLURM and update singularity
1 parent 18d21cc commit 2800d49

File tree

2 files changed

+76
-47
lines changed

2 files changed

+76
-47
lines changed

‎docs/_static/sbatch.slurm

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/bin/bash
2+
#
3+
#SBATCH -J fmriprep
4+
#SBATCH --time=48:00:00
5+
#SBATCH -n 1
6+
#SBATCH --cpus-per-task=16
7+
#SBATCH --mem-per-cpu=4G
8+
#SBATCH -p normal,mygroup # Queue names you can submit to
9+
# Outputs ----------------------------------
10+
#SBATCH -o log/%x-%A-%a.out
11+
#SBATCH -e log/%x-%A-%a.err
12+
#SBATCH --mail-user=%[email protected]
13+
#SBATCH --mail-type=ALL
14+
# ------------------------------------------
15+
16+
BIDS_DIR="$STUDY/data"
17+
DERIVS_DIR="derivatives/fmriprep-1.5.0"
18+
19+
# Prepare some writeable bind-mount points.
20+
TEMPLATEFLOW_HOST_HOME=$HOME/.cache/templateflow
21+
FMRIPREP_HOST_CACHE=$HOME/.cache/fmriprep
22+
mkdir -p ${TEMPLATEFLOW_HOST_HOME}
23+
mkdir -p ${FMRIPREP_HOST_CACHE}
24+
25+
# Prepare derivatives folder
26+
mkdir -p ${BIDS_DIR}/${DERIVS_DIR}
27+
28+
# This trick will help you reuse freesurfer results across pipelines and fMRIPrep versions
29+
mkdir -p ${BIDS_DIR}/derivatives/freesurfer-6.0.1
30+
if [ ! -d ${BIDS_DIR}/${DERIVS_DIR}/freesurfer ]; then
31+
ln -s ${BIDS_DIR}/derivatives/freesurfer-6.0.1 ${BIDS_DIR}/${DERIVS_DIR}/freesurfer
32+
fi
33+
34+
# Make sure FS_LICENSE is defined in the container.
35+
export SINGULARITYENV_FS_LICENSE=$HOME/.freesurfer.txt
36+
37+
# Designate a templateflow bind-mount point
38+
export SINGULARITYENV_TEMPLATEFLOW_HOME="/templateflow"
39+
SINGULARITY_CMD="singularity run --cleanenv -B $BIDS_DIR:/data -B ${TEMPLATEFLOW_HOST_HOME}:${SINGULARITYENV_TEMPLATEFLOW_HOME} -B $L_SCRATCH:/work $STUDY/images/poldracklab_fmriprep_1.5.0.simg"
40+
41+
# Parse the participants.tsv file and extract one subject ID from the line corresponding to this SLURM task.
42+
subject=$( sed -n -E "$((${SLURM_ARRAY_TASK_ID} + 1))s/sub-(\S*)\>.*/\1/gp" ${BIDS_DIR}/participants.tsv )
43+
44+
# Remove IsRunning files from FreeSurfer
45+
find ${BIDS_DIR}/derivatives/freesurfer-6.0.1/sub-$subject/ -name "*IsRunning*" -type f -delete
46+
47+
# Compose the command line
48+
cmd="${SINGULARITY_CMD} /data /data/${DERIVS_DIR} participant --participant-label $subject -w /work/ -vv --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --output-spaces MNI152NLin2009cAsym:res-2 anat fsnative fsaverage5 --use-aroma"
49+
50+
# Setup done, run the command
51+
echo Running task ${SLURM_ARRAY_TASK_ID}
52+
echo Commandline: $cmd
53+
eval $cmd
54+
exitcode=$?
55+
56+
# Output results to a table
57+
echo "sub-$subject ${SLURM_ARRAY_TASK_ID} $exitcode" \
58+
>> ${SLURM_JOB_NAME}.${SLURM_ARRAY_JOB_ID}.tsv
59+
echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode
60+
exit $exitcode

‎docs/singularity.rst

Lines changed: 16 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -315,53 +315,22 @@ The next step is checking the environment and access to *fMRIPrep* resources, us
315315
...
316316

317317

318+
318319
Running Singularity on a SLURM system
319320
-------------------------------------
320-
An example of ``sbatch`` script to run *fMRIPrep* on a SLURM system with Singularity
321-
available is given below: ::
322-
323-
#!/bin/bash
324-
#
325-
#SBATCH -J fmriprep
326-
#SBATCH --array=1-36 # Replace indices with the right number of subjects
327-
#SBATCH --time=48:00:00
328-
#SBATCH -n 1
329-
#SBATCH --cpus-per-task=16
330-
#SBATCH --mem-per-cpu=4G
331-
#SBATCH -p queues,you,can,submit # Partition names, separated by comma
332-
# Outputs ----------------------------------
333-
#SBATCH -o log/%x-%A-%a.out
334-
#SBATCH -e log/%x-%A-%a.err
335-
#SBATCH --mail-user=%[email protected]
336-
#SBATCH --mail-type=ALL
337-
# ------------------------------------------
338-
339-
BIDS_DIR="$PROJECT/data/ds000109"
340-
DERIVS_DIR="derivatives/fmriprep-1.5.0"
341-
342-
mkdir -p $HOME/.cache/templateflow
343-
mkdir -p ${BIDS_DIR}/${DERIVS_DIR}
344-
mkdir -p ${BIDS_DIR}/derivatives/freesurfer-6.0.1
345-
ln -s ${BIDS_DIR}/derivatives/freesurfer-6.0.1 ${BIDS_DIR}/${DERIVS_DIR}/freesurfer
346-
347-
348-
export SINGULARITYENV_FS_LICENSE=$HOME/.freesurfer.txt
349-
export SINGULARITYENV_TEMPLATEFLOW_HOME="/templateflow"
350-
SINGULARITY_CMD="singularity run --cleanenv -B $PROJECT:/project -B $HOME/.cache/templateflow:/templateflow -B $L_SCRATCH:/work $PROJECT/images/poldracklab_fmriprep_1.5.0-2019-09-10-6157fec3d0ea.simg"
351-
352-
353-
subject=$( sed -n -E "$((${SLURM_ARRAY_TASK_ID} + 1))s/sub-(\S*)\>.*/\1/gp" ${BIDS_DIR}/participants.tsv )
354-
cmd="${SINGULARITY_CMD} /project/data/ds000109 /project/data/ds000109/${DERIVS_DIR} participant --participant-label $subject -w /work/ -vv --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --output-spaces MNI152NLin2009cAsym:res-2 anat fsnative fsaverage5 --cifti-output --use-aroma"
355-
356-
# Setup done, run the command
357-
echo Running task ${SLURM_ARRAY_TASK_ID}
358-
echo Commandline: $cmd
359-
eval $cmd
360-
exitcode=$?
361-
362-
# Output results to a table
363-
echo "sub-$subject ${SLURM_ARRAY_TASK_ID} $exitcode" \
364-
>> ${SLURM_JOB_NAME}.${SLURM_ARRAY_JOB_ID}.tsv
365-
echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode
366-
exit $exitcode
321+
An example of ``sbatch`` script to run *fMRIPrep* on a SLURM system [#1]_ is given `below <singularity.html#sbatch-slurm>`__.
322+
The submission script will generate one task per subject using a *job array*.
323+
Submission is as easy as:
324+
::
325+
326+
$ export STUDY=/path/to/some/folder
327+
$ sbatch --array=1-$(( $( wc -l $STUDY/data/participants.tsv | cut -f1 -d' ' ) - 1 )) sbatch.slurm
328+
329+
330+
.. literalinclude:: _static/sbatch.slurm
331+
:language: bash
332+
:name: sbatch.slurm
333+
:caption: **sbatch.slurm**:
334+
367335

336+
.. [#1] assuming that *job arrays* and Singularity are available

0 commit comments

Comments
 (0)