enh: minor fixes and added fmriprep.slurm

Author: oesteban

docs/apps/framework.md

@@ -81,6 +81,46 @@ For instance, *MRIQC* generates group-level reports with the following command-l
 $ mriqc /data/bids_root /data/bids_root/derivatives/ group
 ```
 
+## What are *BIDS Derivatives*?
+
+*NiPreps* generate *derivatives* of the original data, and they fulfill the BIDS specification for the results of Apps that are created for subsequent consumption by other BIDS-Apps.
+These derivatives must follow the BIDS Derivatives specification ([draft](https://bids-specification.readthedocs.io/en/derivatives/)).
+An example of BIDS Derivatives filesystem tree, generated with *fMRIPrep* 1.5:
+```
+derivatives/
+├── fmriprep/
+│ ├── dataset_description.json
+│ ├── logs
+│ ├── sub-01.html
+│ ├── sub-01/
+│ │ ├── anat/
+│ │ │ ├── sub-01_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_dseg.nii.gz
+│ │ │ ├── sub-01_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_space-MNI152_dseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_from-MNI152_to-T1w_mode-image_xfm.h5
+│ │ │ ├── sub-01_from-T1w_to-MNI152_mode-image_xfm.h5
+│ │ │ └── sub-01_from-orig_to-T1w_mode-image_xfm.txt
+│ │ ├── figures/
+│ │ └── func/
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_boldref.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-preproc_bold.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-confounds_regressors.nii.gz
+│ │ └── sub-01_task-rhymejudgment_space-MNI152_desc-brain_mask.nii.gz
+│ ├── sub-02.html
+│ ├── sub-02/
+│ ├── sub-03.html
+│ └── sub-03/
+```
+
 [bids]: https://bids.neuroimaging.io/
 [bidsapps_paper]: https://doi.org/10.1371/journal.pcbi.1005209
 [Singularity]: https://sylabs.io/singularity/

docs/apps/singularity.md

@@ -342,13 +342,17 @@ environment and access to *fMRIPrep* resources, using
 ## Running Singularity on a SLURM system
 
 An example of `sbatch` script to run *fMRIPrep* on a SLURM system[^1] is
-given [below](singularity.html#sbatch-slurm). The submission script will
-generate one task per subject using a *job array*. Submission is as easy
-as:
+given below. The submission script will
+generate one task per subject using a *job array*.
+```Shell
+{!assets/fmriprep.slurm!}
+```
+Submission is then as easy as:
 
 ```Shell
 $ export STUDY=/path/to/some/folder
-$ sbatch --array=1-$(( $( wc -l $STUDY/data/participants.tsv | cut -f1 -d' ' ) - 1 )) sbatch.slurm
+$ sbatch --array=1-$(( $( wc -l $STUDY/data/participants.tsv | cut -f1 -d' ' ) - 1 )) fmriprep.slurm
 ```
 
+
 [^1]: assuming that *job arrays* and Singularity are available

docs/assets/fmriprep.slurm

@@ -0,0 +1,55 @@
+#!/bin/bash
+#
+#SBATCH -J fmriprep
+#SBATCH --time=48:00:00
+#SBATCH -n 1
+#SBATCH --cpus-per-task=16
+#SBATCH --mem-per-cpu=4G
+#SBATCH -p normal,mygroup  # Queue names you can submit to
+# Outputs ----------------------------------
+#SBATCH -o log/%x-%A-%a.out
+#SBATCH -e log/%x-%A-%a.err
+#SBATCH --mail-user=%[email protected]
+#SBATCH --mail-type=ALL
+# ------------------------------------------
+
+BIDS_DIR="$STUDY/data"
+DERIVS_DIR="derivatives/fmriprep-1.5.0"
+LOCAL_FREESURFER_DIR="$STUDY/data/derivatives/freesurfer-6.0.1"
+
+# Prepare some writeable bind-mount points.
+TEMPLATEFLOW_HOST_HOME=$HOME/.cache/templateflow
+FMRIPREP_HOST_CACHE=$HOME/.cache/fmriprep
+mkdir -p ${TEMPLATEFLOW_HOST_HOME}
+mkdir -p ${FMRIPREP_HOST_CACHE}
+
+# Prepare derivatives folder
+mkdir -p ${BIDS_DIR}/${DERIVS_DIR}
+
+# Make sure FS_LICENSE is defined in the container.
+export SINGULARITYENV_FS_LICENSE=$HOME/.freesurfer.txt
+
+# Designate a templateflow bind-mount point
+export SINGULARITYENV_TEMPLATEFLOW_HOME="/templateflow"
+SINGULARITY_CMD="singularity run --cleanenv -B $BIDS_DIR:/data -B ${TEMPLATEFLOW_HOST_HOME}:${SINGULARITYENV_TEMPLATEFLOW_HOME} -B $L_SCRATCH:/work -B ${LOCAL_FREESURFER_DIR}:/fsdir $STUDY/images/poldracklab_fmriprep_1.5.0.simg"
+
+# Parse the participants.tsv file and extract one subject ID from the line corresponding to this SLURM task.
+subject=$( sed -n -E "$((${SLURM_ARRAY_TASK_ID} + 1))s/sub-(\S*)\>.*/\1/gp" ${BIDS_DIR}/participants.tsv )
+
+# Remove IsRunning files from FreeSurfer
+find ${LOCAL_FREESURFER_DIR}/sub-$subject/ -name "*IsRunning*" -type f -delete
+
+# Compose the command line
+cmd="${SINGULARITY_CMD} /data /data/${DERIVS_DIR} participant --participant-label $subject -w /work/ -vv --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --output-spaces MNI152NLin2009cAsym:res-2 anat fsnative fsaverage5 --use-aroma --fs-subjects-dir /fsdir"
+
+# Setup done, run the command
+echo Running task ${SLURM_ARRAY_TASK_ID}
+echo Commandline: $cmd
+eval $cmd
+exitcode=$?
+
+# Output results to a table
+echo "sub-$subject   ${SLURM_ARRAY_TASK_ID}    $exitcode" \
+      >> ${SLURM_JOB_NAME}.${SLURM_ARRAY_JOB_ID}.tsv
+echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode
+exit $exitcode

docs/devs/releases.md

@@ -1,5 +1,5 @@
 
-As of January 2020, *fMRIPrep* has adopted a Calendar Versioning scheme, and with it we are attempting to apply more coherent semantic rules to our releases.
+As of January 2020, *fMRIPrep* has adopted a [Calendar Versioning](https://calver.org) scheme, and with it we are attempting to apply more coherent semantic rules to our releases.
 
 !!! warning "Note"
 	This document is a draft for internal and external comment. Any commitments expressed here are proposals, and should not be relied upon at this time.
@@ -16,19 +16,19 @@ Patch releases are considered bug-fix releases. Each minor release triggers the
 
 These releases must satisfy four conditions:
 
-1. Resolving one or more bugs. These mostly include failures of *fMRIPrep* to complete or producing invalid derivatives (e.g., a NIfTI file of all zeroes).
-1. Derivatives compatibility. If a subject may be successfully run on 20.0.n, then the imaging derivatives should be identical if rerun with 20.0.(n+1), modulo rounding errors and the effects of nondeterministic algorithms. The changes between successful runs of 20.0.n and 20.0.(n+1) should not be larger than the changes between two successful runs of 20.0.n. Cosmetic changes to reports are acceptable, while differing fields of view or data types in a NIfTI file would not be.
-1. API compatibility. Workflow-generating functions, workflow input- and outputnode fields must not change. As an end-user application, this may seem overly strict, but the odds of introducing a bug are much higher in these cases.
-1. User interface compatibility. Substantial changes to *fMRIPrep* command line must not happen (e.g., the addition of a new, relevant flag).
+1. **Resolving one or more bugs.** These mostly include failures of *fMRIPrep* to complete or producing invalid derivatives (e.g., a NIfTI file of all zeroes).
+1. **Derivatives compatibility.** If a subject may be successfully run on 20.0.n, then the imaging derivatives should be identical if rerun with 20.0.(n+1), modulo rounding errors and the effects of nondeterministic algorithms. The changes between successful runs of 20.0.n and 20.0.(n+1) should not be larger than the changes between two successful runs of 20.0.n. Cosmetic changes to reports are acceptable, while differing fields of view or data types in a NIfTI file would not be.
+1. **API compatibility.** Workflow-generating functions, workflow input- and outputnode fields must not change. As an end-user application, this may seem overly strict, but the odds of introducing a bug are much higher in these cases.
+1. **User interface compatibility.** Substantial changes to *fMRIPrep* command line must not happen (e.g., the addition of a new, relevant flag).
 
 Note that not all bugs can be fixed in a way that satisfies all three of these criteria without significant effort. A developer may determine that the bug will be fixed in the next feature release.
 
 Additional acceptable changes within a minor release series:
 
-1. Improved tests. These often come along with bug fixes, but they can be free-standing improvements to the code base.
-1. Improved documentation. Unless the documentation is of a feature that will not be present in a bug-fix release, this is always welcome.
-1. Updates to the Dockerfile that improve operation for Docker and/or Singularity users, but do not risk behavior change. A good example is including more templates to reduce the need for network requests. An example of an update to the Dockerfile that forces a minor release increment is a change in the pinned version of any of the dependencies or the base container image.
-1. Improvements to the fmriprep-docker wrapper. As long as a command-line invocation that worked for the previous version continues to work and produce the same Docker command, there's little chance of harm.
+1. **Improved tests.** These often come along with bug fixes, but they can be free-standing improvements to the code base.
+1. **Improved documentation.** Unless the documentation is of a feature that will not be present in a bug-fix release, this is always welcome.
+1. **Updates to the Dockerfile** that improve operation for Docker and/or Singularity users, but do not risk behavior change. A good example is including more templates to reduce the need for network requests. An example of an update to the Dockerfile that forces a minor release increment is a change in the pinned version of any of the dependencies or the base container image.
+1. **Improvements to the *lightweight wrappers*.** As long as a command-line invocation that worked for the previous version continues to work and produce the same Docker command, there's little chance of harm.
 
 ## Mechanics
 

mkdocs.yml

@@ -35,6 +35,9 @@ markdown_extensions:
       emoji_index: !!python/name:materialx.emoji.twemoji
       emoji_generator: !!python/name:materialx.emoji.to_svg
   - pymdownx.superfences
+  - pymdownx.snippets
+  - markdown_include.include:
+      base_path: docs
 
 plugins:
   - toc-sidebar

requirements.txt

@@ -1,3 +1,4 @@
 mkdocs
 mkdocs-material
+markdown-include
 git+https://github.com/Mandy91/mkdocs-toc-sidebar-plugin.git@drop_python_2.7_support