Merge remote-tracking branch 'upstream/master' into docs/contributing-update

Author: oesteban

CHANGES.rst

@@ -1,3 +1,15 @@
+1.1.2 (July 6, 2018)
+====================
+
+This release incorporates Nipype improvements that should reduce the
+chance of hanging if tasks are killed for excessive resource consumption.
+
+Thanks to Elizabeth DuPre for documentation updates.
+
+* [DOC] Clarify how to reuse FreeSurfer derivatives (#1189)
+* [DOC] Improve command line option documentation (#1186, #1080)
+* [MAINT] Update core dependencies (#1179, #1180)
+
 1.1.1 (June 7, 2018)
 ====================
 

README.rst

@@ -90,6 +90,30 @@ Principles
    the process and decide which subjects should be kept for the group level
    analysis.
 
+
+Limitations and reasons not to use ``fmriprep``
+-----------------------------------------------
+
+1. Very narrow :abbr:`FoV (field-of-view)` images oftentimes do not contain
+   enough information for standard image registration methods to work correctly.
+   Also, problems may arise when extracting the brain from these data.
+   Supporting these particular images is already a future line of the development
+   road-map.
+2. ``fmriprep`` may also underperform for particular populations (e.g. infants) and
+   non-human brains, although appropriate templates can be provided to overcome the
+   issue.
+3. The "EPInorm" approach is currently not supported, although we plan to implement
+   this feature (see `#857 <https://github.com/poldracklab/fmriprep/issues/857>`_).
+4. If you really want unlimited flexibility (which is obviously a double-edged sword).
+5. If you want students to suffer through implementing each step for didactic purposes,
+   or to learn shell-scripting or Python along the way.
+6. If you are trying to reproduce some *in-house* lab pipeline.
+
+(Reasons 4-6 were kindly provided by S. Nastase in his
+`open review <http://academickarma.org/review/j7d5501n779n>`__
+of our `pre-print <https://www.biorxiv.org/content/early/2018/05/06/306951>`__).
+
+
 Acknowledgements
 ----------------
 

docs/_static/OHBM2018-poster.png

@@ -225,11 +225,26 @@ we recommend to include in your paper.
 Posters
 -------
 
-* Organization for Human Brain Mapping 2017 (`pdf <https://f1000research.com/posters/6-1129>`_)
+* Organization for Human Brain Mapping 2018
+  (`Abstract <https://ww5.aievolution.com/hbm1801/index.cfm?do=abs.viewAbs&abs=1321>`__;
+  `PDF <https://files.aievolution.com/hbm1801/abstracts/31779/2035_Markiewicz.pdf>`__)
+
+.. image:: _static/OHBM2018-poster_thumb.png
+   :target: _static/OHBM2018-poster.png
+
+* Organization for Human Brain Mapping 2017
+  (`Abstract <https://ww5.aievolution.com/hbm1701/index.cfm?do=abs.viewAbs&abs=4111>`__;
+  `PDF <https://f1000research.com/posters/6-1129>`__)
 
 .. image:: _static/OHBM2017-poster_thumb.png
    :target: _static/OHBM2017-poster.png
 
+Presentations
+-------------
+
+* Organization for Human Brain Mapping 2018
+  `Software Demonstration <https://effigies.github.io/fmriprep-demo/>`__.
+
 .. include:: license.rst
 
 Other relevant references

docs/_static/OHBM2018-poster_thumb.png

@@ -21,11 +21,12 @@ dependencies:
 - python-dateutil
 - pydot>=1.2.3
 - cython
+- nipype>=1.1.0
 
 - pip:
     - sphinx-argparse
     - doctest-ignore-unicode
     - svgutils
     - nitime
     - nilearn
-    - niworkflows>=0.4.1
+    - niworkflows>=0.4.2

docs/citing.rst

@@ -66,16 +66,35 @@ what is included in the latest Docker images.
 Singularity Container
 =====================
 
-For security reasons, many HPCs (e.g., TACC) do not allow Docker containers, but do allow `Singularity <https://github.com/singularityware/singularity>`_ containers.
+For security reasons, many HPCs (e.g., TACC) do not allow Docker containers, but do
+allow `Singularity <https://github.com/singularityware/singularity>`_ containers.
+
+Preparing a Singularity image (Singularity version >= 2.5)
+----------------------------------------------------------
+If the version of Singularity on your HPC is modern enough you can create Singularity
+image directly on the HCP.
+This is as simple as: ::
+
+    $ singularity build /my_images/fmriprep-<version>.simg docker://poldracklab/mriqc:<version>
+    
+Where ``<version>`` should be replaced with the desired version of fMRIPrep that you want to download.
+
+
+Preparing a Singularity image (Singularity version < 2.5)
+---------------------------------------------------------
 In this case, start with a machine (e.g., your personal computer) with Docker installed.
-Use `docker2singularity <https://github.com/singularityware/docker2singularity>`_ to create a singularity image. You will need an active internet connection and some time. ::
+Use `docker2singularity <https://github.com/singularityware/docker2singularity>`_ to 
+create a singularity image.
+You will need an active internet connection and some time. ::
 
     $ docker run --privileged -t --rm \
         -v /var/run/docker.sock:/var/run/docker.sock \
         -v D:\host\path\where\to\output\singularity\image:/output \
         singularityware/docker2singularity \
-        poldracklab/fmriprep:latest
+        poldracklab/fmriprep:<version>
 
+Where ``<version>`` should be replaced with the desired version of fMRIPrep that you want 
+to download.
 
 Beware of the back slashes, expected for Windows systems.
 For \*nix users the command translates as follows: ::
@@ -84,46 +103,69 @@ For \*nix users the command translates as follows: ::
         -v /var/run/docker.sock:/var/run/docker.sock \
         -v /absolute/path/to/output/folder:/output \
         singularityware/docker2singularity \
-        poldracklab/fmriprep:latest
+        poldracklab/fmriprep:<version>
 
 
 Transfer the resulting Singularity image to the HPC, for example, using ``scp``. ::
 
-    $ scp poldracklab_fmriprep_latest-*.img [email protected]:/path/to/downloads
+    $ scp poldracklab_fmriprep*.img [email protected]:/my_images
+
+Running a Singularity Image
+---------------------------
 
 If the data to be preprocessed is also on the HPC, you are ready to run fmriprep. ::
 
-    $ singularity run path/to/singularity/image.img \
+    $ singularity run --clearenv /my_images/fmriprep-1.1.2.simg \
         path/to/data/dir path/to/output/dir \
         participant \
         --participant-label label
 
-For example: ::
+.. note::
 
-    $ singularity run ~/poldracklab_fmriprep_latest-2016-12-04-5b74ad9a4c4d.img \
+   Singularity by default `exposes all environment variables from the host inside 
+   the container <https://github.com/singularityware/singularity/issues/445>`_.
+   Because of this your host libraries (such as nipype) could be accidentally used 
+   instead of the ones inside the container - if they are included in ``PYTHONPATH``.
+   To avoid such situation we recommend using the ``--clearenv`` singularity flag 
+   in production use. For example: ::
+
+      $ singularity run --clearenv ~/poldracklab_fmriprep_latest-2016-12-04-5b74ad9a4c4d.img \
         /work/04168/asdf/lonestar/ $WORK/lonestar/output \
         participant \
         --participant-label 387 --nthreads 16 -w $WORK/lonestar/work \
         --omp-nthreads 16
 
-.. note::
 
-   Singularity by default `exposes all environment variables from the host inside the container <https://github.com/singularityware/singularity/issues/445>`_.
-   Because of this your host libraries (such as nipype) could be accidentally used instead of the ones inside the container - if they are included in PYTHONPATH.
-   To avoid such situation we recommend unsetting PYTHONPATH in production use. For example: ::
+    or, unset the ``PYTHONPATH`` variable before running: ::
 
-      $ PYTHONPATH="" singularity run ~/poldracklab_fmriprep_latest-2016-12-04-5b74ad9a4c4d.img \
+      $ unset PYTHONPATH; singularity run ~/poldracklab_fmriprep_latest-2016-12-04-5b74ad9a4c4d.img \
         /work/04168/asdf/lonestar/ $WORK/lonestar/output \
         participant \
         --participant-label 387 --nthreads 16 -w $WORK/lonestar/work \
         --omp-nthreads 16
 
+
+.. note::
+
+   Depending on how Singularity is configured on your cluster it might or might not 
+   automatically bind (mount or expose) host folders to the container. 
+   If this is not done automatically you will need to bind the necessary folders using 
+   the ``-B <host_folder>:<container_folder>`` Singularity argument.
+   For example: ::
+
+      $ singularity run --clearenv -B /work:/work ~/poldracklab_fmriprep_latest-2016-12-04-5b74ad9a4c4d.simg \
+        /work/my_dataset/ /work/my_dataset/derivatives/fmriprep \
+        participant \
+        --participant-label 387 --nthreads 16 \
+        --omp-nthreads 16
+
 Manually Prepared Environment
 =============================
 
-.. note::
+.. warning::
 
-   This method is not recommended! Make sure you would rather do this than use a `Docker Container`_ or a `Singularity Container`_.
+   This method is not recommended! Make sure you would rather do this than 
+   use a `Docker Container`_ or a `Singularity Container`_.
 
 Make sure all of fmriprep's `External Dependencies`_ are installed.
 These tools must be installed and their binaries available in the
@@ -149,9 +191,11 @@ FMRIPREP uses FreeSurfer tools, which require a license to run.
 To obtain a FreeSurfer license, simply register for free at
 https://surfer.nmr.mgh.harvard.edu/registration.html.
 
-When using manually-prepared environments, FreeSurfer will search for a license key
-file first using the ``$FS_LICENSE`` environment variable and then in the default
-path to the license key file (``$FREESURFER_HOME/license.txt``).
+When using manually-prepared environments or singularity, FreeSurfer will search 
+for a license key file first using the ``$FS_LICENSE`` environment variable and then 
+in the default path to the license key file (``$FREESURFER_HOME/license.txt``). 
+If using the ``--clearenv`` flag and ``$FS_LICENSE`` is set, use ``--fs-license-file $FS_LICENSE`` 
+to pass the license file location to fMRIPrep.
 
 It is possible to run the docker container pointing the image to a local path
 where a valid license file is stored.
@@ -193,7 +237,7 @@ would be equivalent to the latest example: ::
 External Dependencies
 =====================
 
-``fmriprep`` is implemented using nipype_, but it requires some other neuroimaging
+FMRIPrep is implemented using nipype_, but it requires some other neuroimaging
 software tools:
 
 - FSL_ (version 5.0.9)

docs/environment.yml

@@ -75,3 +75,20 @@ http://neurostars.org/tags/fmriprep/
 To participate in the ``fmriprep`` development-related discussions please use the
 following mailing list: http://mail.python.org/mailman/listinfo/neuroimaging
 Please add *[fmriprep]* to the subject line when posting on the mailing list.
+
+
+Not running on a local machine? - Data transfer
+===============================================
+
+If you intend to run ``fmriprep`` on a remote system, you will need to
+make your data available within that system first.
+
+For instance, here at the Poldrack Lab we use Stanford's
+:abbr:`HPC (high-performance computing)` system, called Sherlock.
+Sherlock enables `the following data transfer options 
+<https://www.sherlock.stanford.edu/docs/user-guide/storage/data-transfer/>`_.
+
+Alternatively, more comprehensive solutions such as `Datalad
+<http://www.datalad.org/>`_ will handle data transfers with the appropriate
+settings and commands.
+Datalad also performs version control over your data.

docs/installation.rst

@@ -106,16 +106,16 @@ in a multiscale, mutual-information based, nonlinear registration scheme.
 In particular, spatial normalization is done using the `ICBM 2009c Nonlinear
 Asymmetric template (1×1×1mm) <http://nist.mni.mcgill.ca/?p=904>`_ [Fonov2011]_.
 
-When processing images from patients with focal brain lesions (e.g. stroke, tumor 
-resection), it is possible to provide a lesion mask to be used during spatial 
+When processing images from patients with focal brain lesions (e.g. stroke, tumor
+resection), it is possible to provide a lesion mask to be used during spatial
 normalization to MNI-space [Brett2001]_.
 ANTs will use this mask to minimize warping of healthy tissue into damaged
-areas (or vice-versa). 
-Lesion masks should be binary NIfTI images (damaged areas = 1, everywhere else = 0) 
+areas (or vice-versa).
+Lesion masks should be binary NIfTI images (damaged areas = 1, everywhere else = 0)
 in the same space and resolution as the T1 image, and follow the naming convention specified in
 `BIDS Extension Proposal 3: Common Derivatives <https://docs.google.com/document/d/1Wwc4A6Mow4ZPPszDIWfCUCRNstn7d_zzaWPcfcHmgI4/edit#heading=h.9146wuepclkt>`_
-(e.g. ``sub-001_T1w_label-lesion_roi.nii.gz``). 
-This file should be placed in the ``sub-*/anat`` directory of the BIDS dataset 
+(e.g. ``sub-001_T1w_label-lesion_roi.nii.gz``).
+This file should be placed in the ``sub-*/anat`` directory of the BIDS dataset
 to be run through ``fmriprep``.
 
 .. figure:: _static/T1MNINormalization.svg
@@ -165,6 +165,15 @@ structural images.
 If enabled, several steps in the ``fmriprep`` pipeline are added or replaced.
 All surface preprocessing may be disabled with the ``--fs-no-reconall`` flag.
 
+.. note::
+    Surface processing will be skipped if the outputs already exist.
+
+    In order to bypass reconstruction in ``fmriprep``, place existing reconstructed
+    subjects in ``<output dir>/freesurfer`` prior to the run.
+    ``fmriprep`` will perform any missing ``recon-all`` steps, but will not perform
+    any steps whose outputs already exist.
+
+
 If FreeSurfer reconstruction is performed, the reconstructed subject is placed in
 ``<output dir>/freesurfer/sub-<subject_label>/`` (see :ref:`fsderivs`).
 
@@ -201,11 +210,6 @@ If T1w voxel sizes are less than 1mm in all dimensions (rounding to nearest
 .1mm), `submillimeter reconstruction`_ is used, unless disabled with
 ``--no-submm-recon``.
 
-In order to bypass reconstruction in ``fmriprep``, place existing reconstructed
-subjects in ``<output dir>/freesurfer`` prior to the run.
-``fmriprep`` will perform any missing ``recon-all`` steps, but will not perform
-any steps whose outputs already exist.
-
 ``lh.midthickness`` and ``rh.midthickness`` surfaces are created in the subject
 ``surf/`` directory, corresponding to the surface half-way between the gray/white
 boundary and the pial surface.
@@ -537,12 +541,23 @@ Calculated confounds include the mean global signal, mean tissue class signal,
 tCompCor, aCompCor, Frame-wise Displacement, 6 motion parameters, DVARS, and, if
 the ``--use-aroma`` flag is enabled, the noise components identified by ICA-AROMA
 (those to be removed by the "aggressive" denoising strategy).
+Particular details about ICA-AROMA are given below.
+
+
+ICA-AROMA
+~~~~~~~~~
+:mod:`fmriprep.workflows.bold.confounds.init_ica_aroma_wf`
+
+When one of the `--output-spaces` selected is in MNI space, ICA-AROMA denoising
+can be automatically appended to the workflow.
 The number of ICA-AROMA components depends on a dimensionality estimate
 made by MELODIC.
 For datasets with a very short TR and a large number of timepoints, this may
 result in an unusually high number of components.
 In such cases, it may be useful to specify the number of components to be
 extracted with ``--aroma-melodic-dimensionality``.
+Further details on the implementation are given within the workflow generation
+function (:mod:`fmriprep.workflows.bold.confounds.init_ica_aroma_wf`).
 
 *Note*: *non*-aggressive AROMA denoising is a fundamentally different procedure
 from its "aggressive" counterpart and cannot be performed only by using a set of noise

docs/usage.rst

@@ -57,8 +57,8 @@ def get_parser():
 
     g_bids = parser.add_argument_group('Options for filtering BIDS queries')
     g_bids.add_argument('--participant_label', '--participant-label', action='store', nargs='+',
-                        help='one or more participant identifiers (the sub- prefix can be '
-                             'removed)')
+                        help='a space delimited list of participant identifiers or a single '
+                             'identifier (the sub- prefix can be removed)')
     # Re-enable when option is actually implemented
     # g_bids.add_argument('-s', '--session-id', action='store', default='single_session',
     #                     help='select a specific session to be processed')
@@ -97,7 +97,7 @@ def get_parser():
         '--ignore', required=False, action='store', nargs="+", default=[],
         choices=['fieldmaps', 'slicetiming'],
         help='ignore selected aspects of the input dataset to disable corresponding '
-             'parts of the workflow')
+             'parts of the workflow (a space delimited list)')
     g_conf.add_argument(
         '--longitudinal', action='store_true',
         help='treat dataset as longitudinal - may increase runtime')
@@ -119,7 +119,9 @@ def get_parser():
              ' - T1w: subject anatomical volume\n'
              ' - template: normalization target specified by --template\n'
              ' - fsnative: individual subject surface\n'
-             ' - fsaverage*: FreeSurfer average meshes'
+             ' - fsaverage*: FreeSurfer average meshes\n'
+             'this argument can be single value or a space delimited list,\n'
+             'for example: --output-space T1w fsnative'
     )
     g_conf.add_argument(
         '--force-bbr', action='store_true', dest='use_bbr', default=None,
@@ -255,9 +257,9 @@ def main():
     log_level = int(max(25 - 5 * opts.verbose_count, logging.DEBUG))
     # Set logging
     logger.setLevel(log_level)
-    nlogging.getLogger('workflow').setLevel(log_level)
-    nlogging.getLogger('interface').setLevel(log_level)
-    nlogging.getLogger('utils').setLevel(log_level)
+    nlogging.getLogger('nipype.workflow').setLevel(log_level)
+    nlogging.getLogger('nipype.interface').setLevel(log_level)
+    nlogging.getLogger('nipype.utils').setLevel(log_level)
 
     errno = 0
 
@@ -341,7 +343,7 @@ def build_workflow(opts, retval):
     from ..utils.bids import collect_participants
     from ..viz.reports import generate_reports
 
-    logger = logging.getLogger('workflow')
+    logger = logging.getLogger('nipype.workflow')
 
     INIT_MSG = """
     Running fMRIPREP version {version}: