doc: finalize rtd

mgxd · mgxd · commit 2ef877f58f3a · 2019-03-27T18:00:36.000-04:00
diff --git a/.gitignore b/.gitignore
@@ -4,3 +4,4 @@
 *.egg-info/
 .idea/
 venvs/
+_build/
diff --git a/README.rst b/README.rst
@@ -1,4 +1,8 @@
-.. topic:: `HeuDiConv`: a heuristic-centric DICOM converter
+=============
+**HeuDiConv**
+=============
+
+`a heuristic-centric DICOM converter`
 
 .. image:: https://img.shields.io/badge/docker-nipy/heudiconv:unstable-brightgreen.svg?logo=docker&style=flat
   :target: https://hub.docker.com/r/nipy/heudiconv/tags/
@@ -19,17 +23,13 @@
 About
 -----
 
-``heudiconv`` is a flexible DICOM converter for organizing brain imaging data 
+``heudiconv`` is a flexible DICOM converter for organizing brain imaging data
 into structured directory layouts.
 
-- it allows flexible directory layouts and naming schemes through 
-customizable heuristics implementations
+- it allows flexible directory layouts and naming schemes through customizable heuristics implementations
 - it only converts the necessary DICOMs, not everything in a directory
 - you can keep links to DICOM files in the participant layout
 - using dcm2niix under the hood, it's fast
-- it can track the provenance of the conversion from DICOM to NIfTI in W3C 
-PROV format
+- it can track the provenance of the conversion from DICOM to NIfTI in W3C PROV format
 - it provides assistance in converting to `BIDS <http://bids.neuroimaging.io/>`_.
-- it integrates with `DataLad <https://www.datalad.org/>`_ to place converted and 
-original data under git/git-annex version control, while automatically 
-annotating files with sensitive information (e.g., non-defaced anatomicals, etc)
+- it integrates with `DataLad <https://www.datalad.org/>`_ to place converted and original data under git/git-annex version control, while automatically annotating files with sensitive information (e.g., non-defaced anatomicals, etc)
diff --git a/docs/api.rst b/docs/api.rst
@@ -0,0 +1,13 @@
+=============
+API Reference
+=============
+
+.. toctree::
+   :maxdepth: 1
+
+   api/bids
+   api/convert
+   api/dicoms
+   api/parser
+   api/queue
+   api/util
diff --git a/docs/api/bids.rst b/docs/api/bids.rst
@@ -0,0 +1,5 @@
+====
+BIDS
+====
+
+.. automodule:: heudiconv.bids
diff --git a/docs/api/convert.rst b/docs/api/convert.rst
@@ -0,0 +1,5 @@
+==========
+Conversion
+==========
+
+.. automodule:: heudiconv.convert
diff --git a/docs/api/dicoms.rst b/docs/api/dicoms.rst
@@ -0,0 +1,5 @@
+======
+DICOMS
+======
+
+.. automodule:: heudiconv.dicoms
diff --git a/docs/api/parser.rst b/docs/api/parser.rst
@@ -0,0 +1,5 @@
+=======
+Parsing
+=======
+
+.. automodule:: heudiconv.parser
diff --git a/docs/api/queue.rst b/docs/api/queue.rst
@@ -0,0 +1,5 @@
+=============
+Batch Queuing
+=============
+
+.. automodule:: heudiconv.queue
diff --git a/docs/api/utils.rst b/docs/api/utils.rst
@@ -0,0 +1,5 @@
+=======
+Utility
+=======
+
+.. automodule:: heudiconv.utils
diff --git a/docs/changes.rst b/docs/changes.rst
@@ -1 +1,5 @@
-.. include:: ../CHANGELOG.md
+=======
+Changes
+=======
+
+.. mdinclude:: ../CHANGELOG.md
diff --git a/docs/conf.py b/docs/conf.py
@@ -41,7 +41,7 @@
 extensions = [
     'sphinx.ext.autodoc',
     'sphinxarg.ext',
-    'recommonmark',
+    'm2r',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -177,3 +177,5 @@
 
 
 # -- Extension configuration -------------------------------------------------
+autodoc_default_options={
+          'members': None}
diff --git a/docs/heuristics.rst b/docs/heuristics.rst
@@ -2,13 +2,13 @@
 Heuristic
 =========
 
-The heuristic file controls how information about the DICOMs is used to convert 
+The heuristic file controls how information about the DICOMs is used to convert
 to a file system layout (e.g., BIDS). ``heudiconv`` includes some built-in
-heuristics, including `ReproIn <https://github.com/ReproNim/reproin/blob/master/README.md>`_ 
+heuristics, including `ReproIn <https://github.com/ReproNim/reproin/blob/master/README.md>`_
 (which is great to adopt if you will be starting your data collection!).
 
-However, there is a large variety of data out there, and not all DICOMs will be 
-covered by the existing heuristics. This section will outline what makes up a 
+However, there is a large variety of data out there, and not all DICOMs will be
+covered by the existing heuristics. This section will outline what makes up a
 heuristic file, and some useful functions available when making one.
 
 
@@ -19,21 +19,20 @@ Components
 ``infotodict(seqinfos)``
 ------------------------
 
-The only required function for a heuristic, `infotodict` is used to both define 
-the conversion outputs and specify the criteria for scan to output association. 
-Conversion outputs are defined as keys, a `tuple` consisting of a template path 
-used for the basis of outputs, as well as a `tuple` of output types. Valid types 
+The only required function for a heuristic, `infotodict` is used to both define
+the conversion outputs and specify the criteria for scan to output association.
+Conversion outputs are defined as keys, a `tuple` consisting of a template path
+used for the basis of outputs, as well as a `tuple` of output types. Valid types
 include `nii`, `nii.gz`, and `dicom`.
 
-.. note::
-    An example conversion key
-    
+.. note:: An example conversion key
+
     ``('sub-{subject}/func/sub-{subject}_task-test_run-{item}_bold', ('nii.gz', 'dicom'))``
 
 
-The ``seqinfos`` parameter is a list of namedtuples which serves as a grouped and 
-stacked record of the DICOMs passed in. Each item in `seqinfo` contains DICOM 
-metadata that can be used to isolate the series, and assign it to a conversion 
+The ``seqinfos`` parameter is a list of namedtuples which serves as a grouped and
+stacked record of the DICOMs passed in. Each item in `seqinfo` contains DICOM
+metadata that can be used to isolate the series, and assign it to a conversion
 key.
 
 A dictionary of {``conversion key``: ``seqinfo``} is returned.
@@ -42,15 +41,15 @@ A dictionary of {``conversion key``: ``seqinfo``} is returned.
 ``create_key(template, outtype)``
 ---------------------------------
 
-A common helper function used to create the conversion key in ``infotodict``. 
+A common helper function used to create the conversion key in ``infotodict``.
 
 --------------------
 ``filter_files(fl)``
 --------------------
 
 A utility function used to filter any input files.
 
-If this function is included, every file found will go through this filter. Any 
+If this function is included, every file found will go through this filter. Any
 files where this function returns ``True`` will be filtered out.
 
 --------------------------
@@ -59,7 +58,7 @@ files where this function returns ``True`` will be filtered out.
 
 A utility function used to filter any DICOMs.
 
-If this function is included, every DICOM found will go through this filter. Any 
+If this function is included, every DICOM found will go through this filter. Any
 DICOMs where this function returns ``True`` will be filtered out.
 
 -------------------------------
diff --git a/docs/index.rst b/docs/index.rst
@@ -10,7 +10,7 @@ Contents
 
 .. toctree::
    :maxdepth: 2
-   
+
    installation
    changes
    usage
diff --git a/docs/requirements.txt b/docs/requirements.txt
@@ -1,3 +1,3 @@
 sphinx-argparse
-recommonmark
+m2r
 -r ../dev-requirements.txt
diff --git a/docs/usage.rst b/docs/usage.rst
@@ -2,7 +2,7 @@
 Usage
 =====
 
-``heudiconv`` processes DICOM files and converts the output into user defined 
+``heudiconv`` processes DICOM files and converts the output into user defined
 paths.
 
 CommandLine Arguments
@@ -13,7 +13,7 @@ CommandLine Arguments
    :prog: heudiconv
    :nodefault:
    :nodefaultconst:
-   
+
 
 Support
 =======
@@ -29,39 +29,51 @@ All previous ``heudiconv`` questions are available here:
 http://neurostars.org/tags/heudiconv/
 
 
-Batch example
-=============
+Batch jobs
+==========
+
+``heudiconv`` can natively handle multi-subject, multi-session conversions,
+although it will process these linearly. To speed this up, multiple ``heudiconv``
+processes can be spawned concurrently, each converting a different subject and/or
+session.
+
+The following example uses SLURM and Singularity to submit every subjects'
+DICOMs as an independent ``heudiconv`` execution.
+
+The first script aggregates the DICOM directories and submits them to
+``run_heudiconv.sh`` with SLURM as a job array.::
+
+    #!/bin/bash
+
+    set -eu
 
-``heudiconv`` can greatly facilitate large scale conversions
+    # where the DICOMs are located
+    DCMROOT=/dicom/storage/voice
+    # where we want to output the data
+    OUTPUT=/converted/data/voice
 
+    # find all DICOM directories that start with "voice"
+    DCMDIRS=(`find ${DCMROOT} -maxdepth 1 -name voice* -type d`)
 
-*******************************************************************************************
-*******************************************************************************************
-*******************************************************************************************
-*******************************************************************************************
+    # submit to another script as a job array on SLURM
+    sbatch --array=0-$len run_heudiconv.sh ${OUTPUT} ${DCMDIRS[@]}
 
-Call `heudiconv` like this:
 
-    heudiconv -d '{subject}*.tar*' -s xx05 -f ~/myheuristics/convertall.py
+The second script processes a DICOM directory with ``heudiconv`` using the built-in
+`reproin` heuristic.::
 
-where `-d '{subject}*tar*'` is an expression used to find DICOM files
-(`{subject}` expands to a subject ID so that the expression will match any
-`.tar` files, compressed or not that start with the subject ID in their name).
-An additional flag for session (`{session}`) can be included in the expression
-as well. `-s od05` specifies a subject ID for the conversion (this could be a
-list of multiple IDs), and `-f ~/myheuristics/convertall.py` identifies a
-heuristic implementation for this conversion (see below) for details.
+    #!/bin/bash
+    set -eu
 
-This call will locate the DICOMs (in any number of matching tarballs), extract
-them to a temporary directory, search for any DICOM series it can find, and
-attempts a conversion storing output in the current directory. The output
-directory will contain a subdirectory per subject, which in turn contains an
-`info` directory with a full protocol of detected DICOM series, and how their
-are converted.
+    OUTDIR=${1}
+    # receive all directories, and index them per job array
+    DCMDIRS=(${@:2})
+    DCMDIR=${DCMDIRS[${SLURM_ARRAY_TASK_ID}]}
+    echo Submitted directory: ${DCMDIR}
 
+    IMG="/singularity-images/heudiconv-0.5.4-dev.sif"
+    CMD="singularity run -B ${DCMDIR}:/dicoms:ro -B ${OUTDIR}:/output -e ${IMG} --files /dicoms/ -o /output -f reproin -c dcm2niix -b --minmeta -l ."
 
-To generate lean BIDS output, consider using both the `-b` and the `--minmeta` flags
-to your heudiconv command. The `-b` flag generates a json file with BIDS keys, while
-the `--minmeta` flag restricts the json file to only BIDS keys. Without `--minmeta`,
-the json file and the associated Nifti file contains DICOM metadata extracted using
-dicomstack.
+    printf "Command:\n${CMD}\n"
+    ${CMD}
+    echo "Successful process"

-Original file line number
+Diff line change
 *.egg-info/
 .idea/
 venvs/
 +_build/