Merge remote-tracking branch 'nipy/master' into ref/modularize-multifile-renamers

# Conflicts:
#	heudiconv/convert.py

Author: tsalo

CHANGELOG.md

@@ -4,6 +4,74 @@ All notable changes to this project will be documented (for humans) in this file
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.8.1] - Date
+
+TODO Summary
+
+### Added
+### Changed
+### Deprecated
+### Fixed
+### Removed
+### Security
+
+
+## [0.8.0] - 2020-04-15
+
+### Enhancements
+
+- Centralized saving of .json files.  Indentation of some files could
+  change now from previous versions where it could have used `3`
+  spaces. Now indentation should be consistently `2` for .json files
+  we produce/modify ([#436][]) (note: dcm2niix uses tabs for indentation)
+- ReproIn heuristic: support SBRef and phase data ([#387][])
+- Set the "TaskName" field in .json sidecar files for multi-echo data
+  ([#420][])
+- Provide an informative exception if command needs heuristic to be
+  specified ([#437][])
+
+### Refactored
+
+- `embed_nifti` was refactored into `embed_dicom_and_nifti_metadata`
+  which would no longer create `.nii` file if it does not exist
+  already ([#432][])
+
+### Fixed
+
+- Skip datalad-based tests if no datalad available ([#430][])
+- Search heuristic file path first so we do not pick up a python
+  module if name conflicts ([#434][])
+
+## [0.7.0] - 2020-03-20
+
+### Removed
+
+- Python 2 support/testing
+
+### Enhancement
+
+- `-g` option obtained two new modes: `all` and `custom`. In case of `all`,
+  all provided DICOMs will be treated as coming from a single scanning session.
+  `custom` instructs to use `.grouping` value (could be a DICOM attribute or
+  a callable)provided by the heuristic ([#359][]).
+- Stop before reading pixels data while gathering metadata from DICOMs ([#404][])
+- reproin heuristic:
+  - In addition to original "md5sum of the study_description" `protocols2fix`
+    could now have (and applied after md5sum matching ones)
+    1). a regular expression searched in study_description,
+    2). an empty string as "catch all".
+    This features could be used to easily provide remapping into reproin
+    naming (documentation is to come to http://github.com/ReproNim/reproin)
+    ([#425][])
+
+### Fixed
+
+- Use nan, not None for absent echo value in sorting
+- reproin heuristic: case seqinfos into a list to be able to modify from
+  overloaded heuristic ([#419][])
+- No spurious errors from the logger upon a warning about `etelemetry`
+  absence ([#407][])
+
 ## [0.6.0] - 2019-12-16
 
 This is largely a bug fix.  Metadata and order of `_key-value` fields in BIDS
@@ -271,6 +339,7 @@ TODO Summary
 [#348]: https://github.com/nipy/heudiconv/issues/348
 [#351]: https://github.com/nipy/heudiconv/issues/351
 [#352]: https://github.com/nipy/heudiconv/issues/352
+[#359]: https://github.com/nipy/heudiconv/issues/359
 [#360]: https://github.com/nipy/heudiconv/issues/360
 [#364]: https://github.com/nipy/heudiconv/issues/364
 [#369]: https://github.com/nipy/heudiconv/issues/369
@@ -279,4 +348,15 @@ TODO Summary
 [#376]: https://github.com/nipy/heudiconv/issues/376
 [#379]: https://github.com/nipy/heudiconv/issues/379
 [#380]: https://github.com/nipy/heudiconv/issues/380
+[#387]: https://github.com/nipy/heudiconv/issues/387
 [#390]: https://github.com/nipy/heudiconv/issues/390
+[#404]: https://github.com/nipy/heudiconv/issues/404
+[#407]: https://github.com/nipy/heudiconv/issues/407
+[#419]: https://github.com/nipy/heudiconv/issues/419
+[#420]: https://github.com/nipy/heudiconv/issues/420
+[#425]: https://github.com/nipy/heudiconv/issues/425
+[#430]: https://github.com/nipy/heudiconv/issues/430
+[#432]: https://github.com/nipy/heudiconv/issues/432
+[#434]: https://github.com/nipy/heudiconv/issues/434
+[#436]: https://github.com/nipy/heudiconv/issues/436
+[#437]: https://github.com/nipy/heudiconv/issues/437

Makefile

@@ -0,0 +1,6 @@
+all:
+	echo 'nothing by default'
+
+prep_release:
+	# take previous one, and replace with the next one
+	utils/prep_release

docs/conf.py

@@ -26,7 +26,7 @@
 # The short X.Y version
 version = ''
 # The full version, including alpha/beta/rc tags
-release = '0.6.0'
+release = '0.8.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/heuristics.rst

@@ -68,3 +68,20 @@ DICOMs where this function returns ``True`` will be filtered out.
 Further processing on ``seqinfos`` to deduce/customize subject, session, and locator.
 
 A dictionary of {"locator": locator, "session": session, "subject": subject} is returned.
+
+---------------------------------------------------------------
+``grouping`` string or ``grouping(files, dcmfilter, seqinfo)``
+---------------------------------------------------------------
+
+Whenever ``--grouping custom`` (``-g custom``) is used, this attribute or callable
+will be used to inform how to group the DICOMs into separate groups. From
+`original PR#359 <https://github.com/nipy/heudiconv/pull/359>`_::
+
+    grouping = 'AcquisitionDate'
+
+or::
+
+    def grouping(files, dcmfilter, seqinfo):
+        seqinfos = collections.OrderedDict()
+        ...
+        return seqinfos  # ordered dict containing seqinfo objects: list of DICOMs

docs/installation.rst

@@ -26,7 +26,7 @@ If `Docker <https://docs.docker.com/install/>`_ is available on your system, you
 can visit `our page on Docker Hub <https://hub.docker.com/r/nipy/heudiconv/tags>`_ 
 to view available releases. To pull the latest release, run::
 
-    $ docker pull nipy/heudiconv:0.6.0
+    $ docker pull nipy/heudiconv:0.8.0
 
 
 Singularity
@@ -35,4 +35,4 @@ If `Singularity <https://www.sylabs.io/singularity/>`_ is available on your syst
 you can use it to pull and convert our Docker images! For example, to pull and 
 build the latest release, you can run::
 
-    $ singularity pull docker://nipy/heudiconv:0.6.0
+    $ singularity pull docker://nipy/heudiconv:0.8.0

docs/tutorials.rst

@@ -13,6 +13,14 @@ other users' tutorials covering their experience with ``heudiconv``.
 
 - `Sample Conversion: Coastal Coding 2019 <http://www.repronim.org/coco2019-training/presentations/heudiconv/#1>`_.
 
+- `A joined DataLad and HeuDiConv tutorial for reproducible fMRI studies <http://www.repronim.org/coco2019-training/04-02-reproin/>`_.
+
+- `The ReproIn conversion workflow overview <https://github.com/repronim/reproin#conversion>`_.
+
+- `Slides <https://docs.google.com/presentation/d/14UNWQVY49c9Xc-7sj1FkoILXnt-wYjW404oqT-FtCW8/edit#slide=id.p>`_ and
+  `recording <https://www.youtube.com/watch?v=j2SKX37-w4c&list=PLs3CA4ShM1DUX0nTMKfoB8Z6kdrZpByLa&index=5&t=0s>`_
+  of a ReproNim Webinar on ``heudiconv``.
+
 .. caution::
     Some of these tutorials may not be up to date with 
     the latest releases of ``heudiconv``.

docs/usage.rst

@@ -82,7 +82,7 @@ The second script processes a DICOM directory with ``heudiconv`` using the built
     DCMDIR=${DCMDIRS[${SLURM_ARRAY_TASK_ID}]}
     echo Submitted directory: ${DCMDIR}
 
-    IMG="/singularity-images/heudiconv-0.6.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.8.0-dev.sif"
     CMD="singularity run -B ${DCMDIR}:/dicoms:ro -B ${OUTDIR}:/output -e ${IMG} --files /dicoms/ -o /output -f reproin -c dcm2niix -b notop --minmeta -l ."
 
     printf "Command:\n${CMD}\n"
@@ -97,7 +97,7 @@ This script creates the top-level bids files (e.g.,
     set -eu
 
     OUTDIR=${1}
-    IMG="/singularity-images/heudiconv-0.6.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.8.0-dev.sif"
     CMD="singularity run -B ${OUTDIR}:/output -e ${IMG} --files /output -f reproin --command populate-templates"
 
     printf "Command:\n${CMD}\n"

heudiconv/bids.py

@@ -171,7 +171,7 @@ def populate_aggregated_jsons(path):
             act = "Generating"
         lgr.debug("%s %s", act, task_file)
         fields.update(placeholders)
-        save_json(task_file, fields, indent=2, sort_keys=True, pretty=True)
+        save_json(task_file, fields, sort_keys=True, pretty=True)
 
 
 def tuneup_bids_json_files(json_files):
@@ -193,7 +193,7 @@ def tuneup_bids_json_files(json_files):
             # Let's hope no word 'Date' comes within a study name or smth like
             # that
             raise ValueError("There must be no dates in .json sidecar")
-        save_json(jsonfile, json_, indent=2)
+        save_json(jsonfile, json_)
 
     # Load the beast
     seqtype = op.basename(op.dirname(jsonfile))
@@ -223,7 +223,7 @@ def tuneup_bids_json_files(json_files):
             was_readonly = is_readonly(json_phasediffname)
             if was_readonly:
                 set_readonly(json_phasediffname, False)
-            save_json(json_phasediffname, json_, indent=2)
+            save_json(json_phasediffname, json_)
             if was_readonly:
                 set_readonly(json_phasediffname)
 
@@ -259,8 +259,7 @@ def add_participant_record(studydir, subject, age, sex):
                         ("Description", "(TODO: adjust - by default everyone is in "
                             "control group)")])),
                 ]),
-                sort_keys=False,
-                indent=2)
+                sort_keys=False)
     # Add a new participant
     with open(participants_tsv, 'a') as f:
         f.write(
@@ -373,8 +372,7 @@ def add_rows_to_scans_keys_file(fn, newrows):
                         ("LongName", "Random string"),
                         ("Description", "md5 hash of UIDs")])),
                 ]),
-                sort_keys=False,
-                indent=2)
+                sort_keys=False)
 
     header = ['filename', 'acq_time', 'operator', 'randstr']
     # prepare all the data rows

heudiconv/cli/run.py

@@ -62,6 +62,7 @@ def process_extra_commands(outdir, args):
         for f in args.files:
             treat_infofile(f)
     elif args.command == 'ls':
+        ensure_heuristic_arg(args)
         heuristic = load_heuristic(args.heuristic)
         heuristic_ls = getattr(heuristic, 'ls', None)
         for f in args.files:
@@ -78,6 +79,7 @@ def process_extra_commands(outdir, args):
                     % (str(study_session), len(sequences), suf)
                 )
     elif args.command == 'populate-templates':
+        ensure_heuristic_arg(args)
         heuristic = load_heuristic(args.heuristic)
         for f in args.files:
             populate_bids_templates(f, getattr(heuristic, 'DEFAULT_FIELDS', {}))
@@ -88,16 +90,21 @@ def process_extra_commands(outdir, args):
         for name_desc in get_known_heuristics_with_descriptions().items():
             print("- %s: %s" % name_desc)
     elif args.command == 'heuristic-info':
-        from ..utils import get_heuristic_description, get_known_heuristic_names
-        if not args.heuristic:
-            raise ValueError("Specify heuristic using -f. Known are: %s"
-                             % ', '.join(get_known_heuristic_names()))
+        ensure_heuristic_arg(args)
+        from ..utils import get_heuristic_description
         print(get_heuristic_description(args.heuristic, full=True))
     else:
         raise ValueError("Unknown command %s", args.command)
     return
 
 
+def ensure_heuristic_arg(args):
+    from ..utils import get_known_heuristic_names
+    if not args.heuristic:
+        raise ValueError("Specify heuristic using -f. Known are: %s"
+                         % ', '.join(get_known_heuristic_names()))
+
+
 def main(argv=None):
     parser = get_parser()
     args = parser.parse_args(argv)
@@ -124,7 +131,6 @@ def main(argv=None):
 
     if args.debug:
         setup_exceptionhook()
-
     process_args(args)
 
 
@@ -154,8 +160,7 @@ def get_parser():
                         'If not provided, DICOMS would first be "sorted" and '
                         'subject IDs deduced by the heuristic')
     parser.add_argument('-c', '--converter',
-                        default='dcm2niix',
-                        choices=('dcm2niix', 'none'),
+                        choices=('dcm2niix', 'none'), default='dcm2niix',
                         help='tool to use for DICOM conversion. Setting to '
                         '"none" disables the actual conversion step -- useful'
                         'for testing heuristics.')
@@ -219,7 +224,7 @@ def get_parser():
                         help='custom actions to be performed on provided '
                         'files instead of regular operation.')
     parser.add_argument('-g', '--grouping', default='studyUID',
-                        choices=('studyUID', 'accession_number'),
+                        choices=('studyUID', 'accession_number', 'all', 'custom'),
                         help='How to group dicoms (default: by studyUID)')
     parser.add_argument('--minmeta', action='store_true',
                         help='Exclude dcmstack meta information in sidecar '
@@ -343,7 +348,8 @@ def process_args(args):
                         seqinfo=seqinfo,
                         min_meta=args.minmeta,
                         overwrite=args.overwrite,
-                        dcmconfig=args.dcmconfig,)
+                        dcmconfig=args.dcmconfig,
+                        grouping=args.grouping,)
 
         lgr.info("PROCESSING DONE: {0}".format(
             str(dict(subject=sid, outdir=study_outdir, session=session))))