Merge remote-tracking branch 'origin/master' into mbme-reproin

* origin/master: (32 commits)
  ENH(TST): skip test_dlad.py if no datalad.api
  ENH: reproin validator config - ignore absence of custom column description
  [DATALAD RUNCMD] Update versions to the next version (as in changelog)
  helper makefile
  ENH: a helper to prepare release
  CHANGELOG for 0.7.0
  BF: #427 Sort of None echotime was not working. Replaced with nan
  DOC: basic doc about "-g custom"
  BF(PY<3.7): just sample the class of a compiled regex
  RF+ENH: allow matching study using regexp on study_description
  ENH: reproin - log applied substitutions
  ENH(TST): test application of "catch all" rule
  RF: reproin - remove binding of fixup data structures in function signatures
  RF+ENH: reproin - allow for an empty key in protocols2fix to provide fixups for any study
  fix: remove filtered DICOMs from file list
  enh: improve regression test
  fix: changes from code review
  RF: return back indexes in comments for seqinfo_fields
  Minor DOC (explicit list of values in numpy doc style) and remove ambiguity in reading comparison
  ci: remove 34 testing, add 37
  ...

 Conflicts:
	heudiconv/convert.py - conflicted with using nan instead of None for EchoTime

Author: yarikoptic

CHANGELOG.md

@@ -4,6 +4,36 @@ All notable changes to this project will be documented (for humans) in this file
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.7.0] - 2020-03-20
+
+### Removed
+
+- Python 2 support/testing
+
+### Enhancement
+
+- `-g` option obtained two new modes: `all` and `custom`. In case of `all`,
+  all provided DICOMs will be treated as coming from a single scanning session.
+  `custom` instructs to use `.grouping` value (could be a DICOM attribute or
+  a callable)provided by the heuristic ([#359][]).
+- Stop before reading pixels data while gathering metadata from DICOMs ([#404][])
+- reproin heuristic:
+  - In addition to original "md5sum of the study_description" `protocols2fix`
+    could now have (and applied after md5sum matching ones)
+    1). a regular expression searched in study_description,
+    2). an empty string as "catch all".
+    This features could be used to easily provide remapping into reproin
+    naming (documentation is to come to http://github.com/ReproNim/reproin)
+    ([#425][])
+
+### Fixed
+
+- Use nan, not None for absent echo value in sorting
+- reproin heuristic: case seqinfos into a list to be able to modify from
+  overloaded heuristic ([#419][])
+- No spurious errors from the logger upon a warning about `etelemetry`
+  absence ([#407][])
+
 ## [0.6.0] - 2019-12-16
 
 This is largely a bug fix.  Metadata and order of `_key-value` fields in BIDS
@@ -271,6 +301,7 @@ TODO Summary
 [#348]: https://github.com/nipy/heudiconv/issues/348
 [#351]: https://github.com/nipy/heudiconv/issues/351
 [#352]: https://github.com/nipy/heudiconv/issues/352
+[#359]: https://github.com/nipy/heudiconv/issues/359
 [#360]: https://github.com/nipy/heudiconv/issues/360
 [#364]: https://github.com/nipy/heudiconv/issues/364
 [#369]: https://github.com/nipy/heudiconv/issues/369
@@ -280,3 +311,7 @@ TODO Summary
 [#379]: https://github.com/nipy/heudiconv/issues/379
 [#380]: https://github.com/nipy/heudiconv/issues/380
 [#390]: https://github.com/nipy/heudiconv/issues/390
+[#404]: https://github.com/nipy/heudiconv/issues/404
+[#407]: https://github.com/nipy/heudiconv/issues/407
+[#419]: https://github.com/nipy/heudiconv/issues/419
+[#425]: https://github.com/nipy/heudiconv/issues/425

Makefile

@@ -0,0 +1,6 @@
+all:
+	echo 'nothing by default'
+
+prep_release:
+	# take previous one, and replace with the next one
+	utils/prep_release

docs/conf.py

@@ -26,7 +26,7 @@
 # The short X.Y version
 version = ''
 # The full version, including alpha/beta/rc tags
-release = '0.6.0'
+release = '0.7.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/heuristics.rst

@@ -68,3 +68,20 @@ DICOMs where this function returns ``True`` will be filtered out.
 Further processing on ``seqinfos`` to deduce/customize subject, session, and locator.
 
 A dictionary of {"locator": locator, "session": session, "subject": subject} is returned.
+
+---------------------------------------------------------------
+``grouping`` string or ``grouping(files, dcmfilter, seqinfo)``
+---------------------------------------------------------------
+
+Whenever ``--grouping custom`` (``-g custom``) is used, this attribute or callable
+will be used to inform how to group the DICOMs into separate groups. From
+`original PR#359 <https://github.com/nipy/heudiconv/pull/359>`_::
+
+    grouping = 'AcquisitionDate'
+
+or::
+
+    def grouping(files, dcmfilter, seqinfo):
+        seqinfos = collections.OrderedDict()
+        ...
+        return seqinfos  # ordered dict containing seqinfo objects: list of DICOMs

docs/installation.rst

@@ -26,7 +26,7 @@ If `Docker <https://docs.docker.com/install/>`_ is available on your system, you
 can visit `our page on Docker Hub <https://hub.docker.com/r/nipy/heudiconv/tags>`_ 
 to view available releases. To pull the latest release, run::
 
-    $ docker pull nipy/heudiconv:0.6.0
+    $ docker pull nipy/heudiconv:0.7.0
 
 
 Singularity
@@ -35,4 +35,4 @@ If `Singularity <https://www.sylabs.io/singularity/>`_ is available on your syst
 you can use it to pull and convert our Docker images! For example, to pull and 
 build the latest release, you can run::
 
-    $ singularity pull docker://nipy/heudiconv:0.6.0
+    $ singularity pull docker://nipy/heudiconv:0.7.0

docs/usage.rst

@@ -82,7 +82,7 @@ The second script processes a DICOM directory with ``heudiconv`` using the built
     DCMDIR=${DCMDIRS[${SLURM_ARRAY_TASK_ID}]}
     echo Submitted directory: ${DCMDIR}
 
-    IMG="/singularity-images/heudiconv-0.6.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.7.0-dev.sif"
     CMD="singularity run -B ${DCMDIR}:/dicoms:ro -B ${OUTDIR}:/output -e ${IMG} --files /dicoms/ -o /output -f reproin -c dcm2niix -b notop --minmeta -l ."
 
     printf "Command:\n${CMD}\n"
@@ -97,7 +97,7 @@ This script creates the top-level bids files (e.g.,
     set -eu
 
     OUTDIR=${1}
-    IMG="/singularity-images/heudiconv-0.6.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.7.0-dev.sif"
     CMD="singularity run -B ${OUTDIR}:/output -e ${IMG} --files /output -f reproin --command populate-templates"
 
     printf "Command:\n${CMD}\n"

heudiconv/cli/run.py

@@ -124,7 +124,6 @@ def main(argv=None):
 
     if args.debug:
         setup_exceptionhook()
-
     process_args(args)
 
 
@@ -154,8 +153,7 @@ def get_parser():
                         'If not provided, DICOMS would first be "sorted" and '
                         'subject IDs deduced by the heuristic')
     parser.add_argument('-c', '--converter',
-                        default='dcm2niix',
-                        choices=('dcm2niix', 'none'),
+                        choices=('dcm2niix', 'none'), default='dcm2niix',
                         help='tool to use for DICOM conversion. Setting to '
                         '"none" disables the actual conversion step -- useful'
                         'for testing heuristics.')
@@ -219,7 +217,7 @@ def get_parser():
                         help='custom actions to be performed on provided '
                         'files instead of regular operation.')
     parser.add_argument('-g', '--grouping', default='studyUID',
-                        choices=('studyUID', 'accession_number'),
+                        choices=('studyUID', 'accession_number', 'all', 'custom'),
                         help='How to group dicoms (default: by studyUID)')
     parser.add_argument('--minmeta', action='store_true',
                         help='Exclude dcmstack meta information in sidecar '
@@ -343,7 +341,8 @@ def process_args(args):
                         seqinfo=seqinfo,
                         min_meta=args.minmeta,
                         overwrite=args.overwrite,
-                        dcmconfig=args.dcmconfig,)
+                        dcmconfig=args.dcmconfig,
+                        grouping=args.grouping,)
 
         lgr.info("PROCESSING DONE: {0}".format(
             str(dict(subject=sid, outdir=study_outdir, session=session))))

heudiconv/convert.py

@@ -2,6 +2,7 @@
 import os
 import os.path as op
 import logging
+from math import nan
 import shutil
 import sys
 
@@ -80,8 +81,8 @@ def conversion_info(subject, outdir, info, filegroup, ses):
 
 
 def prep_conversion(sid, dicoms, outdir, heuristic, converter, anon_sid,
-                   anon_outdir, with_prov, ses, bids_options, seqinfo, min_meta,
-                   overwrite, dcmconfig):
+                    anon_outdir, with_prov, ses, bids_options, seqinfo, 
+                    min_meta, overwrite, dcmconfig, grouping):
     if dicoms:
         lgr.info("Processing %d dicoms", len(dicoms))
     elif seqinfo:
@@ -157,16 +158,17 @@ def prep_conversion(sid, dicoms, outdir, heuristic, converter, anon_sid,
         # So either it would need to be brought back or reconsidered altogether
         # (since no sample data to test on etc)
     else:
-        # TODO -- might have been done outside already!
-        # MG -- will have to try with both dicom template, files
         assure_no_file_exists(target_heuristic_filename)
         safe_copyfile(heuristic.filename, target_heuristic_filename)
         if dicoms:
             seqinfo = group_dicoms_into_seqinfos(
                 dicoms,
+                grouping,
                 file_filter=getattr(heuristic, 'filter_files', None),
                 dcmfilter=getattr(heuristic, 'filter_dicom', None),
-                grouping=None)
+                flatten=True,
+                custom_grouping=getattr(heuristic, 'grouping', None))
+
         seqinfo_list = list(seqinfo.keys())
         filegroup = {si.series_id: x for si, x in seqinfo.items()}
         dicominfo_file = op.join(idir, 'dicominfo%s.tsv' % ses_suffix)
@@ -531,7 +533,7 @@ def save_converted_files(res, item_dicoms, bids_options, outtype, prefix, outnam
 
         # Check for varying echo times
         echo_times = sorted(list(set(
-            b.get('EchoTime', None)
+            b.get('EchoTime', nan)
             for b in bids_metas
             if b
         )))