Merge remote-tracking branch 'origin/master' into fix/group

* origin/master:
  BF: fix logging string + move etelemetry under try/except
  Add stop_before_pixels to the rest of dicom reads
  Boost perspective release date in changelog to today
  doc: closes #401
  ENH(TST): Fix version to older pytest to ease backward compatibility testing
  RF: use tmpdir not tmp_path fixture
  FIX: minor typo in CHANGELOG.md
  Version boost to 0.6.0
  DOC: populate detailed changelog for 0.6.0 and tune up formatting in previous one
  Fix miscellaneous typos in ReproIn heuristic file.
  BF: fix check for the sbatch (SLURM) not being available
  ENH: make test-compare-two-versions take any two worktrees, and just show diff if results already known
  Update heudiconv/convert.py
  apply @mgxd 's suggestions, adding a warning and a timeout environment variable
  need str typecast
  Use empty string not None
  Empty acq_time results in empty cell not 'n/a'
  Acquire a lock prior to top level bids file modification

 Conflicts:
	heudiconv/dicoms.py -- removed both "common base" and "master" conflicting portion of group_dicoms_into_seqinfos, seems to be moved elsewhere and the only change was stopping reading dicom before pixel data, and I think that is already everywhere in this PR anyways

Author: yarikoptic

CHANGELOG.md

@@ -4,15 +4,72 @@ All notable changes to this project will be documented (for humans) in this file
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] - 2019-12-16
+
+This is largely a bug fix.  Metadata and order of `_key-value` fields in BIDS
+could change from the result of converting using previous versions, thus minor
+version boost.
+14 people contributed to this release -- thanks
+[everyone](https://github.com/nipy/heudiconv/graphs/contributors)!
+
+### Enhancement
+
+- Use [etelemetry](https://pypi.org/project/etelemetry) to inform about most
+  recent available version of heudiconv. Please set `NO_ET` environment variable
+  if you want to disable it ([#369][])
+- BIDS:
+  - `--bids` flag became an option. It can (optionally) accept `notop` value
+    to avoid creation of top level files (`CHANGES`, `dataset_description.json`,
+    etc) as a workaround during parallel execution to avoid race conditions etc.
+    ([#344][])
+  - Generate basic `.json` files with descriptions of the fields for
+    `participants.tsv` and `_scans.tsv` files ([#376][])
+  - Use `filelock` while writing top level files. Use
+    `HEUDICONV_FILELOCK_TIMEOUT` environment to change the default timeout value
+    ([#348][])
+  - `_PDT2` was added as a suffix for multi-echo (really "multi-modal")
+    sequences ([#345][])
+- Calls to `dcm2niix` would include full output path to make it easier to
+  discern in the logs what file it is working on ([#351][])
+- With recent [datalad]() (>= 0.10), created DataLad dataset will use
+  `--fake-dates` functionality of DataLad to not leak data conversion dates,
+  which might be close to actual data acquisition/patient visit ([#352][])
+- Support multi-echo EPI `_phase` data ([#373][] fixes [#368][])
+- Log location of a bad .json file to ease troubleshooting ([#379][])
+- Add basic pypi classifiers for the package ([#380][])
+
+### Fixed
+- Sorting `_scans.tsv` files lacking valid dates field should not cause a crash
+  ([#337][])
+- Multi-echo files detection based number of echos ([#339][])
+- BIDS
+  - Use `EchoTimes` from the associated multi-echo files if `EchoNumber` tag is
+    missing ([#366][] fixes [#347][])
+  - Tolerate empty ContentTime and/or ContentDate in DICOMs ([#372][]) and place
+    "n/a" if value is missing ([#390][])
+  - Do not crash and store original .json file is "JSON pretification" fails
+    ([#342][])
+- ReproIn heuristic
+  - tolerate WIP prefix on Philips scanners ([#343][])
+  - allow for use of `(...)` instead of `{...}` since `{}` are not allowed
+    ([#343][])
+  - Support pipolar fieldmaps by providing them with `_epi` not `_magnitude`.
+    "Loose" BIDS `_key-value` pairs might come now after `_dir-` even if they
+    came first before ([#358][] fixes [#357][])
+- All heuristics saved under `.heudiconv/` under `heuristic.py` name, to avoid
+  discrepancy during reconversion ([#354][] fixes [#353][])
+- Do not crash (with TypeError) while trying to sort absent file list ([#360][])
+- heudiconv requires nipype >= 1.0.0 ([#364][]) and blacklists `1.2.[12]` ([#375][])
+
 ## [0.5.4] - 2019-04-29
 
 This release includes fixes to BIDS multi-echo conversions, the
- re-implementation of queuing support (currently just SLURM), as well as
- some bugfixes.
+re-implementation of queuing support (currently just SLURM), as well as
+some bugfixes.
 
- Starting today, we will (finally) push versioned releases to DockerHub.
- Finally, to more accurately reflect on-going development, the `latest`
- tag has been renamed to `unstable`.
+Starting today, we will (finally) push versioned releases to DockerHub.
+Finally, to more accurately reflect on-going development, the `latest`
+tag has been renamed to `unstable`.
 
 ### Added
 - Readthedocs documentation ([#327][])
@@ -189,8 +246,37 @@ TODO Summary
 [DBIC]: http://dbic.dartmouth.edu
 [datalad]: http://datalad.org
 [dcm2niix]: https://github.com/rordenlab/dcm2niix
-[#293]: https://github.com/nipy/heudiconv/issues/293
 [#301]: https://github.com/nipy/heudiconv/issues/301
+[#353]: https://github.com/nipy/heudiconv/issues/353
+[#354]: https://github.com/nipy/heudiconv/issues/354
+[#357]: https://github.com/nipy/heudiconv/issues/357
+[#358]: https://github.com/nipy/heudiconv/issues/358
+[#347]: https://github.com/nipy/heudiconv/issues/347
+[#366]: https://github.com/nipy/heudiconv/issues/366
+[#368]: https://github.com/nipy/heudiconv/issues/368
+[#373]: https://github.com/nipy/heudiconv/issues/373
+[#293]: https://github.com/nipy/heudiconv/issues/293
 [#304]: https://github.com/nipy/heudiconv/issues/304
 [#306]: https://github.com/nipy/heudiconv/issues/306
 [#310]: https://github.com/nipy/heudiconv/issues/310
+[#327]: https://github.com/nipy/heudiconv/issues/327
+[#328]: https://github.com/nipy/heudiconv/issues/328
+[#334]: https://github.com/nipy/heudiconv/issues/334
+[#337]: https://github.com/nipy/heudiconv/issues/337
+[#339]: https://github.com/nipy/heudiconv/issues/339
+[#342]: https://github.com/nipy/heudiconv/issues/342
+[#343]: https://github.com/nipy/heudiconv/issues/343
+[#344]: https://github.com/nipy/heudiconv/issues/344
+[#345]: https://github.com/nipy/heudiconv/issues/345
+[#348]: https://github.com/nipy/heudiconv/issues/348
+[#351]: https://github.com/nipy/heudiconv/issues/351
+[#352]: https://github.com/nipy/heudiconv/issues/352
+[#360]: https://github.com/nipy/heudiconv/issues/360
+[#364]: https://github.com/nipy/heudiconv/issues/364
+[#369]: https://github.com/nipy/heudiconv/issues/369
+[#372]: https://github.com/nipy/heudiconv/issues/372
+[#375]: https://github.com/nipy/heudiconv/issues/375
+[#376]: https://github.com/nipy/heudiconv/issues/376
+[#379]: https://github.com/nipy/heudiconv/issues/379
+[#380]: https://github.com/nipy/heudiconv/issues/380
+[#390]: https://github.com/nipy/heudiconv/issues/390

dev-requirements.txt

@@ -1,4 +1,5 @@
 -r requirements.txt
-pytest
+# Fix version to older pytest to ease backward compatibility testing
+pytest==3.6.4
 tinydb
 inotify

docs/conf.py

@@ -26,7 +26,7 @@
 # The short X.Y version
 version = ''
 # The full version, including alpha/beta/rc tags
-release = '0.5.4'
+release = '0.6.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/installation.rst

@@ -26,7 +26,7 @@ If `Docker <https://docs.docker.com/install/>`_ is available on your system, you
 can visit `our page on Docker Hub <https://hub.docker.com/r/nipy/heudiconv/tags>`_ 
 to view available releases. To pull the latest release, run::
 
-    $ docker pull nipy/heudiconv:0.5.4
+    $ docker pull nipy/heudiconv:0.6.0
 
 
 Singularity
@@ -35,4 +35,4 @@ If `Singularity <https://www.sylabs.io/singularity/>`_ is available on your syst
 you can use it to pull and convert our Docker images! For example, to pull and 
 build the latest release, you can run::
 
-    $ singularity pull docker://nipy/heudiconv:0.5.4
+    $ singularity pull docker://nipy/heudiconv:0.6.0

docs/tutorials.rst

@@ -7,7 +7,7 @@ other users' tutorials covering their experience with ``heudiconv``.
 
 - `YouTube tutorial <https://www.youtube.com/watch?v=O1kZAuR7E00>`_ by `James Kent <https://github.com/jdkent>`_.
 
-- `Walkthrough <http://reproducibility.stanford.edu/bids-tutorial-series-part-2a/>`_ by the `Standard Center for Reproducible Neuroscience <http://reproducibility.stanford.edu/>`_.
+- `Walkthrough <http://reproducibility.stanford.edu/bids-tutorial-series-part-2a/>`_ by the `Stanford Center for Reproducible Neuroscience <http://reproducibility.stanford.edu/>`_.
 
 - `U of A Neuroimaging Core <https://neuroimaging-core-docs.readthedocs.io/en/latest/pages/heudiconv.html>`_ by `Dianne Patterson <https://github.com/dkp>`_.
 

docs/usage.rst

@@ -82,7 +82,7 @@ The second script processes a DICOM directory with ``heudiconv`` using the built
     DCMDIR=${DCMDIRS[${SLURM_ARRAY_TASK_ID}]}
     echo Submitted directory: ${DCMDIR}
 
-    IMG="/singularity-images/heudiconv-0.5.4-dev.sif"
+    IMG="/singularity-images/heudiconv-0.6.0-dev.sif"
     CMD="singularity run -B ${DCMDIR}:/dicoms:ro -B ${OUTDIR}:/output -e ${IMG} --files /dicoms/ -o /output -f reproin -c dcm2niix -b notop --minmeta -l ."
 
     printf "Command:\n${CMD}\n"
@@ -97,7 +97,7 @@ This script creates the top-level bids files (e.g.,
     set -eu
 
     OUTDIR=${1}
-    IMG="/singularity-images/heudiconv-0.5.4-dev.sif"
+    IMG="/singularity-images/heudiconv-0.6.0-dev.sif"
     CMD="singularity run -B ${OUTDIR}:/output -e ${IMG} --files /output -f reproin --command populate-templates"
 
     printf "Command:\n${CMD}\n"

heudiconv/bids.py

@@ -413,7 +413,7 @@ def get_formatted_scans_key_row(dcm_fn):
         acq_time = datetime.strptime(td, '%H%M%S%Y%m%d').isoformat()
     except (AttributeError, ValueError) as exc:
         lgr.warning("Failed to get date/time for the content: %s", str(exc))
-        acq_time = None
+        acq_time = ''
     # add random string
     # But let's make it reproducible by using all UIDs
     # (might change across versions?)

heudiconv/cli/run.py

@@ -246,11 +246,11 @@ def process_args(args):
 
     outdir = op.abspath(args.outdir)
 
-    import etelemetry
     try:
+        import etelemetry
         latest = etelemetry.get_project("nipy/heudiconv")
     except Exception as e:
-        lgr.warning("Could not check for version updates: ", e)
+        lgr.warning("Could not check for version updates: %s", str(e))
         latest = {"version": 'Unknown'}
 
     lgr.info(INIT_MSG(packname=__packagename__,

heudiconv/convert.py

@@ -1,3 +1,4 @@
+import filelock
 import os
 import os.path as op
 import logging
@@ -32,6 +33,7 @@
     compress_dicoms
 )
 
+LOCKFILE = 'heudiconv.lock'
 lgr = logging.getLogger(__name__)
 
 
@@ -211,14 +213,23 @@ def prep_conversion(sid, dicoms, outdir, heuristic, converter, anon_sid,
         clear_temp_dicoms(item_dicoms)
 
     if bids_options is not None and 'notop' not in bids_options:
-        if seqinfo:
-            keys = list(seqinfo)
-            add_participant_record(anon_outdir,
-                                   anon_sid,
-                                   keys[0].patient_age,
-                                   keys[0].patient_sex)
-        populate_bids_templates(anon_outdir,
-                                getattr(heuristic, 'DEFAULT_FIELDS', {}))
+        lockfile = op.join(anon_outdir, LOCKFILE)
+        if op.exists(lockfile):
+            lgr.warning("Existing lockfile found in {0} - waiting for the "
+                        "lock to be released. To set a timeout limit, set "
+                        "the HEUDICONV_FILELOCK_TIMEOUT environmental variable "
+                        "to a value in seconds. If this process hangs, it may "
+                        "require a manual deletion of the {0}.".format(lockfile))
+        timeout = os.getenv("HEUDICONV_LOCKFILE_TIMEOUT", -1)
+        with filelock.SoftFileLock(lockfile, timeout=timeout):
+            if seqinfo:
+                keys = list(seqinfo)
+                add_participant_record(anon_outdir,
+                                       anon_sid,
+                                       keys[0].patient_age,
+                                       keys[0].patient_sex)
+            populate_bids_templates(anon_outdir,
+                                    getattr(heuristic, 'DEFAULT_FIELDS', {}))
 
 
 def convert(items, converter, scaninfo_suffix, custom_callable, with_prov,