Merge branch 'master' into fix/updater-lists

Author: tsalo

.travis.yml

@@ -1,10 +1,11 @@
 # vim ft=yaml
 language: python
+dist: bionic
 python:
-  - 3.5
   - 3.6
   - 3.7
   - 3.8
+  - 3.9
 
 cache:
   - apt

CHANGELOG.md

@@ -4,16 +4,78 @@ All notable changes to this project will be documented (for humans) in this file
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
-## [0.8.1] - Date
+## [0.10.0] - 2021-09-16
 
-TODO Summary
+Various improvements and compatibility/support (dcm2niix, datalad) changes.
 
 ### Added
+
+- Add "AcquisitionTime" to the seqinfo ([#487][])
+- Add support for saving the Phoenix Report in the sourcedata folder ([#489][])
+
 ### Changed
-### Deprecated
+
+- Python 3.5 EOLed, supported (tested) versions now: 3.6 - 3.9
+- In reprorin heuristic, allow for having multiple accessions since now there is
+  `-g all` groupping ([#508][])
+- For BIDS, produce a singular `scans.json` at the top level, and not one per
+  sub/ses (generates too many identical files) ([#507][])
+
+
 ### Fixed
+
+- Compatibility with DataLad 0.15.0. Minimal version is 0.13.0 now.
+- Try to open top level BIDS .json files a number of times for adjustment,
+  so in the case of competition across parallel processes, they just end up
+  with the last one "winning over" ([#523][])
+- Don't fail if etelemetry.get_project returns None ([#501][])
+- Consistently use `n/a` for age/sex, also handle ?M for months ([#500][])
+- To avoid crashing on unrelated derivatives files etc, make `find_files` to
+  take list of topdirs (excluding `derivatives/` etc),
+  and look for _bold only under sub-* directories ([#496][])
+- Ensure bvec/bval files are only created for dwi output ([#491][])
+
 ### Removed
-### Security
+
+- In reproin heuristic, old hardcoded sequence renamings and filters ([#508][])
+
+
+## [0.9.0] - 2020-12-23
+
+Various improvements and compatibility/support (dcm2niix, datalad,
+duecredit) changes.  Major change is placement of output files to the
+target output directory during conversion.
+
+### Added
+
+- #454 zenodo referencing in README.rst and support for ducredit for
+  heudiconv and reproin heuristic
+- #445 more tutorial references in README.md
+
+### Changed
+
+- [#485][] placed files during conversion right away into the target
+  directory (with a `_heudiconv???` suffix, renamed into ultimate target
+  name later on), which avoids hitting file size limits of /tmp ([#481][]) and
+  helped to avoid a regression in dcm2nixx 1.0.20201102
+- [#477][] replaced `rec-<magnitude|phase>` with `part-<mag|phase>` now
+  hat BIDSsupports the part entity
+- [#473][] made default for CogAtlasID to be a TODO URL
+- [#459][] made AcquisitionTime used for acq_time scans file field
+- [#451][] retained sub-second resolution in scans files
+- [#442][] refactored code so there is now heudiconv.main.workflow for
+  more convenient use as a Python module
+
+### Fixed
+
+- minimal version of nipype set to 1.2.3 to guarantee correct handling
+  of DWI files ([#480][])
+- `heudiconvDCM*` temporary directories are removed now ([#462][])
+- compatibility with DataLad 0.13 ([#464][])
+
+### Removed
+
+- #443 pathlib as a dependency (we are Python3 only now)
 
 
 ## [0.8.0] - 2020-04-15
@@ -323,6 +385,12 @@ TODO Summary
 [#366]: https://github.com/nipy/heudiconv/issues/366
 [#368]: https://github.com/nipy/heudiconv/issues/368
 [#373]: https://github.com/nipy/heudiconv/issues/373
+[#485]: https://github.com/nipy/heudiconv/issues/485
+[#442]: https://github.com/nipy/heudiconv/issues/442
+[#451]: https://github.com/nipy/heudiconv/issues/451
+[#459]: https://github.com/nipy/heudiconv/issues/459
+[#473]: https://github.com/nipy/heudiconv/issues/473
+[#477]: https://github.com/nipy/heudiconv/issues/477
 [#293]: https://github.com/nipy/heudiconv/issues/293
 [#304]: https://github.com/nipy/heudiconv/issues/304
 [#306]: https://github.com/nipy/heudiconv/issues/306
@@ -360,11 +428,16 @@ TODO Summary
 [#434]: https://github.com/nipy/heudiconv/issues/434
 [#436]: https://github.com/nipy/heudiconv/issues/436
 [#437]: https://github.com/nipy/heudiconv/issues/437
-[#425]: https://github.com/nipy/heudiconv/issues/425
-[#420]: https://github.com/nipy/heudiconv/issues/420
-[#425]: https://github.com/nipy/heudiconv/issues/425
-[#430]: https://github.com/nipy/heudiconv/issues/430
-[#432]: https://github.com/nipy/heudiconv/issues/432
-[#434]: https://github.com/nipy/heudiconv/issues/434
-[#436]: https://github.com/nipy/heudiconv/issues/436
-[#437]: https://github.com/nipy/heudiconv/issues/437
+[#462]: https://github.com/nipy/heudiconv/issues/462
+[#464]: https://github.com/nipy/heudiconv/issues/464
+[#480]: https://github.com/nipy/heudiconv/issues/480
+[#481]: https://github.com/nipy/heudiconv/issues/481
+[#487]: https://github.com/nipy/heudiconv/issues/487
+[#489]: https://github.com/nipy/heudiconv/issues/489
+[#491]: https://github.com/nipy/heudiconv/issues/491
+[#496]: https://github.com/nipy/heudiconv/issues/496
+[#500]: https://github.com/nipy/heudiconv/issues/500
+[#501]: https://github.com/nipy/heudiconv/issues/501
+[#507]: https://github.com/nipy/heudiconv/issues/507
+[#508]: https://github.com/nipy/heudiconv/issues/508
+[#523]: https://github.com/nipy/heudiconv/issues/523

Makefile

@@ -1,6 +1,18 @@
+PYTHON ?= python3
+
 all:
 	echo 'nothing by default'
 
 prep_release:
 	# take previous one, and replace with the next one
 	utils/prep_release
+
+release-pypi: prep_release
+	# avoid upload of stale builds
+	test ! -e dist
+	# make sure all is still clean/committed
+	! bash -c 'git diff | grep -q .'
+	$(PYTHON) setup.py sdist
+	$(PYTHON) setup.py bdist_wheel
+	twine upload dist/*
+

README.rst

@@ -20,6 +20,10 @@
   :target: http://heudiconv.readthedocs.io/en/latest/?badge=latest
   :alt: Readthedocs
 
+.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1012598.svg
+  :target: https://doi.org/10.5281/zenodo.1012598
+  :alt: Zenodo (latest)
+
 About
 -----
 
@@ -33,3 +37,10 @@ into structured directory layouts.
 - it can track the provenance of the conversion from DICOM to NIfTI in W3C PROV format
 - it provides assistance in converting to `BIDS <http://bids.neuroimaging.io/>`_.
 - it integrates with `DataLad <https://www.datalad.org/>`_ to place converted and original data under git/git-annex version control, while automatically annotating files with sensitive information (e.g., non-defaced anatomicals, etc)
+
+How to cite
+-----------
+
+Please use `Zenodo record <https://doi.org/10.5281/zenodo.1012598>`_ for
+your specific version of HeuDiConv.  We also support gathering
+all relevant citations via `DueCredit <http://duecredit.org>`_.

docs/conf.py

@@ -26,7 +26,7 @@
 # The short X.Y version
 version = ''
 # The full version, including alpha/beta/rc tags
-release = '0.8.0'
+release = '0.10.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/installation.rst

@@ -26,7 +26,7 @@ If `Docker <https://docs.docker.com/install/>`_ is available on your system, you
 can visit `our page on Docker Hub <https://hub.docker.com/r/nipy/heudiconv/tags>`_
 to view available releases. To pull the latest release, run::
 
-    $ docker pull nipy/heudiconv:0.8.0
+    $ docker pull nipy/heudiconv:0.10.0
 
 
 Singularity
@@ -35,4 +35,4 @@ If `Singularity <https://www.sylabs.io/singularity/>`_ is available on your syst
 you can use it to pull and convert our Docker images! For example, to pull and
 build the latest release, you can run::
 
-    $ singularity pull docker://nipy/heudiconv:0.8.0
+    $ singularity pull docker://nipy/heudiconv:0.10.0

docs/usage.rst

@@ -82,7 +82,7 @@ The second script processes a DICOM directory with ``heudiconv`` using the built
     DCMDIR=${DCMDIRS[${SLURM_ARRAY_TASK_ID}]}
     echo Submitted directory: ${DCMDIR}
 
-    IMG="/singularity-images/heudiconv-0.8.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.10.0-dev.sif"
     CMD="singularity run -B ${DCMDIR}:/dicoms:ro -B ${OUTDIR}:/output -e ${IMG} --files /dicoms/ -o /output -f reproin -c dcm2niix -b notop --minmeta -l ."
 
     printf "Command:\n${CMD}\n"
@@ -97,7 +97,7 @@ This script creates the top-level bids files (e.g.,
     set -eu
 
     OUTDIR=${1}
-    IMG="/singularity-images/heudiconv-0.8.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.10.0-dev.sif"
     CMD="singularity run -B ${OUTDIR}:/output -e ${IMG} --files /output -f reproin --command populate-templates"
 
     printf "Command:\n${CMD}\n"

heudiconv/bids.py

@@ -18,11 +18,12 @@
     load_json,
     save_json,
     create_file_if_missing,
-    json_dumps_pretty,
+    json_dumps,
     set_readonly,
     is_readonly,
     get_datetime,
 )
+from . import __version__
 
 lgr = logging.getLogger(__name__)
 
@@ -40,10 +41,47 @@
         ("Description", "md5 hash of UIDs")])),
 ])
 
+#: JSON Key where we will embed our version in the newly produced .json files
+HEUDICONV_VERSION_JSON_KEY = 'HeudiconvVersion'
+
+
 class BIDSError(Exception):
     pass
 
 
+BIDS_VERSION = "1.4.1"
+
+
+def maybe_na(val):
+    """Return 'n/a' if non-None value represented as str is not empty
+
+    Primarily for the consistent use of lower case 'n/a' so 'N/A' and 'NA'
+    are also treated as 'n/a'
+    """
+    if val is not None:
+        val = str(val)
+        val = val.strip()
+    return 'n/a' if (not val or val in ('N/A', 'NA')) else val
+
+
+def treat_age(age):
+    """Age might encounter 'Y' suffix or be a float"""
+    age = str(age)
+    if age.endswith('M'):
+        age = age.rstrip('M')
+        age = float(age) / 12
+        age = ('%.2f' if age != int(age) else '%d') % age
+    else:
+        age = age.rstrip('Y')
+    if age:
+        # strip all leading 0s but allow to scan a newborn (age 0Y)
+        age = '0' if not age.lstrip('0') else age.lstrip('0')
+        if age.startswith('.'):
+            # we had float point value, let's prepend 0
+            age = '0' + age
+    return age
+
+
 def populate_bids_templates(path, defaults={}):
     """Premake BIDS text files with templates"""
 
@@ -53,7 +91,7 @@ def populate_bids_templates(path, defaults={}):
         save_json(descriptor,
               OrderedDict([
                   ('Name', "TODO: name of the dataset"),
-                  ('BIDSVersion', "1.0.1"),
+                  ('BIDSVersion', BIDS_VERSION),
                   ('License', defaults.get('License',
                         "TODO: choose a license, e.g. PDDL "
                         "(http://opendatacommons.org/licenses/pddl/)")),
@@ -87,6 +125,9 @@ def populate_bids_templates(path, defaults={}):
     create_file_if_missing(op.join(path, 'README'),
         "TODO: Provide description for the dataset -- basic details about the "
         "study, possibly pointing to pre-registration (if public or embargoed)")
+    create_file_if_missing(op.join(path, 'scans.json'),
+        json_dumps(SCANS_FILE_FIELDS, sort_keys=False)
+    )
 
     populate_aggregated_jsons(path)
 
@@ -111,7 +152,8 @@ def populate_aggregated_jsons(path):
     # way too many -- let's just collect all which are the same!
     # FIELDS_TO_TRACK = {'RepetitionTime', 'FlipAngle', 'EchoTime',
     #                    'Manufacturer', 'SliceTiming', ''}
-    for fpath in find_files('.*_task-.*\_bold\.json', topdir=path,
+    for fpath in find_files('.*_task-.*\_bold\.json',
+                            topdir=glob(op.join(path, 'sub-*')),
                             exclude_vcs=True,
                             exclude="/\.(datalad|heudiconv)/"):
         #
@@ -120,7 +162,7 @@ def populate_aggregated_jsons(path):
         # TODO: if we are to fix it, then old ones (without _acq) should be
         # removed first
         task = re.sub('.*_(task-[^_\.]*(_acq-[^_\.]*)?)_.*', r'\1', fpath)
-        json_ = load_json(fpath)
+        json_ = load_json(fpath, retry=100)
         if task not in tasks:
             tasks[task] = json_
         else:
@@ -172,10 +214,10 @@ def populate_aggregated_jsons(path):
         placeholders = {
             "TaskName": ("TODO: full task name for %s" %
                          task_acq.split('_')[0].split('-')[1]),
-            "CogAtlasID": "TODO",
+            "CogAtlasID": "http://www.cognitiveatlas.org/task/id/TODO",
         }
         if op.lexists(task_file):
-            j = load_json(task_file)
+            j = load_json(task_file, retry=100)
             # Retain possibly modified placeholder fields
             for f in placeholders:
                 if f in j:
@@ -207,6 +249,10 @@ def tuneup_bids_json_files(json_files):
             # Let's hope no word 'Date' comes within a study name or smth like
             # that
             raise ValueError("There must be no dates in .json sidecar")
+        # Those files should not have our version field already - should have been
+        # freshly produced
+        assert HEUDICONV_VERSION_JSON_KEY not in json_
+        json_[HEUDICONV_VERSION_JSON_KEY] = str(__version__)
         save_json(jsonfile, json_)
 
     # Load the beast
@@ -274,12 +320,13 @@ def add_participant_record(studydir, subject, age, sex):
                             "control group)")])),
                 ]),
                 sort_keys=False)
+
     # Add a new participant
     with open(participants_tsv, 'a') as f:
         f.write(
           '\t'.join(map(str, [participant_id,
-                              age.lstrip('0').rstrip('Y') if age else 'N/A',
-                              sex,
+                              maybe_na(treat_age(age)),
+                              maybe_na(sex),
                               'control'])) + '\n')
 
 
@@ -369,11 +416,6 @@ def add_rows_to_scans_keys_file(fn, newrows):
         os.unlink(fn)
     else:
         fnames2info = newrows
-        # Populate _scans.json (an optional file to describe column names in
-        # _scans.tsv). This auto generation will make BIDS-validator happy.
-        scans_json = '.'.join(fn.split('.')[:-1] + ['json'])
-        if not op.lexists(scans_json):
-            save_json(scans_json, SCANS_FILE_FIELDS, sort_keys=False)
 
     header = SCANS_FILE_FIELDS
     # prepare all the data rows
@@ -404,10 +446,10 @@ def get_formatted_scans_key_row(dcm_fn):
     """
     dcm_data = dcm.read_file(dcm_fn, stop_before_pixels=True, force=True)
     # we need to store filenames and acquisition times
-    # parse date and time and get it into isoformat
+    # parse date and time of start of run acquisition and get it into isoformat
     try:
-        date = dcm_data.ContentDate
-        time = dcm_data.ContentTime
+        date = dcm_data.AcquisitionDate
+        time = dcm_data.AcquisitionTime
         acq_time = get_datetime(date, time)
     except (AttributeError, ValueError) as exc:
         lgr.warning("Failed to get date/time for the content: %s", str(exc))