Merge commit 'v0.10.0-10-g80a6538' (origin/master) into adds_populate_intended_for

Merge is done to update the PR with uptodate CI and testing fixes to simplify local
testing etc

* commit 'v0.10.0-10-g80a6538':
  BF(TST): use caplog instead of capfd for testing if we log a warning
  Travis - Use bionic for the base
  BF(TST): fix the test for annonimization so it actually tests
  ENH: anonymize_sid - do sanity check that anonymization script produced output
  ENH: add HeudiconvVersion to sidecar .json files
  Changelog etc for 0.10.0 release
  Revert "BF(TST): workaround for python 3.5"
  BF(workaround): retry loading json multiple times
  BF: account for datalad.create no_annex removal, deprecated in 0.13.0
  Drop 3.5 (support ended) and add 3.9
  BF: make anonymize_script actually output anything and map determinstically
  RM: reproin - Remove (ancient) DBIC specific craft
  ENH: reproin - consider all accessions (not assume there is 1) in fix_canceled_runs

Author: yarikoptic

.travis.yml

@@ -1,10 +1,11 @@
 # vim ft=yaml
 language: python
+dist: bionic
 python:
-  - 3.5
   - 3.6
   - 3.7
   - 3.8
+  - 3.9
 
 cache:
   - apt

CHANGELOG.md

@@ -4,6 +4,42 @@ All notable changes to this project will be documented (for humans) in this file
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.10.0] - 2021-09-16
+
+Various improvements and compatibility/support (dcm2niix, datalad) changes.
+
+### Added
+
+- Add "AcquisitionTime" to the seqinfo ([#487][])
+- Add support for saving the Phoenix Report in the sourcedata folder ([#489][])
+
+### Changed
+
+- Python 3.5 EOLed, supported (tested) versions now: 3.6 - 3.9
+- In reprorin heuristic, allow for having multiple accessions since now there is
+  `-g all` groupping ([#508][])
+- For BIDS, produce a singular `scans.json` at the top level, and not one per
+  sub/ses (generates too many identical files) ([#507][])
+
+
+### Fixed
+
+- Compatibility with DataLad 0.15.0. Minimal version is 0.13.0 now.
+- Try to open top level BIDS .json files a number of times for adjustment,
+  so in the case of competition across parallel processes, they just end up
+  with the last one "winning over" ([#523][])
+- Don't fail if etelemetry.get_project returns None ([#501][])
+- Consistently use `n/a` for age/sex, also handle ?M for months ([#500][])
+- To avoid crashing on unrelated derivatives files etc, make `find_files` to
+  take list of topdirs (excluding `derivatives/` etc),
+  and look for _bold only under sub-* directories ([#496][])
+- Ensure bvec/bval files are only created for dwi output ([#491][])
+
+### Removed
+
+- In reproin heuristic, old hardcoded sequence renamings and filters ([#508][])
+
+
 ## [0.9.0] - 2020-12-23
 
 Various improvements and compatibility/support (dcm2niix, datalad,
@@ -22,12 +58,12 @@ target output directory during conversion.
   directory (with a `_heudiconv???` suffix, renamed into ultimate target
   name later on), which avoids hitting file size limits of /tmp ([#481][]) and
   helped to avoid a regression in dcm2nixx 1.0.20201102
-- #477 replaced `rec-<magnitude|phase>` with `part-<mag|phase>` now
-  that BIDS supports the part entity
-- #473 made default for CogAtlasID to be a TODO URL
-- #459 made AcquisitionTime used for acq_time scans file field
-- #451 retained sub-second resolution in scans files
-- #442 refactored code so there is now heudiconv.main.workflow for
+- [#477][] replaced `rec-<magnitude|phase>` with `part-<mag|phase>` now
+  hat BIDSsupports the part entity
+- [#473][] made default for CogAtlasID to be a TODO URL
+- [#459][] made AcquisitionTime used for acq_time scans file field
+- [#451][] retained sub-second resolution in scans files
+- [#442][] refactored code so there is now heudiconv.main.workflow for
   more convenient use as a Python module
 
 ### Fixed
@@ -350,6 +386,11 @@ TODO Summary
 [#368]: https://github.com/nipy/heudiconv/issues/368
 [#373]: https://github.com/nipy/heudiconv/issues/373
 [#485]: https://github.com/nipy/heudiconv/issues/485
+[#442]: https://github.com/nipy/heudiconv/issues/442
+[#451]: https://github.com/nipy/heudiconv/issues/451
+[#459]: https://github.com/nipy/heudiconv/issues/459
+[#473]: https://github.com/nipy/heudiconv/issues/473
+[#477]: https://github.com/nipy/heudiconv/issues/477
 [#293]: https://github.com/nipy/heudiconv/issues/293
 [#304]: https://github.com/nipy/heudiconv/issues/304
 [#306]: https://github.com/nipy/heudiconv/issues/306
@@ -391,3 +432,12 @@ TODO Summary
 [#464]: https://github.com/nipy/heudiconv/issues/464
 [#480]: https://github.com/nipy/heudiconv/issues/480
 [#481]: https://github.com/nipy/heudiconv/issues/481
+[#487]: https://github.com/nipy/heudiconv/issues/487
+[#489]: https://github.com/nipy/heudiconv/issues/489
+[#491]: https://github.com/nipy/heudiconv/issues/491
+[#496]: https://github.com/nipy/heudiconv/issues/496
+[#500]: https://github.com/nipy/heudiconv/issues/500
+[#501]: https://github.com/nipy/heudiconv/issues/501
+[#507]: https://github.com/nipy/heudiconv/issues/507
+[#508]: https://github.com/nipy/heudiconv/issues/508
+[#523]: https://github.com/nipy/heudiconv/issues/523

docs/conf.py

@@ -26,7 +26,7 @@
 # The short X.Y version
 version = ''
 # The full version, including alpha/beta/rc tags
-release = '0.9.0'
+release = '0.10.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/installation.rst

@@ -26,7 +26,7 @@ If `Docker <https://docs.docker.com/install/>`_ is available on your system, you
 can visit `our page on Docker Hub <https://hub.docker.com/r/nipy/heudiconv/tags>`_
 to view available releases. To pull the latest release, run::
 
-    $ docker pull nipy/heudiconv:0.9.0
+    $ docker pull nipy/heudiconv:0.10.0
 
 
 Singularity
@@ -35,4 +35,4 @@ If `Singularity <https://www.sylabs.io/singularity/>`_ is available on your syst
 you can use it to pull and convert our Docker images! For example, to pull and
 build the latest release, you can run::
 
-    $ singularity pull docker://nipy/heudiconv:0.9.0
+    $ singularity pull docker://nipy/heudiconv:0.10.0

docs/usage.rst

@@ -82,7 +82,7 @@ The second script processes a DICOM directory with ``heudiconv`` using the built
     DCMDIR=${DCMDIRS[${SLURM_ARRAY_TASK_ID}]}
     echo Submitted directory: ${DCMDIR}
 
-    IMG="/singularity-images/heudiconv-0.9.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.10.0-dev.sif"
     CMD="singularity run -B ${DCMDIR}:/dicoms:ro -B ${OUTDIR}:/output -e ${IMG} --files /dicoms/ -o /output -f reproin -c dcm2niix -b notop --minmeta -l ."
 
     printf "Command:\n${CMD}\n"
@@ -97,7 +97,7 @@ This script creates the top-level bids files (e.g.,
     set -eu
 
     OUTDIR=${1}
-    IMG="/singularity-images/heudiconv-0.9.0-dev.sif"
+    IMG="/singularity-images/heudiconv-0.10.0-dev.sif"
     CMD="singularity run -B ${OUTDIR}:/output -e ${IMG} --files /output -f reproin --command populate-templates"
 
     printf "Command:\n${CMD}\n"

heudiconv/bids.py

@@ -28,6 +28,7 @@
     remove_suffix,
     remove_prefix,
 )
+from . import __version__
 
 lgr = logging.getLogger(__name__)
 
@@ -45,6 +46,10 @@
         ("Description", "md5 hash of UIDs")])),
 ])
 
+#: JSON Key where we will embed our version in the newly produced .json files
+HEUDICONV_VERSION_JSON_KEY = 'HeudiconvVersion'
+
+
 class BIDSError(Exception):
     pass
 
@@ -180,7 +185,7 @@ def populate_aggregated_jsons(path):
         # TODO: if we are to fix it, then old ones (without _acq) should be
         # removed first
         task = re.sub('.*_(task-[^_\.]*(_acq-[^_\.]*)?)_.*', r'\1', fpath)
-        json_ = load_json(fpath)
+        json_ = load_json(fpath, retry=100)
         if task not in tasks:
             tasks[task] = json_
         else:
@@ -235,7 +240,7 @@ def populate_aggregated_jsons(path):
             "CogAtlasID": "http://www.cognitiveatlas.org/task/id/TODO",
         }
         if op.lexists(task_file):
-            j = load_json(task_file)
+            j = load_json(task_file, retry=100)
             # Retain possibly modified placeholder fields
             for f in placeholders:
                 if f in j:
@@ -267,6 +272,10 @@ def tuneup_bids_json_files(json_files):
             # Let's hope no word 'Date' comes within a study name or smth like
             # that
             raise ValueError("There must be no dates in .json sidecar")
+        # Those files should not have our version field already - should have been
+        # freshly produced
+        assert HEUDICONV_VERSION_JSON_KEY not in json_
+        json_[HEUDICONV_VERSION_JSON_KEY] = str(__version__)
         save_json(jsonfile, json_)
 
     # Load the beast

heudiconv/external/dlad.py

@@ -58,7 +58,7 @@ def add_to_datalad(topdir, studydir, msg, bids):
             ds_ = dl.create(curdir_, dataset=superds,
                          force=True,
                          # initiate annex only at the bottom repository
-                         no_annex=isubdir<(len(subdirs)-1),
+                         annex=isubdir==(len(subdirs)-1),
                          fake_dates=True,
                          # shared_access='all',
                          )