feat(schema): Add meta.templates and allow for composing multiple references (#2140)

* feat(schema): Add meta.templates

* fix(test): Replace removed schema location

* feat(schema): Move more imaging data into imaging.yaml

* feat(schema): Remove redundant anatomical derivative rules

* feat(schema): Expand use of raw templates

* rf: Dereference while looking up references

* doc: Document null deletion

* schema: Factor out common PET and MRS entities

Author: effigies

src/metaschema.json

@@ -71,6 +71,9 @@
             "additionalProperties": false
           }
         },
+        "templates": {
+          "type": "object"
+        },
         "versions": {
           "type": "array",
           "items": {

src/schema/README.md

@@ -167,21 +167,43 @@ references (the cases in which they are used will be presented later):
     (which are in turn references to individual values), and the references inside `GeneticLevel.anyOf` indicate that there may be a single
     such value or a list of values.
 
-1.  In [`rules.files.deriv.preprocessed_data`][preprocessed_data]:
+1.  In [`rules.files.deriv.imaging`](./rules/files/deriv/imaging.yaml):
     ```YAML
-    anat_nonparametric_common:
-      $ref: rules.files.raw.anat.nonparametric
+    anat_parametric_volumetric:
+      $ref: rules.files.raw.anat.parametric
       entities:
-        $ref: rules.files.raw.anat.nonparametric.entities
-        space: optional
-        description: optional
+        $ref:
+          - meta.templates.deriv.volumetric.entities
+          - rules.files.raw.anat.parametric.entities
     ```
     Here, the derivative datatype rule starts by copying the raw datatype rule
     `rules.files.raw.anat.nonparametric`.
     It then *overrides* the `entities` portion of that rule with a new object.
-    To *extend* the original `entities`, it again begins
-    by referencing `rules.files.raw.anat.nonparametric.entities`,
-    and adding the new entities `space` and `description`.
+    To *extend* the original `entities`, it composes
+    `meta.templates.deriv.volumetric.entities`
+    and `rules.files.raw.anat.nonparametric.entities`.
+    When multiple references are aggregated, the first reference takes
+    precedence.
+
+    Note also that `value: null` can be used to "delete" a key from a template.
+    For example, in `rules.files.raw.events`:
+
+    ```YAML
+    events__pet:
+      $ref: rules.files.raw.events.events
+      datatypes:
+        - pet
+      entities:
+        $ref: meta.templates.raw.task.entities
+        tracer: optional
+        reconstruction: optional
+        # Most events allow acquisition, PET doesn't
+        acquisition: null
+    ```
+
+    This technique should be used judiciously, preferring semantic clarity to brevity.
+    Templates should be expected to grow as BIDS evolves,
+    and should thus be used only where those changes should propagate.
 
 ### Expressions
 
@@ -1074,5 +1096,4 @@ ensuring consistency across the entire schema directory. Validation of the schem
 incorporated into the CI, so any changes that are inconsistent will be flagged before
 inclusion.
 
-[preprocessed_data]: https://github.com/bids-standard/bids-specification/tree/master/src/schema/rules/files/deriv/preprocessed_data.yaml
 [tabular files]: https://bids-specification.readthedocs.io/en/stable/common-principles.html#tabular-files

src/schema/meta/templates.yaml

@@ -0,0 +1,148 @@
+---
+# This section is unstructured, for inclusion in rules by reference
+
+# Entities that apply to most raw files
+# Include thus:
+#
+# filerule:
+#   entities:
+#     $ref: meta.templates.raw.base.entities
+#     task: optional
+#     ...
+#   datatypes:
+#     ...
+raw:
+  base:
+    entities:
+      subject: required
+      session: optional
+
+  recording:
+    entities:
+      $ref: meta.templates.raw.base.entities
+      acquisition: optional
+      run: optional
+
+  task:
+    entities:
+      $ref: meta.templates.raw.recording.entities
+      task: required
+
+  mri:
+    entities:
+      $ref: meta.templates.raw.recording.entities
+      ceagent: optional
+      reconstruction: optional
+      chunk: optional
+
+  epi:
+    entities:
+      $ref: meta.templates.raw.mri.entities
+      direction: optional
+      part: optional
+
+  pet:
+    entities:
+      $ref: meta.templates.raw.base.entities
+      task: optional
+      tracer: optional
+      reconstruction: optional
+      run: optional
+
+  mrs:
+    entities:
+      $ref: meta.templates.raw.recording.entities
+      task: optional
+      nucleus: optional
+      volume: optional
+      reconstruction: optional
+
+# Entities that apply to most derivative files
+# Include thus:
+#
+# filerule:
+#   entities:
+#     $ref: meta.templates.deriv.base.entities
+#     task: optional
+#     ...
+#   datatypes:
+#     ...
+deriv:
+  base:
+    selectors:
+      - dataset.dataset_description.DatasetType == 'derivative'
+    entities:
+      subject: optional
+      session: optional
+      description: optional
+
+  spatial:
+    $ref: meta.templates.deriv.base
+    entities:
+      $ref: meta.templates.deriv.base.entities
+      space: optional
+
+  # Imaging derivatives
+  volumetric:
+    $ref: meta.templates.deriv.spatial
+    extensions:
+      - .nii.gz
+      - .nii
+      - .json
+    entities:
+      $ref: meta.templates.deriv.spatial.entities
+      resolution: optional
+
+  surface:
+    $ref: meta.templates.deriv.spatial
+    entities:
+      $ref: meta.templates.deriv.spatial.entities
+      hemisphere: optional
+      density: optional
+
+  mask:
+    $ref: meta.templates.deriv.volumetric
+    suffixes:
+      - mask
+    entities:
+      $ref: meta.templates.deriv.volumetric.entities
+      label: optional
+
+  dseg:
+    $ref: meta.templates.deriv.volumetric
+    extensions:
+      - .nii.gz
+      - .nii
+      - .tsv
+      - .json
+    suffixes:
+      - dseg
+    entities:
+      $ref: meta.templates.deriv.volumetric.entities
+      segmentation: optional
+
+  probseg:
+    $ref: meta.templates.deriv.volumetric
+    extensions:
+      - .nii.gz
+      - .nii
+      - .json
+    suffixes:
+      - probseg
+    entities:
+      $ref: meta.templates.deriv.volumetric.entities
+      segmentation: optional
+      label: optional
+
+  dseg_surface:
+    $ref: meta.templates.deriv.surface
+    extensions:
+      - .label.gii
+      - .dlabel.nii
+      - .tsv
+      - .json
+    suffixes:
+      - dseg
+    entities:
+      $ref: meta.templates.deriv.surface.entities
+      segmentation: optional