Replace "inheritance" with "summarization" principle

[yarikoptic]

Status/TODOs:

• already in effect and RECOMMENDED in BIDS 1.0 as of 1.10.1 thanks to
  • [ENH] Inheritance Principle: Advise against value overloading. Plan to deprecate support for overloading for BIDS 2.0 bids-specification#1834
• change to REQUIRED for BIDS 2.0 . May be
  • define on how it could be overridden "explicitly" (in some overrides.json or field within dataset_description.json) to be explicit; although could be also done by bids-validator config kinda to suppress

It is a next step to the discussion which happened in

• Remove inheritance rule #36

On a recent road-trip with @effigies we briefly discussed it and so far did not see a show stopper but it would require more minds to analyze.

ATM one of the problems of inheritance principle is unclear semantic in case of a value to be modified down the hierarchy: order can be unclear in case of multiple "candidate" files, unclear how to "remove" a value, etc.
And overall for a human it is cumbersome to "gather" the final value since for a file down the hierarchy someone needs to go through all possibly inherited files to arrive at the final value. But what if we take my suggestion in aforementioned issue further:

• retain ability to "chain" candidates for metadata from higher to lower levels as in current inheritance principle
• completely disallow overloading the value at lower (deeper in hierarchy) levels Corollaries:
  • if present at different levels (e.g. entire dataset and then specific sidecar .json) - value must be identical/consistent across all levels of inheritance, or otherwise not given at any higher level
  • if particular subject/session has some different value from the others as defined at higher (dataset) level, we need to remove that value from higher level and define at lower (e.g. subject/session) level

It will be a (now doable) job for a validator to ensure that all duplicated (across levels, if any) metadata is consistent.

As a result we would provide user a convenience that looking at top level metadata file provides a "guaranteed" correct metadata across all subject sessions, which is not the case currently as we can change it following the order of inheritance.

• FWIW, we already do something like that in heudiconv, where top level task-*_bold.json files collate all identical values across subject/sessions -- makes it easy to see what is common (e.g. scanner ID etc)
• Conceptually is what we have in BIDS ATM, e.g. participants.tsv summarizes metadata across participants and we expect it to be consistent with possible other phenotypic information to be found in subject/sessions.
  • Hence I think it also relates to BEP036 (Phenotypic Data Guidelines), attn @surchs @ericearl (I just now created @bids-standard/bep036 team) where the idea circles to be able to "segregate" metadata into subject/session level while keeping consistently in the top level (under phetotype/ folder).
• It somewhat would allow for easier composition of Allow composition of a BIDS dataset (dataset level) from smaller (subj or subj/ses) level #59. Again -- metadata present on higher level would remain consistent with the lower, which would be easier to achieve (copy) and ensure (validator).

Attn @Lestropie as he has spent most time to improve Inheritance principle definition, and @dorahermes who is an active proponent and its user: do you think such "simplification" (removal of "value overload") of inheritance would simplify and remain usable? Or may be I do not see some common use case such additional "restriction" would disallow?

I think it might be worth writing some checker and apply it across all openneuro datasets to see if we run into such data "overloads". What would be a tool/functionality which implements inheritance principle already "closest to the bible", e.g. which pretty much would return a list of lists of .json/.tsv files in their "inherited" bundles? (specific code examples would be welcome)

Edits:

• might cause trouble with Allow composition of a BIDS dataset (dataset level) from smaller (subj or subj/ses) level #59 since we do need to overload value. But this file isn't really subject to inheritance principle in it's current formulation although is the information pertinent to all files
• @candleindark proposed: "make it that common metadata MUST be pulled up to the higher level"
  • pros: would make it easier for human "users" of datasets.
  • cons: might make it more difficult to curate such datasets since shifting metadata "down" would be needed more often.

[ericearl]

@yarikoptic Thanks for this thoughtful issue and for the @bids-standard/bep036 team. I can do the check in a month or so for how inheritance is currently being used in OpenNeuro, thanks to Datalad's datalad clone ///openneuro-ability, of course!

I mostly like the idea above, except maybe I'm confused about one thing. Let's say up top there's a task-rest_bold.json with most of the parameters put out by dcm2niix. Then down below in 20 subject's func directories there's a disagreement with FlipAngle or EchoTime or both between these 20 and a separate 5 subject's task-rest bold JSONs. I believe you're saying that EVERY subject's JSON in this scenario has to have FlipAngle and EchoTime in it though only 5 subjects differ. dcm2niix doesn't care that you shouldn't duplicate fields at lower levels. So you end up with a need to filter most of your JSONs in the subjects func folders for common metadata... This might be difficult for many new users especially.

I think whether we move forward with either the inheritance principle OR the summarization principle, the call is for tools to support either of them. If one small set of tools could be created to support either, whichever one makes it to the gate first could be most easily adopted. This is why I've had creating a set of inheritance software tools has been on my BIDS maintainers desirables list for a long time. Thoughts?

[yarikoptic]

Let's say up top there's a task-rest_bold.json with most of the parameters put out by dcm2niix.

to be precise: it is not dcm2niix which places/creates such a file at the top level. It is a BIDS dataset "owner" who decides to take all or some fields from dcm2niix-produced sidecar .json for a specific .nii.gz and place that selection at the top of the dataset. So it is for some script/user to decide which fields to do copy to that file, dcm2niix is not really a "player" here.

I believe you're saying that EVERY subject's JSON in this scenario has to have FlipAngle and EchoTime in it though only 5 subjects differ.

Correct. Even if a single subject differs - such metadata should not be present at the level where it is not common for all levels below. Possible solutions:

• remove that subject/section or scan as it is inconsistent with other scans for the task
• if it is decided permissible to have such "divergent" acquisitions across subjects - contain such metadata only at the subject/session level (e.g. could have task-rest_bold.json on top level with all common, sub-X/task-rest_bold.json with subject specific, sub-X/ses-Y/task-rest_bold.json with subject/session specific, but must not be present with different values across levels).

So you end up with a need to filter most of your JSONs in the subjects func folders for common metadata... This might be difficult for many new users especially.

no -- new users just should not bother creating top level task-rest_bold.json with anything which is not common to all files underneath and they would stay "BIDS compliant". Then they could make use of some tools (e.g. here is function in heudiconv - populate_aggregated_jsons) to collect / rewrite top level .json files with only common metadata, or just write what they know is common (validator would verify that no conflicting/differing values present).

Re support -- correct, tools support would be needed... BUT "summarization" principle is just a more restricted case of inheritance if I see it right, so in principle any tool supporting current inheritance should work with "summarization" without any change.

An line of thought on .tsv + .json duality...

.tsv's we have are pretty much a case of summarization (as placing in a tabular structure within a single file) for entries where metadata could be different (e.g. age for a subject)... i.e. in participants.tsv we summarize commonalities and differences between participants. Overall we get {entitities}.tsv summarizing (flat list of) metadata fields typically (but not necessarily) different between values for that entity. In {entity}-{value}_{suffix}.json files we are providing what is common for that {value} (and paired with datatype {suffix}), and typically when we do not have such {entity}-{value}/ folder level separation (related #54), since then we would place common data and metadata under that folder.
Overall "gather metadata for {entity} of {value}" algorithm should load metadata from {entities}.tsv, and all applicable {entity}-{value}*.json. Any inconsistency in values make "order of loading" important and thus possibly ambiguous. They also make it mandatory to read all the files to get the ultimate value, as opposed to the proposed here case -- first loaded value is "good enough" since they all must be the same: age of participant from participants.tsv should be consistent with any other age loaded from e.g. somewhere in phenotype/ (shhh about multiple sessions etc...)

[Lestropie]

completely disallow overloading the value at lower (deeper in hierarchy) levels.

I'm a fan.
This would:

• Simplify the description of the inheritance principle itself
• Simplify software that wants to read all relevant sidecar data; they could just load all relevant JSONs in any directories in any order into a single dictionary
• Be permissive of having multiple applicable sidecar files within a single directory level, which I would very much like to be able to utilise in BEP016
• Be more faithful to the prospect of creating a piece of software that would analyse a BIDS dataset, identify sidecar data that is consistent and therefore promote it up the inheritance tree, thereby deferring utilisation of the inheritance principle entirely to software.
  (It sounds like you've already got a limited instance of this in heudiconv; I'd like for there to be something that does this across the board)

do you think such "simplification" (removal of "value overload") of inheritance would simplify and remain usable?

I would perhaps pose a different question. There's a bifurcation in opinions on the inheritance principle. I've personally been pushing for making it more powerful, which required improvement to the definition of current behaviour in order to facilitate the subsequent augmentation. Others would prefer that the whole principle disappear entirely, and all metadata relevant to a data file be present in the sidecar file.

The way I would therefore look at this proposal is: if the capacity for value overloading (specifically a present value at a higher level being overridden at a lower level) were to be removed, would this sway those previously opposed to the inheritance principle toward its preservation? So that's actually a question directed not at me but at others.

[yarikoptic]

@Lestropie, a birdie said that you might be participating in BIDS hackathon (if only virtually)? Would you be interested to work on this one. It can already be done as a PR against

• BIDS-2.0 bids-specification#1775

similar to WiP I just started

• BIDS 2̶.̶0̶1.0: flex BIDS layout (bids-2-devel/issues/54) bids-specification#1809

[Lestropie]

I'm not aiming to participant in the hackathon and looking for a project so much as want to take on automating the use of inheritance and see the hackathon as a potential way to motivate project commencement and get other people on board. I want to write it up as a proposal somewhere, but wasn't quite sure where would be best: it's not yet guaranteed that I'll be able to do the Hackathon, and what I have in mind is also not specific to BIDS 2.0. Maybe I should create an empty repository and start listing issues there.

[Lestropie]

See https://github.com/Lestropie/IP-me/issues for my current intentions on the topic.