[ENH] BEP 16: Diffusion modeling derivatives

[arokem]

This transfers work that so far has been done in a separate repository: https://github.com/bids-standard/bids-bep016.

I believe this is not too far from ready to be reviewed by the community, but also seeking input from @Lestropie, @francopestilli, @oesteban, @PeerHerholz.

[arokem]

Forgot to tag @mattcieslak, whose input here also useful. And probably others, please tag and add to this discussion.

[francopestilli]

@effigies @dorahermes @jdtournier @frheault @mattcieslak We are trying to merge the current status of the DWI BEP 16. Would you guys take a look, please? @PeerHerholz

[arokem]

Looking at the required deliverables for BEPs, I see that we might be at the stage to discuss whether we have 1 (the markdown component), and that discussion can lead to work on 2 and 3 (examples, relatively easy; schema, relatively hard, and where we might need some guidance/examples to follow).

[PeerHerholz]

Hey @arokem and everyone,

thx @arokem! Yeah, I think once it's decided that the current markdown and respective information/proposals are in a good status we would create a couple of example datasets (I can update the conversion software respectively). I do agree that the examples should be relatively easy, especially considering that we would provide empty imaging files and focus on the metadata. The schema appears to be a different beast and having someone from the maintainers/developers involved would be great.

[arokem]

One general question: does the validator currently look into the content of data files to validate that their content is sensible? The BEP proposes that data be organized in a particular way within the files (e.g., a particular order of tensor components), and I am wondering whether there is any way that this is currently being validated. If not, maybe that's fine. Ultimately, it is up to users of the spec to produce data that makes sense, but I still wonder whether we can help enforce that.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.71%. Comparing base (73281ae) to head (4cd9b0f).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2211   +/-   ##
=======================================
  Coverage   82.71%   82.71%           
=======================================
  Files          20       20           
  Lines        1608     1608           
=======================================
  Hits         1330     1330           
  Misses        278      278           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[effigies]

One general question: does the validator currently look into the content of data files to validate that their content is sensible?

It validates tsv contents and NIfTI headers.

Validating NIfTI data blocks is likely to be significantly slower, and would probably make sense for a separate tool like miniQC.

What kind of check are you thinking?

[arokem]

One thing that comes to mind is to validate the shape of the data. For example, the tensor should have six elements on the last dimension; CSD coefficients would also have a certain dimensionality that can be checked. This can be done without reading the data. So, if there is precedent for this kind of check, that's good.

[arokem]

Should I be using the macros___make_sidecar_table for the tables that are raising remark linting issues?

[effigies]

Yes, checking NIfTI shapes is doable:

bids-specification/src/schema/rules/checks/func.yaml

Lines 57 to 67 in b9a6673

	BoldNot4d:
	issue:
	code: BOLD_NOT_4D
	message: |
	BOLD scans must be 4 dimensional.
	level: error
	selectors:
	- suffix == "bold"
	- type(nifti_header) != "null"
	checks:
	- nifti_header.dim[0] == 4

You'll want to use make_sidecar_table if it corresponds to a sidecar schema rule, but for describing deeper models, you'll probably want make_metadata_table, e.g.,

bids-specification/src/common-principles.md

Lines 539 to 561 in b9a6673

	<!-- This block generates a metadata table.
	The definitions of these fields can be found in
	src/schema/objects/metadata.yaml
	and a guide for using macros can be found at
	https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
	-->
	{{ MACROS___make_metadata_table(
	{
	"LongName": "OPTIONAL",
	"Description": (
	"RECOMMENDED",
	"The description of the column.",
	),
	"Format": "OPTIONAL",
	"Levels": "RECOMMENDED",
	"Units": "RECOMMENDED",
	"Delimiter": "OPTIONAL",
	"TermURL": "RECOMMENDED",
	"HED": "OPTIONAL",
	"Maximum": "OPTIONAL",
	"Minimum": "OPTIONAL",
	}
	) }}