[ENH] BEP044 - Stim-BIDS #2022

neuromechanist · 2024-12-22T02:29:37Z

This PR addresses #153 by introducing specifications for handling and standardizing stimulus files and their annotations within the BIDS specifications. The changes focus on improving the organization, referencing, and metadata of stimulus files to enhance consistency, reusability, and efficiency.

Important

HTML render of the stimuli page : https://bids-specification--2022.org.readthedocs.build/en/2022/modality-agnostic-files/stimuli.html

Note

The issue: #153
Google Doc
Example PR: bids-standard/bids-examples#433

Known issues:

The validator fails because extensions in /src/schema/rules/files/raw require datatype. Stimuli might be a special data type that can only be present at the root of the dataset. So, the datatype field is missing for now.
There are some style errors by the remark validators

cc: @bids-standard/bep044 and @monique2208

Implement the standardization of stimulus files and their annotations within the BIDS specifications. * **Add new file `src/modality-specific-files/stimuli.md`** - Describe the specifications for the stimuli directory. - Include guidelines for storing stimulus files and their annotations. - Define what goes into `stimuli.tsv/json`, `annotations.tsv/json`, and `stim-<label>.json`. - Use the same style as other modality-specific docs to design the tables, variables, and examples. * **Modify `src/modality-specific-files/task-events.md`** - Add a section detailing the standardization of stimulus files and their annotations within the BIDS specifications. - Include examples of how to use the `stim_file` and `stim_id` columns in `events.tsv` files. - Provide guidelines for storing stimulus files in the `/stimuli` directory. - Expand the definition of the `stim_file` column to include `stim_id`. * **Modify `src/schema/objects/columns.yaml`** - Update the definition of the `stim_file` column to ensure consistency in stimulus file references. - Add the `stim_id` column definition for `events.tsv` files. * **Modify `src/schema/rules/checks/events.yaml`** - Add a check for missing stimulus files declared in `events.tsv`. - Add a check for missing `stim_id` references in `events.tsv`. * **Modify `src/schema/rules/sidecars/events.yaml`** - Specify the `StimulusPresentation` metadata field for `events.tsv` files. - Include the `stim_id` column in the metadata field specifications. * **Modify `src/schema/objects/entities.yaml`** - Add entities described in the document with proper requirement levels and descriptions. * **Modify `src/schema/objects/suffixes.yaml`** - Add suffixes for `{audio, image, video, audiovideo}`. - Include the file extensions and descriptions for each suffix. * **Add new file `src/schema/rules/sidecars/stimulus.yaml`** - Define sidecar tables for `stimuli.tsv/json`, `annotations.tsv/json`, and `stim-<label>.json`. - Use the same style as other modality-specific docs to design the tables. --- For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/neuromechanist/bids-specification?shareId=XXXX-XXXX-XXXX-XXXX).

Add stimuli specifications to BIDS

for more information, see https://pre-commit.ci

src/modality-specific-files/stimuli.md

for more information, see https://pre-commit.ci

…/2022

Remi-Gau · 2025-01-09T18:35:54Z

will do a bit of clean up to get less red in CI and maybe see if we can get the HTML version of the BEP to render

Remi-Gau · 2025-01-09T19:37:11Z

HTML: stimuli page
https://bids-specification--2022.org.readthedocs.build/en/2022/modality-specific-files/stimuli.html

yarikoptic

initial quick pass with little recommendations etc

src/modality-specific-files/stimuli.md

src/modality-specific-files/task-events.md

src/schema/rules/files/raw/audio.yaml

src/schema/objects/suffixes.yaml

src/modality-specific-files/stimuli.md

mkdocs.yml

Quote the complex expression in StimulusIdExists check to prevent YAML from misinterpreting the parentheses and comma as a nested dictionary structure. This was causing test failures across all Python versions in CI.

Handle both suffix-based and stem-based file groups in the filename template rendering code. Previously, the code assumed all groups had suffixes, but stem-based groups like stimuli.tsv and participants.tsv use stem instead.

- Add part__stimuli entity definition for flexible stimulus part labels - Update test to handle stem-based files in stimuli datatype - Update test fixture to use local schema when available - Remove complex entity overrides in favor of context-specific entities

- Add comprehensive pattern matching for stimuli files in template tests - Handle root-level files (stimuli.tsv, annotations.tsv) and entity files - Stimuli files don't follow standard subject/session patterns

- Update metaschema to support stem-based rules in raw files - Add path support to both stemRule and suffixRule for stimuli - Fix stimuli check expression to use valid syntax instead of lambda - Replace for_any/lambda with standard exists/count expression pattern

- Allow stemRule in raw files alongside suffixRule for stimuli support - Add path and entities support to stemRule for completeness - Add path support to suffixRule for custom directory handling - Fix filename rendering to handle stem files without underscore prefix - Maintains backward compatibility with all existing file patterns

Remove invisible Unicode Zero Width Joiner (U+200D) character that was causing UnicodeDecodeError on Windows systems using cp1252 encoding. Maintains all functionality while ensuring Windows compatibility.

- Replace hardcoded examples with filename template macros in stimuli.md - Streamline and clarify stimulus organization section in events.md - Add distinction between stim files and stimuli directory in behavioral experiments

for more information, see https://pre-commit.ci

- Implement make_root_filename_template for directories like /stimuli - Register new macro in main.py following existing patterns - Update stimuli.md to use new root template macro - Addresses placement, prefix, and annotation files issues

- Remove incorrect datatype designation from stimuli rules - Create schema-driven root filename template macro for path-based rules - Update stimuli.md to use proper root template macro - Fix understanding: stimuli is root-level directory, not subject datatype

- Create make_root_filename_template macro for root-level directories like /stimuli - Update bidsschematools to skip path-based rules (without datatypes) in entity table and filename templates - Remove stimuli from entity-table.md as it doesn't follow subject-scoped entity ordering - Dynamically look up entity definitions from schema instead of hardcoding mappings

The test was failing because it tried to access datatypes attribute on all rules, including the new stimuli rules that use path-based organization instead.

… stimuli tables - Add explicit schema path loading to make_sidecar_table to avoid caching issues - Temporarily comment out stimuli.Stimuli tables that need namespace fixes - File tree generation from schema is now working for /stimuli directory

- Fix entity format labels in root template macro (use <label> instead of hardcoded) - Add explicit schema loading to make_columns_table to avoid caching issues - Re-enable all stimuli tables (sidecar and columns) that now work correctly - File tree now properly shows stim-<label>[_part-<label>] format

…macro Add generic root-level template macro for non-datatype directories

- Show .<extension> when more than 2 extensions exist for same suffix - Keep stem files (stimuli.tsv, annotations.tsv) explicit for clarity - Keep events files explicit (only 2 extensions) - Consolidate audio/image/video/audiovideo files (5-6 extensions each)

The <extension> placeholder was being interpreted as an HTML tag and hidden. Now using <extension> for HTML output while keeping <extension> for PDF.

neuromechanist · 2025-09-12T21:45:17Z

Update prior to making PR out of Draft:

Moved Stimuli to Modality Agnostic section.
Added MACROS to render root-level file tree (should hopefully help with other use cases down the road)
Tests are passing for simple (no-annotation) example from [WIP] examples for BEP044, Stim-BIDS bids-examples#433
X Validator should probably be extended to handle event files under stimuli 🤔
Tests are passing

dorahermes

lgtm

yarikoptic · 2025-10-07T20:01:02Z

src/modality-agnostic-files/stimuli.md

+   "raw",
+   path="stimuli")
+}}
+


here it seems to not render entirely as desired - -<label> (and <index>? for part) is missing

neuromechanist added 12 commits December 14, 2024 14:15

edit stimuli description

91df78b

Update task events to include stim-bids spec

0167a41

Add missing entities

3c2357b

handling GH Action errors

b7b19df

yaml-link issues

aa570ac

remark validation

162cd1c

Move extensions to their own place.

1ef76bc

Revise the extensions

e84cac4

YAML lint

1dd9083

Merge pull request #2 from neuromechanist/add-stimuli-specifications

50ad1dd

Add stimuli specifications to BIDS

Merge branch 'bids-standard:master' into master

77f33b0

neuromechanist requested a review from erdalkaraca as a code owner December 22, 2024 02:29

neuromechanist and others added 4 commits December 30, 2024 16:02

Sidecar description is only needed of rhte individual stimulus files.

4978552

[pre-commit.ci] auto fixes from pre-commit.com hooks

80d3077

for more information, see https://pre-commit.ci

tabular information matching the spec.

a335ebf

Edit stimuli.md to reflect the spec more completely

f370c95

VisLab approved these changes Dec 31, 2024

View reviewed changes

neuromechanist and others added 4 commits December 31, 2024 12:24

Implement Kay's comments for clarity

330b69f

[pre-commit.ci] auto fixes from pre-commit.com hooks

ad24800

for more information, see https://pre-commit.ci

Revert adding BEP044 doc, implementing Kay's suggestions.

a5acd54

Merge remote-tracking branch 'upstream/master' into pr/neuromechanist…

238e645

…/2022

Remi-Gau added 4 commits January 9, 2025 19:36

sty

074f5fb

remark

f197bba

fix

02c6f31

fix

3ae9fe1

yarikoptic requested changes Jan 9, 2025

View reviewed changes

Remi-Gau reviewed Jan 10, 2025

View reviewed changes

mkdocs.yml Outdated Show resolved Hide resolved

neuromechanist added 4 commits September 11, 2025 23:26

Remove incorrect stimuli modality mapping

83f620d

Fix stimuli catalog files to use stem pattern

a5a13e6

Separate metadata fields from TSV columns in events rules

a1698d1

Fix YAML and Markdown linting issues

95c76f3

neuromechanist force-pushed the master branch from 95db096 to 95c76f3 Compare September 12, 2025 06:27

neuromechanist and others added 12 commits September 11, 2025 23:32

Fix YAML parsing error in stimuli check rule

b067f3a

Quote the complex expression in StimulusIdExists check to prevent YAML from misinterpreting the parentheses and comma as a nested dictionary structure. This was causing test failures across all Python versions in CI.

Fix rendering for stem-based file groups

c442a51

Handle both suffix-based and stem-based file groups in the filename template rendering code. Previously, the code assumed all groups had suffixes, but stem-based groups like stimuli.tsv and participants.tsv use stem instead.

Update tests to handle stimuli files properly

fc6cff9

- Add comprehensive pattern matching for stimuli files in template tests - Handle root-level files (stimuli.tsv, annotations.tsv) and entity files - Stimuli files don't follow standard subject/session patterns

Fix Windows encoding issue in entities.yaml

f56816c

Remove invisible Unicode Zero Width Joiner (U+200D) character that was causing UnicodeDecodeError on Windows systems using cp1252 encoding. Maintains all functionality while ensuring Windows compatibility.

Improve stimuli specification documentation

016ddb9

- Replace hardcoded examples with filename template macros in stimuli.md - Streamline and clarify stimulus organization section in events.md - Add distinction between stim files and stimuli directory in behavioral experiments

[pre-commit.ci] auto fixes from pre-commit.com hooks

0ee8522

for more information, see https://pre-commit.ci

neuromechanist mentioned this pull request Sep 12, 2025

Add generic root-level template macro for non-datatype directories neuromechanist/bids-specification#3

Merged

neuromechanist added 6 commits September 12, 2025 12:07

fix: Update test_make_filename_template to skip rules without datatypes

4a440e6

The test was failing because it tried to access datatypes attribute on all rules, including the new stimuli rules that use path-based organization instead.

Merge pull request #3 from neuromechanist/feat/generic-root-template-…

c77b7cb

…macro Add generic root-level template macro for non-datatype directories

fix: Escape <extension> placeholder for HTML rendering

bede400

The <extension> placeholder was being interpreted as an HTML tag and hidden. Now using <extension> for HTML output while keeping <extension> for PDF.

neuromechanist requested review from yarikoptic, VisLab and dorahermes September 12, 2025 21:47

dorahermes approved these changes Sep 16, 2025

View reviewed changes

yarikoptic reviewed Oct 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ENH] BEP044 - Stim-BIDS #2022

[ENH] BEP044 - Stim-BIDS #2022

Uh oh!

neuromechanist commented Dec 22, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Remi-Gau commented Jan 9, 2025

Uh oh!

Remi-Gau commented Jan 9, 2025

Uh oh!

yarikoptic left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

neuromechanist commented Sep 12, 2025

Uh oh!

dorahermes left a comment

Uh oh!

yarikoptic Oct 7, 2025

Uh oh!

Uh oh!

[ENH] BEP044 - Stim-BIDS #2022

Are you sure you want to change the base?

[ENH] BEP044 - Stim-BIDS #2022

Uh oh!

Conversation

neuromechanist commented Dec 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Remi-Gau commented Jan 9, 2025

Uh oh!

Remi-Gau commented Jan 9, 2025

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

neuromechanist commented Sep 12, 2025

Uh oh!

dorahermes left a comment

Choose a reason for hiding this comment

Uh oh!

yarikoptic Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

neuromechanist commented Dec 22, 2024 •

edited

Loading