[ENH] microelectrode electrophysiology specification (BEP032) #1705

yarikoptic · 2024-02-14T15:24:13Z

Replaces #1352 submitted from a fork outside of bids-specification.

Add specification for microelectrode electrohpysiology datasets based on the BEP032 proposal. old google doc

Note

We meet regularly and everyone is welcome

Next meeting: insert date on URL to join

Communication channel: https://framalistes.org/sympa/info/neuroscience-data-structure

Tip

HTML preview of this BEP

To use this WiP schema on sample datasets, see https://deno.land/x/[email protected]#modifying-and-building-a-new-schema on how to use "stock" bids-validator with a custom schema. (attn @TheChymera)

DONEs

TODOs

Issues this PR would likely to address

Fixes run node tests in github actions legacy-validator#1375

Issues to see being addressed while working on this BEP (likely to move above) or not (moved below):

Other issues which relate but not in scope here and provided for reference/backreference

Spreadsheet with correspondence to ProbeInterface: https://docs.google.com/spreadsheets/d/1O0bZzD-n4MjR68r1GlcH3d2JLXBLAU1PfsDceD3IPeo/edit?usp=sharing

… _acq-

src/schema/rules/files/raw/channels.yaml

TheChymera · 2024-03-06T16:39:46Z

src/schema/objects/columns.yaml

    for privacy purposes.
  type: number
  unit: year
+alpha_rotation:


I'd really recommend renaming the rotation axes to yaw, roll, and pitch (that would be the analogous angle order). There was no consensus either way on the google docs discussion. Someone said both are confusing, which I guess might be expected, but alpha, beta, gamma, are just more confusing...

I agree -- very adhoc. Let's discuss in google doc

https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

Could you chime in? I think the other guy commenting might be amenable to accepting this as well.

I agree that both are confusing, but at least alpha, beta and gamma are SO confusing that everyone realizes that additional specification is needed to define them properly. With roll, yaw and pitch it seems at first that all is clear, until you have a number of different people go through different use cases. See the more challenging examples that I posted on the google doc under https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

@robertoostenveld sorry, I only now saw the update, reply there. I don't think confusion on purpose is good in this case, because the documentation is very eclectic so we might be sending people down rabbit holes. Wiki, where people will invariably go first, does a particularly poor job explaining both euler and TB angles for the casual non-mathematics-versed user. The only thing that wiki has going for it here are the aircraft animations on the TB page. Yaw, Pitch, Roll, will be intuited correctly as long as we specify the starting postion. That we can do (1) as (I think, it's pretty vague) is currently proposed, aligning the implant with the world coordinate system (meaning most implants will be at yaw 0 pitch -90 roll 0) or (2) relative to the implantation stereotax normal (meaning most implants will be at 0 0 0).

For comparison, Euler commonly has the normal pointing up so most implants will be.... 0 -180 0 🤔

in the last BEP032 meeting we discussed further and wanted to follow this approach now (based on the Allen Inst. standard and IBL standard):

Assumption: x,y,z is posterior, ventral, right (unit needs to be specified).

Translational origin (0,0,0) needs to be defined (typically Bregma for rodents).

Rotational origin (0,0,0) is the probe facing up with the tip facing forward. Rotations are all clockwise in degrees and around the tip. For multi-shank probes, the “tip” of the probe is defined as the end of the left shank if you are looking at the electrodes.

yaw: clockwise when looking down

pitch: In the direction of the electrode face

roll: clockwise when looking down at the probe

The depth (unit needs to be specified) is a translation in the direction that the tip is pointing.

We need to add a “probe_model”, which references probeinterface_library

versioning of probe library files (raised here)

annotation of “tip” position (raised here)

NOTE: We need to change the electrode x,y,z.
X,y,z in BIDS refers to location in brain, not on probe.

src/schema/objects/columns.yaml

TheChymera · 2024-03-12T01:33:27Z

Also relevant if you'd like to comment. → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIzHGpUU

TheChymera · 2024-03-12T01:57:39Z

Also I forgot to link to this when I wrote it → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIGPAMOw

src/schema/rules/files/raw/ephys.yaml

…t was missing

src/schema/objects/datatypes.yaml

src/schema/objects/modalities.yaml

src/schema/objects/suffixes.yaml

…ication into bep032

bendichter · 2025-10-07T22:59:43Z

src/schema/objects/columns.yaml

+    Point of the probe that is described by the probe coordinates and
+    on which the yaw, pitch, and roll rotations are applied.


Suggested change

Point of the probe that is described by the probe coordinates and

on which the yaw, pitch, and roll rotations are applied.

Point of the skull that acts as the origin of the surgical coordinate system. If AP, DV, and ML are present and this field is omitted, it as assumed to be "bregma."

bendichter · 2025-10-07T23:03:25Z

src/schema/objects/entities.yaml

    In the case of long data recordings that exceed a file size of 2Gb,
-    `.fif` files are conventionally split into multiple parts.
-    Each of these files has an internal pointer to the next file.
+    `.fif`, `.nwb`, `.nix` files are conventionally split into multiple parts.


Suggested change

`.fif`, `.nwb`, `.nix` files are conventionally split into multiple parts.

Files with a lot of data may be split into multiple parts.

Some file types, such as `.fif`, have internal pointers to the next file

… BIDS conventions

ree-gupta · 2025-10-14T12:18:17Z

I think any of these is probably fine, ultimately.

If I understand it properly, the idea is for the files in /probes/ to be (in principle) uploadable to a remote store as-is and removed from the dataset. If that's the case, then keeping them as standalone documents and not nested inside a probes.json makes more sense to me.

In terms of connecting them to your probes.tsv file through JSON, you can still use TermURL along with BIDS URIs:
"model": {
    "Levels": {
        "neuronexus1": {
            "Description": "A 1-shank probe",
            "TermURL": "https://raw.githubusercontent.com/SpikeInterface/probeinterface_library/refs/heads/main/neuronexus/A1x32-Poly3-10mm-50-177/A1x32-Poly3-10mm-50-177.json"
        },
        "neuronexus2": {
            "Description": "new experimental probe",
            "TermURL": "bids::probes/neuronexus2.json"
        },
    }
}
bids:: prefix means the file is inside the current dataset, and probes/neuronexus2.json is the root-relative path to the file.

Does that fill in the remaining gaps?

This looks good to me too! I had just a small question about how should the probes directory at root be correctly recognized by the schema/validator? Would you suggest adding it to src.schema.objects.files.yaml? like how stimuli directory is specified?

…a little

yarikoptic · 2025-10-15T14:21:03Z

bids:: prefix means the file is inside the current dataset, and probes/neuronexus2.json is the root-relative path to the file.

@effigies so you suggest to establish yet another folder with structured data but which will not be formalized by BIDS in terms of schema/structure under it? or should we strive to come up with some consistent with BIDS overall and/or like e.g. BEP044 approaches to formalize under stimuli/ - https://bids.neuroimaging.io/extensions/beps/bep_044.html and consider making it probes/probe-<label>_probeinterface.json so we

have it explicitely stating to be in probeinterface format
have probe-<label> to be able to say probe_id (probe-<label>) eventually (and <label> here I think matches the "name")

effigies · 2025-10-15T15:00:29Z

@ree-gupta Apologies for the delay. I had unsubscribed so that I would see mentions but not other activity on this PR, so I did not see your response. Please @ me in the future, as I will be doing so again after today.

I had just a small question about how should the probes directory at root be correctly recognized by the schema/validator? Would you suggest adding it to src.schema.objects.files.yaml? like how stimuli directory is specified?

Yes, that or phenotype are the most direct parallels.

@yarikoptic

so you suggest to establish yet another folder with structured data but which will not be formalized by BIDS in terms of schema/structure under it?

The probes are a separate standard with their own contents and file-naming conventions. The intended progression appears to be:

Stick new probe definitions into /probes and refer to them with bids::probes/*.
Upload probes as-is to the global probe namespace.
Replace the bids::probes/ prefix with a full URL prefix.

Given that they already have a BIDS-incompatible naming convention (there are hyphens in the names), it does not seem particularly appealing to me to put an effort to come up with a mapping. Any tool that wants to find the probe definitions is going to need to resolve a URI in any case, so querying using BIDS entities doesn't seem like it adds much utility.

We can probably check that the file name follows probe interface conventions as a schema check, if not a normal filename rule.

That said, if making the directory more BIDSy feels like a good use of people's time, I'm not here to stop you.

yarikoptic

some recommendations

yarikoptic · 2025-10-15T14:52:36Z