[draft] new reader for curry files, using curry-python-reader code

[dominikwelke]

hi all
as discussed in #13033 here a draft PR to use the official curry reader code.

in a first step i just use the reader as a module (only fixed formatting to force it past pre-commit).
it has some drawbacks (e.g. data is always loaded) and i did not implement all possible data yet (eg hpi, or epoched recordings) but in general it already works pretty well. tested it with all their example data, and one of my own recordings that didnt work in mne before.

it would be great to get some feedback how you want me to proceed with this @drammock @larsoner:

• do we want to stick with the module approach, leave their code untouched and work with the output (would allow easier updating when they push changes)
• or should i merge the code more thoroughly. making it easier to maintain and in terms of clarity

BACKGROUND:
the curry data reader currently cant handle all/newer curry files
plan is port code from the official curry reader into mne-python

for permission see #12855

closes #12795
closes #13033
closes #12855

[agramfort]

is this file the official reader? is this file copied from somewhere?

[dominikwelke]

yes, it's copied from https://github.com/neuroscan/curry-python-reader

the false info you flagged was added by [autofix.ci]
further formatting changes were necessary to pacify pre-commit hook

[dominikwelke]

you discussed this topic with a compumedics dev in #12855

they said they wont supply a pypi or conda version, but we are free to use the code.
the github repo has a BSD3 license applied, but they dont include any note in the file itself

[drammock]

do we want to stick with the module approach, leave their code untouched and work with the output (would allow easier updating when they push changes), or should i merge the code more thoroughly. making it easier to maintain and in terms of clarity

Given that @CurryKaiser refused our offer to help them package up their reader for PyPI / conda-forge, I see two remaining options:

1. "vendor" their code. To make it slightly future-proof, we could write a script (in tools/ I guess) that fetches the latest code from their repo, auto-formats it to make it compatible with MNE's pre-commit requirements, and puts the (formatted but otherwise unmodified) code in mne/io/curry/_vendored.py. (This is basically a manual version of git submodule update because I don't think we should invoke git submodule for this use case.) We then adapt our code in mne/io/curry.py to be a wrapper around their code that basically just gets things into proper MNE container objects; and know that we might need to tweak our wrappers any time the vendored code is updated.

2. Fully incorporate their code. Re-write their reader to align better with our codebase, in terms of variable names, idioms like _check_option or _validate_type, features like preload=False, etc.

Personally I lean toward option 2. I say this because if we're going to try to support curry files, at a minimum we need to be able to fix bugs when they arise, and ideally we should be willing/able to incorporate new features that have high demand from our users (preload=False is an obvious first example). But if we're fixing bugs, do we open PRs to upstream (with no guarantee of responsiveness), or tweak our "thin" wrapper to handle more and more corner cases? Neither option is appealing, so at that point it starts to seem easier to me to just maintain the entire reader ourselves.

[agramfort]

hum... my first reaction is to push a version 0.1 of their package on pypi and rely on this. Basically we maintain a fork and hope that the fork changes are accepted upstream... it feels less hacky and they also have a CI and some testing setup with test_data that I would not duplicate in mne-python...

…

Message ID: ***@***.***>

[drammock]

hum... my first reaction is to push a version 0.1 of their package on pypi and rely on this. Basically we maintain a fork and hope that the fork changes are accepted upstream... it feels less hacky and they also have a CI and some testing setup with test_data that I would not duplicate in mne-python...

that indeed is less hacky than my approach to vendoring. I'd be OK with that outcome, though curious what @larsoner will think.

[larsoner]

I'm fine with that idea but it would be good to get some blessing/permission from them to do this

[drammock]

@CurryKaiser

I'm fine with that idea but it would be good to get some blessing/permission from them to do this

xref to #12855 (comment) where I've asked for confirmation that Compumedics really doesn't want to be the packager and they're OK with us doing it.

[CurryKaiser]

@CurryKaiser

I'm fine with that idea but it would be good to get some blessing/permission from them to do this

xref to #12855 (comment) where I've asked for confirmation that Compumedics really doesn't want to be the packager and they're OK with us doing it.

And nothing has changed, so all good from our side. Sorry we couldn't package it for you. And thank you for working on this!

[dominikwelke]

thanks @CurryKaiser !

ok, sounds like a plan.. I can start working on this again soon, if you give a go @agramfort @drammock @larsoner

I guess the fork should live in the mne-tools org? I have the necessary rights to create it

[larsoner]

Yeah I think so

[drammock]

Yeah I think so

I already made the fork

[agramfort]

ok, sounds like a plan.. I can start working on this again soon, if you give a go @agramfort <https://github.com/agramfort> @drammock <https://github.com/drammock> @larsoner <https://github.com/larsoner>Message ID: ***@***.***>

+1

[drammock]

xref to mne-tools/curry-python-reader#1

[dominikwelke]

i could need some guidance on 2 things:

1. channel locations:
   curry files come with channel locations, and for EEG it was straight forward to build a montage and apply.
   but for MEG it seems i need to use other functions. any pointers would help!
   do i need to populate info["dig"] directly?

2. HPI/CHPI data:
   some MEG files seem to come with these data. how do i store them in the raw object?

a few other things to discuss:

• preload
  easiest would be to not offer preload=False and just load the data to memory.
  a single load_data() call would also be doable with the official reader, but a chunk reader not really (if we dont want to hack it; e.g. load all data and discard large parts). not sure i'm deep enough in the mne codebase to know what the implications are (e.g. computations, plots etc with unloaded data)
  what are your thought?

• epoched files
  the reader code looks as if there could be files with epoched recordings, but there are none among their sample files. do any of you know more about this? otherwise ill ask the curry devs

[dominikwelke]

p.s. and could you remind me how to switch off the CIs when pushing these early commits?

[larsoner]

Push commits with [ci skip] in the commit message and they long / expensive CIs shouldn't run (a few quick ones still will I think)

[larsoner]

... for the cHPI stuff it's probably easiest to load a Neuromag raw FIF file with cHPI info and look at how the info is stored for example in info["hpi_subsystem"]. You can also look at the Info docs, especially the notes. It's not complete but it will help.

For preload, since preload=False is in main it would be a functionality regression to remove it. Once you know how the data are stored on disk and how to read an arbitrary time slice from it, it's really not bad to do the work to make preload=False work. So if you can figure this part out in some function, I can help you fit it into the _read_segment_file code. Since the .py file is only a few hundred lines (a lot of which seems like plotting etc. that we won't use), I'm cautiously optimistic we can figure it out and make it work. And then the python-curry-reader code can really be for reading metadata, annotations, sensor locs, etc. plus knowing where to read the data from disk. We can probably even keep the existing _read_segment_file, it should work in theory...

[dominikwelke]

ok, _read_segment_file does indeed work unchanged.
the reader should now be more/less functional

• I'd still need some guidance on handling/storing the channel locations, esp. for MEG data
• HPI data - looks like I got it wrong - there might not be cHPI data after all, only HPI marker locations provided in different formats depending on the system

[dominikwelke]

@CurryKaiser
thanks for the permission to use the code, also from my side!

in another place you said you might be able to provide us with test files - could we perhaps get a small one with epoched recordings in it (format version shouldn't matter)?
your repository for the python reader contains some test files that the reader interprets as epoched, but they dont seem to really be (perhaps the files were truncated for size)

[CurryKaiser]

Could be that they were truncated, let me check.

[CurryKaiser]

Ok, try these:
EpochedData

[dominikwelke]

thanks for the file @CurryKaiser
fyi, we have now packaged and published the curryreader on PyPI.
it can be installed via pip install curryreader

[dominikwelke]

@drammock @larsoner @agramfort
it is on PyPI but not on conda-forge - how is this case dealt with in MNE? should we also submit it to conda forge?

currently pip install mne[full] fetches it, but conda env create --file environment.yml doesnt

related question:
which pip dependency level in pyproject.toml should this go to? i treated curryreader like the antio package (for ANT neuro files) but this makes it an optional requirement (in mne[full]). i believe this mean it wont be automatically installed when calling pip install mne?

[larsoner]

it is on PyPI but not on conda-forge - how is this case dealt with in MNE? should we also submit it to conda forge?

Yeah use grayskull, it's not too painful, see for example conda-forge/staged-recipes#28279

[dominikwelke]

Yeah use grayskull,

see conda-forge PR: conda-forge/staged-recipes#29754

[dominikwelke]

hi @larsoner @drammock -
this is now doing what it should and it would be great to get some more in-depth revision!
thanks in advance for your feedback :)

specific questions:

• sensor digitization:
  could someone with a better understanding of how MNE stores these have a look?
  curry files can store eeg locations and meg locations, all as metric xyz coordinates (i think). especially in case of meg i'm not 100% confident if i set all FIFF flags correctly, or if there are transformation steps i missed.
  what i implemented is partly copying the previous mne reader version, and a 3d plot of sensor locations (eeg+meg) looks ok.

• tests:
  feel free to suggest improvements, or extensions to better cover my code.

• API 1:
  curry files can store continuous or epoched data. for epoched data, i default to return an instance of Epochs but offer option to return a Raw with annotations (import_epochs_as_annotations=False).
  good like this, or should i change the default / rename the argument?
  (ps. maybe it can also store evoked data - the curryreader returns number of averages per epoch which suggest this might be the case, but i dont have example data for this, so i currently raise NotImplementedError. @CurryKaiser: can you tell us more about this? perhaps provide a small example file with evoked/averaged recordings?)

• API 2:
  apart from reader read_raw_curry i also added defs to read eeg montage (read_dig_curry) and impedance measurements (read_impedances_curry). do i need to add them somewhere to the docs?
  the latter could also be included in https://mne.tools/stable/auto_examples/io/read_impedances.html, i guess?

• potential privacy issue:
  curry files can include amplifier/device information. i currently just dump this to device_info.type.
  in some example files i saw that this info can be very long and include serial numbers and paths to local files.
  as i didnt find any spec for the curry format its not easy to split the string meaningfully, to store this identifiable stuff in the device_info items that are overwritten by MNEs anonymization functions.
  how should i go with this?
  @CurryKaiser: is there a fix structure to the information stored in the AmplifierInfo field that i could use?
  example:

Synamps MEG Headbox 1: Digital(PN:8509 SN:0029) Analog(PN:9227 SN:0020) FW:01 -- Headbox 2: Digital(PN:8509 SN:0063) Analog(PN:9227 SN:0059) FW:01 -- Headbox 3: Digital(PN:8509 SN:0026) Analog(PN:9227 SN:0045) FW:01 -- Headbox 4: Digital(PN:8509 SN:0019) Analog(PN:8548 SN:0012) FW:01 -- Headbox 5: Digital(PN:8509 SN:0078) Analog(PN:8548 SN:0070) FW:01 (SynAmps MEG), LP-Filter: order 2 Butterworth (Configuration: 186 MEG + Quik-Cap Net 128 01_23_2020) || MEG: H1 conf: E:\Curry_Configs\BNI_A_parameters20200220.cfg (Thu Feb 20 17:21:42 2020¶) | H1 sens: ./OrionMEG/MEG_186_system_CURRY_10092019.cfg (Wed Sep 11 02:57:42 2019¶) | H1 geom: ./OrionMEG/MEG_186_10092019 (Mon Sep 30 06:35:12 2019¶)

[dominikwelke]

btw, the style issue says: mne/decoding/base.py:272: unused attribute 'required' (60% confidence)
dont seem to be me?

[larsoner]

sensor digitization:

Yeah I can look a bit deeper... before I do, what's the situation on main? Are sensor positions loaded when you read_raw_curry?

The short version in case you want to think about if things are done correctly:

1. EEG-like sensors are stored in head coord frame, i.e., the right-handed RAS coordinate frame formed by LPA, RPA, and Nasion.
2. MEG-like sensor positions are stored in the MEG coord frame, which has a device-dependent origin (and up/orientation) but is otherwise RAS.
3. info["dev_head_t"] stores the coordinate transformation between the head and MEG coordinate frame. This should be non-None for recordings with humans.

Is this how things work in this PR (and on main)?

[read_impedances_curry] could also be included in https://mne.tools/stable/auto_examples/io/read_impedances.html, i guess?

Sure!

store this identifiable stuff in the device_info items that are overwritten by MNEs anonymization functions.

Seems reasonable to me

curry files can store continuous or epoched data. for epoched data, i default to return an instance of Epochs but offer option to return a Raw with annotations (import_epochs_as_annotations=False).

Why have an option to import as raw? To me this seems like a more general problem that we might want to have an Epochs method epochs.as_raw() that does the wrapping back to a 2D array/structure with annotations or whatever. Can we live without this for now and see if people end up wanting it?

[larsoner]

btw, the style issue says: mne/decoding/base.py:272: unused attribute 'required' (60% confidence)

Only tangentially related... vulture uses hueristics and I think by you removing the required from some function in the curry code, it now sees code in mne/decoding.py with a .required as being unused anywhere 🤷 To work around this just you should be able to add _.required to the vulture_allowlist.py and be good

[dominikwelke]

curry files can store continuous or epoched data. for epoched data, i default to return an instance of Epochs but offer option to return a Raw with annotations (import_epochs_as_annotations=False).

Why have an option to import as raw? To me this seems like a more general problem that we might want to have an Epochs method epochs.as_raw() that does the wrapping back to a 2D array/structure with annotations or whatever. Can we live without this for now and see if people end up wanting it?

i can remove the option, no prob.

[dominikwelke]

store this identifiable stuff in the device_info items that are overwritten by MNEs anonymization functions.

Seems reasonable to me

the point was: its not straightforward to parse and split the string ;)
fwiw there could be anything n there

a few options are:

1. leaving as is, leading to cases with incomplete anonymization
2. acting as if there were structure, and split the strings based on the limited examples i saw
3. ignoring this info and leaving device_info empty
4. dumping everything in one of the fields of device info that's overwritten instead of device_info.type

[dominikwelke]

sensor digitization:

Yeah I can look a bit deeper... before I do, what's the situation on main? Are sensor positions loaded when you read_raw_curry?

yes sensor locations are also loaded in the legacy version. as i said, i reuse parts of this code.

in the PR i do set the coordinate frames as intended (head for eeg, device for meg) and for eeg im generally quite confident everything is fine (including RPA, LPA and Nasion).
for meg not so much - i did set a head-dev transform but i want feedback. thanks for having a look!

[dominikwelke]

@larsoner -
vulture is indeed happy now.
new issue in check_neuromag2ft arose

[larsoner]

For the device stuff, for now we could just put it all in [info["device_info"]["model"]`](https://mne.tools/stable/generated/mne.Info.html)

It's a bit of a misnomer but I think it's a reasonable first step. We can always improve it later by splitting the string or whatever. Another option would be info["description"] but it's nice to leave that for more experiment-related stuff.

For M/EEG alignment, is there a good example or dataset with M/EEG Curry data I can use to check things?

[larsoner]

... this failure seems real:

https://github.com/mne-tools/mne-python/actions/runs/16425669566/job/46415245479?pr=13176

_________________________ test_read_raw_curry_epoched __________________________
mne/io/curry/tests/test_curry.py:86: in test_read_raw_curry_epoched
    ep = read_raw_curry(epoched_file)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<decorator-gen-416>:12: in read_raw_curry
    ???
mne/io/curry/curry.py:588: in read_raw_curry
    fname = _check_curry_filename(fname)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mne/io/curry/curry.py:52: in _check_curry_filename
    raise FileNotFoundError("no curry data file found (.dat or .cdt)")
E   FileNotFoundError: no curry data file found (.dat or .cdt)

[dominikwelke]

... this failure seems real:

this is the epoched file I recently asked you to merge into mne-testing-data..
how can i push the CIs to use the most recent dataset version? i had looked for it but didnt find.

[dominikwelke]

For the device stuff, for now we could just put it all in [info["device_info"]["model"]`]

..that wouldnt solve the anonymisation issue, though. anonymize_info only overwrites serialand site

i could also just add a user-warning, if not empty?
smth like "there can be serial numbers etc in info['device_info']; overwrite if you want to anonymize the dataset."

[larsoner]

Maybe put it in site, then? I don't think it matters too much where we put it

For the testing dataset you change

mne-python/mne/datasets/config.py

Line 90 in bf57d9c

testing="0.161",

mne-python/mne/datasets/config.py

Line 118 in bf57d9c

hash="md5:a32cfb9e098dec39a5f3ed6c0833580d",

[dominikwelke]

For M/EEG alignment, is there a good example or dataset with M/EEG Curry data I can use to check things?

this one (from the official curryreader testing data): HPI.zip

thinking about it, would be good to include this in our testing data. it can improve some other tests, too.
will open a PR

it also a case that has additional HPI data that i cant really make much sense of - first thought it was cHPI, but not so sure anymore after what i read in the curryreader code

[dominikwelke]

@larsoner @drammock this is still waiting for review

[larsoner]

@larsoner @drammock this is still waiting for review

Ahh okay the comment from 2 weeks ago I read as meaning you were going to add to test data, modify code, etc. and then I assumed you would ping for review in another comment. Looks like you modified the code but didn't ping for review until today. Most maintainers don't get notified when commits are made for PRs -- it's waaaay too noisy -- so I didn't realize you had pushed. I should be able to look soon!

[larsoner]

Before I look at the MEG data (hopefully Monday), noticed a lot of red in the tests that I figured I'd ask about now

[larsoner]

I was hoping existing tests would be touched/changed as little as possible... Can you explain why this test had to be removed? Or could it be put back and still work?

[larsoner]

Same thing here (and below) ... why did this have to be removed?

[larsoner]

@dominikwelke so this code, which uses sample's MRI as a surrogate (which isn't a great match but that's okay):

import mne
subjects_dir = mne.datasets.sample.data_path() / "subjects"
raw = mne.io.read_raw_curry("~/Desktop/HPI/HPI.cdt")
print(raw.info["dev_head_t"])
trans = mne.transforms.Transform("mri", "head")
mne.viz.plot_alignment(raw.info, coord_frame="meg", dig=True, meg=("helmet", "sensors"), subject="sample", subjects_dir=subjects_dir, surfaces=dict(head=0.2), trans=trans)

fails on main but on this PR:

So a couple of things are noticeable here. First, the helmet doesn't match the sensors. This is because the sensor defs are wrong:

>>> raw.info["chs"][0]
{'loc': array([-0.04734   ,  0.1175    ,  0.01531   ,  0.        , -0.32730447,
        0.94491893, -0.93714016, -0.32973247, -0.11421394, -0.34895319,
        0.88552147,  0.30673016]), 'unit_mul': 0 (FIFF_UNITM_NONE), 'range': 1.0, 'cal': 1.0, 'kind': 1 (FIFFV_MEG_CH), 'coil_type': 3024 (FIFFV_COIL_VV_MAG_T3), 'unit': 112 (FIFF_UNIT_T), 'coord_frame': 1 (FIFFV_COORD_DEVICE), 'ch_name': 'M01', 'scanno': 1, 'logno': 1}

This does not appear to be a Neuromag system I think (184 channels), so the coil_type is wrong here.

Second, the dev_head_t is None, which shouldn't be the case for most datasets. I saw during reading:

found 2 cHPI samples for 10 coils

I haven't looked into what this means... but if there were 5 HPI coils, and they were defined in the head coordinate frame (5 points) and the MEG/device coordinate frame (same matched 5 points) then these can be used to get a proper info["dev_head_t"] using mne.transforms._fit_matched_points. I don't know the Curry format well enough to know if this is what's needed, but it's possible...