Exdirio #371
Exdirio #371
Conversation
|
I understand that a new version of the NWB standard, together with a new Python library that can both read and write NWB files, will be released at the SfN meeting. I have put further work on NWBIO on hold until the new version comes out. Concerning multiple Segments, in NWBIO I mapped Neo Segments to NWB Epochs. Concerning signals, I mapped AnalogSignal to NWB's MultiChannelTimeSeries. On the subject of this PR, is it ready to merge, or did you just create it as a focus for discussion? |
|
To be clear, this PR is a focus for discussion, about the exdir backend. We
thought that you might be interested in such and thus have a possibility to
create a dataset with the respective IO and look at the API of exdir
through this PR.
We made this IO for two purposes, to test exdir and store our data in the
NWB format without HDF5 until some official python API for NWB was
released. The IO as it is now have several caveats e.g. no linking, not
possible to store only segments, the reader search for the naming of groups
and not the NWB type etc.
Glad to hear that a release of NWB is expected at SfN, maybe we could meet
there for discussion if you are also going? Anyway, if the exdir backend is
to be implemented it could be optional in the NEO NWBIO, where one simply
exchange h5py with exdir.
Concerning multiple segments, we also thought about mapping this to epochs.
However, I am a bit concerned about distinguishing a "segment epoch" from a
"regular epoch".
Concerning signals, I was unaware of the type MultiChannelTimeSeries.
2017-08-28 16:25 GMT+02:00 Andrew Davison <[email protected]>:
… I understand that a new version of the NWB standard, together with a new
Python library that can both read and write NWB files, will be released at
the SfN meeting. I have put further work on NWBIO on hold until the new
version comes out.
Concerning multiple Segments, in NWBIO I mapped Neo Segments to NWB Epochs.
Concerning signals, I mapped AnalogSignal to NWB's MultiChannelTimeSeries.
On the subject of this PR, is it ready to merge, or did you just create it
as a focus for discussion?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#371 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADQn8GfKek28QhRDUyxa72fv-tg654zUks5scs3tgaJpZM4OmoPr>
.
--
Mikkel Elle Lepperød <http://cinpla.org/people/>
Centre for Integrative Neuroplasticity (CINPLA <http://cinpla.org>)
University of Oslo (UiO <http://uio.no>)
Kristine Bonnevies hus, room 2513
|
|
@lepmik
I propose to postpone this PR for later. Best Samuel |
|
Hi! We are in contact with NWB NeurodataWithoutBorders/pynwb#629 discussing ways to implement Exdir as an alternative backend. I think if the exdir should be combined with the nwb structure in a neo io it should be in the nwbio or similar. If you are interested in something similar to the old hdf5io we could use exdir there as well. Anyway, this PR was mainly intended to give you a taste of exdir, so you may close it at your convenience. |
|
@lepmik : this PR is pretty old for now. |
|
Just close it, thanks.
man. 24. jan. 2022, 12:42 skrev Garcia Samuel ***@***.***>:
… @lepmik <https://github.com/lepmik> : this PR is pretty old for now.
Unless you want to port it to rawio I propose to close for now.
—
Reply to this email directly, view it on GitHub
<#371 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA2CP4GWHMEBY74V3IBQYZDUXU3K7ANCNFSM4DU2QPVQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
6 participants
Dear NEO team,
We are currently developing an alternative to HDF5 called exdir. Shortly it is an API that allows you to store data and metadata in an hierarchical structure similar to HDF5. Groups are represented as directories with data sets stored in .npy files and metatada in .yaml files. We aim to keep the simplicity of HDF5 and alleviate known issues such as mentioned in this blog post with a syntax equal to h5py. We are currently finishing the documentation and writing a paper detailing the project.
We use exdir on daily basis with NEO and ExdirIO which conforms to the NWB structure. As exdir is getting more mature we feel that it is time to get feedback on this IO. To use the NWB structure and NEO together was not straightforward and we had to make some compromises. Some of the limitations which maybe should be implemented in this IO is:
One NWB dataset is stored as a block, but can only contain one
Segment. We have been a bit back and forth with this, but since you need aBlockto contain aChannelIndexa NWB dataset should be represented as aBlock. To have multipleSegmentsone could define this asEpochs, but those would have to point to multiple datasets or similar which we felt was to much work to implement for the time being.Unitscan only contain a singleSpikeTrain, this follows mainly from the singleSegmentlimitation.AnalogSignalsare either stored as LFP ifAnalogSignal.description == 'LFP'or just as aAnalogSignalwhich is not fully defined in the NWB structure. Here NWB also defines filtered signals, which is unclear to me how should be implemented in this IO.We tried to run NWBIO to compare output but was unfortunately not able to do that due to errors. Ideally the NWBIO and the ExdirIO could be merged, where the user can chose which backend to use (exdir or h5py). We were originally hoping to base this ExdirIO on the NWB backend, but that was not possible since no nwb python api existed at that time. We are hoping to get an nwb exdir python package implemented in the future.