Request : Add compatibility with next gen BRW files

[b-grimaud]

Hi,

3Brain is currently moving towards BRW v4.x, compatible with newer versions of their software but most importantly adapted to multiwell chips, which means that recordings from multiple parallel chips can be stored in the same file.
This implies a major rework of the underlying HDF5 architecture, and the current BiocamIO reader is not compatible.

I got the OK from 3Brain to share their documentation on that file format, but it would still probably be better to not have it publically hosted.

Is there someone I can reach out to that could take a look at that documentation and possibly figure out how to update the reader ?

Thanks !

[JuliaSprenger]

Hi @BptGrm,
thanks for reaching out. This already the right place for this discussion. Do you also already have a dataset in the new format? With a tiny test dataset to be added to GIN we could start extending the current BiocamIO.

[JuliaSprenger]

@alejoe91 Do you have an idea on how to extend this?

[alejoe91]

Hi @JuliaSprenger

Unfortunately not. I think that one will need to explore the new h5 test files and see where they save the relevant stuff. It would be great to ask support from someone from 3Brain directly.

[JuliaSprenger]

@BptGrm Any news on this from your side? As mentioned above, feel free to contribute a small test dataset to GIN to start extending the neo IO.

[b-grimaud]

Hi, thanks for bringing this up again !

I submitted a very basic PR to the HerdingSpikes spike sorter to read newer BrainWave files. This might be a starting point but it's definetly not compatible with the multi-well recording that this format is meant to support.

I dug pretty deep in the file structure at the time but there were still some aspects that I couldn't quite figure out, namely how they manage data chunks.

I tried reaching out to 3Brain about this issue a while ago, but I didn't get any answers. As I mentionned in the original post of this issue, they did send me some documentation on the format that I could send through email.

I have a few recordings in this format that I could share, I'll get started on the GIN upload procedure soon.

[JuliaSprenger]

Hi @BptGrm Thanks for the update. This sounds like it will still take some time and a lot of tinkering to figure out the new format without someone from 3Brain helping out. Unless you require neo supporting the new format I would say we postpone this until 3Brain contributes or someone knowing the format in more detail has a chance to look at it.

[b-grimaud]

I'll ask 3Brain again and see if I can get a more positive answer.

In the meantime, if anyone desperately need to open newer BrainWave file through neo, there's another open source project called Xenon LFP analysis which contains a downsampling tool that can convert those files to the older format. Setting the 'new' sampling rate to the original should do the trick.

[Lychfindel]

Hi all,
I am also very interested for the support of brw 4.x.
I could also access the official documentation of the format, which are very extensive and include also some code snippet to extract the information in python.
How is proceeding the implementation? I'm willing to help and contribute if there is the need.

[b-grimaud]

@Lychfindel As of #1326 it is now supported.

[alejoe91]

@Lychfindel you need to install NEO from sources!

[Lychfindel]

Thank you! This is a great news!
Unfortunately the import of my dataset is still raising errors.
After investigating a little bit I understood that the error comes from the type of raw data saved: while the new implementation handles the .brw file with data type "Raw" (i.e. with no compression applied), it does not handle the other two supported raw data types "WaveletBasedEncodedRaw" and "EventsBasedSparseRaw", which involve some type of data compression.
Do you think it would be possible to include also these raw types even if they are not completely raw? It should be possible to reconstruct the raw signal (with some data loss) even from these types

[b-grimaud]

Looking through the docs again, those formats are quite a bit more complicated.
3Brain does provide some code examples for both types, but I don't know how much work it would be to transpose them into NEO.
The PDF is available on the NeuralEnsemble GIN.