neo, blackrockio, for version 0.41, comments mask is 0xFFFF, not 0xFFF

[neoana]

neo, blackrockio, for version 0.41, comments mask is 0xFFFF, not 0xFFF.
and for version Neo 0.5.0-alpha 1, blackrockio, comments data is not read

[samuelgarcia]

I can push 0xFFFF in 0.4.x

In neo-0.5.x (master) the blackrockIO is a different beast completly rewritten to deal with many version of the file format.

We would be happy to get a contribution for including again event comments .

I ask authors.

@JuliaSprenger :
@mdenker :
In old BlackrockIO event generated by comments were catch by this (line 267 in neo0.4.x):

        mask = (data['id']==0xFFF) 
        dt_comments = [('samplepos', 'uint32'),
                    ('id', 'uint16'), 
                    ('charset', 'uint8'), 
                    ('reserved0', 'uint8'), 
                    ('color', 'uint32'), 
                    ('comment', 'S{0}'.format(h['packet_size']-12)),
                ]
        data_comments = data.view(dt_comments)[mask]

I don't see equivalent things in the new BlackrockIO.
Maybe I am missing it.
Do you have this in your dev version of this IO ?

[mdenker]

Hi, I'll have to look into it, but on line 947, there is at least corresponding structure defined:

        'Comments': {
            'a': [
                ('timestamp', 'uint32'),
                ('packet_id', 'uint16'),
                ('char_set', 'uint8'),
                ('flag', 'uint8'),
                ('data', 'uint32'),
                ('comment', 'S{0}'.format(data_size - 12))]},

and on line 899, there is a similar bug with the 0xFFF

        'Comments': {
            'a': (packet_ids == 0xFFFF)},

However, I am not yet sure, whether these fields are acutally used -- I'll dig deeper in the code later.

[samuelgarcia]

Thank you. I was not able to make the link.

@Deckard123321: could you provide a file were comments is read in 0.4.X and not read in 0.5.x ?

[mdenker]

I think that one would have to enter a bit of new code around line 2059 following in function read_segment(), which only reads nev_data["NonNeural"] Events so far...

[neoana]

1. For version 0.4.x, first ,you should change this "mask = (data['id']==0xFFFF) " you can read "comments" data by this: from neo import io r = io.BlackrockIO(filename) seg = r.read_segment() print seg.eventarrays # both "digital event" and "comments" are all in seg.eventarrays 2. For version 0.5.x, on line 947, "comments" structure are defined, but it is not used in later code. I suggest, like version 0.4.x, "comments" data can also be put in "event" container with "digital event" in version 0.5.x. I added some functions in blackrockio.py, and could read "comments" data in version 0.5.x: 1. line 1008 to line 1009: 'comment_unit': pq.CompoundUnit("1.0/{0} * s".format( self.__nev_basic_header['timestamp_resolution'])), 2. line 1727 to line 1746: def __read_comment(self, n_start, n_stop, data, lazy=False): comment_unit = self.__nev_params('comment_unit') if lazy: times = [] labels = np.array([],dtype='S') else: times = data['timestamp']*comment_unit labels = data['comment'] mask = (times>=n_start) & (times<=n_stop) if np.sum(mask)>0: cm = Event( times = times[mask].astype(float), labels=labels[mask], name='comment' ) if lazy: cm.lazy_shape = np.sum(mask) else: cm = None return cm 3. line 2089 to line 2096: if load_comments: cm = self.__read_comment( n_start=n_start, n_stop=n_stop, data=nev_data['Comments'], lazy=lazy) if cm is not None: seg.events.append(cm) 4. line 2166 to line 2169: def read_block( self, index=None, name=None, description=None, nsx_to_load='none', n_starts=None, n_stops=None, channels='none', units='none', load_waveforms=False, load_events=False, load_comments=False,lazy=False, cascade=True): 5. line 2286: load_comments=load_comments, 6. line 1972 to line 1975: def read_segment( self, n_start, n_stop, name=None, description=None, index=None, nsx_to_load='none', channels='none', units='none', load_waveforms=False, load_events=False,load_comments=False, lazy=False, cascade=True): Then, "comments" data can be read like this: r = io.BlackrockIO(filename) bl = r.read_block(load_events=True,load_comments=True,channels=channels,units='all') seg = bl.segments[0] # "digital event" and "comments" are both in seg.events print seg.events Maybe there are other better ways. At 2017-03-10 18:06:50, "Garcia Samuel" <notifications@github.com> wrote: Thank you. I was not able to make the link. @Deckard123321: could you provide a file were comments is read in 0.4.X and not read in 0.5.x ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[neoana]

1. For version 0.4.x, first ,you should change this "mask = (data['id']==0xFFFF) " you can read "comments" data by this: from neo import io r = io.BlackrockIO(filename) seg = r.read_segment() print seg.eventarrays # both "digital event" and "comments" are all in seg.eventarrays 2. For version 0.5.x, on line 947, "comments" structure are defined, but it is not used in later code. I suggest, like version 0.4.x, "comments" data can also be put in "event" container with "digital event" in version 0.5.x. I added some functions in blackrockio.py, and could read "comments" data in version 0.5.x: 1. line 1008 to line 1009: 'comment_unit': pq.CompoundUnit("1.0/{0} * s".format( self.__nev_basic_header['timestamp_resolution'])), 2. line 1727 to line 1746: def __read_comment(self, n_start, n_stop, data, lazy=False): comment_unit = self.__nev_params('comment_unit') if lazy: times = [] labels = np.array([],dtype='S') else: times = data['timestamp']*comment_unit labels = data['comment'] mask = (times>=n_start) & (times<=n_stop) if np.sum(mask)>0: cm = Event( times = times[mask].astype(float), labels=labels[mask], name='comment' ) if lazy: cm.lazy_shape = np.sum(mask) else: cm = None return cm 3. line 2089 to line 2096: if load_comments: cm = self.__read_comment( n_start=n_start, n_stop=n_stop, data=nev_data['Comments'], lazy=lazy) if cm is not None: seg.events.append(cm) 4. line 2166 to line 2169: def read_block( self, index=None, name=None, description=None, nsx_to_load='none', n_starts=None, n_stops=None, channels='none', units='none', load_waveforms=False, load_events=False, load_comments=False,lazy=False, cascade=True): 5. line 2286: load_comments=load_comments, 6. line 1972 to line 1975: def read_segment( self, n_start, n_stop, name=None, description=None, index=None, nsx_to_load='none', channels='none', units='none', load_waveforms=False, load_events=False,load_comments=False, lazy=False, cascade=True): Then, "comments" data can be read like this: r = io.BlackrockIO(filename) bl = r.read_block(load_events=True,load_comments=True,channels=channels,units='all') seg = bl.segments[0] # "digital event" and "comments" are both in seg.events print seg.events Maybe there are other better ways. At 2017-03-10 18:06:50, "Garcia Samuel" <notifications@github.com> wrote: Thank you. I was not able to make the link. @Deckard123321: could you provide a file were comments is read in 0.4.X and not read in 0.5.x ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[samuelgarcia]

Thank you for going into the code.

Could send a Pull Request that could encompass all patches ?

I can do the PR for 0.4.x (more easy for me!!).