ENH: Array data in `.tsv` cells

[sappelhoff]

Potentially to be dealt with:

• [FIX] Do not require notch frequencies to be parsed as numbers, accommodating multiples #1605

Discussion originated in this thread:

• Example of the HED schema library for SCORE implementation bids-examples#324 (comment)

Currently, cells in BIDS .tsv files may take on numeric or string values (see tabluar data).

Here I list some examples:

• a string n/a, reflecting a missing value or that a value is not applicable for the present cell
• a string mystr, that is further specified in the column_name.Levels object in a .json file that accompanies the .tsv file, where column_name refers to the name of the column in the .tsv file under which the string mystr occurs
• floating point numbers, reflecting time since acquisition for the onset column in events.tsv
  • often associated with a unit, defined in column_name.Units, analogously to the mystr example above

However, for some cases, it would be desirable to define an array of values in a particular .tsv cell. For example when a given event (in a row) in an events.tsv file is associated with multiple channels from a neural recording (e.g., EEG, iEEG, NIRS, ...):

This event at onset X and with duration Y is associated with channels A, B, and C

Currently, there is no formal way to specify such an array in BIDS. We think that it could be helpful for several data modalities (now and in the future) to have such a concept to work with.

In bids-standard/bids-examples#324 (comment), @effigies suggests that we use JSON arrays for this case (see his comment for advantages over more basic structures, like a comma-separated string).

An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).

I think if we were to adopt JSON arrays as a valid input for .tsv cells, we would would benefit by (at least initially) restricting the kinds of values that may be listed in a given JSON array. Values are defined as such:

A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.

For our present purposes, I think it would suffice to confine ourselves to:

• string
• number
• boolean (true/false)

---

What are your opinions? Do you see other use cases apart from annotations in ephys data (in events.tsv files)? Please comment and advise.

[VisLab]

I always like to see what I'm in for downstream, so I did a couple of simple tests of the alternatives on Python (pandas DataFrame) and Matlab (Table), which I believe will be the two most common downstream data structures to hold tabular data for users.

After view the results (below and attached files), I would prefer option 3 of just having channels be [x , y, z]. It seems that the tabular input for standard tools sees the [ ] and immediately thinks this is a string.

Here is my code and the results. The three test files are attached. All of the Matlab results have outer { } because they all come back as a string in a cell array of size 1 x 1.

table1 = readtable('test1_events.tsv', 'Delimiter', 'tab', 'FileType', 'text');
table2 = readtable('test2_events.tsv', 'Delimiter', 'tab', 'FileType', 'text');
table3 = readtable('test3_events.tsv', 'Delimiter', 'tab', 'FileType', 'text');

Python:

table1 = pd.read_csv('test1_events.tsv', delimiter='\t')
table2 = pd.read_csv('test2_events.tsv', delimiter='\t')
table3 = pd.read_csv('test3_events.tsv', delimiter='\t')

Table	Row	Input	Python	Matlab
1	0	["P1", "P2", "P3"]	'["P1", "P2", "P3"]'	{'["P1", "P2", "P3"]'}
1	1	n/a	nan	{'n/a'}
1	2	["CP1"]	'["CP1"]'	{'["CP1"]'}
1	3	[1, 2, 3]	'[1, 2, 3]'	{'[1, 2, 3]'}
1	4	[1, "CP1", 2]	'[1, "CP1", 2]'	{'[1, "CP1", 2]'}
2	0	'[P1, P2, P3]'	'\'[P1, P2, P3]\''	{''[P1, P2, P3]''}
2	1	'n/a'	nan	{'n/a'}
2	2	'[CP1]'	'\'[CP1]\''	{''[CP1]''}
2	3	'[1, 2, 3]'	'\'[1, 2, 3]\''	{''[1, 2, 3]''}
2	4	'[1, CP1, 2]'	'\'[1, CP1, 2]\''	{''[1, CP1, 2]''}
3	0	[P1, P2, P3]	'[P1, P2, P3]'	{'[P1, P2, P3]'}
3	1	n/a	nan	{'n/a'}
3	2	[CP1]	'[CP1]'	{'[CP1]'}
3	3	[1, 2, 3]	'[1, 2, 3]'	{'[1, 2, 3]'}
3	4	[1, CP1, 2]	'[1, CP1, 2]'	{'[1, CP1, 2]'}
test_event_files.zip

@christinerogers @arnodelorme

[christinerogers]

Thanks @sappelhoff and @VisLab . My EEGNet colleague Tyler @Andesha Collins will weigh below, and I'll re-iterate part of our concern with embedding json in a tsv --:

It adds challenges for researchers trying to BIDS-ify their data, since it's harder to read and format in a .tsv cell, more prone to formatting errors, and can't be validated as easily (afaik) - a recipe for frustration and attrition in data sharing.

I understand the reasons why embedded json can be preferred, and at the same time it would be nice to also aim for reducing barriers in onboarding many new research groups to BIDS (as my group does with large open-source projects like LORIS and data-sharing platforms like EEGNet).

p.s. @sappelhoff per my original question in the BIDS Maintainers meeting March 9 - We can't be the only corner of the BIDS world looking at this, can we? who else should be consulted in from other specs?

[Remi-Gau]

Note to self: check how octave + bids-matlab will deal with this.

[effigies]

Some notes on Kay's comment:

I would suggest that we don't allow heterogeneous arrays, and instead say the type is "array of strings". Encoded in JSON it would be:

onset       duration    trial_type  channels
3.452       n/a         blink       ["P1", "P2", "P3"]
6.112       0           show        n/a
6.82        0.2         response    ["CP1"]
7.345       n/a         start       ["1", "2", "3"]
8.543       5.0         mixed       ["1", "CP1", "2"]
9.112       0           show        []

(I've added another row to show an empty list.)

There are two approaches with Pandas (note that the reprs don't make clear the string versus object; if "P1" shows up, the whole thing is a string; if P1 shows up it is in a list of strings):

Convert on load

>>> import pandas as pd
>>> import json
>>> import math
>>> def loader(val):
...     if val == 'n/a':
...         return None
...     return json.loads(val)
... 
>>> table = pd.read_csv('test4_events.tsv', delimiter='\t', converters={'channels': loader})
>>> table
   onset  duration trial_type      channels
0  3.452       NaN      blink  [P1, P2, P3]
1  6.112       0.0       show          None
2  6.820       0.2   response         [CP1]
3  7.345       NaN      start     [1, 2, 3]
4  8.543       5.0      mixed   [1, CP1, 2]
5  9.112       0.0      mixed            []

1, 2 and 3 are strings here.

Load then convert

>>> import pandas as pd
>>> import json
>>> def from_default(val):
...     if isinstance(val, str):
...         return json.loads(val)
...     return val
... 
>>> table = pd.read_csv('test4_events.tsv', delimiter='\t')
>>> table
   onset  duration trial_type            channels
0  3.452       NaN      blink  ["P1", "P2", "P3"]
1  6.112       0.0       show                 NaN
2  6.820       0.2   response             ["CP1"]
3  7.345       NaN      start     ["1", "2", "3"]
4  8.543       5.0      mixed   ["1", "CP1", "2"]
5  9.112       0.0      mixed                  []
>>> table.channels = table.channels.apply(from_default)
>>> table
   onset  duration trial_type      channels
0  3.452       NaN      blink  [P1, P2, P3]
1  6.112       0.0       show           NaN
2  6.820       0.2   response         [CP1]
3  7.345       NaN      start     [1, 2, 3]
4  8.543       5.0      mixed   [1, CP1, 2]
5  9.112       0.0      mixed            []

1, 2 and 3 are also strings here.

---

The other thing we considered was a simple comma-separated string, so it would be

onset       duration    trial_type  channels
3.452       n/a         blink       P1,P2,P3
6.112       0           show        n/a
6.82        0.2         response    CP1
7.345       n/a         start       1,2,3
8.543       5.0         mixed       1,CP1,2
9.112       0           mixed       n/a

This would be easy enough to deal with simply by using ','.split(val) in our converter. Note, that there is no distinction between the integer 1 and the string "1". If channels for some reason had numeric names, pandas would by default convert 1 to an integer, but 1,2 to a string.

When suggesting JSON-encoded strings, I assumed that these would not be hand-edited but generated from spreadsheets or annotation tools. If this will be tooling mediated, the JSON will be better structured and the validator could distinguish numbers from strings and give errors. If people are going to be writing these by hand, then I think the comma-separated strings will be the easiest.

[robertoostenveld]

Just some wild thoughts:

Besides it being JSON, what would be the advantage of using the [ and ] brackets around the list? And why use " quotes around each item by default? The 2nd example in the previous comment is simpler and also looks fine to me (assuming that the horizontal spaces correspond to tabs, which they technically don't in this case). I realize that I am touching here upon human- versus machine-readability.

Possible answers to myself:

The [] brackets facilitate parsing nested lists like [1, 2, 3, [4.1, 4.2]], which would be impossible without them.

The " quotes allow specifying characters that otherwise would be parsed incorrectly, like ["{", "(", "["], or strings containing a tab. This would not be specific to the lists, it would also apply if I now wanted to represent an item with a tab in it, where the tab needs to be escaped somehow to prevent it from being interpreted as column separator.

Why would we use a different separator inside the list (comma) than outside the list (tab); that could also both be a tab. Yup, I realize this makes parsing harder, both for human and machine. But would that mean that if we have a nested list, we would use yet another separator (e.g., ;) at the 2nd level?

I like the idea what the downstream consequences are when people use software to process the data. Some softwares that I suggest to consider (besides native Python, Python+Pandas, MATLAB ) are R, SPSS, Excel, ...

[Andesha]

I hadn't personally considered nested lists as part of my suggestion based on the simple List format...

@robertoostenveld Is that something that we would need to support? 🤔

[jadesjardins]

The comma separated list inside of brackets seems to be the most simple (and safest) representation that fits the range of use cases that I am aware of.. from what I can tell, things like adding the quotations around items inside the brackets would only allow the users to do things like have commas in channels names for example (which I think I am okay with restricting).

[effigies]

Talking with @sappelhoff, one of the problems with comma-separated channels is that in practice channels might have commas in their names. I can see two approaches that would allow custom delimiters that are only incrementally more complex than comma-separated strings.

Syntax	Example	With channels containing commas
Comma-separation	P1,P2,P3	Not allowed
Wrapped delimiters	,P1,P2,P3,	|P1,2|P2|P3|
Prefixed delimiter	,:P1,P2,P3	|:P1,2|P2|P3

These could be implemented:

def wrapped_delimiters(val):
    if isinstance(val, str):
        delimiter = val[0]
        if not delimiter.isalnum() and val[-1] == delim:
            return val[1:-1].split(delimiter)
    return val

def prefixed_delimiters(val):
    if isinstance(val, str) and len(val) > 2:
        if val[1] == ":":
            delimiter = val[0]
            return val[2:].split(delimiter)
    return val

Unless there's some globally safe delimiter (or we're willing to force people to mass-rename channels in their data), A dynamic delimiter or escape characters are likely the only options. One additional step of complexity is an overridable default:

Syntax	Example	With channels containing commas
Overridable delimiter	P1,P2,P3	|P1,2|P2|P3| or |:P1,2|P2|P3

[robertoostenveld]

I hadn't personally considered nested lists as part of my suggestion based on the simple List format...

@robertoostenveld Is that something that we would need to support? 🤔

No, I do not see the use case for it at the moment. But then, I also did not see the use case in the past for lists within one cell of the TSV table.

Nevertheless, "Simple is better than complex", so I think that we should not plan for the future, but do what makes sense now. Personally I like the comma-separated list and if I need it I would simply avoid comma's in channel names (I do that anyway, I only use [A-Za-z0-9], - and _). And the overridable default (e.g., with |) makes that even better.

[sappelhoff]

Thanks for the suggestion, Chris. I like the idea of delimited lists where the delimiter is wrapped and may be freely selected from a list of defined delimiters (like ,|;.+- or so). I am not sure I like the idea of just having comma-separated lists as the default and only optionally allowing for "overridable delimiters via wrapping (or prefixing)", because this seems like an unneeded branching to me: It's not a hassle to do ,P1,P2, instead of P1,P2.

[VisLab]

I'm glad this discussion is progressing towards a resolution. I don't see a use case for nested lists at this time and don't think we should address this until there is a need for it.

Either the wrapped delimiter solution or the overridable delimiter solution is fine.