Best Practices for Packaging Data Segments of a Single Experiment into NWB

[nehatk17]

It's typical for many researchers to segment or split data that comes from a single experiment, especially video data, into temporally continuous sections for ease of analysis (easier to work with many smaller size files than one huge file). For example, when that video data is annotated in a tool like SLEAP, and exported into ndx-pose, you end up with several .nwb files that are all part of the same experiment. What would be the best way to package this kind of data into NWB for uploading to DANDI Archive? As I see it there are three options:

1. somehow concatenate all the videos (and other possibly temporally segmented acquisition/processed data - auditory, physiological, neural) and .nwb files that contain ndx-pose data into a single nwb (since this is a single experiment).
2. keep the different temporal segments in different nwb files (seems to go against one experiment session per nwb)
3. keep the different temporal segments in different containers within a single nwb file, though I didn't see examples of this structure in DANDI

Other options?

[oruebel]

Other options?

I believe those are the three main option. I think the preferred option depends a bit on the use case and effort required:

1. somehow concatenate all the videos (and other possibly temporally segmented acquisition/processed data - auditory, physiological, neural) and .nwb files that contain ndx-pose data into a single nwb (since this is a single experiment).

From the perspective of a data reuser, I think this would be a good option as it saves them the step of assembling the parts. However, whether to do this depends I think on effort and cost. E.g., as you mentioned, the initial split was likely done for a reason (e.g,. the video being to large), and so that may still apply. Aside from size of the NWB, video (or other files), I think this depends on how complex it is to combine the parts.

2. keep the different temporal segments in different nwb files (seems to go against one experiment session per nwb)

This is in principal fine. I would mainly choose this option if the other parts of the data (e.g., the neural data) are segmented in the same way. In this way, a single file would still represent a coherent range in time from a session/experiment, even if the recording session itself still continued. This approach can be useful when dealing with very long recording session (e.g., many hours or days), and dividing the data into time segments helps keep file sizes manageable.

3. keep the different temporal segments in different containers within a single nwb file, though I didn't see examples of this structure in DANDI

I think this is in principal valid. I think this could be a useful option if the other parts of the data (e.g., the neural data) is not being split into segments. I.e., by placing all the parts into the same file you can still maintain the logic that the NWB file represents a coherent range in time from the experiment (even if parts of the data being split across multiple TimeSeries that represent different ranges in time from that session). Since the data is being placed in the same file, I think it would be more useful for users if the data can be "re-assembled" during conversion (i.e., option 1), however, if the effort to do so during conversion is too high, then I think this is option is ok.

In general, it is useful to have the NWB file represent a coherent part of the experiment (often some range in time). I.e., when you have multiple data streams (e.g., neural data along behavior data), then if that data needs to be divided up in time, then those different streams should ideally be broken up in time in the same way. If different data streams must be split up differently in time, then I think it can be useful to also split those different data streams across different NWB files to avoid confusion.