[discussion]: Where is data (resources) to be stored?

[joelostblom]

What would you like to discuss?

Maybe this is already documented and I'm missing it, but I don't currently understand where the actual data (resources) are to be stored/uploaded for individual data packages. Currently, the data packages that sprout creates are tightly integrated with git/github, but that's traditionally not friendly toward working with large binary data files like parquet. Is the idea to use something like GitHub LFS. https://dvc.org/, or on premises cloud storage for storing the data resources while all the metadata and python package related files are tracked via Git?

I noticed that there was mention on LFS in one of the sample projects seedcase-project/example-rhesus-monkeys#24

[lwjohnst86]

Yea, we are thinking LFS. But we haven't yet actually tested this out with data in a "real" environment. We tried it in GenomeDK, which is what Steno will start using for storing their data.

[joelostblom]

From reading https://sprout.seedcase-project.org/docs/guide/resources, I'm also a bit unsure how many copies of the data is being stored. It seems like there are 2-3 copies to me currently:

1. It sounds like one copy of the raw data is being stored at its original location, but then also optionally included in a raw folder in the data package.
2. A tidy version of the raw data is stored in the resources folder
3. A separate type of file/location is called "batch data" and seems to be used both as a backup and for tracking changes when new data is coming into the the project.

Maybe I'm misunderstanding something here, but it seems expensive to have the storage go up 2-3x. I don't fully see the advantage that this bring either, so maybe we can clarify that for me either here or on a future call. I would instinctively have thought that either just the cleaned data, or the script for cleaning the data with a link to the raw data, would have been stored inside the Data Package.

[K-Beicher]

From a data manager perspective I would like to have the full cleaned data set in a separate location, as well as the raw data files stored somewhere (at least temporarily), I would also like to have a change log in case any data changes between one version and the next.
One of the things we are talking about in ON LiMiT is to see what distance to a gym does to compliance, thus it would be important for me to know if the participant moves during the study, and I would need to keep a record of where they have lived as well as where they now live.

[joelostblom]

Thanks for the reply @K-Beicher ! I think I'm still a bit lost here, I'll try to develop more in response to what you wrote:

From a data manager perspective I would like to have the full cleaned data set in a separate location, as well as the raw data files stored somewhere (at least temporarily), I would also like to have a change log in case any data changes between one version and the next.

I agree that the raw data should be stored somewhere; probably the ideal location is outside the Data Package but with a persistent link to it if possible (even if access to the link requires escalated permissions). From there, my expectation would be that there is one cleaned/processed data file per collection raw data files describing the same data. So if raw data is recorded/uploaded in batches, then there would be multiple raw data files stored (somewhere) and then a script that cleans and combines these raw data files into a single parquet file that is stored/published as part of the Data Package.

I don't see why we need to convert the raw data to a so called "batch file", store that, and then also store the concatenation of all the batch files under a separate file name. The cleaning steps would need to happen directly on the raw data anyways, not between the batch file and the processed data file, right? I don't see the need to store this intermediate batch data that is cleaned but not yet combined into a single file.

Here is the example from the docs where the batch file is the exact same as the data resource file (and even if there were two batch files, then the delta between those and the processed data file is just a join which to me is a ling in a data processing script, not one more file stored on disk):

One of the things we are talking about in ON LiMiT is to see what distance to a gym does to compliance, thus it would be important for me to know if the participant moves during the study, and I would need to keep a record of where they have lived as well as where they now live.

I understand that keeping a record of home address changes is important for the research question. In my mind, I would expect this to be stored as an additional observation, rather than an update to an existing observation (ie a row is added to the data rather than editing the "home address" cell of an existing row). For more complex scenarios where regular updates to existing data needs to happen, I would expect that some fully fledged data versioning system is used.

---

Having all that said, maybe there is just something I don't understand about how data is often used in the projects you collaborate with.

[signekb]

Thanks for starting this discussion, @joelostblom ! 💭

As I remember it, the initial choice to include both a raw/ folder and a batch/ folder dates back to a time where we didn't expect the data to be Parquet and read into Polars dataframes. At this time, the raw/ folder was for the actual raw files, for example a csv, xlsx, json, or a similar data format. This came from us using the frictionless-py package to extract properties from different file formats (e.g., #1017), instead of a Polars dataframe as we do now (#1203). We then "loaded" these files "into Sprout" by written them as Parquet files into the batch/ folder. If an error was found in the raw data during this process (e.g., data didn't match properties), an error would be thrown and the raw data wouldn't be written as a Parquet into the batch/ folder. At this point in time, we

We later decided to simplify this process with a "hard requirement of tidy data" loaded as a Polars data frame instead of different file formats (e.g., see #1193 and #1203).

I can see that we actually renamed the raw/ folder to batch/ (#1127 and #1128) but that the raw/ folder is described as "optional" in the third step in the list overview in the resources guide, so I understand your confusion. I also agree that having (technically) three copies of the data seems unideal from a storage perspective.

At this point, I'm not sure that we need both the raw/, batch/, and data.parquet. Since we expect the data to be tidy when it "enters" Sprout, we could supply users with the structure of having their data in raw/ and then let them decide how they want to do the cleaning and tidying as long as they keep the files in there. Then, in the main script in their Data Package, they'll load their data from raw into a Polars dataframe (how they acheive that is none of our business) and use Sprout's functionality to extract properties, perform checks, etc., and then finally join and save the resource data to data.parquet in the end. This way, we might not need the batch/ folder?

I do have a feeling that I might have forgotten something about the need for a batch/ folder though. I suspect/hope that @lwjohnst86 or @martonvago has a clearer memory of this?

[joelostblom]

Thanks for adding that historical perspective @signekb! It's helpful to know how the current approach came to be and evolved over time.

At this point, I'm not sure that we need both the raw/, batch/, and data.parquet. Since we expect the data to be tidy when it "enters" Sprout, we could supply users with the structure of having their data in raw/ and then let them decidir how they want to do the cleaning and tidying as long as they keep the files in there. Then, in the main script in their Data Package, they'll load their data from raw into a Polars dataframe (how they acheive that is none of our business) and use Sprout's functionality to extract properties, perform checks, etc., and then finally join and save the resource data to data.parquet in the end. This way, we might not need the batch/ folder?

This sounds similar to how I imagine things. With the addition that I could see arguments for having the raw folder optional even if the batch folder is removed, e.g. if the raw data is huge or contains sensitive attributes it might be tricky/impossible to include it. For the purposes of making analysis reproducible and data easier to (re)-discover, I think the most important data to share as part of the Data Package is the cleaned data.parquet. In my eyes, having access to the raw data would mainly be useful to see if something went awry during cleaning or to extend an analysis by including additional observations or variables.

[martonvago]

I also remember pretty much what Signe said!

Basically that the initial idea was to store raw files however they came in (though we mostly considered CSVs) and then process them into a clean Parquet file. But then it turned out that processing CSV into Parquet following the rules of the Frictionless Framework is actually quite a big task, so we said we would take a Polars data frame as input instead. So in this most recent version we don’t require or manage the raw folder, but it is true that the data must come from somewhere.

I think the reason behind keeping the batches in the batch folder was that we wanted to preserve the precise history of the development of the data (without keeping track of raw data). So keep old values and mistakes, with a view to maybe being able to restore old versions at specific time points if needed. If LFS and co can do that (and we can get it to host data on the secure server) then I see the argument for dropping the batch folder altogether. But, yeah, maybe Luke remembers if there were other reasons for needing the batches.

[lwjohnst86]

Great question and good comments all around! I'll add on to what the others wrote. I also recognize that a lot of these discussions should be put into the website some how. But that's for later.

A disclaimer, I also don't love the ergonomics with the flows, but this stuff is complicated and complex. As far as I know, there isn't a super elegant way of handling all these different input data sources. There are literally dozens of successful companies whose only product is helping companies manage all their different data sources. So this problem isn't unique to us.

First, a big step back, since I'm sensing that some of your (@joelostblom) comments come from your experiences using data for research analysis and questions. So a couple of things:

1. Data Package is a machine readable standard for connecting metadata to the data. It isn't a format for strictly for helping with the analysis phase of a research project (though knowing the metadata does help). So Data Package's are less to do with analysis of data, and more about structuring and storing metadata.
2. The Seedcase Project first and foremost is about data management and engineering. It has nothing directly to do with analysis. There's more than enough software that targets the analysis side of things. We're interested in building up packages to help with data infrastructure, organisation, and management to improve how people manage data, share it, and document it, so that analysts at the end can make better informed decisions about how and what to analyze by learning and understanding the context that the data was created for and from (the metadata).
3. We happen to build upon/form our foundation on the Data Package standard (because it is the best that we've found). So what others might do with Data Packages isn't necessarily what we are using it for.
4. Lastly, we can't assume we can share the data itself, at least not easily and not in its entirety. This relates to the rule of least privileges that also applies to data (via GDPR), meaning a researcher/user should only be given the data they need for their specific purpose, and no more than that. Because of GDPR, we would, at a minimum, only be able to share the datapackage.json file publicly. So researchers and analysts will very likely never interact with the Seedcase-structured Data Package. Managers and data engineers on the other hand will.

With these things in mind, how we design things is vastly different if we are designing for analysis or for management. For instance, having duplicate/redundant data doesn't make sense from an analysis perspective. But from a storage, management, and auditing perspective it is necessary to have some redundancy/duplication.

Let's break it down a bit to showcase some of the needs here (at least in our situations):

1. We can't assume that data stored in data collection/entry servers will be available for very long. These servers are meant for collection/entry, not for long term storage. And in fact, for us, REDCap (data capture) can only store data from up to 5 years, but for some of our collaborators they aim to store data for up to 20 or more years.
2. Data entered into collection/entry servers can and likely will be changed (e.g. with a data entry error that the collectors fix). We need to have some recordkeeping on our end to identify if data was changed and when/where. This is also for auditing and legal purposes. We need to be able to show if data was changed and when and in what way.
3. Not all data can be pinged via an API. Some will come in as a raw form that will need to be more extensively processed.

In the ideal case, we directly fetch from REDCap via API for a Data Resource, like so:

flowchart TB
	A[Participant<br>fills in survey] --> B["Collection server<br>[REDCap]"]
	B -->|"fetch (monthly?) via<br>API and process<br>into tidy form"| C["Storage server<br>[GenomeDK, Sprout]"]
	C -->|"write_resource_batch()"| D["Stored in Resource 1<br>[Batch, checks]"]
	D -->|"join_resource_batches()<br>write_resource_data()"| E["Stored as<br>Data Resource 1<br>[resolved duplicates]"]

Loading

But, we don't always get data in such structured, relatively tidied form as from REDCap. If we get it from lab machines, it will come in a much more rough, raw state. In that case:

flowchart TB
	A[Machine runs<br>sample] --> B["Collection server<br>[REDCap?, Sharepoint?]"]
	B -->|"fetch via<br>API or manual?"| C["Storage server<br>[GenomeDK, Sprout]"]
	C -->|"write to 'raw/'?"| D1["Stored in raw state"]
	D1 -->|"read and process<br>write_resource_batch()"| D["Stored in Resource 2<br>[Batch, checks"]
	D -->|"join_resource_batches()<br>write_resource_data()"| E["Stored as<br>Data Resource 2<br>[resolved duplicates]"]

Loading

This rough, raw data will come in a variety of formats, like CSV or xlsx (maybe?). We want to keep the full data from e.g. lab machines so that it can be processed via code (rather than the manual process data entry/technicians might use), so that we have a record of the original data if anything happens.

So, some Resources will only have batches, while others may have a raw as well as batches.

Part of the reason we went with batches in Parquet format is to make sure the data that we keep for record-keeping and auditing complies with the Data Resource schema, otherwise it gets to be a major pain to have to check each batch file against the standard everytime we join the batches together into the final Data Resource. And Parquet ensures that data has explicit types, nor can it be easily modified (not like a CSV).

I've been waiting to see how we handle things with the ON-LiMiT data before revisiting the design of Sprout, especially when it comes to raw/ data. We might need to include some basic code to store it in a resources/<id>/raw/ folder to keep it contained to the Data Resources file.

As for the storage space question, Parquet files are very compressed and space efficient, as are CSV files that are compressed. And considering there will be only one Seedcase Data Package per study, space isn't really a problem. For researchers who want to use data in a Data Package, they will receive data via our (planned) Propagate package, which will be a (likely) a single Parquet file (since a single file is easier for analysis than mulitple files).

[joelostblom]

Thanks a lot for the comprehensive reply! Much appreciated! Your emphasis on the difference between analytics and the goals of Seedcase will be helpful for me to keep in mind when thinking about Seedcase.

So what others might do with Data Packages isn't necessarily what we are using it for. ... So researchers and analysts will very likely never interact with the Seedcase-structured Data Package. Managers and data engineers on the other hand will.

It's helpful to know that you are using Data Package structure as a foundation, but won't necessarily be having the same end goals in terms of packages being shared for analysis. I think that the sentence in the docs overview that talks about data becoming more discoverable and (re)usable for later analyses made me think that this is something that will be shared with analysts, but now I understand it as that the discoverability and (re)usability are for data engineers/managers instead. Maybe the wording there could change a bit?

Data entered into collection/entry servers can and likely will be changed (e.g. with a data entry error that the collectors fix). We need to have some recordkeeping on our end to identify if data was changed and when/where. This is also for auditing and legal purposes. We need to be able to show if data was changed and when and in what way.

I appreciate that there is a need for record keeping for legal purposes, but I'm still unsure if the most desirable to do that record keeping is to duplicate the data each time an update is made such as a small error fix. I would have expected it to be more helpful to run a diff on the updated and original data file and then store that diff together with the updated file. So essentially a dedicated data version control system which keeps an audit trail on exactly what changed in the data with each new upload and who did it, instead of simply storing a new file with each update. Although you mention that storage is not an issue, I could easily see the storage snowball if a large file is updated several times and each update mandates that a full copy of the data is stored again. I realize that this is outside what I regularly work with so maybe I just don't understand some part of it, but thought I would share my thoughts.

Part of the reason we went with batches in Parquet format is to make sure the data that we keep for record-keeping and auditing complies with the Data Resource schema, otherwise it gets to be a major pain to have to check each batch file against the standard everytime we join the batches together into the final Data Resource. And Parquet ensures that data has explicit types, nor can it be easily modified (not like a CSV).

To me, a diff-based audit trail for the processed data as described in the previous paragraph would meet this need more directly that separate data files (but I'm not familiar with the legal data requirements you're referring to).

I've been waiting to see how we handle things with the ON-LiMiT data before revisiting the design of Sprout,

Yeah that makes sense; it would be helpful to see a full example of how it works in practice when implemented in a project.

...considering there will be only one Seedcase Data Package per study...

In my mind, I thought there would be one Data Package per dataset (or set of related data files) rather than per study, so this comment raises a new question for me: What happens when the same dataset is used for multiple studies? I'm thinking of researchers creating a massive dataset with some new technique or large expensive survey approach that several groups then benefit from. Is each group meant to make their own data package?

[lwjohnst86]

Related to the full data being stored each time. No, that isn't the goal. Each update would only be an update of what was changed (how we exactly implement that depends heavily on how we resolve the specific workflow for raw data flow, which will become clearer as we actually work with flowing data sources into Seedcase Data Package structure).

Maybe I'm also not clear what you mean by diff-based audit trail. I think we are actually thinking the same thing, just using different words.

"Study" vs "dataset" might also be a language/domain specific difference. And the fact that "study" is a bit vague. When I use the word study, I mean something like a clinical trial or an epidemiological study or a clinical study. These studies would only ever make one dataset (based on the same participants). But yea, a study could also be three PhD studies using the same dataset. Maybe better words might be "cohort" or "intervention" or "trial", rather than "study".

[joelostblom]

Related to the full data being stored each time. No, that isn't the goal. Each update would only be an update of what was changed ... Maybe I'm also not clear what you mean by diff-based audit trail. I think we are actually thinking the same thing, just using different words.

Yup that sounds similar to what I'm thinking. Essentially something akin to how git works: it doesn't store the previous and current version of a file, but rather the current version plus a description of the differences from the previous version of the file.

With such a diff-based/git-like version control system for the data, it seems to me that any changes to the data over time would be captured by the history/log and there would not be a need to also keep a separate batch file for each new upload of the data (whether that upload is the addition of observations of fixing a previous error).

"Study" vs "dataset" might also be a language/domain specific difference. And the fact that "study" is a bit vague. When I use the word study, I mean something like a clinical trial or an epidemiological study or a clinical study. These studies would only ever make one dataset (based on the same participants). But yea, a study could also be three PhD studies using the same dataset. Maybe better words might be "cohort" or "intervention" or "trial", rather than "study".

👍