Provenance for CSV file data

[dbirman]

Add some provenance information so we can track where CSV files came from / were generated from

[saskiad]

Do you have thoughts on how to do this?

[dbirman]

Maybe a JSON file that maps csv file names -> a URL where the file was downloaded from, then when we generate a Registry object we can add a field for provenance? I don't think this is a critical thing just making sure we have a path back to where things came from.

The file I generated for the mouse anatomy was built from a script, so provenance alone doesn't reproduce it. Is that something to worry about or not bother with?

[dyf]

I don't think a URL is sufficient.

We could put CSV generator scripts into the repo (a la brainglobe_atlas_api).

I'm on the fence about including them in test coverage. It would be nice if they were reproducible, but they often have bespoke dependencies and manual steps. Also importantly they refer to third-party APIs and we can't control uptime/stability of those. So I lean toward just having them there and not adding CI.

[saskiad]

(I'm moving this to a discussion while we figure out what we want here)

[saskiad]

We have a couple types of CSV files we're working with. Ones that are lists that we make and maintain - eg. our list of Modalities or Organizations. Others are lists that we pull from existing registries or ontologies. e.g. anatomy ontology, brain atlas,
For the latter, I want us to be able to say what "registry" we're using (registry might not be the right word, but for now I'm sticking with it) and have that be the provenance. Eg. CCFv3. EMAPA (the second A there indicates it's a specific version). This does assume that the csv we have accurately reflects that registry, and perhaps that isn't a fair assumption?
But so maybe we need a field for "registry version" (again, we can use a better word that registry, but something like that)
For the former, the lists of organizations, we want to be updating this as we add new collaborators and manufacturers. And we want other users (e.g. external users) to be able to maintain their own list of organizations