Pseudo-versioning of metadata changes

[tmchartrand]

Getting down in writing some discussions with @dbirman and @dyf -
In general we treat data assets as immutable, and this is reflected in the processing schema which identifies a DataAsset input by its s3 URI only. However, we actually expect QC/Curation metadata to change, and even core metadata can change for a hotfix or schema update.

This creates a reproducibility challenge for downstream analysis that uses metadata, which we expect to be common for Curation at least (ie the average of some cell-wise metric across a curated population of cells within an asset).

Open questions:

• is full reproducibility at this level needed (ie ability to rerun any past analysis with same result)? If so, would need to either embed full history in all metadata records, or embed the full input metadata in each analysis/processing record.
• Alternatively, is it enough to be able to check whether a given analysis/processing result is outdated with respect to metadata inputs (ie they have changed since the last run)? In this case the current QC schema approach of just recording change dates is probably sufficient. We might want to embed the relevant change date in the input DataAsset record of the processing, but could also calculate it on the fly when running such a query.

[tmchartrand]

One point to consider is there may be performance implications on queries that need to be run quite frequently in the analysis architecture - not having thought through this thoroughly, I imagine it might be helpful to bubble the change date up to a higher schema level as well and index the DB on it for instance.