Requirement: Include both a format and a data/publication version number

[krischer]

Include both a format and a data/publication version number.

[chad-earthscope]

I support the addition of both a data format and data publication version.

The motivation for the format version is to make the format self describing and for identification, i.e. as a signature to match. It would also allow future evolution of the fundamental portions of the format.

The data publication version was discussed during the previous evaluation. With miniSEED 2.x there is no versioning built into the format. Some data centers used the "data quality" identifier as a crude form of versioning, but this is extremely limited with only 4 "levels" and a vague implication of "quality". Tracking data versions, which are a reality in modern data management and use, is especially important for scientific data. Including the capability to identify versions directly in the format allows for basic versioning and can be used by systems external to the format for extended, version-specific metadata.

[andres-h]

Data/publication version number should be an optional (IRIS) extension. Linear version numbers do not support "forks" where data has been modified in multiple datacentres. I would not hardcode this feature into the standard, because something more clever might be needed in future.

[crotwell]

Format version is critical for sure.

Andres has a good point about linear version numbers not working well with forks. If two data centers both receive version 7 of the data, each does something and then has a different version 8.

The alternatives are to either name-space the data version (perhaps within the additional headers) or to declare that the data version has no meaning beyond the context of the datacenter where it was created.

[chad-earthscope]

... or to declare that the data version has no meaning beyond the context of the datacenter where it was created.

That is exactly the conclusion we got to in the previous conversation last July on this topic. In the case of the IRIS DMC, I think we would work with those that contribute data so that the version is done by the owner whenever possible.

A system that identifies relative relationships between versions across forks and data centers would require some sort of central registry or much more complexity.

I suspect a data publication version in a record would be useful for many data centers, justifying a fixed 1 byte it would use, but it would be OK to use an optional header for this if that's where the consensus lands.

[krischer]

Summary

(Please let me know if I missed a point or misunderstood something)

Please vote on:

1. Do we want to include the actual data format version to enable self-identification and versioning of the data format? (Yes/No)
2. Do we want a single byte "data publication version" somewhere in each record? This would be a linear version number without a lot of additional semantics largely useful internally for data centers. (Yes/No)
3. Do we want a more complex "data publication version" which must include things like namespaces. (Yes/No)

[crotwell]

1 yes
2 yes
3 no, or at least not as a required header field. No objection to a standardized key that could be used in the optional part of the header as in #14

[chad-earthscope]

1. Yes
2. Yes
3. Not as a requirement.

[kaestli]

1. Yes
2. No (a data stream which was modified should get a different streamID, not a different version number, but a streamID pretending it to be the same. What if "version" tag varies between records of the "same" stream?. Using the streamID for to point to metadata allows to further describe the version/modification there)
3. (this should be answered in the streamID discussion)

[ozym]

1. Yes
2. Yes, although I could see this used as a mechanism to determine data providence within the collection systems (e.g. daisy chained data feeds) rather than version per se
3. No

[claudiodsf]

1. Do we want to include the actual data format version to enable self-identification and versioning of the data format? (Yes/No)

Yes

2. Do we want a single byte "data publication version" somewhere in each record? This would be a linear version number without a lot of additional semantics largely useful internally for data centers. (Yes/No)

Yes

3. Do we want a more complex "data publication version" which must include things like namespaces. (Yes/No)

No

[ihenson-bsl]

1. Yes
2. Yes
3. No

[ValleeMartin]

1. Yes
2. Yes
3. No