Requirement: A new method of identifying a time series

[krischer]

A new method of identifying a time series is required for NGF: It should be adequate to meet the need to deploy multiple sensors, retain semantic meaning where possible and support a significant increase in the number of sensors deployed as a single project.

• Suggestion: Use a variable length time series identifier allowing significantly more flexibility in the way individual elements can be identified.
• Suggestion: Create an FDSN name space allowing straightforward adoption by other communities in a simpler and more decoupled manner.

[chad-earthscope]

I am strongly in favor of defining a single time series identifier for two main reasons:

• We would only need a single, likely variable length, field with an "identifier" in the format. A (large) maximum length would be the only limiting factor defined by the low level format. Then discussions about how the identifiers are constructed and how codes can be expanded are independent of the low level format. We could, of course, have variable length fields for each identifier, but that adds complexity that is unnecessary in my opinion.

• Such an identifier provides a huge degree of flexibility for currently needed and future changes. We will be able to expand, for example, channel codes without a limit imposed by the low level format. Furthermore, if we incorporate a "namespace" into the identifier we then allow the possibility to adopt evolved or complete different identification schemes in the future.

While this would set the stage for having a complete new identifier scheme, we have a large volume of miniSEED 2.x that uses the old identifiers that must be accommodated. To this end, IRIS proposed a solution during the previous technical discussion, which has been modified slightly since then, to define an identifier constructed as a Uniform Resource Name (URN) with the following pattern:

FDSN:<network>_<station>_<location>_<channel>

where the network, station and channel codes are required to be non-empty and the location code may be empty. The 3 underscore (ASCII 95) delimiters must always be present.

Example identifiers:
FDSN:IU_COLA_00_BHZ (where network=IU, station=COLA, location=00 and channel=BHZ)
FDSN:NL_HGN__LHZ (where network=NL, station=HGN, location is empty and channel=LHZ)

The "FDSN:" namespace would identify this combination of SEED codes. Alternates schemes could be defined or adopted in the future.

For reference the current working draft of this proposal is attached:
FDSN Identifiers - 2018-1-3.pdf

There are a few areas that are known to still need work:

• Rules/convention for location codes. This is not fully defined in the working draft and needs further discussion.
• The draft includes our concept of 3 or 4 character channel codes where an extra character could be used to identifier more instrument types. It is unclear whether this concept can accommodate all needed instruments.

[crotwell]

I am also in favor of the single identifier. It has many advantages, but also a couple disadvantages worth noting.

An advantage of a single identifier is that the likely most common operation on miniseed data, matching of records with channels, is a single string comparison instead of 4 currently. The truncation of null bytes or searching for the '~' step is also reduced from 4 to 1, perhaps making extraction quicker.

But a single identifier will use additional bytes. The IRIS proposal for example has effectively 8 extra bytes compared with existing miniseed2. Also, extraction of the network or station code becomes a more expensive string splitting operation. I think the tradeoff is acceptable, but the disadvantages are worth noting.

Instead of "FDSN:" perhaps defining that the stored version identifier that starts with ':' implies 'FDSN:' would save 4 bytes. All other non-fdsn namespaces would have to be fully specified.

[jfclinton]

@chad-iris

For reference the current working draft of this proposal is attached:
FDSN Identifiers - 2018-1-3.pdf

is there going to be an opportunity to comment on this working draft? If the consensus is to move to a single identifier that simply extends SNCL, the actual proposed extension requires significant discussion [1]. Does it fit into this stage of the discussion, or when would you see this taking place?

John

[1] e.g. Generally I agree with what is being proposed in the working draft, though I think the channel code could be extended beyond 4 characters to add some additional information about synthetic / processed data without having to scamper off to an alternative URL. For example, it would be useful to an additional new data type code to indicate whether data is raw / processed or synthetic (default is raw). Then e.g. a synthetic BHN stream can be identified as BHN-X or a strong motion channel converted to acceleration can be identified as HGZ-Y. At the moment, using X as the band code looses all this information for processed / synthetic streams.
Also, since we have the opportunity, I would propose considering splitting the band code to separate out the ranges of sampling rates and the ranges of sensor corner frequencies.

[chad-earthscope]

@jfclinton:

is there going to be an opportunity to comment on this working draft? If the consensus is to move to a single identifier that simply extends SNCL, the actual proposed extension requires significant discussion [1]. Does it fit into this stage of the discussion, or when would you see this taking place?

Yes, absolutely. From the NGF perspective, the important part is whether we agree to this kind of identifier. If we collectively agree, then the definition of NGF can move forward (imposing, perhaps, only a maximum identifier length of 255) while the discussion of what form the extensions take being split-off into a separate conversation.

I suggest one of these options:

1. create another space to discuss the form of the identifier and expansion of the codes independently of NGF. This could just be another GitHub project under the FDSN account.
2. use this issue to gauge acceptance of a SNCL-mappable, single identifier form. When the evaluation is closed at the end of the month, if member consensus is to use such a form we create another space to discuss the form of the identifier and the expansion of the codes.
3. use this issue to discuss single identifier form and expansion of codes.

Those are in my order of preference. I volunteer to create another GitHub project for FDSN identifiers and create issues to discuss form and expansion and rules for each of the 4 codes (network, station, location, and channel) if there is agreement to do this.

Even if the consensus is to keep 4 fields for each of the 4 codes we need to discuss how to expand them.

Expanding the codes is a very important topic, of all the changes we are discussing it is the one that will effect end-users the most in my opinion. It merits a separate conversation that is not muddled with the rest of the NGF details.

[jfclinton]

Expanding the codes is a very important topic, of all the changes we are discussing it is the one that will effect end-users the most in my opinion. It merits a separate conversation that is not muddled with the rest of the NGF details.

I agree its very important, and it also couples into some of the discussions in other conversations, e.g. if we agree that identification of processed data is part of the new naming convention, then we can agree and close #10 , so we need to begin discussing it soon. I propose we move forward on this with Chad's first suggestion - its going to get too messy to fold this entire topic into the single issue here.

[crotwell]

Can we keep it here instead of a second github? Add as many issues as you think you need, but following similar discussions arbitrarily split into 2 repositories makes it harder to follow I feel.

[krischer]

Can we keep it here instead of a second github? Add as many issues as you think you need, but following similar discussions arbitrarily split into 2 repositories makes it harder to follow I feel.

Done in #27-#30.