Requirement: Support variable length records for efficiency and flexibility

[krischer]

Support variable length records for efficiency and flexibility (e.g. real time streaming with lower latency).

[andres-h]

I agree that variable record length is needed in some use cases, but there are better ways for lower latency than reducing record size.

[krischer]

Adding a footer to support incomplete record streaming might be better suited for discussion in #25 and is independent of a variable record length.

[crotwell]

I think there is a need for variable length on the larger end as well, useful for saving processed data in fewer large chunks instead of breaking in to many small records. The current miniseed power of 2 system is too coarse.

The size of the variable record and the maximum number of samples header need to be compatible. Previous discussions were for a Uint16, so 64K max for both.

[krischer]

Summary

(Please let me know if I missed a point or misunderstood something)

Please vote on (compatibility of the number of samples field is implied):

1. Do we want variable length headers without being restricted to a length of 2^X?
2. What should the maximum length be? Likely determined by the data type used to store the maximum length. (Uint16=65536 bytes / Uint32=4294967296 bytes / other suggestion).

[kaestli]

(does "no" to one mean no variable length headers [except for the ID voted elsewhere], or headers only of length 2^X?)

[crotwell]

I think this is "variable length records", not headers? So existing is like 512 or 4096 and BGF records could be 593 or 64201 bytes depending on need.

1 yes
2 Uint32 - with the caveat that datacenters may choose to limit input/output record sizes based on their needs.

[chad-earthscope]

1. Yes, variable length records.
2. UINT16.

[krischer]

(does "no" to one mean no variable length headers [except for the ID voted elsewhere], or headers only of length 2^X?)

As @crotwell says - this issue is about the length of complete records.

[kaestli]

1. yes
2. UINT32

(but: has anybody practically checked the additional overhead of e.g. finding a specific time interval in a dayfile?)

[ozym]

1. Yes
2. Uint16

[claudiodsf]

1. Do we want variable length headers without being restricted to a length of 2^X?

Yes (no restriction)

2. What should the maximum length be? Likely determined by the data type used to store the maximum length. (Uint16=65536 bytes / Uint32=4294967296 bytes / other suggestion).

Uint32

[ihenson-bsl]

1. yes
2. Uint16

[ValleeMartin]

1. Yes
2. Uint32