Data Type Extension: single point definition

[DennisHeimbigner]

It appears that a data type extension must be re-defined at every use (specifically in each array's metadata).
It would certainly be useful if a data type extension could be defined once somewhere and used where needed.

[jbms]

I'm not sure I understand what you're asking.

A data type extension just means adding support to the spec or implementation for a new data type. It is referenced in the metadata by name, just like any existing data type

[DennisHeimbigner]

Sorry, let me try to clearer.
Consider this example from the spec:

"data_type": {
        "name": "datetime",
        "configuration": {
            "unit": "ns"
        }
    },

which is defined in the metadata for an array.
The question is: suppose I want to have two arrays of type "datetime",
Do I need to repeat the above declaration in each array?

[d-v-b]

The question is: suppose I want to have two arrays of type "datetime",
Do I need to repeat the above declaration in each array?

Yes, because in principle the arrays could have different data types. How else should the data type of an array be specified, if not in the metadata for the array?

[DennisHeimbigner]

But does each reference to "datetime" need to include the "configuration" or is can
just the name "datetime" be used; similar to how I can just say "float64"?

[jbms]

datetime hasn't been standardized so it is kind of hypothetical at this point. But the intent of configuration is to allow parameterized data types without having to encode all of the parameters as a single string. Until there actually are any parameterized data types, though, you might just ignore the possibility in your implementation. That is what I've done in tensorstore.

[DennisHeimbigner]

The specific example -- datetime -- is irrelevant,
but I was planning in adding a type "char" as an
extension in support of NCZarr, so I have an interest
in this issue.

[jbms]

Is the idea that char will be a fixed-length string, and you would then use a configuration option to indicate the length in bytes, e.g. "data_type": {"name": "char", "configuration": {"length": 10}} ?

[jbms]

See also zarr-developers/zeps#47 regarding a variable-length string proposal.

[DennisHeimbigner]

No, char will be an 8-bit unsigned integer whose purpose is to hold a single character in some encoding,
typically ASCII or ISO-LATIN-8859, or such. I do not want to use "uint8" because I need to identify the type
for NCZarr.

[jbms]

No, char will be an 8-bit unsigned integer whose purpose is to hold a single character in some encoding, typically ASCII or ISO-LATIN-8859, or such. I do not want to use "uint8" because I need to identify the type for NCZarr.

I see, in that case you would not need any configuration options, unless you want to use a configuration option to indicate the character encoding.

[DennisHeimbigner]

Yes, with the option of specifying the encoding.

[jbms]

I think there is in general a question as to what should go into the data type configuration and what should go into a separate attribute to indicate "units".

For example, if we are storing a temperature in degrees C, we would probably not have a separate "degrees C stored as float64" data type. Instead, we would store it with a data type of "float64" and use some other attribute to indicate that the unit is "degrees C".

For datetime, a unit rather than a separate data type would also seem to me to be cleaner, but many data storage formats, including zarr v2 as implemented by zarr-python, do have a separate data type for datetime.

[DennisHeimbigner]

We seem to have strayed from my original question, namely, does the full datatype definition, including
"configuration" need to be included in every array in which it is used?
I gather from the above discussion that there is currently no answer to this question,
and an answer must wait until there is an actual use case.

[jbms]

If the data type has no configuration options, then it can be specified as a plain string, {"data_type": "float64", ...}. If the data type has configuration options, like {"data_type": {"name": "char", "configuration": {"encoding": "ascii"}}, ...}, then indeed those configuration options must be specified each time the data type is specified. There isn't any way to specify "default" configuration options for a data type, like saying that any time we reference char we should assume a configuration of {"encoding": "ascii"}.

[DennisHeimbigner]

An alternative is to reify the type and declare it once in, say the zarr.info of a group, and then
allow arrays to reference it by name with out the configuration.
The problem with repeating the whole type declaration repeatedly is well-known in programming
language semantics as the structural type equality problem.
Consider two type declarations in two different arrays.
Just because the declarations look identical is no reason to to assume they refer to the same type.