[Feature]: Support zarr-python v3

[rly]

What would you like to see added to HDMF-ZARR?

Zarr-Python is moving closer to a v3 release which has significant changes in the API that result in errors in hdmf-zarr. It is not clear how stable the current alpha release is right now. But we should add support for Zarr Python v3 soon. See #200 for more details.

Is your feature request related to a problem?

No response

What solution would you like?

^

Do you have any interest in helping implement the feature?

Yes, but I would need guidance.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this change was not already requested?

[bendichter]

Zarr-Python v3 is slated for release today (source)!

I was looking through the release notes for v3 and noticed some major changes, in particular this section of the Zarr v3 spec:

Scope reduction
In order to facilitate implementation of the core specification across different programming languages, the scope of the specification has been reduced relative to Zarr v2. Note in particular the following:

• The set of core data types is reduced relative to Zarr v2. Only fixed size integer and floating point data types are defined, in addition to a Boolean data type and a fixed length raw data type. Any data types relating to storage of textual data have been left for definition within an extension specification. The “object” data type, which was never included within the Zarr v2 spec but which was implemented in the Python implementation, is also not included – it is suggested that a better approach be found for supporting variable length data types, and defined within a data type extension specification.

The Zarr team has also provided a guide for migrating from v2 to v3, with the following section:

The following features that were supported by Zarr-Python 2 have not been ported to Zarr-Python 3 yet:

• Structured arrays / dtypes (#2134)
• Fixed-length string dtypes (#2347)
• Datetime and timedelta dtypes (#2616)
• Object dtypes (#2617)
• Ragged arrays (#2618)
• Groups and Arrays do not implement __enter__ and __exit__ protocols (#2619)
• Big Endian dtypes (#2324)
• Default filters for object dtypes for Zarr format 2 arrays (#2627)

Any data types relating to storage of textual data have been left for definition within an extension specification

So if I am reading this right, does that mean that all strings must be in attributes? That's going to be a pretty big breaking change for us, as there are several places where we have string datasets which are required. I suppose we could use the raw type and use magic attributes? Either that or define an "extension specification." If they have a good formal way of creating these specifications we might also be able to use it for links and references, thus making our brand of zarr a bit more in-line with the official zarr standard, but this seems like it could be a nontrivial amount of work.

[oruebel]

I was looking through the release notes for v3 and noticed some major changes, in particular this section of the Zarr v3 spec:

I agree, that reduction in scope will make things tricky for string datasets, compound datasets, and references. In particular for strings it would be great to not have to role our own custom solution, to both maintain interoperability and avoid having too much custom functionality.

Either that or define an "extension specification."

Yes, if "extension specifications" are possible then I think that would nice. Strings are a common need so one would hope that there will be standard extension for this.

If they have a good formal way of creating these specifications we might also be able to use it for links and references, thus making our brand of zarr a bit more in-line with the official zarr standard, but this seems like it could be a nontrivial amount of work.

Yes, that would be nice.

[oruebel]

zarr-developers/zarr-specs#325

[alxmrs]

Hey Ryan, hey Oliver. I'd be happy to join working on this. After reading a few issues, it seems like a Zarr v3 migration is planned before hdmf-zarr mergest with hdmf? I'd be happy to take a look at the migration issue to support "string datasets, compound datasets, and references".

WRT the extension option -- it would be cool to extend the nwb protocol to create a "NeuroZarr" extension! I'd be happy to assist with this as well (should it be the design choice we opt for).

Are either of you available to meet? Maybe it would be helpful to discuss the outstanding technical challenges synchronously.

[rly]

Hi @alxmrs , thanks for your interest! I'll reach out to you by email to set up a quick meeting with myself, @oruebel , and @mavaylon1 to discuss the migration and how best we can work together. I see an email address on your github profile - should I reach you there?

[alxmrs]

Yes! Thanks, that is perfect :)

[yarikoptic]

Any ETA for this to make it feasible for downstream tools to do support v3 without conflicting with requirement here to have older Zarr library?

[rly]

One solution is to unpin zarr, add an error on import of this library if zarr >= 3, and tell people in instructions to also install zarr < 3. That doesn't feel right to me. This package just does not work with zarr >= 3.

For the DANDI use case, I would instead recommend that nwbinspector make the hdmf-zarr installation optional and the nwbinspector docs to say that if you are inspecting a zarr store, you need to install hdmf-zarr separately. What do you think? This won't work if DANDI wants to support both NWB zarr<3 files and other zarr>=3 files though...

[d-v-b]

are strings the main sticking point for zarr-python 3 adoption? because we are cooking a big refactor of data types in zarr-python that will make data types extensible, and there's a concurrent effort to resolve the specification side of string dtypes over in the zarr-extensions repo. I'd love to get your feedback on either of those efforts.