ZEP0002 Review

[normanrz]

Hi everyone,

I hope you’re doing well.

Thank you all for taking the time to review the ZEP0001 and V3 specification. The V3 specification is approved and accepted by the ZSC, ZIC and the Zarr community.

The initial discussion on sharding dates back to 11/2021; please see zarr-developers/zarr-python#877. There have been major developments since the proposal of sharding, some of them are:

• 08/2022 → Submission of sharding for a ZEP, i.e. ZEP0002; see ZEP 2 - Sharding storage transformer zeps#13
• 08/2022 → Prototype implementation of sharding as storage transformer in Zarr-Python (2022/08); see Sharding storage transformer for v3 zarr-python#1111
• 03/2023 → Pivoting to implement sharding as codec rather than storage transformer; see sharding as a codec rather than array storage transformer #220 and meeting notes
• 03/2023 → Updated prototype implementation of sharding as a codec in Zarrita; see here
• 07/2023 → Added index_codecs to the sharding codec

Now, we want to put forth the ZEP0002 - Sharding Codec for voting.

We have created this issue to track the approvals from the ZSC, ZIC and the broader Zarr community.
Specific technical feedback on sharding should be made via narrowly scoped issues on the zarr-specs repository that link to this issue.

Now, according to the section, ‘How does a ZEP becomes accepted’ - ZEP0000, a ZEP must satisfy three conditions for approval:

• Unanimous approval of the Zarr Steering Council
• Majority approval of the Zarr Implementations Council
• And, no vetos from the Zarr Implementations Council

As an implementation council member, you have three options for your vote:

• YES - Approve the ZEP0002. This indicates that your implementation intends to implement support for sharding codec.
• ABSTAIN - This would be appropriate if you do not want to implement the sharding codec.
• VETO - As a ZIC member, you can veto a ZEP, including this one.

We request you, the ZIC, and the ZSC review the ZEP0002 and let us know your thoughts. We’ve listed steps to read and understand the sharding completely. They are as follows:

• Please read and review the ZEP0002. This document contains various vital sections, e.g. Motivation and Scope, Usage and Impact, Backward Compatibility and a detailed description with illustrations.
• After this, please review the specification document here: https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/v1.0.html.
• Gather your thoughts
• Optional but recommended: begin implementing the ZEP experimentally.
• Cast your vote by dropping a comment on the issue here.

We understand that the whole process takes time, so we’ve decided to have a relaxed timeline for ZEP0002 voting. We’d appreciate your vote by 31 October 2023, 23:59:59 AoE.

Example implementations

• zarrita by scalable minds
• zarrita.js by Trevor Manz
• tensorstore by Google
• neuroglancer by Google
• Webknossos by scalable minds

Please let us know if there are any questions. Thank you for your time.

Voting status:

Github user	Project	Vote
@joshmoore	ZSC	YES
@ryan-williams	ZSC	YES
@alimanfoo	ZSC	YES
@rabernat	ZSC	YES
@jakirkham	ZSC	YES
@andersy005	freeman-lab/zarr-js	YES
@axtimwalde	saalfeldlab/n5-zarr	YES
@aschampion	sci-rs/zarr	YES
@meggart	JuliaIO/Zarr.jl	YES
@jbms	google/tensorstore	YES
@constantinpape	constantinpape/z5	ABSTAIN
@WardF	Unidata/netcdf-c + Unidata/netcdf-java	YES
@davidbrochart	xtensor-stack/xtensor-zarr	ABSTAIN
@grlee77	zarr-developers/zarr-python	YES
@manzt	gzuidhof/zarr.js	YES

[normanrz]

CC: @zarr-developers/implementation-council @zarr-developers/steering-council

[clbarnes]

Just a quick sanity check - this spec depends heavily on storage backends supporting Range requests; suffixes in particular (for getting the shard location footer). Over on this issue apache/arrow-rs#4611 , it's been suggested that common stores like S3 don't support suffixes. I don't have a store/ account I could use to test - could anyone give it a go? If not it's obviously a HEAD then a non-suffix GET Range request in serial, which isn't ideal. But possibly not that big an issue when accounting for the lack of multipart range support.

From apache/arrow-rs#4612 it looks like multipart ranges aren't supported by any cloud provider (have yet to interact with a server implementation which does tbh, and you need to jump through a lot of hoops to interpret the response as, according to the HTTP spec, they can basically send you whatever they want) and that particular library gets around it by making a bunch of different requests in parallel. Presumably that isn't a blocker in and of itself, although making hundreds or thousands of requests to pull down a region of a single shard is hardly ideal, especially as any given backend could just decide to send you the entire shard, or indeed just about any subset of it, with each request!

[normanrz]

Just a quick sanity check - this spec depends heavily on storage backends supporting Range requests; suffixes in particular (for getting the shard location footer). Over on this issue apache/arrow-rs#4611 , it's been suggested that common stores like S3 don't support suffixes. I don't have a store/ account I could use to test - could anyone give it a go? If not it's obviously a HEAD then a non-suffix GET Range request in serial, which isn't ideal. But possibly not that big an issue when accounting for the lack of multipart range support.

All major cloud providers (including S3, GCS, Azure) and static file HTTP server applications support requesting suffixes.
Here is an example against S3:

curl -H 'Range: bytes=-524292' https://static.webknossos.org/data/zarr_v3/l4_sample/color/1/c/0/3/3/1 | wc -c

From apache/arrow-rs#4612 it looks like multipart ranges aren't supported by any cloud provider (have yet to interact with a server implementation which does tbh, and you need to jump through a lot of hoops to interpret the response as, according to the HTTP spec, they can basically send you whatever they want) and that particular library gets around it by making a bunch of different requests in parallel. Presumably that isn't a blocker in and of itself, although making hundreds or thousands of requests to pull down a region of a single shard is hardly ideal, especially as any given backend could just decide to send you the entire shard, or indeed just about any subset of it, with each request!

Multipart ranges are not as widely supported. I know that S3 doesn't support it. Issuing single range requests per inner chunk is equivalent to using Zarr without sharding. So, I would argue sharding doesn't make the situation worse. On the contrary, implementations can choose to coalesce the byte ranges of multiple chunks into single requests to reduce the number of requests. This works especially well, if the chunks are laid out in an order that matches the common access pattern (e.g. Z-ordering). Implementations can also download entire shards. In our standard configuration that means reducing the number of requests by a factor of ~32,000.

[clbarnes]

Issuing single range requests per inner chunk is equivalent to using Zarr without sharding.

Yes, so long as the existence of sharding doesn't change users' heuristics on (sub)chunk layout. I suppose it comes down to whether you see sharding as a way to coalesce your small chunks into single files, or as a way to access inner regions of your large chunks!

In our standard configuration that means reducing the number of requests by a factor of ~32,000.

The "worst case" I was thinking about was when you need about half the chunk - clients may want some internal strategy trading off making thousands of small requests to download just what you need, or downloading considerably more than you need in 1 request to then slice the result.

[JackKelly]

clients may want some internal strategy trading off making thousands of small requests to download just what you need, or downloading considerably more than you need in 1 request to then slice the result

Yes, and perhaps the Zarr implementation would allow the user to specify the threshold. e.g. "I'm reading data from a RAID6 array of HDDs, which has very high bandwidth but also terrifyingly high latencies for random reads, so it's faster to read sequentially, even if I throw away 90% of the chunks after reading them from disk".

[constantinpape]

Hi everyone,
I will go on vacation next week, and don't have time to really read up on the changes here.
Overall I am very excited about sharding and its adoption in zarr, as this is a feature that I would extensively use myself / in our group down the line. I just have one high-level question before a formal vote:
How do the changes here affect zarr without sharding? I assume the default will still be to use normal chunking, and libraries that don't support sharding yet can just use the rest of the zarr spec as usual? (And would need to be adapted to raise an appropriate exception if it encounters a sharded zarr array)

[normanrz]

Sharding is specified as a new codec. That means that Zarr without sharding is not affected. Libraries that don't support it can still use the normal chunking. Of course, libraries that don't support sharding will not be able to open/read arrays that have been created with sharding.

[constantinpape]

Thanks for the clarification @normanrz !
Than I vote ABSTAIN (as I don't have the capacity to implement it in z5 myself), but with full support for going this direction!

[manzt]

I vote YES! Currently migrating zarrita.js (future for zarr.js) to ZEP0001 and hope to have the capacity to support ZEP0002.

[jbms]

I vote yes for tensorstore and neuroglancer!

[meggart]

I vote YES for Zarr.jl , it is definitely something I would need and use, but can not promise an implementation time line.