Skip to content

Commit 37b0acc

Browse files
committed
add v3 blog post
1 parent d713c62 commit 37b0acc

File tree

1 file changed

+71
-0
lines changed

1 file changed

+71
-0
lines changed
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
---
2+
layout: post
3+
title: "Zarr protocol v3 design update"
4+
date: 2019-06-19
5+
categories: zarr specs
6+
---
7+
8+
Today I put together some [slides summarising the current state of
9+
exploratory work on the Zarr v3 protocol
10+
spec](https://zarr-developers.github.io/slides/v3-update-20190619.html). The
11+
purpose of this blog post is to share those slides more widely, and to
12+
provide some context explaining why work has started on a v3 spec.
13+
14+
## Why work on a v3 spec?
15+
16+
The [current (v2) Zarr
17+
spec](https://zarr.readthedocs.io/en/stable/spec/v2.html) is
18+
implemented in a number of software libraries, and is a stable and
19+
robust protocol that is used in production in a number of different
20+
scientific communities. If you need to store and compute in parallel
21+
against large array-like data, it's a good solution. So why start
22+
thinking about a new protocol version?
23+
24+
### Language-agnostic
25+
26+
One reason is that the v2 protocol is somewhat Python-centric, and
27+
includes some features which are not straightforward to implement in
28+
other languages. This has meant that implementations do not all
29+
support the same feature set. It would be good to have a minimal v3
30+
protocol spec that could be fully implemented in any language, so all
31+
implementations have parity around a core feature set.
32+
33+
### Unifying Zarr and N5
34+
35+
Another reason is that we would like to merge development efforts
36+
between the Zarr and N5 communities, and so a goal for the v3 spec is
37+
to unify the two approaches and provide a common implementation
38+
target.
39+
40+
### Extensibility
41+
42+
A third reason is that a number of different groups have started
43+
experimenting and extending the Zarr protocol in interesting ways, but
44+
it's not always clear how to extend the v2 protocol to support new
45+
features. It would be good if the v3 spec provided a variety of clear
46+
extension points and extension mechanisms.
47+
48+
### Cloud storage
49+
50+
Finally, while the v2 spec can be used very effectively with
51+
distributed storage systems like Amazon S3 or Google Cloud Storage,
52+
there is room for improvement, particularly regarding how metadata is
53+
stored and organised.
54+
55+
## Zarr v3 design update
56+
57+
I you are interested in knowing more about the current status of work
58+
on the v3 spec, please take a look at the [v3 design update
59+
slides](https://zarr-developers.github.io/slides/v3-update-20190619.html). The
60+
slides use reveal.js and have both horizontal and vertical
61+
navigation - if you haven't seen that before, then navigate downwards
62+
first wherever you can, before navigating to the right.
63+
64+
As I mention in the slides, the current v3 spec is just a straw man,
65+
meant to illustrate some ideas and potential solutions, but everything
66+
is up for discussion. So if you have any comments or ideas, please do
67+
get in touch, anyone is welcome to participate.
68+
69+
----
70+
71+
Blog post written by [Alistair Miles](https://github.com/alimanfoo).

0 commit comments

Comments
 (0)