|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: "Zarr protocol v3 design update" |
| 4 | +date: 2019-06-19 |
| 5 | +categories: zarr specs |
| 6 | +--- |
| 7 | + |
| 8 | +Today I put together some [slides summarising the current state of |
| 9 | +exploratory work on the Zarr v3 protocol |
| 10 | +spec](https://zarr-developers.github.io/slides/v3-update-20190619.html). The |
| 11 | +purpose of this blog post is to share those slides more widely, and to |
| 12 | +provide some context explaining why work has started on a v3 spec. |
| 13 | + |
| 14 | +## Why work on a v3 spec? |
| 15 | + |
| 16 | +The [current (v2) Zarr |
| 17 | +spec](https://zarr.readthedocs.io/en/stable/spec/v2.html) is |
| 18 | +implemented in a number of software libraries, and is a stable and |
| 19 | +robust protocol that is used in production in a number of different |
| 20 | +scientific communities. If you need to store and compute in parallel |
| 21 | +against large array-like data, it's a good solution. So why start |
| 22 | +thinking about a new protocol version? |
| 23 | + |
| 24 | +### Language-agnostic |
| 25 | + |
| 26 | +One reason is that the v2 protocol is somewhat Python-centric, and |
| 27 | +includes some features which are not straightforward to implement in |
| 28 | +other languages. This has meant that implementations do not all |
| 29 | +support the same feature set. It would be good to have a minimal v3 |
| 30 | +protocol spec that could be fully implemented in any language, so all |
| 31 | +implementations have parity around a core feature set. |
| 32 | + |
| 33 | +### Unifying Zarr and N5 |
| 34 | + |
| 35 | +Another reason is that we would like to merge development efforts |
| 36 | +between the Zarr and N5 communities, and so a goal for the v3 spec is |
| 37 | +to unify the two approaches and provide a common implementation |
| 38 | +target. |
| 39 | + |
| 40 | +### Extensibility |
| 41 | + |
| 42 | +A third reason is that a number of different groups have started |
| 43 | +experimenting and extending the Zarr protocol in interesting ways, but |
| 44 | +it's not always clear how to extend the v2 protocol to support new |
| 45 | +features. It would be good if the v3 spec provided a variety of clear |
| 46 | +extension points and extension mechanisms. |
| 47 | + |
| 48 | +### Cloud storage |
| 49 | + |
| 50 | +Finally, while the v2 spec can be used very effectively with |
| 51 | +distributed storage systems like Amazon S3 or Google Cloud Storage, |
| 52 | +there is room for improvement, particularly regarding how metadata is |
| 53 | +stored and organised. |
| 54 | + |
| 55 | +## Zarr v3 design update |
| 56 | + |
| 57 | +I you are interested in knowing more about the current status of work |
| 58 | +on the v3 spec, please take a look at the [v3 design update |
| 59 | +slides](https://zarr-developers.github.io/slides/v3-update-20190619.html). The |
| 60 | +slides use reveal.js and have both horizontal and vertical |
| 61 | +navigation - if you haven't seen that before, then navigate downwards |
| 62 | +first wherever you can, before navigating to the right. |
| 63 | + |
| 64 | +As I mention in the slides, the current v3 spec is just a straw man, |
| 65 | +meant to illustrate some ideas and potential solutions, but everything |
| 66 | +is up for discussion. So if you have any comments or ideas, please do |
| 67 | +get in touch, anyone is welcome to participate. |
| 68 | + |
| 69 | +---- |
| 70 | + |
| 71 | +Blog post written by [Alistair Miles](https://github.com/alimanfoo). |
0 commit comments