Skip to content

Commit eb5bea0

Browse files
committed
edit later sections
1 parent 57957a3 commit eb5bea0

File tree

1 file changed

+44
-39
lines changed

1 file changed

+44
-39
lines changed

_posts/2019-05-02-zarr-2.3-release.md

Lines changed: 44 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -118,53 +118,58 @@ ds = cat.atmosphere.gmet_v1.to_dask()
118118
...and [here's the underlying catalog
119119
entry](https://github.com/pangeo-data/pangeo-datastore/blob/aa3f12bcc3be9584c1a9071235874c9d6af94a4e/intake-catalogs/atmosphere.yaml#L6).
120120

121+
121122
## Compatibility with N5
122123

123124
Around the same time that development on Zarr was getting started, a
124-
separate team led by Stephan Saafeld at the Janelia research campus
125-
was experiencing similar frustrations trying to use HDF5 for storing
126-
brain imaging data, and developed a software library called
127-
N5. Although N5 was originally implemented in Java, it has strong
128-
similarities to Zarr in the approach it takes to storing both metadata
129-
and data chunks. We became aware of each others' work around a year
130-
ago and have been learning about the two approaches.
131-
132-
The bottom line is that there is a lot of commonality and we are
133-
working jointly to bring the two approaches together. As a first
134-
experimental step towards that goal, the Zarr 2.3 release includes a
135-
storage adapter which allows reading and writing of data on disk in
136-
the N5 format. Here's an example:
125+
separate team led by [Stephan
126+
Saafeld](https://www.janelia.org/lab/saalfeld-lab) at the Janelia
127+
research campus was experiencing similar challenges storing and
128+
computing with large amounts of neural imaging data, and developed a
129+
software library called [N5](https://github.com/saalfeldlab/n5). N5 is
130+
implemented in Java but is very similar to Zarr in the approach it
131+
takes to storing both metadata and data chunks, and to decoupling the
132+
storage backend to enable efficient use of cloud storage.
133+
134+
There is a lot of commonality between Zarr and N5 and we are working
135+
jointly to bring the two approaches together. As a first experimental
136+
step towards that goal, the Zarr 2.3 release includes an [N5 storage
137+
adapter](https://zarr.readthedocs.io/en/stable/api/n5.html#zarr.n5.N5Store)
138+
which allows reading and writing of data on disk in the N5
139+
format.
137140

138-
{% highlight python %}
139-
TODO
140-
{% endhighlight %}
141141

142142
## Future developments
143143

144144
There is a growing community of interest around new approaches to
145-
storage of array-like data, and we'd like to do what we can to build
146-
connections and share knowledge and ideas between these
147-
communities. We've started a [regular teleconference](@@TODO) which is
148-
open to anyone to join, and there is a new [gitter channel](@@TODO)
149-
for general discussion.
145+
storage of array-like data, particularly in the cloud. For example,
146+
Theo McCaie from the UK Met Office Informatics Lab recently wrote a
147+
series of blog posts about the challenges involved in [storing 200TB
148+
of "high momentum" weather model data every
149+
day](https://medium.com/informatics-lab/creating-a-data-format-for-high-momentum-datasets-a394fa48b671). This
150+
is an exciting space to be working in and we'd like to do what we can
151+
to build connections and share knowledge and ideas between
152+
communities. We've started a [regular
153+
teleconference](https://github.com/zarr-developers/zarr/issues/315)
154+
which is open to anyone to join, and there is a new [gitter
155+
channel](https://gitter.im/zarr-developers/community) for general
156+
discussion.
150157

151158
The main focus of our conversations so far has been setting up work
152159
towards development of a new set of specifications that support the
153-
features of both Zarr and N5, and the requirements of key user groups,
154-
including Earth and climate sciences, bioimaging and microscopy, and
155-
data derived from large scale DNA and RNA sequencing experiments. It
156-
is still relatively early days and there are lots of open questions to
157-
work through, both on the technical side and in terms of how we
158-
organise and coordinate efforts. However, the community is very
159-
friendly and supportive, and anyone is welcome to participate, so if
160-
you have an interest please do consider getting involved.
161-
162-
If you would like to stay in touch with where things are going, keep
163-
an eye on the [zarr](@@TODO) and [zarr-specs](@@TODO) GitHub
164-
repositories, join the [gitter channel](@@TODO), and please feel free
165-
to raise issues or add comments if you have any questions or ideas.
166-
167-
## Further reading
168-
169-
* [Zarr Python 2.3 release notes](@@TODO)
170-
* anything else...
160+
features of both Zarr and N5, and provide a platform for exploration
161+
and development of new features, while also identifying a minimal core
162+
protocol that can be implemented in a range of different programming
163+
languages. It is still relatively early days and there are lots of
164+
open questions to work through, both on the technical side and in
165+
terms of how we organise and coordinate efforts. However, the
166+
community is very friendly and supportive, and anyone is welcome to
167+
participate, so if you have an interest please do consider getting
168+
involved.
169+
170+
If you would like to stay in touch with or contribute to new
171+
developments, keep an eye on the
172+
[zarr](https://github.com/zarr-developers/zarr) and
173+
[zarr-specs](https://github.com/zarr-developers/zarr-specs) GitHub
174+
repositories, and please feel free to raise issues or add comments if
175+
you have any questions or ideas.

0 commit comments

Comments
 (0)