@@ -118,53 +118,58 @@ ds = cat.atmosphere.gmet_v1.to_dask()
118118...and [ here's the underlying catalog
119119entry] ( https://github.com/pangeo-data/pangeo-datastore/blob/aa3f12bcc3be9584c1a9071235874c9d6af94a4e/intake-catalogs/atmosphere.yaml#L6 ) .
120120
121+
121122## Compatibility with N5
122123
123124Around the same time that development on Zarr was getting started, a
124- separate team led by Stephan Saafeld at the Janelia research campus
125- was experiencing similar frustrations trying to use HDF5 for storing
126- brain imaging data, and developed a software library called
127- N5. Although N5 was originally implemented in Java, it has strong
128- similarities to Zarr in the approach it takes to storing both metadata
129- and data chunks. We became aware of each others' work around a year
130- ago and have been learning about the two approaches.
131-
132- The bottom line is that there is a lot of commonality and we are
133- working jointly to bring the two approaches together. As a first
134- experimental step towards that goal, the Zarr 2.3 release includes a
135- storage adapter which allows reading and writing of data on disk in
136- the N5 format. Here's an example:
125+ separate team led by [ Stephan
126+ Saafeld] ( https://www.janelia.org/lab/saalfeld-lab ) at the Janelia
127+ research campus was experiencing similar challenges storing and
128+ computing with large amounts of neural imaging data, and developed a
129+ software library called [ N5] ( https://github.com/saalfeldlab/n5 ) . N5 is
130+ implemented in Java but is very similar to Zarr in the approach it
131+ takes to storing both metadata and data chunks, and to decoupling the
132+ storage backend to enable efficient use of cloud storage.
133+
134+ There is a lot of commonality between Zarr and N5 and we are working
135+ jointly to bring the two approaches together. As a first experimental
136+ step towards that goal, the Zarr 2.3 release includes an [ N5 storage
137+ adapter] ( https://zarr.readthedocs.io/en/stable/api/n5.html#zarr.n5.N5Store )
138+ which allows reading and writing of data on disk in the N5
139+ format.
137140
138- {% highlight python %}
139- TODO
140- {% endhighlight %}
141141
142142## Future developments
143143
144144There is a growing community of interest around new approaches to
145- storage of array-like data, and we'd like to do what we can to build
146- connections and share knowledge and ideas between these
147- communities. We've started a [ regular teleconference] ( @@TODO ) which is
148- open to anyone to join, and there is a new [ gitter channel] ( @@TODO )
149- for general discussion.
145+ storage of array-like data, particularly in the cloud. For example,
146+ Theo McCaie from the UK Met Office Informatics Lab recently wrote a
147+ series of blog posts about the challenges involved in [ storing 200TB
148+ of "high momentum" weather model data every
149+ day] ( https://medium.com/informatics-lab/creating-a-data-format-for-high-momentum-datasets-a394fa48b671 ) . This
150+ is an exciting space to be working in and we'd like to do what we can
151+ to build connections and share knowledge and ideas between
152+ communities. We've started a [ regular
153+ teleconference] ( https://github.com/zarr-developers/zarr/issues/315 )
154+ which is open to anyone to join, and there is a new [ gitter
155+ channel] ( https://gitter.im/zarr-developers/community ) for general
156+ discussion.
150157
151158The main focus of our conversations so far has been setting up work
152159towards development of a new set of specifications that support the
153- features of both Zarr and N5, and the requirements of key user groups,
154- including Earth and climate sciences, bioimaging and microscopy, and
155- data derived from large scale DNA and RNA sequencing experiments. It
156- is still relatively early days and there are lots of open questions to
157- work through, both on the technical side and in terms of how we
158- organise and coordinate efforts. However, the community is very
159- friendly and supportive, and anyone is welcome to participate, so if
160- you have an interest please do consider getting involved.
161-
162- If you would like to stay in touch with where things are going, keep
163- an eye on the [ zarr] ( @@TODO ) and [ zarr-specs] ( @@TODO ) GitHub
164- repositories, join the [ gitter channel] ( @@TODO ) , and please feel free
165- to raise issues or add comments if you have any questions or ideas.
166-
167- ## Further reading
168-
169- * [ Zarr Python 2.3 release notes] ( @@TODO )
170- * anything else...
160+ features of both Zarr and N5, and provide a platform for exploration
161+ and development of new features, while also identifying a minimal core
162+ protocol that can be implemented in a range of different programming
163+ languages. It is still relatively early days and there are lots of
164+ open questions to work through, both on the technical side and in
165+ terms of how we organise and coordinate efforts. However, the
166+ community is very friendly and supportive, and anyone is welcome to
167+ participate, so if you have an interest please do consider getting
168+ involved.
169+
170+ If you would like to stay in touch with or contribute to new
171+ developments, keep an eye on the
172+ [ zarr] ( https://github.com/zarr-developers/zarr ) and
173+ [ zarr-specs] ( https://github.com/zarr-developers/zarr-specs ) GitHub
174+ repositories, and please feel free to raise issues or add comments if
175+ you have any questions or ideas.
0 commit comments