Skip to content

Commit c196e56

Browse files
committed
final edits
1 parent 2984f7f commit c196e56

File tree

1 file changed

+44
-16
lines changed

1 file changed

+44
-16
lines changed

_posts/2019-05-02-zarr-2.3-release.md

Lines changed: 44 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -138,24 +138,39 @@ adapter](https://zarr.readthedocs.io/en/stable/api/n5.html#zarr.n5.N5Store)
138138
which allows reading and writing of data on disk in the N5
139139
format.
140140

141-
## Support of the buffer protocol
142-
143-
The Python buffer protocol for those that are familiar is a specification for
144-
how to share large blocks of memory between different libraries without
145-
copying. This protocol has evolved over time from its original introduction in
146-
Python 2 and later revamped implementation added in Python 3 (with backports to
147-
Python 2.6 and 2.7). Due to the changes in its behavior from Python 2 to Python
148-
3 and what objects supported which implementation of the buffer protocol, it
149-
was a bit challenging to leverage effectively in Zarr. This despite the value
150-
of having zero-copy when storing data or performing filtering and compression.
151-
152-
Thanks to some under-the-hood changes in Numcodecs 0.6 and Zarr 2.3, the buffer
153-
protocol is now cleanly supported for Python 2/3 in both libraries when working
154-
with data. In addition to improved memory handling and performance, this should
155-
make it easier for users developing their own stores, compressors, and filters
156-
to use with Zarr. Also it has cutdown on the amount of code specialized for
141+
142+
## Support for the buffer protocol
143+
144+
Zarr is intended to work efficiently across a range of different
145+
storage systems with different latencies and bandwidth, from cloud
146+
object stores to local disk and memory. In many of these settings,
147+
making efficient use of local memory, and avoiding memory copies
148+
wherever possible, can make a substantial difference to
149+
performance. This is particularly true within the
150+
[Numcodecs](http://numcodecs.rtfd.io) package, which is a companion to
151+
Zarr and provides implementations of compression and filter codecs
152+
such as Blosc and Zstandard. A key aspect of achieving fewer memory
153+
copies has been to leverage the Python buffer protocol.
154+
155+
The [Python buffer
156+
protocol](https://docs.python.org/3/c-api/buffer.html) is a
157+
specification for how to share large blocks of memory between
158+
different libraries without copying. This protocol has evolved over
159+
time from its original introduction in Python 2 and later revamped
160+
implementation added in Python 3 (with backports to Python 2.6 and
161+
2.7). Due to the changes in its behavior from Python 2 to Python 3 and
162+
what objects supported which implementation of the buffer protocol, it
163+
was a bit challenging to leverage effectively in Zarr.
164+
165+
Thanks to some under-the-hood changes in Zarr 2.3 and Numcodecs 0.6,
166+
the buffer protocol is now cleanly supported for Python 2/3 in both
167+
libraries when working with data. In addition to improved memory
168+
handling and performance, this should make it easier for users
169+
developing their own stores, compressors, and filters to use with
170+
Zarr. Also it has cutdown on the amount of code specialized for
157171
handling different Python versions.
158172

173+
159174
## Future developments
160175

161176
There is a growing community of interest around new approaches to
@@ -190,3 +205,16 @@ developments, keep an eye on the
190205
[zarr-specs](https://github.com/zarr-developers/zarr-specs) GitHub
191206
repositories, and please feel free to raise issues or add comments if
192207
you have any questions or ideas.
208+
209+
210+
## And finally... SciPy!
211+
212+
If you're coming to SciPy this year, we're very pleased to be giving a
213+
talk on Zarr on [day 1 of the conference (Wednesday 10
214+
July)](https://www.eiseverywhere.com/ehome/381993). Several members of
215+
the Zarr community will be at the conference, and there are sprints
216+
going on after the conference in a number of related areas, including
217+
an Xarray sprint on the Saturday. Please do say hi or [drop us a
218+
comment on this
219+
issue](https://github.com/zarr-developers/zarr/issues/396) if you'd
220+
like to connect and discuss anything.

0 commit comments

Comments
 (0)