@@ -138,24 +138,39 @@ adapter](https://zarr.readthedocs.io/en/stable/api/n5.html#zarr.n5.N5Store)
138138which allows reading and writing of data on disk in the N5
139139format.
140140
141- ## Support of the buffer protocol
142-
143- The Python buffer protocol for those that are familiar is a specification for
144- how to share large blocks of memory between different libraries without
145- copying. This protocol has evolved over time from its original introduction in
146- Python 2 and later revamped implementation added in Python 3 (with backports to
147- Python 2.6 and 2.7). Due to the changes in its behavior from Python 2 to Python
148- 3 and what objects supported which implementation of the buffer protocol, it
149- was a bit challenging to leverage effectively in Zarr. This despite the value
150- of having zero-copy when storing data or performing filtering and compression.
151-
152- Thanks to some under-the-hood changes in Numcodecs 0.6 and Zarr 2.3, the buffer
153- protocol is now cleanly supported for Python 2/3 in both libraries when working
154- with data. In addition to improved memory handling and performance, this should
155- make it easier for users developing their own stores, compressors, and filters
156- to use with Zarr. Also it has cutdown on the amount of code specialized for
141+
142+ ## Support for the buffer protocol
143+
144+ Zarr is intended to work efficiently across a range of different
145+ storage systems with different latencies and bandwidth, from cloud
146+ object stores to local disk and memory. In many of these settings,
147+ making efficient use of local memory, and avoiding memory copies
148+ wherever possible, can make a substantial difference to
149+ performance. This is particularly true within the
150+ [ Numcodecs] ( http://numcodecs.rtfd.io ) package, which is a companion to
151+ Zarr and provides implementations of compression and filter codecs
152+ such as Blosc and Zstandard. A key aspect of achieving fewer memory
153+ copies has been to leverage the Python buffer protocol.
154+
155+ The [ Python buffer
156+ protocol] ( https://docs.python.org/3/c-api/buffer.html ) is a
157+ specification for how to share large blocks of memory between
158+ different libraries without copying. This protocol has evolved over
159+ time from its original introduction in Python 2 and later revamped
160+ implementation added in Python 3 (with backports to Python 2.6 and
161+ 2.7). Due to the changes in its behavior from Python 2 to Python 3 and
162+ what objects supported which implementation of the buffer protocol, it
163+ was a bit challenging to leverage effectively in Zarr.
164+
165+ Thanks to some under-the-hood changes in Zarr 2.3 and Numcodecs 0.6,
166+ the buffer protocol is now cleanly supported for Python 2/3 in both
167+ libraries when working with data. In addition to improved memory
168+ handling and performance, this should make it easier for users
169+ developing their own stores, compressors, and filters to use with
170+ Zarr. Also it has cutdown on the amount of code specialized for
157171handling different Python versions.
158172
173+
159174## Future developments
160175
161176There is a growing community of interest around new approaches to
@@ -190,3 +205,16 @@ developments, keep an eye on the
190205[ zarr-specs] ( https://github.com/zarr-developers/zarr-specs ) GitHub
191206repositories, and please feel free to raise issues or add comments if
192207you have any questions or ideas.
208+
209+
210+ ## And finally... SciPy!
211+
212+ If you're coming to SciPy this year, we're very pleased to be giving a
213+ talk on Zarr on [ day 1 of the conference (Wednesday 10
214+ July)] ( https://www.eiseverywhere.com/ehome/381993 ) . Several members of
215+ the Zarr community will be at the conference, and there are sprints
216+ going on after the conference in a number of related areas, including
217+ an Xarray sprint on the Saturday. Please do say hi or [ drop us a
218+ comment on this
219+ issue] ( https://github.com/zarr-developers/zarr/issues/396 ) if you'd
220+ like to connect and discuss anything.
0 commit comments