Commit 52c93c3
Use (new) buffer protocol in
* Use (new) buffer protocol in `Pickle.decode`
On Python 3, `pickle.loads` is able to take anything that conforms to
the (new) buffer protocol. So there has been no issue giving it an
`ndarray` to work with. Unfortunately, on Python 2, `pickle.loads`
requires a `bytes` object specifically and is not able to take any type
implementing the (new) buffer protocol. So we have been going ahead and
coercing everything to `bytes` on Python 2.
However it turns out that `cStringIO`'s `StringIO` on Python 2 does
support the buffer protocol. Thus a `StringIO` object can be created on
Python 2 without copying the data. While this still cannot be used with
`pickle.loads`, it can be used with `pickle.load`, which special cases
reading from `StringIO` leveraging the read function, which amounts to
sharing a pointer between `StringIO` and `pickle.loads`. Thus achieving
a no copying unpickler for Python 2.
ref: http://www.hydrogen18.com/blog/unpickling-buffers.html
ref: https://github.com/python/cpython/blob/2.7/Modules/cStringIO.c#L716
ref: https://github.com/python/cpython/blob/2.7/Modules/cStringIO.c#L681
ref: https://github.com/python/cpython/blob/2.7/Modules/cPickle.c#L614
ref: https://github.com/python/cpython/blob/2.7/Modules/cStringIO.c#L160
* Link this PR to `Pickle.decode` release notePickle.decode on Python 2 (#150)1 parent 0c264a1 commit 52c93c3
2 files changed
+7
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
| 52 | + | |
| 53 | + | |
51 | 54 | | |
52 | | - | |
| 55 | + | |
53 | 56 | | |
54 | | - | |
| 57 | + | |
55 | 58 | | |
56 | | - | |
57 | 59 | | |
58 | 60 | | |
59 | 61 | | |
| |||
0 commit comments