-
-
Notifications
You must be signed in to change notification settings - Fork 33.1k
Description
Bug report
Bug description:
Is framing supposed to be implemented in the C Pickle Accelerator? PEP 3154 describes framing and some of the rules for it. It also says
A well-written C implementation doesn’t need additional memory copies for the framing layer, preserving general (un)pickling efficiency.
I'll be honest I don't super understand what this means, but it doesn't seem that it's saying it shouldn't be implemented in C. Framing seems to be implemented in the C accelerator when dumping/saving a pickle, but not loading. The load_frame
function in fact only does one thing when it encounters the FRAME
opcode - it only checks that frame_len
bytes exist in the stream, then continues to do what it's doing. It never reads frame_len
bytes into a buffer in memory to prevent lots of small reads by future opcodes (from what I can tell).
Because framing isn't really implemented in _pickle.c
, there are a few pickles that fail to be loaded in pickle.py
, but are just fine in _pickle.c
. For example:
- Python checks that up to
frame_len
bytes exist in the stream. If the argument for theFRAME
opcode is larger than the number of bytes available, the Python implementation will simply read as many bytes as possible into the memory buffer and continue on, while the C implementation is looking for at a minimumframe_len
bytes in the stream. - Breaking an opcode argument across frames (which is specifically prohibited by PEP 3154) is not checked in
_pickle.c
, and does not error out. - Overlapping frames are checked for in
pickle.py
but are not checked in_pickle.c
and those pickles do not error out.
Is this intended behavior? I tried looking through old issues to see if there has been any discussion about this, but I haven't found any.
CPython versions tested on:
3.11
Operating systems tested on:
Linux
Metadata
Metadata
Assignees
Labels
Projects
Status