Document pickle behavior for subclasses of dicts/lists

[birkenfeld]

BPO	4712
Nosy	@birkenfeld, @rhettinger, @pitrou, @avassalotti, @mitsuhiko, @ericsnowcurrently, @serhiy-storchaka
Dependencies	bpo-3635: pickle.dumps cannot save instance of dict-derived class that overrides getattribute

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2008-12-21.17:24:03.973>
labels = ['easy', 'type-bug', 'docs']
title = 'Document pickle behavior for subclasses of dicts/lists'
updated_at = <Date 2015-11-26.22:11:25.100>
user = 'https://github.com/birkenfeld'

bugs.python.org fields:

activity = <Date 2015-11-26.22:11:25.100>
actor = 'serhiy.storchaka'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2008-12-21.17:24:03.973>
creator = 'georg.brandl'
dependencies = ['3635']
files = []
hgrepos = []
issue_num = 4712
keywords = ['easy']
message_count = 4.0
messages = ['78154', '83098', '255434', '255439']
nosy_count = 7.0
nosy_names = ['georg.brandl', 'rhettinger', 'pitrou', 'alexandre.vassalotti', 'aronacher', 'eric.snow', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4712'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

Linked PRs

• gh-48962: Fix duplicate values in the list when deepcopying a subclass list #101466

[birkenfeld]

When unpickling dict subclasses, the dict is filled via setitem before
__setstate__ is called. This, and other behavior around subclasses of
classes that have special pickle behavior should be documented.

[rhettinger]

The behavior is a PITA. It means that dict subclasses the redefine
__setitem__ have unpleasant pickling challenges. The __setitem__
insertions are called before the subclass can initialize.

The workaround involves a funky dance using __reduce__. See
collections.OrderedDict::reduce() for an example.

[serhiy-storchaka]

The copy module uses the same __reduce__ protocol, but reconstruct the object in different order, first set state, then add items. This discrepancy causes a difference between results of pickle/unpickle and copy operations. Example:

>>> class L(list):
...     def __getstate__(self):
...         return list(self)
...     def __setstate__(self, state):
...         self[:] = state
... 
>>> import copy, pickle
>>> pickle.loads(pickle.dumps(L([1, 2])))
[1, 2]
>>> copy.deepcopy(L([1, 2]))
[1, 2, 1, 2]

This was happened with xml.dom.minicompat.NodeList (bpo-10131).

Definitely one of pickle's or copy's behavior should be changed. But what?

[serhiy-storchaka]

The order in copy was changed by bpo-1100562.

[furkanonder]

The copy module uses the same reduce protocol, but reconstruct the object in different order, first set state, then add items. This discrepancy causes a difference between results of pickle/unpickle and copy operations. Example:

>>> class L(list):
...     def __getstate__(self):
...         return list(self)
...     def __setstate__(self, state):
...         self[:] = state
... 
>>> import copy, pickle
>>> pickle.loads(pickle.dumps(L([1, 2])))
[1, 2]
>>> copy.deepcopy(L([1, 2]))
[1, 2, 1, 2]

This was happened with xml.dom.minicompat.NodeList (bpo-10131).

Definitely one of pickle's or copy's behavior should be changed. But what?

@serhiy-storchaka I solved the problem of duplicate values in list while copying the subclass of list. You can see my PR in here.