implementation details in sys module

[fijall]

BPO	10994
Nosy	@loewis, @brettcannon, @arigo, @terryjreedy, @pitrou, @ambv

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-01-24.13:43:39.461>
labels = ['type-bug', 'docs']
title = 'implementation details in sys module'
updated_at = <Date 2013-01-22.12:49:37.730>
user = 'https://bugs.python.org/fijall'

bugs.python.org fields:

activity = <Date 2013-01-22.12:49:37.730>
actor = 'ezio.melotti'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2011-01-24.13:43:39.461>
creator = 'fijall'
dependencies = []
files = []
hgrepos = []
issue_num = 10994
keywords = []
message_count = 14.0
messages = ['126925', '126926', '126927', '126928', '127018', '127023', '127025', '127026', '127027', '127028', '127029', '127878', '127900', '136703']
nosy_count = 8.0
nosy_names = ['loewis', 'brett.cannon', 'arigo', 'terry.reedy', 'pitrou', 'docs@python', 'lukasz.langa', 'fijall']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10994'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

[fijall]

sys module documentation (as it is online) has some things that in my opinion should be marked as implementation details, but are not. Feel free to counter why not.

Some of them has info it should be used for specialized purposes only, but IMO it's not the same as not mandatory for other implementations.

Temporary list:

_clear_type_cache

dllhandle

getrefcount

getdlopenflags (?)

getsizeof - it might be not well defined on other implementations

setdlopenflags

api_version

[pitrou]

Well, getsizeof is not better-defined under CPython than elsewhere. It just gives a hint.
Agreed about the other.

[fijall]

I suppose wrt getsizeof it's more of "if you provide us with a reasonable expectations, we can implement this" other than anything else.

[pitrou]

I suppose wrt getsizeof it's more of "if you provide us with a
reasonable expectations, we can implement this" other than anything
else.

The expectation is that it returns the memory footprint of the given
object, and only it (not taking into account sharing, caching,
dependencies or anything else). For example, an instance will not count
its attribute __dict__. But a str object will count its object header
plus the string payload, if the payload is private.

Of course, you are free to tweak these semantics for the PyPy
implementation.

[arigo]

The expectation is that it returns the memory footprint of the given
object, and only it (not taking into account sharing, caching,
dependencies or anything else).

It would be nice if this was a well-defined definition, but unfortunately it is not. For example, string objects may appear different from the user's point of view (e.g. as seen by id() and 'is') but share the implementation's data; they may even share only a part of it (if ropes are enabled). Conversely, for user-defined objects you would typically think not to count the "shape" information, which is usually shared among several instances -- but then you risk a gross under-estimation in the (rarer) cases where it is not shared.

Another way to look at the "official" definition is to return the size of the object itself and none of its dependencies, because in theory they might be shared; but that would make all strings, lists, tuples, dicts, and so on have a getsizeof() of 8 or 12, which is rather useless.

I hope this clarifies fijal's original comment: "it might be not well defined on other implementations."

[pitrou]

> The expectation is that it returns the memory footprint of the given
> object, and only it (not taking into account sharing, caching,
> dependencies or anything else).

It would be nice if this was a well-defined definition, but
unfortunately it is not.

I didn't claim it was. Actually, if you read the rest of my message, I
did mention that PyPy could tweak the semantics if it made more sense.
So, of course, the more sharing and caching takes place, the less
obvious these semantics are, but even with CPython they are not obvious
anyway. It's not supposed to be an exact measurement for the common
developer, rather a hint that experts can use to tweak their data
structures and algorithms; you need to know details of your VM's
implementation to use that information.

[fijall]

I can hardly think about a specification that would potentially help me identify actual sizes. Even as a rough estimation. Which experts you had in mind?

[pitrou]

Which experts you had in mind?

People who know how the Python implementation works.

[fijall]

> Which experts you had in mind?

People who know how the Python implementation works.

I'm serious. What semantics would make sense to anyone? Even if you know implementation quite well a single number per object does not provide enough information.

[brettcannon]

You could return -1 for everything. =)

In all seriousness, it could simply be proportional. IMO as long as people realize if a list takes up less space than a dict then the numbers seem fine to me.

[pitrou]

Even if you know implementation quite well a single number per object
does not provide enough information.

Enough information for what? It can certainly provide information about
the overhead of that particular object (again, regardless of sharing).

[loewis]

I can propose a specification of getsizeof: if you somehow manage to traverse all objects (without considering an object twice), and sum up the getsizeof results, you should end up with something close to, but smaller than the actual memory consumption. How close is a quality-of-implementation issue (so always returning 0 would be correct-but-useless).

It may be that implementations can also support counting certain hidden memory usage (headers, blocks shared across instances that are not objects themselves). Such functions would should have different names and interfaces (e.g. sys.gethiddenblocks(o) may return a list of (address, size) pairs); CPython doesn't provide any such function (although sys.mallocoverhead might be useful).

In any case: I'm not convinced that it is useful to mark functions as CPython-specific in the documentation. This clutters the documentation, and is of interest only for language lawyers. So if implementation details are to be documented, I'd prefer this to happen in a separate document.