Intern dataclass field names to improve performance

[XuehaiPan]

Feature or enhancement

Proposal:

Interning string literals can speed up dictionary lookup. Ref: sys.intern

sys.intern(string)

Enter string in the table of “interned” strings and return the interned string – which is string itself or a copy. Interning strings is useful to gain a little performance on dictionary lookup – if the keys in a dictionary are interned, and the lookup key is interned, the key comparisons (after hashing) can be done by a pointer compare instead of a string compare. Normally, the names used in Python programs are automatically interned, and the dictionaries used to hold module, class or instance attributes have interned keys.

We already intern the typename and field names for namedtuple.

cpython/Lib/collections/__init__.py

Line 384 in a971574

typename = _sys.intern(str(typename))

cpython/Lib/collections/__init__.py

Line 424 in a971574

field_names = tuple(map(_sys.intern, field_names))

We can make similar improvements for dataclass.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-112653: Intern dataclass field names to improve performance #112657

[serhiy-storchaka]

In namedtuple() the type name and the field names are interned because they are provided as string arguments. The field names can even be the result of str.split(), so they are definitely not interned. But all variable names and attribute names are interned. I expected the field names in a dataclass be interned. If it is not so, it should be solved on the lower level.

[XuehaiPan]

But all variable names and attribute names are interned.

I can confirm this if we create dataclasses from type annotations:

import dataclasses
import sys


@dataclasses.dataclass
class Foo:
    my_field_0: str


field_name = dataclasses.fields(Foo)[0].name  # 'my_field_0'

assert sys.intern(field_name) is field_name  # the field name in the annotation is interned

---

In namedtuple() the type name and the field names are interned because they are provided as string arguments. The field names can even be the result of str.split(), so they are definitely not interned. But all variable names and attribute names are interned.

If it is not so, it should be solved on the lower level.

The field names for a dataclass can also be the result of other functions. There is a way to create dataclasses dynamicly using dataclasses.make_dataclass(), which does not use the @dataclass.dataclass decorator. Just like collections.namedtuple() vs. typing.NamedTuple.

import dataclasses
import sys


Foo = dataclasses.make_dataclass('Foo', [(f'my_field_{0}', str)])


field_name = dataclasses.fields(Foo)[0].name  # 'my_field_0'

assert sys.intern(field_name) is not field_name  # the field name is not interned