Skip to content

[mypyc] Intern a few more strings #20537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ilevkivskyi merged 1 commit into from
Jan 10, 2026
+33 −12
Merged

[mypyc] Intern a few more strings #20537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions mypyc/lib-rt/exc_ops.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,11 +123,11 @@ static PyObject *CPy_GetTypeName(PyObject *type) {
PyObject *module = NULL, *name = NULL;
PyObject *full = NULL;

module = PyObject_GetAttrString(type, "__module__");
module = PyObject_GetAttr(type, mypyc_interned_str.__module__);
if (!module || !PyUnicode_Check(module)) {
goto out;
}
name = PyObject_GetAttrString(type, "__qualname__");
name = PyObject_GetAttr(type, mypyc_interned_str.__qualname__);
if (!name || !PyUnicode_Check(name)) {
goto out;
}
Expand Down
1 change: 1 addition & 0 deletions mypyc/lib-rt/librt_strings.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,5 +449,6 @@ static PyModuleDef librt_strings_module = {
PyMODINIT_FUNC
PyInit_strings(void)
{
intern_strings();
return PyModuleDef_Init(&librt_strings_module);
}
Comment on lines 450 to 454
Copy link
Collaborator Author

@cdce8p cdce8p Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few of the changed functions changed here are also called by librt. Therefore we need to initialize them here as well, not just for mypyc.

Technically this is a small performance penalty for other projects which only rely on librt. If that's relevant, we could consider separating the strings used by just mypyc and those used by both mypyc and librt.

Overall this will likely still be a net improvement though, since the strings are initialized / converted to a PyObject only once instead of for each function call.

/CC @ilevkivskyi

Copy link
Member

@ilevkivskyi ilevkivskyi Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the expected order of magnitude of the penalty? If it is under 1ms I think it is OK.

Also cc @JukkaL for visibility.

Copy link
Member

@ilevkivskyi ilevkivskyi Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(to be clear my question is specifically about import time, not the net performance)

Copy link
Collaborator Author

@cdce8p cdce8p Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure tbh, though all the function does is create a PyObject for each string in static_data.c and immortalize it. https://github.com/python/mypy/blob/master/mypyc/lib-rt/static_data.c

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_InternFromString

20 changes: 10 additions & 10 deletions mypyc/lib-rt/misc_ops.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ static bool _CPy_IsSafeMetaClass(PyTypeObject *metaclass) {
// manage to work with TypingMeta and its friends.
if (metaclass == &PyType_Type)
return true;
PyObject *module = PyObject_GetAttrString((PyObject *)metaclass, "__module__");
PyObject *module = PyObject_GetAttr((PyObject *)metaclass, mypyc_interned_str.__module__);
if (!module) {
PyErr_Clear();
return false;
Expand Down Expand Up @@ -242,7 +242,7 @@ PyObject *CPyType_FromTemplate(PyObject *template,
sizeof(PyTypeObject) - sizeof(PyVarObject));

if (bases != orig_bases) {
if (PyObject_SetAttrString((PyObject *)t, "__orig_bases__", orig_bases) < 0)
if (PyObject_SetAttr((PyObject *)t, mypyc_interned_str.__orig_bases__, orig_bases) < 0)
goto error;
}

Expand Down Expand Up @@ -285,7 +285,7 @@ PyObject *CPyType_FromTemplate(PyObject *template,

// Reject anything that would give us a nontrivial __slots__,
// because the layout will conflict
slots = PyObject_GetAttrString((PyObject *)t, "__slots__");
slots = PyObject_GetAttr((PyObject *)t, mypyc_interned_str.__slots__);
if (slots) {
// don't fail on an empty __slots__
int is_true = PyObject_IsTrue(slots);
Expand All @@ -298,7 +298,7 @@ PyObject *CPyType_FromTemplate(PyObject *template,
PyErr_Clear();
}

if (PyObject_SetAttrString((PyObject *)t, "__module__", modname) < 0)
if (PyObject_SetAttr((PyObject *)t, mypyc_interned_str.__module__, modname) < 0)
goto error;

if (init_subclass((PyTypeObject *)t, NULL))
Expand Down Expand Up @@ -458,7 +458,7 @@ CPyPickle_GetState(PyObject *obj)
{
PyObject *attrs = NULL, *state = NULL;

attrs = PyObject_GetAttrString((PyObject *)Py_TYPE(obj), "__mypyc_attrs__");
attrs = PyObject_GetAttr((PyObject *)Py_TYPE(obj), mypyc_interned_str.__mypyc_attrs__);
if (!attrs) {
goto fail;
}
Expand Down Expand Up @@ -734,7 +734,7 @@ int CPyStatics_Initialize(PyObject **statics,
// Call super(type(self), self)
PyObject *
CPy_Super(PyObject *builtins, PyObject *self) {
PyObject *super_type = PyObject_GetAttrString(builtins, "super");
PyObject *super_type = PyObject_GetAttr(builtins, mypyc_interned_str.super);
if (!super_type)
return NULL;
PyObject *result = PyObject_CallFunctionObjArgs(
Expand Down Expand Up @@ -889,7 +889,7 @@ CPy_CallReverseOpMethod(PyObject *left,
PyObject *CPySingledispatch_RegisterFunction(PyObject *singledispatch_func,
PyObject *cls,
PyObject *func) {
PyObject *registry = PyObject_GetAttrString(singledispatch_func, "registry");
PyObject *registry = PyObject_GetAttr(singledispatch_func, mypyc_interned_str.registry);
PyObject *register_func = NULL;
PyObject *typing = NULL;
PyObject *get_type_hints = NULL;
Expand All @@ -902,7 +902,7 @@ PyObject *CPySingledispatch_RegisterFunction(PyObject *singledispatch_func,
// passed a class
// bind cls to the first argument so that register gets called again with both the
// class and the function
register_func = PyObject_GetAttrString(singledispatch_func, "register");
register_func = PyObject_GetAttr(singledispatch_func, mypyc_interned_str.register_);
if (register_func == NULL) goto fail;
return PyMethod_New(register_func, cls);
}
Expand All @@ -923,7 +923,7 @@ PyObject *CPySingledispatch_RegisterFunction(PyObject *singledispatch_func,
func = cls;
typing = PyImport_ImportModule("typing");
if (typing == NULL) goto fail;
get_type_hints = PyObject_GetAttrString(typing, "get_type_hints");
get_type_hints = PyObject_GetAttr(typing, mypyc_interned_str.get_type_hints);

type_hints = PyObject_CallOneArg(get_type_hints, func);
PyObject *argname;
Expand All @@ -944,7 +944,7 @@ PyObject *CPySingledispatch_RegisterFunction(PyObject *singledispatch_func,
}

// clear the cache so we consider the newly added function when dispatching
PyObject *dispatch_cache = PyObject_GetAttrString(singledispatch_func, "dispatch_cache");
PyObject *dispatch_cache = PyObject_GetAttr(singledispatch_func, mypyc_interned_str.dispatch_cache);
if (dispatch_cache == NULL) goto fail;
PyDict_Clear(dispatch_cache);

Expand Down
10 changes: 10 additions & 0 deletions mypyc/lib-rt/static_data.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,13 @@ mypyc_interned_str_struct mypyc_interned_str;
int
intern_strings(void) {
INTERN_STRING(__init_subclass__, "__init_subclass__");
INTERN_STRING(__module__, "__module__");
INTERN_STRING(__mro_entries__, "__mro_entries__");
INTERN_STRING(__mypyc_attrs__, "__mypyc_attrs__");
INTERN_STRING(__name__, "__name__");
INTERN_STRING(__orig_bases__, "__orig_bases__");
INTERN_STRING(__qualname__, "__qualname__");
INTERN_STRING(__slots__, "__slots__");
INTERN_STRING(__radd__, "__radd__");
INTERN_STRING(__rsub__, "__rsub__");
INTERN_STRING(__rmul__, "__rmul__");
Expand All @@ -42,12 +47,17 @@ intern_strings(void) {
INTERN_STRING(clear, "clear");
INTERN_STRING(close_, "close");
INTERN_STRING(copy, "copy");
INTERN_STRING(dispatch_cache, "dispatch_cache");
INTERN_STRING(get_type_hints, "get_type_hints");
INTERN_STRING(keys, "keys");
INTERN_STRING(items, "items");
INTERN_STRING(join, "join");
INTERN_STRING(register_, "register");
INTERN_STRING(registry, "registry");
INTERN_STRING(send, "send");
INTERN_STRING(setdefault, "setdefault");
INTERN_STRING(startswith, "startswith");
INTERN_STRING(super, "super");
INTERN_STRING(throw_, "throw");
INTERN_STRING(translate, "translate");
INTERN_STRING(update, "update");
Expand Down
10 changes: 10 additions & 0 deletions mypyc/lib-rt/static_data.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,12 @@ int intern_strings(void);

typedef struct mypyc_interned_str_struct {
PyObject *__init_subclass__;
PyObject *__module__;
PyObject *__mro_entries__;
PyObject *__mypyc_attrs__;
PyObject *__orig_bases__;
PyObject *__qualname__;
PyObject *__slots__;
PyObject *__name__;
PyObject *__radd__;
PyObject *__rsub__;
Expand All @@ -38,12 +43,17 @@ typedef struct mypyc_interned_str_struct {
PyObject *clear;
PyObject *close_;
PyObject *copy;
PyObject *dispatch_cache;
PyObject *get_type_hints;
PyObject *keys;
PyObject *items;
PyObject *join;
PyObject *register_;
PyObject *registry;
PyObject *send;
PyObject *setdefault;
PyObject *startswith;
PyObject *super;
PyObject *throw_;
PyObject *translate;
PyObject *update;
Expand Down