PyImport_ImportModule

[mwriter]

Bug report

Bug description:

import sys

import json
print(json) # <module 'json'>
print(sys.modules["json"]) # <module 'json'>

import ctypes
print(ctypes.pythonapi.PyImport_ImportModule(ctypes.c_char_p(b"json"))) # 2356734321

sys.modules = sys.modules.copy()

import email
print(email) # <module 'email'>
print(sys.modules["email"]) # <module 'email'>

print(ctypes.pythonapi.PyImport_ImportModule(ctypes.c_char_p(b"email"))) # KeyError: 'email'

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

No response

[ZeroIntensity]

I'll investigate this a little. At first, I thought this was just user-error, but all PyImport_ImportModule does is call builtins.__import__, which does seem to work with the reproducer, so I'm not exactly sure what's going on.

[mwriter]

so I'm not exactly sure what's going on

I'm researching the development of event audit ideas. And these probably won't be the last problems found :)

[ZeroIntensity]

A fair warning: issues resulting from manual manipulation of sys.modules probably won't get fixed. I'll see if there's something obvious that can be worked around here, but it might very well be a wontfix.

[mwriter]

I'll see if there's something obvious that can be worked around here, but it might very well be a wontfix.

My job is to write about strange things. But the verdict is yours :)
And the manipulation of sys.modules does not affect the python code version in any way.
Since the behavior is very different for actually the same import, this is very similar to a bug.

[ZeroIntensity]

And the manipulation of sys.modules does not affect the python version in any way.

It does, the interpreter relies on Python modules like traceback--if you break sys.modules and stop those imports from working, there's nothing we can do.

[mwriter]

At first, I thought this was just user-error, but all PyImport_ImportModule does is call builtins.__import__

Also

import sys

import builtins

name = "json"
print(builtins.__import__(name)) # <module 'json'>
print(ctypes.pythonapi.PyImport_ImportModule(ctypes.c_char_p(name.encode()))) # 3536781451

sys.modules = sys.modules.copy()

name = "email"
print(builtins.__import__(name)) # <module 'email'>
print(ctypes.pythonapi.PyImport_ImportModule(ctypes.c_char_p(name.encode()))) # KeyError: 'email'

[mwriter]

I thought this was just user-error, but all PyImport_ImportModule does is call builtins.__import__, which does seem to work with the reproducer, so I'm not exactly sure what's going on.

Do you know about any internal caches?
I have a suspicion that PyImport_ImportModule is working with the previous version of sys.modules before copying.

[mwriter]

This is definitely a bug :)

import sys
import builtins

sms = sys.modules
sys.modules = sms.copy()

name = "email"

m = builtins.__import__(name)
print(m) # <module 'email'>
sms[name] = m
print(ctypes.pythonapi.PyImport_ImportModule(ctypes.c_char_p(name.encode()))) # 456145783

[ZeroIntensity]

Ah, I found the issue--it's a wontfix. The interpreter uses two references to the modules dictionary: the actual one on the interpreter state, and then sys.modules is simply a reference to that. Reassigning sys.modules doesn't update the one on the interpreter state, so there ends up being two mismatched copies of it. Python code uses sys.modules, and then C code uses the interpreter state.

The fix would be to allow changes made on sys to directly affect the sysdict, but that's a). a lot of work for a niche feature and b). unsafe anyway, because the user could put something other than a dictionary in there, which can segfault depending on what the finalizer does (sysdict gets cleared after everything else in the interpreter, so lots of C API calls fail).