Skip to content

gh-131656: Solved the cross-interpreter support of multiprocessing module #131657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

gh-131656: Solved the cross-interpreter support of multiprocessing module #131657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0c8e908
Update spawn.py
pypy66 Mar 24, 2025
e1352db
Update spawn.py
pypy66 Mar 24, 2025
5760514
📜🤖 Added by blurb_it.
blurb-it[bot] Mar 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions Lib/multiprocessing/spawn.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,10 @@ def get_preparation_data(name):
if util._logger is not None:
d['log_level'] = util._logger.getEffectiveLevel()

sys_path=sys.path.copy()
sys_path=[]
for path in sys.path:
if sys.base_prefix not in os.path.abspath(path):
sys_path.append(path) # For cross-interpreter support
try:
i = sys_path.index('')
except ValueError:
Expand Down Expand Up @@ -226,7 +229,11 @@ def prepare(data):
util.get_logger().setLevel(data['log_level'])

if 'sys_path' in data:
sys.path = data['sys_path']
sys_path = data['sys_path'].copy()
for path in data['sys_path']:
if path in sys.path:
sys_path.remove(path)
sys.path.extend(sys_path) # For cross-interpreter support
Copy link
Member

@gpshead gpshead Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be needed with the user supplied sys_path manipulation function in the parent. that can already know what the entire sys.path of the set_executable interpreter should be and pre-populate the value being sent to the child. no in child filtering or extend needed.

Copy link
Author

@pypy66 pypy66 Mar 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'll do it.

Copy link
Author

@pypy66 pypy66 Mar 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gpshead But how can the parent process detect the appropriate sys.path of the child since the child may has its own configs (e.g.HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.12\PythonPath on Windows) and its own implementation of how to read these configs that differs from the parent.
Firstly, I've decided to simply replace the base path. For instance, D:\Python313\Lib is simply replaced with D:\Python312\Lib, and D:\Python313\DLLs is replaced with D:\Python312\DLLs,
but if the Python 3.12 is installed inside Cygwin and its sys.path is D:\cygwin\usr\local\lib\python3.12, there is no way to detect it.
So, I suppose that merging the paths is a better and more elegant solution:

# In the parent process
sys_path=[] # The paths being sent to the child
for path in sys.path:
    if sys.base_prefix not in os.path.abspath(path): # Filter out the path that is not related to the parent (e.g. current working directory) by simply using `sys.base_prefix`
        sys_path.append(path)
...

In the child:

for path in data['sys_path']:
    if path in sys.path:
        sys_path.remove(path) # remove duplicate paths
sys.path.extend(sys_path)

However, my code using sys.base_prefix may be not a elegant way since you cannot use not in in Linux. Could you suggest a better way, please?


if 'sys_argv' in data:
sys.argv = data['sys_argv']
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2025-03-24-12-37-35.gh-issue-131656.7PiFWH.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed the cross-interpreter issue of subprocess module by modifying :func:`multiprocessing.spawn.prepare` and :func:`multiprocessing.spawn.get_preparation_data`. Contributed by pypy66.
Loading