Skip to content

Fix multiprocessing testcase #7743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sfc-gh-truwase merged 8 commits into from
Jan 15, 2026
Merged

Fix multiprocessing testcase #7743

sfc-gh-truwase merged 8 commits into from
Jan 15, 2026

Conversation

@k-artem
Copy link
Contributor

@k-artem k-artem commented Dec 22, 2025 •
edited
Loading

Need to force context re-creating inside testcase otherwise runtime exception raises:

    def set_start_method(self, method, force=False):
        if self._actual_context is not None and not force:
>           raise RuntimeError('context has already been set')
E           RuntimeError: context has already been set

Command for verification:
pytest -v tests/unit/runtime/test_ds_initialize.py

@k-artem
Fix multiprocessing testcase
cce52e5 
Need to force context re-creating inside testcase otherwise runtime exception raises:
```
    def set_start_method(self, method, force=False):
        if self._actual_context is not None and not force:
>           raise RuntimeError('context has already been set')
E           RuntimeError: context has already been set
```

Signed-off-by: Artem Kuzmitckii <[email protected]>
@k-artem
Copy link
Contributor Author

k-artem commented Jan 5, 2026

@sfc-gh-truwase please have a look

@sfc-gh-truwase
Copy link
Collaborator

sfc-gh-truwase commented Jan 9, 2026

@k-artem can you share a bit about the error that this fixes? In my experience set_start_method is very flaky and environment-dependent.

@tohtana, @loadams, any thoughts?

@k-artem
Copy link
Contributor Author

k-artem commented Jan 12, 2026

@k-artem can you share a bit about the error that this fixes? In my experience set_start_method is very flaky and environment-dependent.

@tohtana, @loadams, any thoughts?

sure, on my side it’s consistently reproducible. torch.multiprocessing is a wrapper around Python’s multiprocessing, and its start method can only be set once per process via set_start_method and after that the multiprocessing context is locked and cannot be changed. However, the test calls set_start_method three times with different methods via @pytest.mark.parametrize('method', ['spawn', 'fork', 'forkserver']) , which triggers an exception after first call. In my fix, I force the multiprocessing context to be recreated between runs.

tests/unit/runtime/test_ds_initialize.py::test_start_method_safety[spawn] PASSED                                                                                                                          [ 33%]
tests/unit/runtime/test_ds_initialize.py::test_start_method_safety[fork] FAILED                                                                                                                           [ 66%]
tests/unit/runtime/test_ds_initialize.py::test_start_method_safety[forkserver] FAILED                                                                                                                     [100%]

=================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________________________ test_start_method_safety[fork] _________________________________________________________________________________________

method = 'fork'

    @pytest.mark.parametrize('method', ['spawn', 'fork', 'forkserver'])
    def test_start_method_safety(method):
        import torch.multiprocessing as mp
        print(os.getpid())
>       mp.set_start_method(method)

tests/unit/runtime/test_ds_initialize.py:33:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <multiprocessing.context.DefaultContext object at 0x7620fe759280>, method = 'fork', force = False

    def set_start_method(self, method, force=False):
        if self._actual_context is not None and not force:
>           raise RuntimeError('context has already been set')
E           RuntimeError: context has already been set

/usr/lib/python3.12/multiprocessing/context.py:247: RuntimeError
--------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------

@k-artem
Copy link
Contributor Author

k-artem commented Jan 15, 2026

hi @sfc-gh-truwase please approve if no objections

@sfc-gh-truwase
Copy link
Collaborator

sfc-gh-truwase commented Jan 15, 2026

@k-artem, the approval from @loadams is actually sufficient.

@sfc-gh-truwase sfc-gh-truwase merged commit 7bbb6c7 into deepspeedai:master Jan 15, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@loadams loadams loadams approved these changes

@sfc-gh-truwase sfc-gh-truwase sfc-gh-truwase approved these changes

@tjruwase tjruwase Awaiting requested review from tjruwase tjruwase is a code owner

@tohtana tohtana Awaiting requested review from tohtana tohtana is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@k-artem @sfc-gh-truwase @loadams