Twisted concurrency model #816
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Julian
reviewed
Sep 12, 2019
| show_help( | ||
| "Options affecting multiprocessing must only be specified " | ||
| "in a configuration file.\n" | ||
| "Options affecting multiprocessing and twisted must " |
There was a problem hiding this comment.
(Will probably not be very useful reviewing much of this, but it'd be a bit nicer UX-wise if rather than saying "A or B" here you used .format and inserted in options.concurrency, i.e. the actual one of the two of those that the user really did use, instead of including ones they didn't)
nedbat
force-pushed
the
master
branch
2 times, most recently
from
0c14f1a to
82169a6
Compare
|
Is this still valid? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds
--concurrency=twistedwhich causes coverage to be tracked across child processes created with Twisted'sreactor.spawnProcess.I'm not entirely happy with the sitecustomize management code. Also it seems like this could probably easily be a
--concurrency=execfeature instead, with a the sitecustomize being written and PYTHONPATH updated early. Then all processes forked & exec'd will get the logic. It's less clear when to do cleanup but it's probably still possible.Also this approach comes with the downside that it obscures any existing sitecustomize. I don't really know what people use sitecustomize for so I'm not sure how big of a deal this is.
Any other solution for fork&exec seems like a challenge because there is no general way to inject some code into a process you've just exec'd. exec is inherently about giving up control of what's running to some other program. There might be some platform-specific tricks but that doesn't seem like an appealing avenue.
While working on this, https://pypi.org/project/coverage_enable_subprocess/ was also pointed out to me and it seems to accomplish roughly the same thing. It uses the pth file approach instead but this means it requires a virtualenv (or for you to splat junk into your probably-OS-managed Python install which is not a great idea). It also collects coverage from all Python programs instead of all programs in a tree under the one you start but I don't know if that's much of a practical difference.