Add support for loading .pth files in st2 venv
#6183
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
importing site has side effects which were messing up sys.path I wasn't sure what else to import, so I imported sys. An import is required to trigger the exec().
avoid duplicate entries and hopefully avoid adding system paths before the st2 paths.
The system lib paths are already in the sys.path before site gets imported. Then, all .pth files append after that. So, this is expected.
By default, pants+pex adds `-sE` to the virtualenv/bin scripts which prevents PYTHONPATH and other PYTHON* vars from affecting the program. st2-run-pack-tests uses PYTHONPATH, however, so bypass the script if it is hermetic.
If the virtualenv is built by pants, the PEP 660 wheels will not include any of the scripts (like st2-run-pack-tests). Only entry_point scripts will be installed in the virtualenv. So, do not assume that a dev venv will have st2-run-pack-tests. Also, adjust path assumptions to work with either kind of venv.
pull-request-size
bot
added
the
size/L
nzlosh
approved these changes
Apr 19, 2024
amanda11
approved these changes
Apr 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In our standard st2 virtualenv there are a variety of
.pthfiles that add modules to thesys.pathwhen python starts up (via thesitemodule: https://docs.python.org/3/library/site.html):Today, however, none of those
.pthfiles affect pack venvs because the parent st2 venv is not in the standard list of site-packages locations.We probably haven't encountered issues with the lack of
.pthfile loading in pack venvs becauseoslo.configandrepoze.lruare the only things that could have an impact, and the impact would be minimal at best.This becomes more pronounced when using a virtualenv generated by pants+pex (instead of via our Makefile) because all of our first party sources get installed with PEP 660 wheels. The legacy calls to
python setup.py developtrigger creating a symlink to st2common, st2auth, st2* and the runner directories under the virtualenv's site-packages directory. PEP 660 wheels use.pthfiles instead of symlinks to achieve the same thing.So, we have to ensure
.pthfiles get loaded whenever interacting with a pack venv. This PR does that by injecting a.pthfile into the pack venv that uses thesitemodule to load any.pthfiles from the parent st2 venv.Also, I adjusted the GHA CI workflow to work when our sources are added to the venv via PEP 660 editable wheels. Apparently, PEP 660 only installs entry point scripts, but most of our
st2*/bin/*scripts are justscriptsnotentry_points. https://peps.python.org/pep-0660/#limitationsFinally, I adjusted how
st2-run-pack-testsso that it can handle hermetic scripts. A hermetic script has a python shebang with the-sEargs that do not load user site-packages (-s) and ignore allPYTHON*vars (-E) likePYTHONPATH. Once we update to a version of pants that includes pantsbuild/pants#20763, we can just disable how pants+pex changes the virtualenv shebang lines. Until then, we need to runnosetestsin a way that letsPYTHONPATHcontinue to affect it.I extracted this from #6130 where I was testing what changes are required to use a pants-built venv to run tests.