fix(py_venv): Repair runfiles-relative venv roots #579
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
arrdem
changed the title
arrdem
changed the title
thesayyn
reviewed
Jun 9, 2025
arrdem
added a commit
that referenced
this pull request
Jun 10, 2025
Identified in manual testing of v1.6.0-rc0. Because of overly conservative removals of `realpath` in #579, an issue is exposed where `$VIRTUAL_ENV` would be an un-resolved relative path both under Bazel and more significantly once a user activates a linked venv. This isn't so bad for binaries which usually don't `chdir`, but it's a problem for shells. Since MacOS doesn't ship a `realpath` which can be configured to ignore symlinks, we can't just `realpath` the runfiles dir or the virtualenv home. But once we've configured a Python interpreter what we can do is use `os.path.abspath`. Unlike `realpath`, `abspath` does not attempt to resolve symlink path segments. It just uses `normpath` to eliminate relative path segments. This allows us to compute an absolute path in a portable way, once we get an appropriate interpreter up and running. --- ### Changes are visible to end-users: no ### Test plan - Manual testing; please provide instructions so we can reproduce: ``` ❯ bazel run //examples/py_binary:py_binary.venv INFO: Analyzed target //examples/py_binary:py_binary.venv (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //examples/py_binary:py_binary.venv up-to-date: bazel-bin/examples/py_binary/py_binary.venv INFO: Elapsed time: 0.785s, Critical Path: 0.02s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action INFO: Running command line: bazel-bin/examples/py_binary/py_binary.venv Linking: /private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/py_binary/.py_binary.venv -> /Users/arrdem/Documents/work/aspect/rules_py/examples/py_binary/.py_binary.venv To activate the virtualenv run: source /Users/arrdem/Documents/work/aspect/rules_py/examples/py_binary/.py_binary.venv/bin/activate Link is up to date! ❯ source /Users/arrdem/Documents/work/aspect/rules_py/examples/py_binary/.py_binary.venv/bin/activate ❯ env | grep -e PYTHON -e RUNFILES -e VIRTUAL -e VENV | sort PYTHONEXECUTABLE=/Users/arrdem/Documents/work/aspect/rules_py/examples/py_binary/.py_binary.venv/bin/python PYTHONHOME=/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/py_binary/py_binary.venv.runfiles/python_toolchain_aarch64-apple-darwin RUNFILES_DIR=/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/py_binary/py_binary.venv.runfiles RUNFILES_MANIFEST_FILE=/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/py_binary/py_binary.venv.runfiles/MANIFEST RUNFILES_REPO_MAPPING=/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/py_binary/py_binary.venv.runfiles/_repo_mapping VIRTUAL_ENV_DISABLE_PROMPT=1 VIRTUAL_ENV=/Users/arrdem/Documents/work/aspect/rules_py/examples/py_binary/.py_binary.venv ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Due to differences between the MacOS and Linux implementations of Bazel's sandboxing (reported as #573) the current implementation of manipulating
argv[0]to make an interpreter see itself as having a path of.runfiles/.../.venv/bin/python3isn't effective.Specifically on Linux it appears that we're more eagerly resolving symlinks.
This means that
bazel-bin/.../*.runfiles/*/.venv/bin/python3is being immediately resolved to an/execroot/bazel-bin/...entry which contains only generated files. An interpreter with relative path based import roots within the execroot will be able to import relocated sources (copied into the venv) but will not be able to import from source files or external generated files.Effectively spawning the interpreter is escaping the
.runfilestree which breaks a bunch of our other assumptions.The repair is to come up with a more portable way to identify the interpreter without escaping from the runfiles sandbox.
As a solution this PR makes the interpreter shim depend both on the
$VIRTUAL_ENVvariable which also serves as an anchor to the "real".runfilestree. Due to the removals ofrealpathcalls in theactivatescript, it's possible that we could provide fallback behavior by introspecting$0, but that verges on #581 so punting for now.Changes are visible to end-users: no
Test plan