You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fixes a user-visible bug present in v1.5.0 which could cause the Python
shim to skip over viable interpreters and fail to resolve an appropriate
one.
### Changes are visible to end-users: yes
- Searched for relevant documentation and updated as needed: yes
- Breaking change (forces users to change their own code or config): no
- Suggested release notes appear below: yes/no
### Details
The Python interpreter shim exists to inspect the `pyvenv.cfg` sentinel
file which defines a virtualenv, extract the interpreter version for the
virtualenv and then traverse the `$PATH` attempting to identify a viable
interpreter of the required version which it will chainload.
As a reminder a venv will look something like
```
|-- .venv
| |-- bin
| | |-- activate
| | |-- pip
| | |-- pip3
| | |-- pip3.12
| | |-- python <- our shim
| | |-- python3 -> python
| | `-- python3.12 -> python
| |-- include
| | `-- python3.12
| |-- lib
| | `-- python3.12
| | `-- site-packages
| | |-- pip
| | `-- pip-24.0.dist-info
| |-- lib64 -> lib
| `-- pyvenv.cfg
`-- src
`-- app
`-- __main__.py
```
The trick is that since the interpreter shim is emplaced in the venv as
`bin/python` and symlinked to by the other versioned Python files we
can't simply `exec python${VERSION}` since when the venv is activated
and `$VIRTUAL_ENV/bin` is prepended to the `$PATH` that would cause an
infinite self-loop.
Enabling the `debug` config on the shim crate and running the
internal_venv example @ v1.5.0 produces the following (simplified)
output
```shellsession
$ bazel run //examples/py_venv:internal_venv
INFO: Analyzed target //examples/py_venv:internal_venv (457 packages loaded, 56604 targets configured).
INFO: Found 1 target...
Target //examples/py_venv:internal_venv up-to-date:
bazel-bin/examples/py_venv/internal_venv
INFO: Elapsed time: 15.549s, Critical Path: 4.20s
INFO: 4 processes: 2 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/examples/py_venv/internal_venv
# First attempt
[aspect] Current executable path: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/.internal_venv/bin/python"
[aspect] Found pyvenv.cfg at: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/.internal_venv/pyvenv.cfg"
[aspect] version_info from pyvenv.cfg: 3.9.0
[aspect] Parsed target Python version (major.minor): 3.9
[aspect] Found potential Python interpreters in PATH with matching version:
[aspect] - "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/bin/python3.9"
[aspect] - "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/python_toolchain_aarch64-apple-darwin/bin/python3.9"
[aspect] - "${HOME}/.local/pyenv/shims/python3.9"
[aspect] Attempting to execute: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/bin/python3.9" with argv[0] as "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/.internal_venv/bin/python" and args as ["-B", "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/say.py"]
# Second attempt
[aspect] Current executable path: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/bin/python3.9"
[aspect] Found pyvenv.cfg at: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/pyvenv.cfg"
[aspect] version_info from pyvenv.cfg: 3.9.0
[aspect] Parsed target Python version (major.minor): 3.9
[aspect] Found potential Python interpreters in PATH with matching version:
[aspect] - "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/python_toolchain_aarch64-apple-darwin/bin/python3.9"
[aspect] - "${HOME}/.local/pyenv/shims/python3.9"
[aspect] Attempting to execute: "${HOME}/.local/pyenv/shims/python3.9" with argv[0] as "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/bin/python3.9" and args as ["-B", "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/say.py"]
# Third attempt
[aspect] Current executable path: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/bin/python3.9"
[aspect] Found pyvenv.cfg at: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/pyvenv.cfg"
[aspect] version_info from pyvenv.cfg: 3.9.0
[aspect] Parsed target Python version (major.minor): 3.9
[aspect] Found potential Python interpreters in PATH with matching version:
[aspect] - "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/python_toolchain_aarch64-apple-darwin/bin/python3.9"
[aspect] - "${HOME}/.local/pyenv/shims/python3.9"
Error: × Unable to find another interpreter at index 2
```
The key detail of which is
```
# First attempt
[aspect] Found pyvenv.cfg at: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/.internal_venv/pyvenv.cfg"
# Second attempt
[aspect] Found pyvenv.cfg at: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/pyvenv.cfg"
# Third attempt
[aspect] Found pyvenv.cfg at: "${EXECROOT}/aspect_rules_py/bazel-out/fastbuild/bin/examples/py_venv/.internal_venv/pyvenv.cfg"
```
What's happening here is that under `bazel run` the first `exec()`
invocation is of the interpreter as it exists in a `.runfiles` tree
under `bazel-out`. However that runfiles tree consists of symlinks to
actual outputs elsewhere, so the observed path of the `pyvenv.cfg`
(which is _not_ canonicalized and is located relative to the nominal
interpreter path) appears to change between attempts going from the
non-canonicalized `.runfiles/` entry to the "actual" build output dir.
This causes the second attempt to change its filtering rules, and skip
the viable toolchain-provided interpreter present in the `$PATH`.
The fix is to canonicalize both the `$PATH` entries and the
`pyvenv.cfg`'s paths when filtering out possibilities to ensure that the
filtering behavior is stable between attempts.
With that fix applied the shim correctly identifies the required
interpreter on the first try.
```shellsession
$ bazel run -c dbg //examples/py_venv:internal_venv
INFO: Analyzed target //examples/py_venv:internal_venv (0 packages loaded, 56606 targets configured).
INFO: From Compiling Rust rlib py (4 files):
INFO: Found 1 target...
Target //examples/py_venv:internal_venv up-to-date:
bazel-bin/examples/py_venv/internal_venv
INFO: Elapsed time: 129.494s, Critical Path: 75.62s
INFO: 472 processes: 166 internal, 306 darwin-sandbox.
INFO: Build completed successfully, 472 total actions
INFO: Running command line: bazel-bin/examples/py_venv/internal_venv
[aspect] Current executable path: "/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-dbg/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/.internal_venv/bin/python"
[aspect] Found pyvenv.cfg at: "/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-dbg/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/.internal_venv/pyvenv.cfg"
[aspect] version_info from pyvenv.cfg: 3.9.0
[aspect] Parsed target Python version (major.minor): 3.9
[aspect] Ignoring dir "/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-dbg/bin/examples/py_venv/.internal_venv/bin"
[aspect] Found potential Python interpreters in PATH with matching version:
[aspect] - "/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/external/python_toolchain_aarch64-apple-darwin/bin/python3.9"
[aspect] - "/Users/arrdem/.local/pyenv/shims/python3.9"
[aspect] Attempting to execute: "/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/external/python_toolchain_aarch64-apple-darwin/bin/python3.9" with argv[0] as "/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-dbg/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/.internal_venv/bin/python" and args as ["-B", "/private/var/tmp/_bazel_arrdem/93bfea6cdc1153cc29a75400cd38823a/execroot/aspect_rules_py/bazel-out/darwin_arm64-dbg/bin/examples/py_venv/internal_venv.runfiles/aspect_rules_py/examples/py_venv/say.py"]
---
virtualenv: ${RUNFILES}/aspect_rules_py/examples/py_venv/.internal_venv/lib/python3.9/site-packages/_virtualenv.py
sys.prefix: ${RUNFILES}/aspect_rules_py/examples/py_venv/.internal_venv
sys.path:
- ${BAZEL_EXECROOT}/aspect_rules_py/bazel-out/darwin_arm64-dbg/bin/examples/py_venv
- ${RUNFILES}/python_toolchain_aarch64-apple-darwin/lib/python39.zip
- ${RUNFILES}/python_toolchain_aarch64-apple-darwin/lib/python3.9
- ${RUNFILES}/python_toolchain_aarch64-apple-darwin/lib/python3.9/lib-dynload
- ${RUNFILES}/aspect_rules_py/examples/py_venv/.internal_venv/lib/python3.9/site-packages
- ${RUNFILES}/aspect_rules_py
- ${RUNFILES}/aspect_rules_py/examples/py_venv
- /Users/arrdem/.local/lib/python3.9/site-packages
- ${RUNFILES}/python_toolchain_aarch64-apple-darwin/lib/python3.9/site-packages
site.PREFIXES:
- ${RUNFILES}/aspect_rules_py/examples/py_venv/.internal_venv
- ${RUNFILES}/python_toolchain_aarch64-apple-darwin
- ${RUNFILES}/python_toolchain_aarch64-apple-darwin
___________________________________________
| hello py_venv! (built at <BUILD_TIMESTAMP>) |
===========================================
\
\
^__^
(oo)\_______
(__)\ )\/\
||----w |
|| ||
```
### Test plan
- [x] The examples now actually work (sigh)
- [x] The examples are now exercised in our release integration testing
---------
Co-authored-by: Sahin Yort <[email protected]>
0 commit comments