pythonTerminalEnvVarActivation shadows pyenv PATH

[kbehlers]

Behaviour

When the pythonTerminalEnvVarActivation experiment is active, I can no longer use pyenv-virtualenv commands for activating and deactivating venvs in the VSCode terminal. Instead the interpreter selected in the Python extension always stays activated in the terminal.

The pyenv-virtualenv commands give the appearance of working (they change my zsh command prompt showing the active venv) but running python commands like python --version or pip freeze show that the python interpreter being used isn't the one shown in my command prompt but instead the one selected using the Python extension in VSCode.

It seems possibly related to how pythonTerminalEnvVarActivation prepends to PATH.

Steps to reproduce:

1. Make sure settings.json has

"python.experiments.optInto": ["pythonTerminalEnvVarActivation"]

2. Install pyenv and pyenv-virtualenv
3. Install two versions of python with pyenv (e.g. pyenv install 3.10 pyenv install 3.9) to make it easier to see when the wrong venv is active
4. Create a virtualenv for each version of python

pyenv virtualenv 3.9 first_example_env
pyenv virtualenv 3.10 second_example_env

5. Choose first_example_env as the python interpreter in VSCode using Python: Select Interpreter
6. Open a VSCode terminal
7. echo $PATH (this shows vscode specific paths when the experiment is enabled)
8. python --version (should say Python 3.9.*)
9. pyenv deactivate (doesn't end up deactivating as it should but gives the appearance of doing so by changing the command prompt)
10. pyenv activate second_example_env (changes the command prompt, doesn't actually change the venv though)
11. python --version (should say Python 3.10.* but still says Python 3.9.* because the environment didn't really switch)
12. Use Python: Select Interpreter and choose second_example_env
13. Reload the VSCode terminal
14. Repeat above steps, except this time second_example_env will be active no matter what

If I instead add:

"python.experiments.optOutFrom": ["pythonTerminalEnvVarActivation"]

to settings.json, I can activate and deactivate venvs using pyenv-virtualenv just as I would expect, where the venv actually switches in the terminal.

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2024-04-11 11:24:39.228 [info] Prepending environment variable PATH in collection with /Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin: {"applyAtShellIntegration":true,"applyAtProcessCreation":true}
2024-04-11 11:24:39.228 [info] Setting environment variable VIRTUAL_ENV in collection to /Users/username/.pyenv/versions/3.9.17/envs/first_example_env {"applyAtShellIntegration":true,"applyAtProcessCreation":true}
2024-04-11 11:24:39.228 [info] Prepending environment variable PS1 in collection with (first_example_env)  {"applyAtShellIntegration":true,"applyAtProcessCreation":false}
2024-04-11 11:24:39.228 [error] Failed to initialize deactivate script /bin/zsh [Error: "/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh/envVars.txt" file not created
	at Timeout.<anonymous> (/Users/username/.vscode/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:271175)
	at listOnTimeout (node:internal/timers:569:17)
	at process.processTimers (node:internal/timers:512:7)]

[anthonykim1]

hello @kbehlers Thanks for filing the issue. Is this only happening with pyenv?? Is it fine with .venv

[kbehlers]

@anthonykim1 Hi! This is only with pyenv, specifically pyenv-virtualenv.

Standard .venv doesn't have the issue, and looking into it that is probably because the source .venv/bin/activate command prepends the venv to PATH itself..

pyenv-virtualenv seems to be relying on the existing shim already in the PATH, rather than prepending the venv each time, so the shadowing occurs because the VS Code experiment seems to prepend the venv from the interpreter selected in the Python extension.

Exploring the behavior a bit more, if I first source deactivate instead of pyenv deactivate then that clears the venv from the interpreter selected in the Python extension from PATH and I can use pyenv-virtualenv normally again.

[anthonykim1]

Hello @kbehlers Thanks much for further looking into this.

I think it can be one or either of two problems:

1. I'm also thinking about the shadowing scenarios of prepending possbily causing shadowing of pyenv adding to existing shim already in PATH. I believe we prepend to PATH all the time. When you take a look at your $PATH after you try to activate pyenv and see pyenv venv prompt, does your PATH contain pyenv related code?

2. My second guess is the way we "deactivate" for pyenv. I will need to further look into deactivate scenario for pyenv. (Im going to try to figure out if python extension does sort of different deactivation for pyenv compared to traditional .venv and if that is correctly working.

Good to hear that there is at least some workaround if you source deactivate first.

I'll let you know if I get update on this.

[anthonykim1]

Related: #21906 Seems to be the PR where we started to use prepend to PATH. With its purpose of attempting to avoid replacing. since pyenv asks their user to manipulate PATH in init script.

[anthonykim1]

@kbehlers can you share what your PATH looks like?
We are trying to figure out which deactivation script is running when yousource deactivate vs. pyenv deactivate

[kbehlers]

@anthonykim1 Sure!

Opted out from the experiment

/opt/homebrew/Cellar/pyenv-virtualenv/1.2.1/shims:/Users/username/.pyenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Opted into the experiment

/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin:/opt/homebrew/Cellar/pyenv-virtualenv/1.2.1/shims:/Users/username/.pyenv/shims:/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin

You can see when opted-in it has some interesting prepend and append behavior

[anthonykim1]

Thanks much for providing PATH. Greatly appreciated!
The prepending is expected as: #21906 and #22905

Im not able to repro the behavior on my machine exactly but I do get "pyenv-virtualenv: no virtualenv has been activated." when I try pyenv deactivate when opting into the experiment. But then that still maintains Python from Pyenv virtualenv. Only when I source deactivate, it seems to bring back to "correct" Python.

Still lost on why prepending would "not allow" pyenv deactivate to work because your pyenv shim is still technically in your PATH even after Python extension prepend our deactivate script in the PATH.