Link multiple Python versions

[Feuermurmel]

I'd like to use multiple Python versions at the same time. I see that Homebrew has packages for Python ranging from 3.7 to 3.10. If I install one of those, I get symlinks to the installed binaries in $HOMEBREW_PREFIX/bin. All in two variants, the generic one ending just in 3 and the more specific one ending in e.g. 3.9. E.g. python3 and python3.9, pip3 and pip3.9 etc.

Once one version has been linked, is it possible to link additional versions via only the specific version number so that I'd e.g. have symlinks at python3 and python3.9 linking to version 3.9 and an additional symlink at python3.8 linking to version 3.8?

I've been doing this with MacPorts and would like to replicate my environment with Homebrew (if possible without additional package managers like pyenv).

[carlocab]

It's not currently possible, but I believe it is possible in principle if the various Python formulae were fixed up to support this.

Dealing with the various executables (python3, pip3, etc) is fairly straightforward, but there are some conflicts that will require a bit more work:

❯ brew link --force --dry-run --overwrite python@3.10
Would remove:
/usr/local/bin/2to3 -> /usr/local/Cellar/python@3.9/3.9.9/bin/2to3
/usr/local/bin/idle3 -> /usr/local/Cellar/python@3.9/3.9.9/bin/idle3
/usr/local/bin/pip3 -> /usr/local/Cellar/python@3.9/3.9.9/bin/pip3
/usr/local/bin/pydoc3 -> /usr/local/Cellar/python@3.9/3.9.9/bin/pydoc3
/usr/local/bin/python3 -> /usr/local/Cellar/python@3.9/3.9.9/bin/python3
/usr/local/bin/python3-config -> /usr/local/Cellar/python@3.9/3.9.9/bin/python3-config
/usr/local/bin/wheel3 -> /usr/local/Cellar/python@3.9/3.9.9/bin/wheel3
/usr/local/share/man/man1/python3.1 -> /usr/local/Cellar/python@3.9/3.9.9/share/man/man1/python3.1
/usr/local/lib/pkgconfig/python3-embed.pc -> /usr/local/Cellar/python@3.9/3.9.9/lib/pkgconfig/python3-embed.pc
/usr/local/lib/pkgconfig/python3.pc -> /usr/local/Cellar/python@3.9/3.9.9/lib/pkgconfig/python3.pc
/usr/local/Frameworks/Python.framework/Headers -> /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Headers
/usr/local/Frameworks/Python.framework/Python -> /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Python
/usr/local/Frameworks/Python.framework/Resources -> /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Resources
/usr/local/Frameworks/Python.framework/Versions/Current -> /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/Current

This might require special handling of the contents of *.framework directories in brew.

[Feuermurmel]

Mhhh, I'm on the lookout for a setup that doesn't require any modification to the packages provided by the core repository. I'd imagine that a setup using brew link --force would break when updating any of the installed Python packages.

[carlocab]

I suspect you may have misunderstood what I said. I wasn't suggesting you make local modifications to anything you have installed, or that you use brew link --force.

[Feuermurmel]

No, I didn't read you as saying that I should do this to achieve my goal. But it made me think about the possibility of using brew link --force --overwrite to achieve my goal, albeit with the downsides I theorized about. All is well. ☺️ I'm looking into pyenv now.

[dwt]

The python makefile support the altinstall target, that create just binaries with the full version number attached. This makes it trivial to link multiple major python versions. Changing the formulas to to either build this target with a flag or to just link those binaries into the path should both be possible. I have no clue wether the Homebrew developers would accept such a change.

If you want to see how that might look, I have scripted this for myself as I wanted this at a time when it didn't look like HomeBrew would ever support something like this: https://gist.github.com/dwt/02d8a5c035e9cd3b1da8c55609d3f73b

I'd be happy to help with the patches to homebrew - given the maintainers express interest in taking this in.

[dwt]

I don't have experience in this, as I don't build my python versions as framework versions. However, the framework folder layout is explicitly built to support different concurrent versions and has supported different python versions in the past (before python3k).

That being said: what exactly are you worrying about? That python won't find it's resources correctly if built as a framework? If that's the case that would be a bug upstream, as the framework built is (to my knowledge) an officially supported feature of python.

[Feuermurmel]

MacPorts also uses the framework build of Python, so the problem is solvable. @carlocab If you look at the paths inside of Python.framework in the list of paths you posted, you can see that these are actually just symlinks into directories that are version-specific:

$ ls -l opt/homebrew/Cellar/python\@3.10/3.10.1/Frameworks/Python.framework
total 0
lrwxr-xr-x  1 cs  staff   24 13 Jan 18:45 Headers -> Versions/Current/Headers
lrwxr-xr-x  1 cs  staff   23 13 Jan 18:45 Python -> Versions/Current/Python
lrwxr-xr-x  1 cs  staff   26 13 Jan 18:45 Resources -> Versions/Current/Resources
drwxr-xr-x  4 cs  staff  128 13 Jan 18:45 Versions
$ ls -l opt/homebrew/Cellar/python\@3.10/3.10.1/Frameworks/Python.framework/Versions
total 0
drwxr-xr-x  10 cs  staff  320 13 Jan 18:45 3.10
lrwxr-xr-x   1 cs  staff    4 13 Jan 18:45 Current -> 3.10

The non-version-specific paths are not used by the installed Python at all, these are just there for other applications or frameworks that don't care about which version they get.

[carlocab]

Yes, of course the problem is solvable. My question was whether make altinstall solves it for us (it seems to for everything else), or if we'll need to handle it ourselves. I wouldn't be shocked if make altinstall didn't do the right thing here -- it's not likely to be a very well-trodden install path.

I'd like to avoid duplicating code across multiple Python formulae (we have plenty of that already), and handling frameworks for versioned formulae seems generally useful (given the directory structure of a framework which you've already noted), so if we need to handle it, it's probably best done in brew rather than in particular formulae.

[dwt]

I'm slightly confused - that should be easily testable by just doing a framework build altinstall and see what happens? Anything that I'm missing?

[carlocab]

It is indeed testable that way.

[viking1304]

I honestly don't understand why would anyone like to have that kind of mess on his development machine. I defiantly would not.

If you are suggesting this just because you are trying to avoid venv because of activate/deactivate, I am almost sure that you are not aware that you can use any python app from any venv without activating that venv.

All my Python projects use venv, but I haven't use activate/deactivate for last few years.

For example:

~/Python/Projects/MyProject/venv/bin/python -m pip install -r ~/Python/Projects/MyProject/requirements.txt

will install requirements to python instance in ~/Python/Projects/MyProject/venv/ without activation

and

~/Python/Projects/MyProject/venv/bin/python test.py

will properly run test.py without activation of venv first.

[dwt]

@viking1304 you do realise that this is about different python versions not virtualenvs? Having different python versions installed in parallel is for example a requirement to test your library works correctly with all python versions it supports.

[dwt]

And most 'automated' ways to install multiple python versions do horrible things to your shell environment variables to support enabling and switching between versions, which is guaranteed to fail as soon as you have a slightly more complex shell setup.

Just plain installing the different versions with altinstall is by far the most preferable way to have convenient access to different python versions.

[Feuermurmel]

@viking1304 Sorry, but you completely missed my point. The first sentence in in my initial post:

I'd like to use multiple Python versions at the same time.

Virtualenvs have nothing to do with that. With virtualenv (or python -m venv) you can still only use Python versions that are already installed on the machine.

Why do you assume that I trying to avoid virtualenvs? I have not mentioned that in my initial post. I am using these different Python versions almost exclusively from within virtualenvs, but I develop software that requires Python >= 3.8 as well as software that is deployed with version 3.6 and so I want to run tests on that version. How should I do that when I can only install a single Python version at the same time? Do you suggest using multiple development machines for that?

[viking1304]

@dwt Absolutely, but my message was for @Feuermurmel. I didn't even read your messages before (but I did now).

I agree with you to some extent, but you should be aware of few things.

the framework folder layout is explicitly built to support different concurrent versions and has supported different python versions in the past (before python3k).

If you check System/Library/Frameworks/Python.framework/ folder, you will see that all versions now point to 2.7.

❯ ls -la /System/Library/Frameworks/Python.framework/Versions/
drwxr-xr-x   7 root  wheel  224 Jan 22 08:42 .
drwxr-xr-x   5 root  wheel  160 Jan 22 08:42 ..
lrwxr-xr-x   1 root  wheel    3 Jan 22 08:42 2.3 -> 2.7
lrwxr-xr-x   1 root  wheel    3 Jan 22 08:42 2.5 -> 2.7
lrwxr-xr-x   1 root  wheel    3 Jan 22 08:42 2.6 -> 2.7
drwxr-xr-x  11 root  wheel  352 Jan 22 08:42 2.7
lrwxr-xr-x   1 root  wheel    3 Jan 22 08:42 Current -> 2.7

Also, as you already wrote, this approach is not used for python3 by default anymore (I am talking about macOS system installation at least). So I really do not see any real benefit of suggested changes. Maybe I am missing something obvious.

Now let's get back to venv. As you probably know venv depends on the "base" python version that was used to create it. I am using many venvs with totally different python versions on my development machine.

I just made extremely simple example to demonstrate few points.

#!/usr/bin/env bash

brew install python@3.7 &>/dev/null
brew install python@3.8 &>/dev/null
/usr/local/opt/python@3.7/bin/python3 -m venv ~/Projects/test37/venv
/usr/local/opt/python@3.8/bin/python3 -m venv ~/Projects/test38/venv
~/Projects/test37/venv/bin/python3 --version
~/Projects/test38/venv/bin/python3 --version

If you save this as pytest.sh and run it, you will get this result

❯ ./pytest.sh
Python 3.7.12
Python 3.8.12

So, test37 was run using 3.7.12 and test38 was run using 3.8.12 even activate wasn't used, no new symlink was created nor any shell variable was changed.

My point was that you need to use complete path to original python only once (when you creating a venv) and than you can use relative path to run your py scripts from shell.

EDIT: my message was accidentally sent before I finished, so I had to edit it.

EDIT2: I almost forgot. You can symlink 3.x versions and just use any of them as base interpreter (works both in shell and in pycharm)

❯ ln -s /usr/local/opt/python@3.7/bin/python3 /usr/local/bin/python3.7
❯ ln -s /usr/local/opt/python@3.8/bin/python3 /usr/local/bin/python3.8

[dwt]

That use case is exactly the goal. And what we want is to not have to get the full paths every time, to make it easy to use tools like tox which automate creation of different venvs for different python versions and running tests in them.

That would be really simple, if we could just add those python-versions all to the path, and that wold be especially easy if all but the main python version was built with altinstall as you only get the version suffixed executables then.

[Feuermurmel]

@viking1304 I don't understand how virtualenvs move into the picture for you. I have not mentioned vritualenvs in my initial post. Using or not using virtualenvs is othogonal to my issue.

What I need is to have both e.g. python3.6 and python3.10 on my $PATH at the same time. This is something that is expected e.g. by Tox and build scripts used by some of the projects I work on. I know that I can hack this by placing some symlinks at the right places, this is absolutely not a question of me not seeing how to solve the problem. I'm looking for some software that deals with that instead of me. MacPorts did that for me for ~12 years without a hitch but I am looking for alternatives.

On a side not: I noticed that it's confusing if your replies are not in the same thread that they are related to. Please try to reply in the same thread instead of creating a new answer to reply.

[viking1304]

I don't understand how virtualenvs move into the picture for you. I have not mentioned vritualenvs in my initial post. Using or not using virtualenvs is othogonal to my issue.

What I need is to have both e.g. python3.6 and python3.10 on my $PATH at the same time. This is something that is expected e.g. by Tox and build scripts used by some of the projects I work on. I know that I can hack this by placing some symlinks at the right places, this is absolutely not a question of me not seeing how to solve the problem. I'm looking for some software that deals with that instead of me. MacPorts did that for me for ~12 years without a hitch but I am looking for alternatives.

If I sounded arrogant or disrespectful to you or @dwt it any of my statements, I apologize to both of you.

I really didn't had any intention to prove that I am the only one right here while everyone else is wrong. I just pointed out that you can use venvs and have different python versions in any of them and that in most cases you don't need to have any of them in path. That's all.

If Homebrew maintainers decide that they should use altinstall for all other version except 3.10 and to link them all, I have no complains, but as nvm user, I will always prefer similar approach more than to have python@3.7 and python@3.8 in my path.

On a side not: I noticed that it's confusing if your replies are not in the same thread that they are related to. Please try to reply in the same thread instead of creating a new answer to reply.

Agree.

[Feuermurmel]

@viking1304 Thanks for your reply!

I honestly don't understand why would anyone like to have that kind of mess on his development machine.

That part was a bit degrading. But all is well, you already apologized! Thanks.

Some of the reasons to have the binary of different versions of Python directly on my $PATH are: Running the interpreter on the command line to quickly test something, running Tox, and having shell scripts in company projects that contain lines like this:

python3.6 -m venv venv

But I would also like Python to get more self-aware and allow e.g. configuring a version in a project configuration file that then would dictate which interpreter is being used when just running python from the command line within that directory (and IDEs could also use that information).

[carlocab]

Opened Homebrew/homebrew-core#107517 to try to implement this.

[nedwilson]

Hey all, I can certainly see the reasoning for this change, but in the event that we do not want to use python@3.10, and have a lot of scripts that run on cron jobs that require /usr/local/bin/python3 be python3.9, this breaks a lot of stuff. I discovered this the hard way when I upgraded something unrelated earlier today. Is there a way to force /usr/local/bin/python3 to point to an arbitrary version of python? Currently the only way to do this with Homebrew is to link python@3.10. The other way is to link python@3.9 and then make the symlinks the old-fashioned, hard way.

[SMillerDev]

Why not just use #!/bin/env python3.9?

[nedwilson]

Thank you for the suggestion. Unfortunately, it's a change that would have to be implemented across hundreds of scripts on probably 40-50 workstations, many of which I'm not responsible for maintaining. For my purposes, it would be much easier to either run something like "brew switch python 3.9" or just not upgrade homebrew at all. I realize that the "brew switch" command has been removed.

[carlocab]

Why would you need to link python@3.10? It is already linked, unless you previously unlinked it.

You can also add

$(brew --prefix python@3.9)/libexec/bin

to your PATH if you wish to call python3.9 as python3.

[nedwilson]

When I run: brew install python@3.9 The default behavior should not be to install Python 3.10 and link all of the unversioned symlinks to the version of Python that I did not ask it to install. This breaks stuff, especially for dependencies that need to be installed via pip. The default behavior should be to install the version of Python that was asked for, no?

…

On Aug 31, 2022, at 5:11 PM, Carlo Cabrera ***@***.***> wrote: Why would you need to link ***@***.***? It is already linked, unless you previously unlinked it. You can also add $(brew --prefix ***@***.***)/libexec/bin to your PATH if you wish to call python3.9 as python3. — Reply to this email directly, view it on GitHub <#2776 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADUMFGYRDCJCBTEO4E3KDILV37YEFANCNFSM5L4QT5SQ>. You are receiving this because you commented.

[carlocab]

When I run: brew install ***@***.***

Hard to say what should happen since GitHub has mangled the command you wrote.

[nedwilson]

Hah, looks like GitHub thought it was a personal email address.